The Second Post Clarifies that.
Without strife, your victory has no meaning.
Without strife, you do not advance.
Without strife, there is only stagnation.

I honestly think it's because Bill's wife Alice was part of cult and set that whole night up to test Bill and see how far he would go. I know it's been talked about before on other threads but it does make sense to me.
If you remember, Alice was mainly the one who started with the flirting and trying to temp men at the Christmas party.
Did anyone else also notice how that hooker Domino knew Bill's last name? He never told her his name!!!
The mask is missing out of his costume when he clearly put it back with the rest of the costume in the bag. Looks as though Alice had taken it put it on the bed near the end of the film , but strange that she wouldn't just confront him about it right away if she found the costume and didn't seam worried about it before?
Also, when Bill comes home after the party that night and walks in the bedroom , Alice is in bed laughing supposedly having that sex dream… but if you look closely it seams that her eyes are open just before she starts laughing. Then like 2 seconds later when she wakes up she all of a sudden seams terrified of the dream she had.
If you notice the daughter Helena also has the exact same hair color as Mandy and Domino and there are many symbols throughout the story as well…and it really comes across like they are trying to show off the way Alice is brushing her daughter's hair and the how much the little girl loves to shows off her Barbie doll, it really seams to symbolizing something like maybe Alice is trying to prepare her daughter for the lifestyle as well?
I really think Alice played a part in the night.

I'm going to start a new thread, then.
"if only you could see what I've seen with your eyes!"
Roy Batty

Alice, Domino, Sally, Marion, Lolita and Mandy. The relationship between those characters and all the men in the fim. I'll leave it at that for now. A. Because we are way off topic and B. It's been covered in other threads.
Suffice to say: it is about
service
.
Buy The Ticket, Take The Ride

It is very much a contemplation on the relationship between fulfillment and wealth, and how relationships (sexual, non sexual) factor into various motivations.
Can you elaborate on this?
"if only you could see what I've seen with your eyes!"
Roy Batty

It is very much a contemplation on the relationship between fulfillment and wealth, and how relationships (sexual, non sexual) factor into various motivations. It does very much advocate honesty, as you suggest, and employs Ziegler as a kind of paradox to Bill: the happy go lucky liar and maybe murderer. A symbol of the New York/U.S/World elite that clearly disgusted Kubrick. And then Kubrick looked at the layers of envy and emulation, ultimately coming around to the audience with a challenge by way of the film's title.
Kubrick's most dense and satisfying film, after initially appearing to be the opposite for most people upon first viewing. Multiple viewings are required.
The plot cannot be untangled from the themes and subtext and even the form itself, so people looking for surface level narrative answers won't be satisfied - you have to incorporate film language into your understanding of the film, or you will be as lost as Bill, trapped behind bars, looking into mirrors..
Buy The Ticket, Take The Ride

I remember one of the things that struck me about the film upon initial viewing was the way people kept repeating themselves, or repeating what was said to them, e.g., Domino says, "What do you want to do?" and Bill replies, "What do I want to do? Hmmm." Or Red Cloak says, "Remove your clothing!" and Bill answers, "Remove my clothing??"
That led me to see what else was repeating, visually, thematically, etc. and I saw that the entire film was one story told twiceone version all lights and happiness, the other all dark and creepy. Probably neither version is "true"things are really somewhere in the middle for the Dr. Bills of the world.
There are perhaps not really these strange, ritualistic orgies of debauched flesh and unbridled greed going on, but neither are the super wealthy your friends. Chances are coming into contact with such people leads to unhappiness, and your best defense against abuse is to be as honest and open with your real friends - with yourself - as you can.
"if only you could see what I've seen with your eyes!"
Roy Batty

I agree with most of that. Notice that Milich's store is like a connection point between the two parties, as far as mise-en-scene goes.
So the point being that the veil of the upper class is basically a facade that actually facilitates their nihilistic hedonism. The whole illusion survives because of the envy of the lower class, proping it up ala sevant and master. Willing servant, thus one of the moments that was censored originally with the servant on all fours and the party goers (prostitutes?) banging on his back. Servant with a black mask.
Kubrick is also mirroing this in an uncanny way via the audiences expectations of seeing Nicole and Tom naked and having sex. It's funny that so many people come away from this disappointed, not realising they are mirroring Bill's sexual frustrations and voyueristic experience.
Buy The Ticket, Take The Ride

Really, I think whether or not Ziegler is a host or a guest at these functions is irrelevant. He is Bill's sponsor or patronthe only one among the so-called elites who recognizes Bill, literally and metaphorically. You recall Bill telling Alice he doesn't know a soul at the first party.
The two parties are really only one party, and the party is really a metaphor for society, or the class system, the upper reaches of which have only one person who Bill knows who likewise knows Bill.
I have no doubt that Ziegler is not at the tippity-top of the social stratathere's always another rung to climb.
But I think the real purpose of Ziegler pointing out that Nick, and by extension Bill, made him look like an ____, is to merely serve as a mirror to the earlier exchange between them regarding Mandy's illness. That Bill, or whoever that may be privy to Zieglers embarrassments, has the
potential
to make Ziegler look like an ___. That these things must be kept "just between us." The second exchange is the same as the first. But like the second party is merely the first stripped of all pretentious, so goes this second exchange. These things must be kept "just between us
or else
."
"if only you could see what I've seen with your eyes!"
Roy Batty

Good point meemee, but Ziegler is also a guest at the second one - taking a step up the social ladder just as Bill was at Ziegler's.
Plus, Nick (Bill actually) made Victor look like an beep
Buy The Ticket, Take The Ride

If you think of the orgy as a thematic reviewing of the first party at Ziegler house, albeit one stripped of its facade to reveal the truth about the guests, then it makes sense that Bill would be recognized and acknowledged by, I presume, Ziegler and his wife: they are his hosts after all.
"if only you could see what I've seen with your eyes!"
Roy Batty

He arrived in a cab and walked into the foyer without a mask on. He also handed his coat over with a receipt from Rainbow Fashions with his name on it.

I'm not sure how that relates to my post since I simply suggest that he was recognized because he (Bill) was the one who made it all up in his head just like in the novel on which the film is based "Dream novel".
If it's all in his mind, then everyone at the party will know who he is - mask or no mask.
But if you wanna find a possible rat, then try having a look at Ziegler when he walks in a circle around the red pool table while intimidating bill. Compare that to the ritual scene where the leader walks around in a circle wearing a red cape.

Interesting idea listed above about the previous owner of the costume shopthe fact that perhaps he attended these rituals and used the same mask as Bill.
The type of mask Bill wears might also have revealed his identity. His mask is essentially a double-mask. By that I mean, it is a full mask that covers his entire face, yet there is also a second mask on top of it that covers his eyes and the bridge of his nose. So this double-mask might symbolize Bill's presence at the ritual. He is masking his identity twice, (lowly doctor pretending to be someone rich and powerful, who then wears the requisite mask for the ritual). I believe the masked woman also wears a similar 'double-mask' as well, except it features an added headdress. So perhaps she recognizes Bill as a fellow imposter because they are both wearing double-masks.
Also, about the nodit is important to note that beside the nodding character is a smaller figure, clearly a woman, judging by her petite physique. This woman has a sad mask with a teardrop. This might indicate that the nodding figure and his 'wife', are Carl and Marion. The teardrop indicating her sadness trapped in a relationship with a man she no longer loves.

The answer is in the title of Schnitzler's original work "Dream novel". The events are not to be taken literally - dreams was the central theme of his work and you have to consider that he was a doctor and a close friend of Freud.
This is how the novel is described in Publishers Weekly
Schnitzler's characters ultimately return from these sleeping and waking "dreams,"
Here Scorsese explains directly how and why Kubrick took that idea and put it into the film
(don't worry, I'm not stalking you)

those explanations are too vague.

When he went to rent the costume he said he was the doctor of someone. Perhaps that person also happened to be in these sort of rituals (and that would explain the misterious ways the russian speaks to Bill). I think that maybe the russian gave him the mask that this person had previously used before moving to chicago. So the people on the second floor that looked at him perhaps thought that he was that person, but if they also happened to know he moved to Chicago they immediatly realized it was not him so it had to be someone elseI know my theory lacks substance but perhaps someone can elaborate someone with it

I doubt it's a Kubrick error. Like him or not, he meticulously plotted out every scene of every movie. If you see the documentary "Kubrick's Boxes," you will see just how much research and planning went into every nuance. Also, interviews with anyone who worked with him.
Given that, and some of the insightful answers above, here's my theory:

1. I agree with others that "Fidelio" was not the "insider" password. Only one, or maybe a handful, of the "help" had it, and they were all accounted for. So, they knew to check his pockets.
2. He arrived at the gate, and door, without mask. That was probably a huge tipoff. Now, Ziegler did know a bunch of the mucky-mucks who were also members, but then, he was no doubt invited to join.
3. "The Nod from the balcony:" It could've been just a reassurance to an obviously new member. Or, word could've quickly gone out via whispers who the interloper was. I think it was the former, and that it was Ziegler and his lady for the evening. They probably discussed the problem of Bill intruding and knew what was to happen.
4. "Nod," cont'd: Another reason for it could've been: "We know you're not one of us, watch what we do about it." Bill's mask was bought at a costume shop. The others all had a design theme. How do I know? Because in the credits, the designer was of "Venetian masks," and the Trivia section shows Kubrick selected them from a previous production about a ball in Venice. Bill's mask, however, was designed for this movie, and modeled after Ryan O'Neal's face. So, maybe the couple on the balcony thought they recognized Ryan O'Neal! (kidding)
5. "Nod," cont'd: As a filmmaker, Kubrick might also have been showing us that Bill had no idea how to respond to any of this; how far out of his element he was. He had the twin feelings of fear of being recognized, vs how to respond to a possible signal among members. Thus, putting US off guard.
6. Mandy's "redemption" of Bill: I believe Ziegler that the whole thing was staged. She was substituted, remember, as his consort, then Bill was removed before anything could happen. Now whether she was drugged on purpose, or not, I don't know. But I suspect she was, in fact, an addict. As a former beauty queen, she had, in a way, sunk very low to be a consort for hire at these parties, and for the likes of Ziegler. So depression and dependence could have taken their toll.
   In any case, I doubt Bill's recognition is any lapse on Kubrick's part. It may not even be material, just something to set a tone of nervousness on our part.
   The important thing was, the mucky-mucks recognized him as an intruder, quickly found out who he was, and wanted to throw a scare into him, which they did.
   Ziegler, probably under orders, also had him followed to ensure he didn't cause any damage.
   Same with Nick, who probably loathed the moment he trusted Bill with the password.
   I wondered what they did with/to the cab driver to ensure his silence.

Well, if Kubrick said they didotherwise, just think, in such a lurid setting where everyone wears a mask to hide their identity, nodding at someone as if you know who they are would be a big faux pas!

Yes, Kubrick put it there. And Hack You can't figure out why.
awww
Buy The Ticket, Take The Ride

Your theory seems far fetched. The plot point is there, they recognized him. I didn't put it there, glorified hack Kubrick did.

What is interesting is that there are two of the blank masked men (one on the balcony, one below during that scene) - one of them 'offers' a woman (that sounds quite like Domino) to Bill..
Buy The Ticket, Take The Ride

ThE fidelio theory makes sense but I am specifically talking about the couple at the balcony that recognize him.
There is nothing that irrefutably implies that they recognized who he was, rather they took notice of him because he stood out. And one tested him with a simple nod as if it were some type cryptic sign of recognition among members, a nice little ruse which Bill fell for when he nodded back. From then on they were watching him closelyMy theory.
But a better question, is how the hell did Mandy recognize him??

Not at all, in all my years watching films, true films mind you, I've come to realize that Kubrick belongs to the Bieber and Bay crowd that fashions themselves to be intelligent. But they truly are not.