I took it as trying to hide from a jealous husband, but I did wonder if they were going for some subtext or something.
So many stories, so little time.

I almost thought it was a gay reference, though i don't think "in the closet" had that meaning when this was made.

I think it's an inside joke for him as in his younger days, he actually did hide in a closet from a jealous husband(Groucho hadn't known the woman was married until the husband came home). To read the full story, and so many others, read his autobiography, GROUCHO AND ME.
"Outside of a dog, a book is man's best friend. Inside of a dog, it's too dark to read."
Groucho

I suspect it is a reference to a similar scene in Monkey Business, in which Groucho kept slipping in and out of a closet to hide from a gangster.

No. It's a reference to something you might find yourself doing if you're "the other man".