Yes, there are moments in the film a character could just leap at Jack, grab on, and pound wildly. Course then there would be no slapstick or the film would be over.
http://www.freewebs.com/demonictoys/

A blind man who's adept at martial arts would have much better odds against the Invisible Man.

The Invisible Man kind of reminds me of the Joker in The Dark Knight. Both are entertaining performances and fun villains, but they seem a little too invincible. It's like logic doesn't apply to either one of them until the film decides that it's time for evil to be defeated.
They come up with several means of negating the Invisible Man's advantage, yet he hovers around the film like a free-floating ghost and you can't touch him unless you can physically see where he's standing. The second movie in the series almost seemed like a deconstruction of the first film. Blow a puff of smoke in his face or turn out the lights and that advantage is gone. The ending also seemed like a forced attempt at pulling a War of the Worlds type of ending. It's somewhat fitting, but it seems a little underwhelming after all of the more extravagant attempts they made at defeating him.
It's still a good movie, but I can't help but nitpick over these little details.
"In literature, it's called plagiarism. In the movies, it's homage" ~ Roger Ebert

Try and grab someone who you can't see.