I wholly believe Ms. Irving deserved her Oscar nomination - I love the innocence and purity of heart that she brings to Hadass. I can't presently think of another performance I've seen that so perfectly encapsulates these facets of human nature.
As to her Razzie nomination, my theory is this:
Admittedly, Hadass has a moment or two of timid awkwardness during the course of Yentl. If I recall from my first viewing of the film, this was what I initially noticed. But through another viewing (and much pondering of Ms. Irving's performance), I was able to look past Hadass' timidity and see what I see now. It seems that the Razzie voters were, simply, unable (or unwilling) to do the same.
Ultimately, to each their own, but I will always hold Ms. Irving's performance as Hadass very close to my heart

The razzies will just go after high profile people for a little publicity, I think. I remember them picking Richard Burton as the worst actor of all time, which was really absurd too. He had some incredible performances to go along with the bad ones.

I just watched watched Yentl (last night to be honest) and I think she earned the Oscar nomination. She plays a very innocent, withdrawn character and it had to take a lot of skill to hold back eveything. As for the razzie I can't see why she was nominated.