Because he is a man who proudly wanted to deliver us all into communist/socialist slavery, that's more than enough reason to hate him or anyone else in my book.
If he proudly supported the Klu Klux Klan, would you be making the same argument? I dare say not nor should you.
By the way, please also note that Venezuela, for which he boldly and openly promoted the Hugo Chavez's communist takeover of, is now in total collapse and people are resorting to eating dogs, zoo animals and anything edible just to stay alive. Oddly enough the once very loud and opinionated Mr. Penn doesn't have a single thing to say about it now, as he hides in his mansion.

Coz he's a liberal left idiot who spouts sh-t, and he is butt ugly too.

Because he was friends with Hugo Chavez, who was a dictator. He's also friends with the Castro family, also dictators. He wanted people who called his buddy Hugo such to be arrested. He supports ideas which are communistic in nature. Communism is a failed and flawed ideology. One where we go "from each according to his ability, to each according to his need."
"In his house at R'lyeh, dead Cthulhu waits dreaming".