I think the studio were pushing the show into the direction of pure horror, which is not where Serling wanted to go with it. That , and the falling ratings, led to his decision to walk away from the show.
He sold the concept back to CBS, which is why they were able to produce the remake series in 1985.

Yeah, it would be boss to have him as a professor. I'd freak out!

I know that Serling wasn't happy with the direction the show was going. If you watch some of the later episodes, he wasn't even on the set, he would be in another seperate room. Apparently, there came a point where they had to literally beg him to narrate them. The same thing happened with Night Gallery. If Serling couldn't have it his way, he didn't want to do it at all. Unfortuantely, that's a problem many writers still have to this day with networks who love to tamper with screenplays and such.
I think he also began losing interest after the third season and wanted to focus more on teaching. Man, how cool would that be to have him as a professor? Anyway, I think the show was simply cancelled by the network due to the ratings beginning to fall.