Both of them, particularly Neil were mostly bitter because they didn't get what they wanted at Christmas causeing them to lose hope about Santa.
Quite unfair if you ask me! Why take it out on the child?????

He even changed his appearance to fit the lie is what they think

They thought his dad was lying to him that he was santa. He was saying they went to the North pole. He seemed delusional

No, not in the beginning of the film. We see Charlie sad to leave Neil's car. We see him say to his mother "Do I gotta stay?" when she dropped him off for Christmas Eve. He demands to be picked up at sun-up. He brags to Scott about Neil being a doctor and being a really good cook. That's why they were all surprised when at "Career Day" at school, Charlie picks Scott to speak to the class instead of Neil.

Scott was always more important to Charlie than Neil. He's his father. Neil is just the guy banging mommy.

He seemed to be the only mad that Charlie still believed in Santa, and his wife just went along with him.
No, he was mad that because Charlie DID believe in Santa and it was connected to his dad, that Scott was suddenly more important to Charlie than Neil was.