She's meant to be ordinarily insecure and none-too-brightexactly the opposite of Marty's hyper-neurotic and controlling sister. To her, Marty is a Prince Charming, and she's willing to overlooka lot. Of course, she doesn't get the chance to figure out how she'd deal with his abnormal family relations long-term. But in the short term, she finds everything about the family so intimidating that she just wants to retreat back to the simple life they've built together. Maybe it's clearer on stage (as opposed to in the film version) that none of this is exactly meant to be straight-up psychological realism.

I think she was more interested in getting Marty away from his crazy family. I think she might have had still had feelings from him but she was being hit from all sides.

We don't really know that she would have stayed with him, ultimately. Everything that happened was pretty much chaotic and in-the-moment, she felt guilt over banging his brother and she viewed his family as insane, manipulative and possibly mentally abusive so she then viewed him as a victim and she probably felt the need to "save" him. Plus, she clearly had some issues of her own and she truly did love him. Not everyone reacts to a "no duh" situation in the same way.
Jackie still fitting into that suit isn't a reach at allhave you noticed how tiny Parker Posey is?
It wasn't me who was murdered, was it?

True same goes for 90210, in that case.

I don't think I ever bought into Spelling's performance because she was the daughter of a powerful producer so you always wonder whether she got the role due to favors, or what. I've always had that problem with her.
Having said that, she did a fine job. And to address your point, the character is definitely mousey.
I've seen things that would make you want to write a book on how to puke.