What happens if he became ultra violent if she broke character. She is asking to stop while trying to stay in character.

Interesting! The House of Yes equals post-war America. I can dig it.
END THE WARS. TAX THE RICH.
THIS ISN'T ROCKET SCIENCE.

In my interpretation, what really matters is that the interpretation of the crimes a hand are left as ambiguous as the JFK assassination was to everyone else. The video tapes notwithstanding. The HOUSE in question is a symbol for america. Burying his body in the yard 'next to daddy' also notwithstanding.
"The more real things get, the more like myths they become. " R.W. Fassbinder

I actually found a clip of the ending:
I'm not sure how to see the whole "who is at fault" thing here. It's been a long time since I've seen the movie, so bear with me if I'm forgetting anything here.
The easy way out is to say that Marty is the one who is at fault here and that he pushed Jackie-O into a sexual relationship or that he should have at least had the sanity to know not to take that step.
Then again, Marty might not be as stable as he appears to be. Not everyone's mental instability is as readily apparent as Jackie-O's is. Marty went out of his way to try to be normal- that might be a sign that he was just as unstable as Jackie-O was. She just wore it on her sleeve while he buried it deep inside of him. If he's as insane as Jackie-O in his own way, then wouldn't that kind of make the blame 50/50?

People really think that? Wow. That thought never even entered my mind. It's not even really about what she's saying but how she says it. How can you think she wanted to stop what they were doing when she was looking at him like that? I agree with the OP.

There are two levels to it. She's obviously being very coy and seductive with the way she says it, egging him on, but there's something about the way she says "Stop it" the last time at the very end that makes me think on some level she knows what they're doing is not right, and Marty, being the relatively saner and less impulsive one, has the power to stop it.
Even if that's not what she is deliberately saying or implying, I do think the makers of the film/play are saying it, and indicting Marty by putting such emphasis on that line. That was the tipping point; he had the chance to put a stop to what they were doing right there, and chose not to.
Get on up.