That's an excellent interpretation, crow_jayne! Freddy romanticized his former life, and now, by association, he despised everything that reminded him about it because he can never gain it back again. And to try to pick up any pieces of his life since his divorce would require taking some major responsibilities that he was too immature to face.
I have to wonder, though, if his former life was really as ideal as that conversation suggests or if it just looks so much better now because things got seriously off-track when their daughter was killed.

You're right. It seems off the wall at first, but Freddy & wife are sharing their common experience, when suddenly she becomes the observer & he's her object of pity. Still off the wall,(A sane man might say: "Oh, so that's how you feel", and then walk away.), but the character IS off the wall, not the screenwriter.

Freddy needs to regain his wife and family. He has never made any attempt at making peace with the death of his little girl. His life is empty. His own sons call him by his first name (they call their stepfather "Dad") Freddy's life is made of one night stands with strippers. He lives his aging life in an alcoholic haze of hate and the only thing he has to live for is the day the man who killed his daughter gets out of jail so that he can take the gun he's been polishing up year after year over to the newly released convict home and kill him. Freddy has this fantasy that if he destroys the origin of his pain the pain will go away and he can go back to before and do things differently. Freddy has mis-placed anger and a wound that has never allowed to heal. Basically that drunk driver took his whole life away. He has a love /hate relationship with his former. Love, for what they had been through together and hate because she has been able to go on and regain some sense of sanity and balance after the traumatic death of their daughter. While in the cafe Freddy is remembering and reminding his former wife of how good their romantic relationship was and how happy they both were. He felt strong and in control back then. Her reply to this: "You were so strong then Freddy. I felt so protected but look at you now Freddy, you are so weak now I feel so sorry for youI feel sorry so for you" Put yourself in Freddy's shoes. How would you react?
Whither goest thou, America, in thy shiny car in the night? ~~ Jack Kerouac