He's still upset the Beta Unit stole his truck and crashed it into the Zando-Zan's space beep
HAHA! I get that line! Let's see if others do!
(Hint: it's a sci-fi reference!)

He's still upset the Beta Unit stole his truck and crashed it into the Zando-Zan's space beep

since he wanted to move away somewhere with Robin, isolated and away from everyone from both sides. Not unlike Romeo and Juliet only Robin doesn't die
Brian was a smooth talking opportunist and manipulator, which is why during his exchange onboard the mothership with Diana, she remarks that it's clear how he's managed to rise so quickly through the ranks. Like recognises like. Dangerous Liaisons is more apt than Romeo & Juliet.
When Diana asks if he's attracted to Robin, Brian jumps at the chance to ingratiate himself into her favours and affections by declaring a preference for Diana! Later, when the pair review the "sex tape", Brian callously remarks that he'd prefer to forget the experience.
Talking about wanting to move away with Robin was pure spiel, yet another self-serving act of manipulation in order to exploit a young girl's emotions so she'd help him escape from the base.

It might be a rare oversight or it might be the level of brilliance in every nuance of the 1983 miniseries, from what I recall. Brian never once came across as two-faced, even when talking with Diana on her mothership. It would have bordered on cliche if he had. He just followed orders and may have been intrigued by the idea when told. TFB carries the continuity over, from what I recall, ultimately making him being a casualty on the Visitors' side, since he wanted to move away somewhere with Robin, isolated and away from everyone from both sides. Not unlike Romeo and Juliet only Robin doesn't die And it helps establish the ill-fated TV show as to how not all of them are either pro-Leader or 5th Column, but one just being in the middle.