O'Brien's fate, in so far as his going to prison, depends on what South African law never a model of civil liberties was in 1965. South West Africa, where the film takes place (since 1990 the independent nation of Namibia), was run by South Africa at that time.

The most that could get him off would be an insanity plea that would see him imprisoned in a mental hospital, but as I've come to understand it, once he becomes sane, he's off to prison.

I think he stayed for a couple of reasons:
(1) He was indeed losing his mind and was determined to remain where he'd be the "alpha male" over
every
species in the desert.
(2) Back in civilization he would surely have been prosecuted for both murder and attempted murder. By sending the doctor off into the desert at gunpoint, he could be charged with attempted murder (luckily the doctor ran into some Bushmen and survived). In his fight with the old German man at the plane, he was trying to kill him when the man suffered an apparent heart attack and died. But as this occurred in the course of an assault, it's felony murder, and O'Brien could be tried (and executed) for that crime. Yes, he could claim that he acted in self-defense, but the Stanley Baker character saw most of what happened from the dune, and O'Brien (Stuart Whitman) then forced
him
to walk off into the desert for a
second
charge of attempted murder. There was just too much accumulated evidence for him to lie about what had happened and get away with it. Even if one accepted a mitigating circumstance of diminished capacity (i.e., he's going crazy), it wouldn't have been compelling or accurate enough to save him from
some
punishment.

Interesting view.
Thats why this film is so good.
It is open to interpretation.

I thought it had more to do with his alpha-male status. As in he would lose it if he went back to civilization.

Im not sure, mate
I saw this film more than 25 years ago
But I clearly remember Stanley Baker making clear to the O'Brien character at some point that if he has got the guts to return, thats ok, but everyone would know about what he did, and I remember Stanley being very provocative.
It struck me hard at that time, cos this film took an entire different road than I expected at first(O'Brien, being the american and all that wouldve been the leading man, and hero of the expedition - Baker wouldve been only baboon meat or smtg like it).
So thats why I remember it.
But Maybe Im thinking of a different scene, before the helicopter scene.
aybe Baker made it clear before that. And when he just yelled at O'Brien telling him they would expect him to come for some more minutes, it wouldve been only implied that he had a chhoice to make, since two other people that knew he was the bad guy would survive too.

No he does not say that he will be prosecuted. What Bain yells to O'Brien after the helicopter has landed is 'O'Brien you've got half an hour to get here, they can't wait any longer than that.'
But O'Brien has no intention of leaving and the helicopter takes off without him.
Besides it would be difficult to prove that O'Brien had committed an indictable offense. He didn't actually murder anyone, Bondrachai was found and rescued by African tribesmen and a reasonable argument could be made he he asked Bondrachai (with some persuasion) to attempt to find a way back into civilisation to save the rest of the stranded party (as Sturdevan had earlier tried to do). Grimmelman's death was caused by a heart attack allegedly caused by a confrontation between him and O'Brien.
It could be argued that all of O'Brien's flawed actions were taken at a time of duress and were aimed at the eventual rescue of the stranded party.

The anti-hero played by Stanley Baker yells at him and makes it clear that he will be prosecuted, if he returns.