I actually really like the psychiatrist scene. It’s compellingly delivered by the actor, it helps you to digest the revelations about Norman and adds important information about his other crimes - the
scale
of his insanity is revealed, and it builds suspense and makes us eager to see Norman one last time, knowing what he truly is (and the pay-off doesn’t disappoint). It also gives Lila some closure as she’s able to finally understand what happened to her Sister, the mystery was killing her.
Don’t mind Abergast falling, just seems like a stylised old-movie effect, like the fake backgrounds during driving scenes. Unconvincing but part of the charm.
For the most part I found John Gavin to be fine, although when he’s stalling Norman to give Lila time to search the house he comes on pretty strong and adversarial too soon. I’d like him to have been more chilled out and able to relax Norman before getting confrontational.
This isn’t a ‘scene’ but one thing that didn’t quite work for me was mother’s voice. I just don’t believe that Norman could sound so female. I needed to see Perkins speak in that voice to believe Norman was capable of mimicking an elderly woman so well. Feels like a bit of a cheat to use a real woman’s voice.

In my opinion the "worst" scene is when Norman knocks out Sam Loomis with an object (not even sure what it was). Something about the way it was shot didn't seem to be very believable for me.
I take your point, but Hitchcock tried a little "editing trick" here:
We see before the fight Norman desperately looking out the parlor window and saying "Where's that girl you came here with?" Norman AND Sam turn their heads to look in opposite directions its a weirdly balanced shot.
Then Hitchcock cuts to a very short shot of: Lila heading down the main stairs of the Bates House(as if to leave).
Then Hitchcock cuts BACK to Norman and Sam and the fight is already well underway. Hitchcock "cut off" the start of the fight. Norman and Sam are locked deep in struggle, have been for awhileand THEN Norman knocks him out.
With what I have always called "an ornate Victorian object." Now I think its an ashtray.
The "jolt" of the cut back into the Norman/Sam fight does "look funny" and it is an example of many "jagged edits" in Psycho that may have cost it a Best Film Editing Oscar nomination. WRONGLY.
BTW, in Van Sant's remake, he has Norman take a roundhouse swing at Sam's noggin with a golf club. That could kill a guy

Also there's the fact that Sam apparently gets up immediately and chases Norman to the cellar in time to rescue Lila. Although I guess it took Norman time to put on his Mother outfit and that gave Sam the leeway he needed.
Hitchcock reportedly worked with his actors and a stopwatch to give Norman enough time to run and get dressed, in many scenes.
For instance, when Arbogast returns to the motel, he rummages around the office and parlor for awhile before going up the hill.
We realize later that this gave Norman time to run up the hill(after seeing Arbogast drive up; he ran through an open passage at the L of the motel that Lila later uses for her climb to the house), run up the stairs, get dressed, and be waiting for Arbogast to climb the stairs(what great stamina Norman has, after all that uphill running he can still violent run out and stab a man.)
Picture Sam's "recovery and run" from his perspective:
Think of how long Lila spent going down to the fruit cellar two different levels and a long walk and the opening of doors.
During that time, Sam woke up(obviously Norman's blow was glancing, just didn't "take" which is very realistic.) And ran up to the house long BEFORE Lila started screaming.
Its possible that Sam SAW Norman heading into the fruit cellar in the dress and followed him down rather than being drawn by Lila's scream(which would be too late.)

Also there's the fact that Sam apparently gets up immediately and chases Norman to the cellar in time to rescue Lila. Although I guess it took Norman time to put on his Mother outfit and that gave Sam the leeway he needed.

In my opinion the "worst" scene is when Norman knocks out Sam Loomis with an object (not even sure what it was). Something about the way it was shot didn't seem to be very believable for me.

I think that all those scenes work (as I think you do, too ) in the context of the film as a whole.
Psycho
is a remarkably consistent and elegant "all of a piece" movie, and take away any of those seemingly less skillfully directed, written or acted scenes from that whole and the film isn't nearly as good (even the greatest of symphonies have their "slow" movements, and for a reason; and great novels, too, need things like lists, exposition, and the occasional flowery use of words, phrases that are like arabesques, to keep the wagons rolling,–works of art are seldom logical, or based on easy books of rules or technique; and genius makes its own rules).

Interesting. I might add that I don't think Psycho is a movie in which EVERY scene is greator even supposed to be.
As an analogy to screen acting, Spencer Tracy( a great actor) said "I must not give the same power of acting to every scene. I do some scenes at a normal level of moderate commitment, and save my strongest acting for the big scenes that should have it."
But every scene has something of interest to you it, and every scene has Hitchcock's vision in it, all the way.
For instance, in the California Charlie scene, note how:
The cop across the street is viewed in the distance as an implacable object.
The camera moves with Charlie and Marion as she looks back at the copand then CHARLIE looks back at the cop(Hitchcock moving camera POV)
Marion walking along the row of cars to make her choice is intercut with the "travelling close-ups" of each license plate, until she picks "NFB."
How Charlie's smiling face drains down to no emotionand then extreme distruston Marion agreeing right away to pay $700. (Hitchcock called this direction to his actor, John Anderson, as "negative acting" or "draining the face of emotion to get an emotional effect." Charlie's change in face creates paranoia on the part of Marion AND the audience.
Upon Marion's driving away from the lot(after the unseen mechanic yells "HEY!" and scares all of us), California Charlie, the cop, and the mechanic "all line up" in a perfectly balanced, perfectly composed row of "suspicious males." (Marion's NOT going to get away with this, and the solidarity of the masculinity of her watchers is disturbing, in a "pack of males versus a lone female" way.)

Actuallyi agree with you(except for the hotel room and California Charlie scenes)

The hotel room scene, the real estate office scene, the cop stop scene, the California Charlie scene, the dusk-til-night rainstorm with voices scene, the drive up to the Bates Motel in the rain scene, the hardware store first scene, the talk between Arbogast and Norman scene
and a few more.

Take away the parlor scene, shower scene, stairs scene, cellar scene and the ending then really, what are the good scenes on this film?

And for the record, I don't think John Gavin did a bad job, Sam was just never a very compelling character.

That's because the screenplay (maybe at Hitchcock's order) took out the background information on Sam. The original book builds him up into a deliberate foil for Norman: the man who talks about character and relationships vs the recluse; the man with healthy hobbies (classical music) vs the man obsessed with pornography and the occult; the man who plans his way out of economic difficulty vs the man who stays mired in it.

Films are like people, they all have flaws.
And for the record, I don't think John Gavin did a bad job, Sam was just never a very compelling character.
But no,
that fall down the steps is terrible.
I don't care what Al was going for, it didn't work.
i told you not to stop, now lets go - Apocalypse Now

Years of reading articles and reviews (both professional and "regular people's on the internet) have singled out these three elements of Psycho as being particularly "bad":
ONE: The shrink scene at the end(this usually hits Number One)
TWO: The shot of Arbogast falling down the stairs("fake and laughable" say rather sophisticated CGI-hounds)
THREE: The John Gavin performance(evidently, Hitchcock called him "the stiff" on the set, and it stuck.)
I think all three of those elements are just fine, and very much a part of the greatness of Psycho on purpose in the case of the shrink scene and Arbogast's fall(these were the scenes EXACTLY as Hitchcock wanted them), and by accident with Gavin(Hitchcock chafed against Gavin's being forced on him by superagent Lew Wasserman instead of first choice Stuart Whitman, but Gavin was good in his role and right for the film.)
And put another way: Psycho is an acknowledged, seminal classic that has made most "Best Films of All Time" lists no lower than Number 20, often in the top 10, and, on one key occasion, at Number One(AFI's Greatest Thrillers of All Time List, aka "100 Years, 100 Thrills.")
And you don't reach that level of achievement with a movie with "worst scenes."
Psycho is a classic, a near-perfect film in which whatever flaws there may be in the shrink scene, Arbogast's fall, or John Gavin's performance, they are "absorbed" into the overall greatness of the film itself.
Points in passing:
To those who think the shrink just tells us what we already know at the end, no. He tells us these crucial things revealed nowhere else in the movie before his scene:
ONE: Norman killed his mother, and her lover. (And "matricide is the most unbearable crime of all, most unbearable to the son who commits it.")
TWO: Norman stole Mother's corpse, stuffed it, preserved it with taxidermy chemicals "to keep it as long as it would keep" and, of course, kept Mom around the house.
THREE: Norman killed two young girls before he killed Marion. Marion wasn't his "first time."
Crucial information, delivered to us in a powerful and straightforward manner along with a historically-allowed-by-the-censors-after-being-prohibited discussion of transvestism, and a pretty darn good analysis of how Norman's psychosis worked ("He was often only mother, but he was never only Norman.") In 1960, what the shrink had to say was mind-blowing, never heard in the American movies before.
Arbogast's fall down the stairs was stylized on purpose, and I'm here to tell you I was around in the sixties when the film hit TV, and kids talked about that fall and NO ONE could see the process shot. They thought the actor really fell. (Check out a character's similar fall down the staircase in Psycho III to see how badly this can be done.)
John Gavin's performance is good in one key physical way: he's so damn TALL he towers over Norman at the end and can believably subdue him (less so with short and wiry Viggo Mortensen versus giant Vince Vaughn in the remake.) And I like Gavin's vocal acting choice: tense and rather scared for such a big man.

The scene on the steps was stylized, but I think it works.
I'd agree the police psychiatrist scene at the end is "stuck on" but it still doesn't bother me.
LBJ's mistress on JFK:

Probably just about everybody will choose the psychiatrist's know-it-all speech at the end. It was the one scene that was improved in the "re-make", where the actor made it look like how he was thinking aloud rather than delivering a rehearsed speech.