I think the biggest difference between the two (in my opinion) is that CtB had much better "battle" action especially when it came to the Riders of Doom both in the beginning and end. Most of the individual, one-on-one action was either very short (as you said) & chopped/edited (such as the pit fights) or felt a little
too
choreographed to me. The only real exception for me was the final showdown with Rexor as mentioned previously.
CtD on the other hand has much better personal and intimate fight scenes. The one scene you mentioned between Conan and the Queen's guard (played by his good friend, Sven Ole Thorsen how many times has he been killed by Arnie? lol) is just absolutely top-notch. The ringing of steel, the speed and impact of their swords it is truly an awesome fight. Even Conan and Bombaata versus the Keepers of the Horn had some great action; when Bombaata picks up that scrawny dude by the neck and lifts him two feet off the ground, I always laugh at just what that guy was thinking! I can't help but enjoy the light-hearted approach this film took and the way it made you want to cheer almost during the fight scenes.
The notable exception is of course Toth-Amon controlling the "Ape-Man" creature. The professional wrestling and almost cartoonish feeling of this fight is just ridiculous for me. I will say though that 9 times out of 10, it is better to have a real object there with with to interact. A "real" Dagoth to slash at and as in CtB, an actual snake prop for Conan to hack the head off. The days before CGI became a crutch almost.

I've liked all the Robert E. Howard movies I've seen, even
Red Sonja
.
Steadfast he opened his mouth, but nothing came out. What a wash out.

They're a great pair of films, both with their faults and their strengths. Purists of the first film who skip the sequel are doing themselves a disservice.
"I said no camels, that's five camels, can't you count?"

I think the fight sequences in CtB are fantastic. Even the
pre
-fight sequences are fantastic. Conan locking eyes with Rexor at the palace (
You
) is better than all of CtD put together.

I've always been a firm supporter of Conan the Destroyer as I grew up watching it along with Barbarian but I have to disagree with you. You could argue that the fights are flashier when compared to the original but you simply cannot match the raw ferocity and brutality of Conan the Barbarian.