The Pink Panther is the first one to watch. The diamond is stolen for the first time, you meet Charles Litton the phantom, Clouseau's wife who later married Litton (her name escapes me at the moment), and Litton's nephew, played by Robert Wagner. You'll meet them again in Trail and Curse. Litton, the diamond stolen, and the wife reappear in 'Return.'
However, the first movie drags quite a bit in many places. Litton's the main character while Clouseau is a supporting role. Clouseau, while bumbling and clumsy, doesn't have the mannerisms or accent found in later movies.
A Shot in the Dark is really where the series takes off, where Clouseau, a main character, is fully born. Dreyfuss, Cato and Francois also make their debut as regular foils for Clouseau.
So watch the first movie, but maybe fast forward, or speed up, scenes without Clouseau, if you're bored by them.