It's a damn shame his films struggle to break even. He's easily one of the best working directors, out there right now, and what he makes is quite a cut above the rest IMO. If I'm correct, The Master and Inherent Vice were both privately financed by the same person. Seems odd since There Will be Blood was his highest grossing film. The Master was dropped by Universal studios initially, according to wikipedia.
Guess the stuff he makes doesn't have much of an audience

They also opened in less than half of the theaters of TWBB.
There Will Be Blood
's widest release had 1,620 theaters to
Inherent Vice
's 653 theaters and
The Master
's 864 theaters.

His post-there will blood movies were very low grossed.

"Nobody" gives a damn about most of his work.
Hard Eight
Budget: $3 million | Worldwide Gross:
$222,559
Boogie Nights
Budget: $15 million | Worldwide Gross:
$43,101,594
Magnolia
Budget: $37 million | Worldwide Gross:
$48,451,803
Punch-Drunk Love
Budget: $25 million | Worldwide Gross:
$24,665,649
There Will Be Blood
Budget: $25 million | Worldwide Gross:
$76,181,545
The Master
Budget: $32 million | Worldwide Gross:
$28,258,060
Inherent Vice
Budget: $20 million | Worldwide Gross:
$14,710,975

I loved both, There Will Be Blood and Inherant Vice. Two movies that most people I know didn't really care for.

A polarizing film, an artistic one especially, is not uncommon in a director's filmography, nor does it equate to an "evaporating" audience by default.
Howard Hughes was Italian?

Do you recall how your audience responded to The Master?

That's just, like, your opinion, man.
Honestly without any graphs or polls concluding this, your just stating something subjective. You and your friends may not like his films as much, but there's nothing stating that there's an "evaporating fanbase".
Some people loved Inherent Vice and hated The Master. Despite polarizing opinions, his films continue to get accolades.
Personally, I didn't really like there will be blood a whole lot, a film many adore who don't even know his filmography, yet i loved the master on first viewing.
So show some evidence outside of the opinion of a handful of overly critical film Geeks such as ourselves, and then maybe you'll have a solid stance.

I was shocked he got financing for another film. I'm still not positive that the new one won't get cancelled.