It refers to the passing of old colonialism and customs and the end of Britains neutrality about Hitler.
A bygone era for England.

A hint is given in the final scene of the film (which I don't want to spoil).
In my opinion, this movie is all about the meaning of life: what do you live for? Are we giving ourselves to what fully accomplishes us or not? The "remains of the day" is what is left of a life approaching its end. The chances that one didn't take are long gone. What is left is memories, and maybe not even the courage to regret the lost occasions of life. Why then people look forward to the evening on a working day? The best of the day is gone, it is in the present moment that we say who we are, and the arrow of time points only in one direction.

Indeed, and this essential scene was shot for the film, but never used. The final script draft, by Ruth Prawer Jhabvala, stupidly omitted the most important scene in the story - on seeing this, Hopkins was incensed and insisted that it be included or he would walk. James Ivory shot it, but betrayed Hopkins by leaving it out of the edit. Now this almost-perfect film has no moment of revelation, no emotional release, and no relevance to the title, and entire point, of the original story.
You can see the scene on the deleted scenes of the Blu-ray/DVD. Or here:

It's actually a bit of an artifact title. In the book, after Mrs. Benn gets on the bus, Stevens meets a man who tells him that he should try looking forward rather be miserable because he's wasted his life. Upon hearing this, he resolves to "make the best of what remains of [his] day", meaning his life.

The remains are the smaller portion of what is left and the day is Mr Steven's life.

Note that "some ppl think the evening is the best part of the day". Some do. Some do not. You decide.
That could be another double meaning also: if the last years of one's life could be considered as (potentially) the best, or not.

This message has been deleted.

This message has been deleted.

The title sums it all. You should read the book, to really understand what it's all about.
It's quite similar to about Schmidt, about a man trying to find something of 'meaning' for his life, and at the end realizing that he must do all he can to accomplish as much as he can at the "remains of his day" - the ending days/years of his life.
The only difference between this and "About Schmidt" is that the protagonist in this film is very selfless why the protagonist in "About Schmidt" is very selfish. In the end, we realize an overabundance of either leads to an unfulfilling life, which is a dilemma both men face in the end.

Somehow i remember in some movie a butler saying something like "If you won't be needing me any more, sir, I'll take the remains of the day."
I had thought it was this movie, but I looked for Hopkins saying that and didn't see it.
So in answer I think the remains of the day are something a house servant has as time for themselves.

Stevens might find that a bit too on-the-nose.

A Polish translator of Ishiguro's book has given a title (another re-translation :)) "The Crumbs of the Day".

I know you wrote that comment years ago, but a belated "brilliant insight".

Just finished watching this film this afternoon and was struck by some dialogue near the end of the film. SPOILER coming.
Miss Kenton and Stevens have met after several years at a seaside resort for tea. Afterward they are strolling on the busy promenade at dusk. Suddenly, the lights along the promenade come on and several people begin cheering, to which Miss Kenton says something to the effect of, "I'm not sure why people always cheer when the lights come on in the evening. I think I read somewhere that for most people the evening is the best part of the day," i.e., the remains of the day.

The remains mean that all that's left for Mr. Stevens and Miss Kenton are the remembrances about what they lived all those years on Darlington Hall.