Bump.

CORRECTION: It's not that the crawlers are banned (my apologies), it's that most of the directories are protected.

robots.txt for IMDb propertiesUser-agent: *Disallow: /boardDisallow: /boards

This means that ALL crawlers and scanners are prevented from downloading the entire board directory and all sub-directories below. See the actual file for more.

Ah. Oh well bye bye IMDB Forums you will be sadly missed

Regardless of what you might have read, unless the site admin modifies their robots.txt file, there's nothing we can do other than email them and request an SQL dump. Neither of which is likely to happen.
Read the file below:
http://www.imdb.com/robots.txt
You can plainly see that almost all engines and crawlers are banned, and unless the IMDb administrator changes that file, no program or script you try to use to download the forum data will work. You'll grab the main page, but that's about it.
More here:
http://www.imdb.com/board/bd0000001/nest/265784055?d=265829976#265829976

What settings did you use?
No matter what I fiddled with it would only take the first page of anything I tried to save.

You can't copy an entire forum with it. It works better on some sites than others. I tried copying a 3 page thread here with it and it picked up two pages, but not the third.

Hey there that program works fast.
But it only seems to copy the first page of a forum and not the pages in the links when you click on a discussion. Or did I miss a setting that would copy all the pages linked to the main forum page?

OK found it and I'm off to try.
There's a few boards I just have to keep for myself. Some of the discussions I really enjoyed so it would be nice to keep them.

They should but it would probably cost money wouldn't it?

IMDB should allow archive.org to do it

WinHTTrack Copier you could use to archive specific movie forum pages.

So would a script or webcrawler be good for archiving the movie forums for movies you have rated and discussed?
That's the kind of thing I was wondering about, for all the movie boards I have participated in.

Please see
/board/bd0000001/view/265787442
and check
/board/bd0000100/threads/
.

Hm. Please standby. Archiving the topical message boards is somewhat trivial. It's the title boards and name boards that would take forever to scan and download.

Check you PM. Need more help.
http://www.my-diary.org/users/851091

Well, the idea is to download every page of the thread listing on a given board and then download every page of every thread directly referenced in the downloaded board pages. It's just iteration after iteration. In some ways it is easier to use scripts, but a web crawler can be configured to do it. I think we're being granted temporary permission to do this, when it'd ordinarily be unwelcome. The subject belongs on the "Computers and Software" board:
/board/bd0000100/threads/
.

I need help with thisI want to save everything! Once they realize I wasn't the one posting harsh comments and give me back my archives.
http://www.my-diary.org/users/851091

Oh OK
I'll go find one thanks.

You use an automated script or a webcrawler.