Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can I back up an old vBulletin forum without admin access?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 74 posts

Re: Ask HN: How can I back up an old vBulletin forum without admin access?

#21

Post by Jorge1o1 »

spike021 wrote: post #6
TedDoesntTalk wrote: post #4

The format you want is WARC. Even the Library of Congress uses it. There are many many WARC scrapers. I'd look at what the Internet Archive recommends. A quick search turned up this from the Archive Team and Jason Scott https://github.com/ArchiveTeam/grab-site ( https://wiki.archiveteam.org/index.php/Who_We_Are ) but I found that in less than 15 seconds of searching so do your own diligence.

Thanks for the links and info. Totally fair on my not sharing those in my original post. I was coming at it thinking in terms of still making it useful to others. WARC seems like a good first step. I'm just not sure what the intermediate steps would be to get something usable like a vBulletin -- basically with the intention of being able to continue sharing the archived stuff with users who may not be as technical an…

You can try https://replayweb.page/ as a test for viewing a WARC file. I do think you'll run into problems though with wanting to browse interconnected links in a forum format, but try this as a first step.

One potential option but definitely a bit more work would be, once you have all the warc files downloaded, you can open them all in python using the warctools module and maybe beautifulsoup and potentially parse/extract all of the data embedded in the WARC archives into your own "fresh" HTML webserver.

https://github.com/internetarchive/warctools

Re: Ask HN: How can I back up an old vBulletin forum without admin access?

#23

Post by ComputerGuru »

Backing up to WARC, HTML, whatever is great for posterity but not much more than that.

Assuming you're a member of the organization and therefore licensed to use the content (but merely unable to access it): Purely hypothetically speaking, if an admin is this mia and obviously not on top of the job, the odds are probably high that they've neglected maintenance. Old PHP server running out-of-date PHP applications... not the most secure combo in the world. I wouldn't be surprised if there were some magic strings you could send to the server to get it to regurgitate the contents of the database in a more developer-friendly, strongly-typed fashion which you could import to myBB or XenForo and continue chugging along..

Re: Ask HN: How can I back up an old vBulletin forum without admin access?

#25

Post by ComputerGuru »

duskwuff wrote: post #18
ZoomerCretin wrote: post #14

Earlier quoted context omitted.

> there was a project to be a vB->Discourse converter God, why? I can't be the only person to find Discourse inferior to vBulletin.

vBulletin is commercial software; Discourse is open source. While I'd agree that Discourse isn't ideal, it's one of the better options available in the open source world.

I'd take myBB over Discourse.

Re: Ask HN: How can I back up an old vBulletin forum without admin access?

#27

Post by mmh0000 »

quesera wrote: post #10

I would make a read-only archive first, using `wget --mirror`. This will fix relative paths, download assets, etc and can be published as-is on a new site. I'm ignoring copyright questions in the interest of archiving fragile data. Then I'd use an HTML parser against the local archive to extract the individual posts, if the additional work was justified.

It has been a VERY long time since I've done this with `wget`, but here's what I pulled out of my notes.

I used to use this to mirror Slashdot when I was travelling for work and had limited/no internet:

```

  wget --ignore-length --no-remove-listing --no-check-certificate --recursive --page-requisites --span-hosts --convert-links --no-verbose --tries=3 --timeout=60 --level=1 --user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0" --html-extension --exclude-directories=my --directory-prefix="./slashdot/" -e robots=off --cookies --load-cookies slashdot-cookies.txt --keep-session-cookies http://slashdot.org/
```

I suspect that still works, just needs the user-agent updated.

Re: Ask HN: How can I back up an old vBulletin forum without admin access?

#28

Post by ChrisArchitect »

TedDoesntTalk wrote: post #4

The format you want is WARC. Even the Library of Congress uses it. There are many many WARC scrapers. I'd look at what the Internet Archive recommends. A quick search turned up this from the Archive Team and Jason Scott https://github.com/ArchiveTeam/grab-site ( https://wiki.archiveteam.org/index.php/Who_We_Are ) but I found that in less than 15 seconds of searching so do your own diligence.

Related:

An Introduction to the WARC File

https://news.ycombinator.com/item?id=39183670

Re: Ask HN: How can I back up an old vBulletin forum without admin access?

#29

Post by boomer918 »

ComputerGuru wrote: post #23

Backing up to WARC, HTML, whatever is great for posterity but not much more than that. Assuming you're a member of the organization and therefore licensed to use the content (but merely unable to access it): Purely hypothetically speaking, if an admin is this mia and obviously not on top of the job, the odds are probably high that they've neglected maintenance. Old PHP server running out-of-date PHP applications... n…

What you're suggesting could get OP in trouble if caught. Not sure it's worth it, but then again I'm not that into cars.
Post reply on HN