Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thank you for helping us increase our bandwidth

blog.archive.org

Post reply on HN
Print view View original

161–170 of 207 posts

Re: Thank you for helping us increase our bandwidth

#161

Post by MaxBarraclough »

jonah-archive wrote: post #146
bits_n_bytes wrote: post #136

Earlier quoted context omitted.

I understand money is tight for a non-profit, but why are bandwidth costs a gating factor? Both your upstreams, Cogent and Hurricane Electric, offer 100G ports at fiveish grand per month in carrier neutral DCs. Given that your budget is in the millions, an outlay of this magnitude doesn't seem wildly out of the question. If you can explain what the problems are in getting more bandwidth, I'd be more than happy to see…

Happy to chat in much greater detail over email if you like (in my profile) -- transit specifically is not generally our limiting cost (though of course in the context we operate in, every penny counts -- e.g. we run our own DCs, ambiently cooled, single-feed grid power with minimal backing, etc). We generally run very close to the edge, capacity-wise, in order to get the most out of what we have -- in this case, dem…

I'm sure I'm not the only one who'd be interested in reading more of this. Is there a reason your conversation can't proceed on HackerNews?

Re: Thank you for helping us increase our bandwidth

#162

Post by sktrdie »

Why don't they put their data dumps in an SQLite database and indexed for full-text-search https://sqlite.org/fts5.html

Then put the file in a torrent. Let the users seed it.

Users can use sqltorrent (https://github.com/bittorrent/sqltorrent) to query the db without downloading the entire torrent - essentially it knows to download only the pieces of the torrent to satisfy the query.

Every time a new dump is published by internet archive, the peers can change to the new torrent and reuse the pieces they already have - since SQLite is indexed in an optimal way to reduce file changes (and hence piece changes) when the data is updated.

I talk a bit about it here: https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4

Would save Internet Archive lots of bandwidth and hassle

Re: Thank you for helping us increase our bandwidth

#163

Post by contravariant »

sktrdie wrote: post #162

Why don't they put their data dumps in an SQLite database and indexed for full-text-search https://sqlite.org/fts5.html Then put the file in a torrent. Let the users seed it. Users can use sqltorrent ( https://github.com/bittorrent/sqltorrent ) to query the db without downloading the entire torrent - essentially it knows to download only the pieces of the torrent to satisfy the query. Every time a new dump is publish…

Seems that you're being downvoted at the moment because it looks like you're making fun of archive.org, but that article is actually rather interesting.

How does this compare to something like IPFS?

Re: Thank you for helping us increase our bandwidth

#164

Post by Aeolun »

mostlysimilar wrote: post #2

archive.org feels like an irreplaceable treasure, the Wayback Machine alone is a time capsule of our digital history. I donate to them monthly and know a lot of other people do as well, so I don't worry much about their financial stability. I'm more worried about external pressures taking content down. I hope the data is backed up six ways to sunday, and that somewhere there's a plan to make it all accessible if Inte…

It’s the only organization I actually donate to. Partially because it was so easy to set up a monthly donation ;)

Re: Thank you for helping us increase our bandwidth

#165

Post by Aeolun »

MaxBarraclough wrote: post #161
jonah-archive wrote: post #146

Earlier quoted context omitted.

Happy to chat in much greater detail over email if you like (in my profile) -- transit specifically is not generally our limiting cost (though of course in the context we operate in, every penny counts -- e.g. we run our own DCs, ambiently cooled, single-feed grid power with minimal backing, etc). We generally run very close to the edge, capacity-wise, in order to get the most out of what we have -- in this case, dem…

I'm sure I'm not the only one who'd be interested in reading more of this. Is there a reason your conversation can't proceed on HackerNews?

Or at least update us on the conclusion :)

Re: Thank you for helping us increase our bandwidth

#166

Post by Aeolun »

sktrdie wrote: post #162

Why don't they put their data dumps in an SQLite database and indexed for full-text-search https://sqlite.org/fts5.html Then put the file in a torrent. Let the users seed it. Users can use sqltorrent ( https://github.com/bittorrent/sqltorrent ) to query the db without downloading the entire torrent - essentially it knows to download only the pieces of the torrent to satisfy the query. Every time a new dump is publish…

A 50PB sqlite database? I don’t think any filesystem would be happy with that.

Re: Thank you for helping us increase our bandwidth

#167

Post by sktrdie »

Aeolun wrote: post #166
sktrdie wrote: post #162

Why don't they put their data dumps in an SQLite database and indexed for full-text-search https://sqlite.org/fts5.html Then put the file in a torrent. Let the users seed it. Users can use sqltorrent ( https://github.com/bittorrent/sqltorrent ) to query the db without downloading the entire torrent - essentially it knows to download only the pieces of the torrent to satisfy the query. Every time a new dump is publish…

A 50PB sqlite database? I don’t think any filesystem would be happy with that.

Indeed. But nothing nothing stops them from storing torrents of torrents. The search index would be just that, the text index. Which would point to another torrent storing the actual content. Assets would point to yet other torrents.

Would be interesting to learn how it's currently partitioned. I would mimic the same portioning system but use torrent instead so users can help with hosting. And use sqltorrent to serve queries efficiently.

Re: Thank you for helping us increase our bandwidth

#168

Post by jfkebwjsbx »

TeMPOraL wrote: post #140
ramraj07 wrote: post #90

Earlier quoted context omitted.

That's a fraction of the AWS bills for many startups arguably doing absolutely nothing

They have VC money to burn. Archive.org doesn't. Also, those backups would be (relatively) cheap to keep, but not necessarily to restore.

I would guess restore wouldn't be a problem. AWS or whoever would do it for free given it is a non-profit (in case of a disaster only, of course).

Re: Thank you for helping us increase our bandwidth

#170

Post by microcolonel »

sp332 wrote: post #98
microcolonel wrote: post #68

Earlier quoted context omitted.

If I'm reading that correctly, it would only cost a bit over 500 bucks a month to host that whole archive on BackBlaze B2. Furthermore it would not be so hard to translate Archive.org items to IPFS objects, if there were an effort to pin a significant number of them to storage and network.

The effort is the issue here. There was this comment back when IA.BAK was in design phase https://news.ycombinator.com/item?id=9148576 And then this is all there was to show for it: https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....

Yeah, I noticed that being a problem. My biggest problems with IPFS are hardly mentioned in their updates, and it's hard to tell if they have any interest.

I had constant issues with objects simply never (hours and many requests) being found, despite being pinned in several places; and the daemon sucked resources away from the system at an alarming rate back when I was trying properly.

With all that being said, maybe now is the time to look at it properly, the budgets are there. Maybe Juan, _prometheus, can find somebody to at least PoC this important application.

Post reply on HN