Live data from Hacker News

Thank you for helping us increase our bandwidth

blog.archive.org

101–110 of 207 posts

Re: Thank you for helping us increase our bandwidth

#101

archive.org feels like an irreplaceable treasure, the Wayback Machine alone is a time capsule of our digital history. I donate to them monthly and know a lot of other people do as well, so I don't worry much about their financial stability. I'm more worried about external pressures taking content down. I hope the data is backed up six ways to sunday, and that somewhere there's a plan to make it all accessible if Inte…

They have lots of copyrighted content. Such as virtually every commercial retro video game ever made it seems.

I personally think it’s great but surely companies aren’t too pleased? How does archive.org avoid being sued into oblivion?

Re: Thank you for helping us increase our bandwidth

#102
post #43

The article doesn't mention why archive.org demand went up so much after COVID. Does anyone know?

I don't know if this explains all of it, but they launched a National Emergency Library, where they pooled the DRM'd versions of ebooks from lots of libraries and removed one little bit of the DRM restrictions. http://blog.archive.org/2020/03/24/announcing-a-national-eme...

Re: Thank you for helping us increase our bandwidth

#103

I'm really surprised they don't use more CDN for this. Anyone know the reason why it isn't served by something like cloudflare?

They serve everything in-house and are their own AS; this allows them to provide stronger guarantees of reader privacy than sites that use a CDN.

> They serve everything in-house and are their own AS

Forgive me, but what is an AS?

Re: Thank you for helping us increase our bandwidth

#104

They increased the bandwidth and traffic immediately filled the new capacity. While this could be just a faster web site attracting more users / users were giving up due to slow loads before, it could also indicate that they have a lot of automated traffic that will consume whatever resources are available and can be throttled without endangering their mission. I hope they'll look into it, find a way to identify that…

We don't save logs of who hits us, but we watch who hits us. It's not bots. But good thinking.

Re: Thank you for helping us increase our bandwidth

#105

They increased the bandwidth and traffic immediately filled the new capacity. While this could be just a faster web site attracting more users / users were giving up due to slow loads before, it could also indicate that they have a lot of automated traffic that will consume whatever resources are available and can be throttled without endangering their mission. I hope they'll look into it, find a way to identify that…

We don't save logs of who hits us, but we watch who hits us. It's not bots. But good thinking.

How do you know? Not doubting you, just curious as to how you figure out if a request is from a bot or not.

Re: Thank you for helping us increase our bandwidth

#106
post #17

60 Gbit/s of continuous traffic is a lot. If I'm reading the graphs right, Wikipedia "only" has 13.4 Gbit/s of outbound traffic [1]. Of course still well below the single-digit Tbit/s traffic of large internet exchanges [2] [3], but still unexpectedly large. [1]: adding up the outbound numbers for each datacenter: 1.888 + 8.003 + 810 + 1.958 + 807: https://grafana.wikimedia.org/d/000000605/datacenter-global-... [2]:…

60 Gbps is about 1/3 of the traffic served by a single Netflix CDN node.

I'd say the "bytes" being transferred are very different.

Netflix has a (relatively) small number of large content pieces, and a CDN node would hold the hottest items.

IA has 430 billion individual pieces of data, on average much smaller than a piece of Netflix content.

So each "byte" transferred from IA is "more work" to produce, and less likely to come from some kind of cache.

Re: Thank you for helping us increase our bandwidth

#107

Earlier quoted context omitted.

They serve everything in-house and are their own AS; this allows them to provide stronger guarantees of reader privacy than sites that use a CDN.

> They serve everything in-house and are their own AS Forgive me, but what is an AS?

Autonomous System. https://en.wikipedia.org/wiki/Autonomous_system_(Internet)

An AS is an entity significant enough to be reasoned about at the BGP level.

Re: Thank you for helping us increase our bandwidth

#108

I donate to the Internet Archive (recurring). I just hope they stick to their core mission, and don't get sidetracked like Wikipedia has. They offer a valuable service and I believe will help historians in the future understand this point in the internet's life. PS - Don't forget to pick an Amazon Smile charity and use the Smile.Amazon sub-domain. Donated almost $20 to the EFF that way.

How has Wikipedia got sidetracked?

Most of wikimedia's budget goes towards non-wikipedia related expenses: https://www.washingtonpost.com/news/the-intersect/wp/2015/12...

Re: Thank you for helping us increase our bandwidth

#109
post #12

Earlier quoted context omitted.

I’m sorry to hear this. I will make an additional donation on your behalf. I can appreciate how authors feel, but they must also realize copyright laws are asymmetrical (tilted heavily towards copyright owners) and in these times, many have no access to their local public library. They are essentially robbed of access to the book materials their tax dollars (which pays an author) have paid for during this time. It’s…

I also find it amusing that sofware developers do not recognize any more how we are on the same side in this fight. Imagine if we were still selling boxed software and they decided that in light of COVID-19 they just hand it out with a two week limit. It's an imperfect comparison but the current situation is not a reason to just discard the law. It might not be the best law, the place to fight that is in Congress and…

You mean entrepreneurs, not software developers. I can't think of anything about software development makes it inherently commercial.

Re: Thank you for helping us increase our bandwidth

#110
post #107

Earlier quoted context omitted.

> They serve everything in-house and are their own AS Forgive me, but what is an AS?

Autonomous System. https://en.wikipedia.org/wiki/Autonomous_system_(Internet) An AS is an entity significant enough to be reasoned about at the BGP level.

Thanks!
Post reply on HN