Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keep Our Servers Running

blog.archive.org

Post reply on HN
Print view View original

291–298 of 298 posts

Re: Keep Our Servers Running

#291

Post by what »

miki123211 wrote: post #141
eis wrote: post #118

Earlier quoted context omitted.

This is completely normal. You can't just issue a chargeback because you feel like you want your money back after the fact. They want to know what went wrong. It could be fraud but it could be also misleading checkout experience and other reasons. The merchant is charged often a fee for a chargeback on the order of $15 and it might result in penalties. So, if you feel like you've been charged unfairly (e.g. product n…

Exactly this. In both US and EU, chargebacks are a general consumer protection tool, implemented by card networks to make you less afraid of using them with potentially-untrustworthy merchants, even when not legally required. They can be used for any kind of legitimate grievance, outright fraud being just one of them. If you book a hotel and the room is not up to the standards on their website, take pictures, take sc…

> If you book a hotel and the room is not up to the standards on their website, take pictures, take screenshots and chargeback.

This is unhinged. Then I can chargeback at McDonald’s because the burger doesn’t look the same as the picture. I still ate it and you probably still stayed in the room.

Re: Keep Our Servers Running

#292

Post by Barbing »

justin66 wrote: post #238
Barbing wrote: post #222

Earlier quoted context omitted.

This is really interesting to me. I wonder how many more hours of downtime The Archive would have experienced in 2025 if no one was ever woken up out of bed to fix it. (If anyone was) I also wonder the cost differential. Got me thinking about these incredible beautiful museums that are closed all night long… you’d still want someone there early enough to catch if the roof leaked so nobody missed their opening-hour to…

World class art museums have some staff on site 24/7, FWIW. > I wonder how many more hours of downtime The Archive would have experienced in 2025 if no one was ever woken up out of bed to fix it. (If anyone was) Wasn't 2025 the year a kooky pro-Palestinian hacker collective decided IA was a good target for their idiocy? There was a great deal of overtime involved there.

Great point on the museums!

Re: Keep Our Servers Running

#294

Post by throwaway2037 »

what wrote: post #290
throwaway2037 wrote: post #272

Earlier quoted context omitted.

> I'm all for ending copyright I don't understand this sadly, common sentiment on HN. How will authors and artists that produce copyrightable artefacts earn income via royalties without strong copyright laws? Are there any highly developed nations that have abandoned copyright? Or any that have weak/non-existent copyright laws and thriving arts and letters? I doubt it.

They don’t think you should profit off your work at all. They expect everything for free.

What bothers me most is the off-handedness of it all. They pop into the conversation like whack-a-mole a say "end copyright", yet offer no reasonable alternatives. Are we suddenly going to stop paying authors for their artistic works? It is crazy. Copyright laws are fundamental to create thriving creative industries.

Re: Keep Our Servers Running

#295

Post by ProllyInfamous »

detourdog wrote: post #286
ProllyInfamous wrote: post #276

Earlier quoted context omitted.

I have a combined 2800BTUs of Monero heating capability, which is equivalent to a standard US "spaceheater" running on medium heat. Localized beneath my main office computer desk, this allows me to further reduce heating of my entire domicile.

That’s pretty impressive I’m going to look into it.

For the past several years, this annualizes to about three weeks of free winter heating (via bespoke XMR). I mine solo, but I imagine if you "pooled" it would be similar incomestream (but more reliable). I have had a year of "no winnings," but I persist with the odds to warm myself and the network.

----

Half of my kilohashing runs on obsolete i5 equipment (==free-to-me, minus geektime) – Monero Foundation strives to operate with GPU- & ASIC- –resistance; processors haven't gotten that much faster (in the past decade), and a surplus i5 isn't that much slower than a new $500 Ryzen (~2/3rds hashrate).

Monero's Windows7 client still works (fully supported), so perhaps you now have a use for your dusty-old laptop?

Should you need to upgrade the hard drive (mining currently requires full-blockchain, which is ~300GB and growing), you might as well experience linux (installing UbuntuOS is easy once you've created the USBmedia).

It is definitely not worth purchasing additional equipment, with residential rates, just to mine monero. I like warm toes, and have lots of spare CPU cores.

----

Reminder: this is only "free electricity" if you're offsetting RESISTIVE HEATING (i.e. not a heatpump). To actually reduce your powerbill and save the planet (the smarter choice), you should install a heatpump for wintertime heating.

This isn't a "guaranteed moneymaker," it's to heat your geeky underdesk toes – you are just making a fancier resistive spaceheater, which occassionally pays for itself.

Re: Keep Our Servers Running

#296

Post by detourdog »

ProllyInfamous wrote: post #295
detourdog wrote: post #286

Earlier quoted context omitted.

That’s pretty impressive I’m going to look into it.

For the past several years, this annualizes to about three weeks of free winter heating (via bespoke XMR). I mine solo, but I imagine if you "pooled" it would be similar incomestream (but more reliable). I have had a year of "no winnings," but I persist with the odds to warm myself and the network. ---- Half of my kilohashing runs on obsolete i5 equipment (==free-to-me, minus geektime) – Monero Foundation strives to…

The building is heated by heat pumps my studio is one large room in the building 60’x 30’ 25’ (20m x 10m x 8m). This would be where some oomph is needed.

Re: Keep Our Servers Running

#297

Post by anandology »

raybb wrote: post #285
anandology wrote: post #280

Earlier quoted context omitted.

I was one of the initial developers of Open Library and wrote most of it's backend code. I doubt any of the performance issues could be because of code being sync. Open Library is IO bound and it is most likely spending time waiting for db to respond. Was there any performance analysis done that indicated that the "code being sync" the main bottleneck? Happy to help.

The issue wasn't the latency of a single DB query (though many were improved/cached), but sync worker saturation: When traffic spikes, slow network calls to archive.org (for availability) and Solr tie up all available sync workers. The queue backs up, and suddenly every page becomes slow because there are no free workers left to handle requests. This was handled from both sides: handling high-concurrency waiting bett…

I see the problem. But I feel switching to async is too big a change and there could be simpler ways to address this.

1. Solr was used (I think still continue to be used) for many things it is not quite the right fit. It was done then because, it was simpler to do. For example, it is used to get list of editions when a work is loaded. Limit Solr only to search and figure out a better way to manage the other pieces, would be a good start. Search could be made a different service and the frontend could call it via API. Both of these together will remove Solr from the critical path.

2. The availability status of archive.org could come from a database and a backend task can continue to update it very frequently.

Re: Keep Our Servers Running

#298

Post by mekarpeles »

anandology wrote: post #297
raybb wrote: post #285

Earlier quoted context omitted.

The issue wasn't the latency of a single DB query (though many were improved/cached), but sync worker saturation: When traffic spikes, slow network calls to archive.org (for availability) and Solr tie up all available sync workers. The queue backs up, and suddenly every page becomes slow because there are no free workers left to handle requests. This was handled from both sides: handling high-concurrency waiting bett…

I see the problem. But I feel switching to async is too big a change and there could be simpler ways to address this. 1. Solr was used (I think still continue to be used) for many things it is not quite the right fit. It was done then because, it was simpler to do. For example, it is used to get list of editions when a work is loaded. Limit Solr only to search and figure out a better way to manage the other pieces, w…

Wonderful to see @raybb and @anandology in this thread.

Lots of operational challenges come up when running a service for 14M patrons.

And Open Library in particular has a handful of challenges. 1. It's database has grown significantly (800+ GB) and Anand is right that IO (even on SSDs) is a challenge. The `thing` (infobase/infogami) triple-store design is well thought out and gets us a lot, and any system has to be tuned as it scales to hundreds of millions of rows. One strategy here is being smarter about cache and also shifting some of the load from psql to solr. Rishabh and others volunteers have been amazing assets as we've moved in this direction. Jim Champ on staff has been helping me tune psql, pgbouncer, and some of our high IO crons to improve raw db performance. 2. Limited hardware resources. We're trying to move some of our services within the Internet Archive's kubernetes cluster and we've done a great job migrating towards a world where everything is dockerized. It used to be a very painful process for our team of 3 to handle server ops, upgrades, and networking for nearly 15 manually orchestrated servers. One of the bare-metal racks running much of Open Library is significanly oversubscribed on vCPUs and so moving services off to free space and eliminate steal is critical for us right now. Our main web server (ol-www0) suffers from up to 20% steal and we're seeing a lot of congestion before requests even get to our web nodes (app servers). We have a plan and it takes time. 3. Open Library is still dependent on Archive.org for many lookups -- like book availability (which Ben Deitch has been helping me and Drini move into solr). When there are network issues and a network requests takes 5+ seconds, every web.py worker on that thread grinds to a halt and Ray's work moving us to FastAPI has made a significant impact 4. Solr. Drini has been heroic at restructuring our setup to use replicated solr in a way that has increased performance and relieved some of the pressure on our main cluster. This was a huge bottleneck for us this time last year and we've taken a lot of steps to ameliorate our situation. See: https://blog.openlibrary.org/2025/09/12/open-library-search-... 5. Raw spikes in traffic. We are seeing massive amounts of traffic that slams our book pages, increasing the pain of all the above. It saturates our limited resources, puts more strain on our database, ties us web workers... It makes modsecurity even more expensive. Part of the solutions is being more clever about provisioning, part of the solution is using fail2ban to prevent bad traffic from subtracting from the experience of the patrons who depend on us. Part of the solution is caching and optimizing our database to scale with load.

There isn't just one solution and the same 3 engineers on staff (and the support of a completely stellar community of dedicated volunteers fellows and leads) are doing our best to balance ops improvements with the necessary "product" and design improvements necessary that ensure we're useful to people to begin with.

I hope this gives the world a bit more of a glimpse how we operate and what some of our challenges are. We're an open source project and our goal is to share as many learnings as we can and to build something useful, sustainable, and beneficial for the community at large.

Thank you Ray, Anand, Drini, Jim, Lokesh, Lisa, Charles, and so many dozens more for your tremendous work (present and past) and thank you for being in our corner.

Post reply on HN