71–80 of 241 posts
What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…
I had nothing but respect for the whole team. Dedicating their time to build everything from scratch, not to mention that they maintain everything for free.. It's a cool project, not sure if there's a way for anyone to contribute. I"ll join the discord afterwork to see if they need any extra hand. Gee, how do these people find other people online to work on all of the cool projects. I would love to join rather than p…
Okay, but isn't most of their content stolen? Why would you want to contribute to that?
Earlier quoted context omitted.
K isn't the abbreviation for kilo, so if you're going to rag on the fellow for the 'b', then you should at least be asking what a Kelvin*bit is.
640KiB is very little and I'm wondering if it's a typo, given that the servers had 1-2TiB available. Postgres 9.0 released in 2010 already had 32MiB as the default for shared_buffers (with a minimum of 128KiB): https://www.postgresql.org/docs/9.0/runtime-config-resource.... and 8.1 released in 2005 used 8MB (1000*8KiB): https://www.postgresql.org/docs/8.1/runtime-config-resource....
but, wtf do i know, i'm the crazy guy who tries to interpret comments generously.
Not sure I'm missing something here. Surely you could sample some prod traffic and then replay it with one of the many load test tools out there. You might lose in the geographical distribution, but load testing a web server with 2k TPS sounds a bit trivial.
I had nothing but respect for the whole team. Dedicating their time to build everything from scratch, not to mention that they maintain everything for free.. It's a cool project, not sure if there's a way for anyone to contribute. I"ll join the discord afterwork to see if they need any extra hand. Gee, how do these people find other people online to work on all of the cool projects. I would love to join rather than p…
Okay, but isn't most of their content stolen? Why would you want to contribute to that?
Earlier quoted context omitted.
Does it serve 20-40 hi resolution images and uploads per user?
I am not sure how to interpret this para: > In practice, we currently see peaks of above 2000 requests every single second during prime time. That is multiple billions of requests per month, or more than 10 million unique monthly visitors. And all of this before actually serving images. If I am reading that correctly, 2000r/s does not include images, and makes it unclear if $1500/month does.
It's a nice article, I guess, but the site is down (the one discussed in the article, not the blog post itself) for me.
You probably have Verizon - they've started null routeing traffic to sites "like this". https://old.reddit.com/r/mangadex/comments/nvj7qf/is_verizon...
That's disappointing. If only I had some choice to ISPs, then I could express my disappointment by voting with my wallet…
I've done things at scale (5-10K req/s) on a budget ($1000 USD) and I've done things at much smaller scales that required a much larger budget. _How_ you hit scale on a budget is one part of the equation. The other part is: what you're doing. Off the top of my head, the "how" will often involve the following (just to list a few): 1 - Baremetal 2 - Cache 3 - Denormalize 4 - Append-only 5 - Shard 6 - Performance focuse…
I've done things at scale (5-10K req/s) on a budget ($1000 USD) and I've done things at much smaller scales that required a much larger budget. _How_ you hit scale on a budget is one part of the equation. The other part is: what you're doing. Off the top of my head, the "how" will often involve the following (just to list a few): 1 - Baremetal 2 - Cache 3 - Denormalize 4 - Append-only 5 - Shard 6 - Performance focuse…
The 1-7 list you mention definitely deserves it’s own blogpost and how to implement these. I’m currently not using any of these except 1, and probably don’t need the rest for a while but I do want to know what I should do when I need it. For example: what and how should things be cached? When and how to denormalize, why is it needed? Why append-only and how? Never ‘sharded’ before, no idea how that works. Heard some…
Sharding sucks, but if your database can't fit on a single machine anymore, you do what you've got to do. The basic idea is instead of everything in one database on one machine (or well redundant group of machines anyway), you have some method to decide for a given key what database machine will have the data. Managing the split of data across different machines is, of course, tricky in practice; especially if you need to change the distribution in the future.
OTOH, Supermicro sells dual processor servers that go up to 8 TB of ram now; you can fit a lot of database in 8 TB of ram, and if you don't keep the whole thing in ram, you can index a ton of data with 8 TB of ram, which means sharding can wait. In contrast, eBay had to shard because a Sun e10k, where they ran Oracle, could only go to 64 GB of ram, and they had no choice but to break up into multiple databases.