411–420 of 499 posts
> Appears that the router in Atlanta announced bad routes (effectively a route leak). Only impacted our backbone. Not all of our PoPs are connected to our backbone, so some would not have seen an issue. Appears to have impacted about 50% of our traffic for a bit over 20 min.
Earlier quoted context omitted.
Eggs and baskets etc etc
Good DNS practice (at least when I did system admin 10 years ago) was ALWAYS having a secondary at some other location/network. Why do we just put some info in Cloudflare and call it good these days?
Earlier quoted context omitted.
Tested with tor and it's right. Some exit nodes aren't affected.
Hm, maybe it's just the SRE in me talking, but if major chunks of the internet being entirely inaccessible doesn't count as an "outage", what does?
Earlier quoted context omitted.
Don't think so. They used to use Cloudflare but stopped. To my knowledge, it's a single server without a database (using the filesystem as a database).
So HN is serving 5.5M page view daily (excluding API access ) on a single server without CDN and without a database? Holy crap I am thinking either there is some magic or everything we are doing in the modern web are wrong. Edit: The number is from Dang [1] > These days around 5.5M page views daily and something like 5M unique readers a month, depending on how you try to count them. [1] https://news.ycombinator.com/i…
Earlier quoted context omitted.
Don't think so. They used to use Cloudflare but stopped. To my knowledge, it's a single server without a database (using the filesystem as a database).
So HN is serving 5.5M page view daily (excluding API access ) on a single server without CDN and without a database? Holy crap I am thinking either there is some magic or everything we are doing in the modern web are wrong. Edit: The number is from Dang [1] > These days around 5.5M page views daily and something like 5M unique readers a month, depending on how you try to count them. [1] https://news.ycombinator.com/i…
Earlier quoted context omitted.
Don't think so. They used to use Cloudflare but stopped. To my knowledge, it's a single server without a database (using the filesystem as a database).
So HN is serving 5.5M page view daily (excluding API access ) on a single server without CDN and without a database? Holy crap I am thinking either there is some magic or everything we are doing in the modern web are wrong. Edit: The number is from Dang [1] > These days around 5.5M page views daily and something like 5M unique readers a month, depending on how you try to count them. [1] https://news.ycombinator.com/i…
I don't know what the server's specs are but I'm sure it must be quite beefy and have quite a few cores, so let's say that it runs about 10 billions instructions per second. That means a budget of about one million instructions per page load in this pessimistic estimate.
The original PlayStation's CPU ran at 33MHz and most games ran at 30fps, so about 1million cycles per fully rendered frame. The CPU was also MIPS and had 4KiB of cache, so it did a lot less with a single cycle than a modern server would. Meanwhile the HN servers has the same instruction budget to generate some HTML (most of which can be cached) and send it to the client.
A middle of the line modern desktop CPU can nowadays emulate the entire PlayStation console on a single core in real time, CPU, GPU and everything else, without even breaking a sweat.
>Holy crap I am thinking either there is some magic or everything we are doing in the modern web are wrong.
Magic, clearly.
We can't keep going on like this. The vulnerability of centralised internet infrastructure is a huge problem for everyone. Somebody, somewhere, really ought to sort it all out
10-20 minute router misconfigurations and subsequent fixes are sometimes a fact of life. big network infrastructure is complicated, and sometimes the best laid route tables of mice and men do go abloop and die. Outages happen no matter what the infrastructure is. There's no solution, they're just something you need to recognize and handle, which Cloudflare seemingly did relatively quickly here.
Level 3 or Telia going offline is perfectly survivable for any customer who has multiple upstreams.
Earlier quoted context omitted.
the DNS itself is not as easy to duplicate across multiple provider, with CF DNS down having a backup CDN wouldn't have helped
This isn't true... you can certainly do redundant dns with automatic failover between providers. Just set up NS records pointing to different providers.
We were down (downforeveryoneorjustme.com) completely, but back up now (as of a few minutes ago). Our domain wasn't even resolving; we use Cloudflare for frontend and DNS. We had a surge of people checking if Discord was down on our site, then I noticed everything went down shortly after. Discord is still the top check right now. I can't ever remember hitting these kind of traffic numbers before.