[deleted]
Google Cloud networking issues in us-east1
291–300 of 341 posts
Re: Google Cloud networking issues in us-east1
#292Wow. GCP is always a networking issue. Their QA on networking changes needs work. Maybe they should spend 20% on it.
Re: Google Cloud networking issues in us-east1
#293Earlier quoted context omitted.
It is - but all companies I’ve ever worked for are. I’m not convinced the letter of the policy is much different.
There are definitely major differences between the FAANGs: what was the last time you saw an Apple employee commenting on anything on HN?
Apple employees comment on Hacker News all the time: they just don’t identify themselves as speaking for the company and make sure to only talk about publicly available information.
Re: Google Cloud networking issues in us-east1
#294Earlier quoted context omitted.
I work for AWS. There is typically a balance that has to be struck when sharing information with customers. I would imagine this goes for most companies, which is why it isn't until a post-mortem that the messaging is fully refined.
True, but I'd argue that the "Customer Obsession" priciple would drive you to attempt some sort of good-faith effort towards real-time communication. Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. Service teams don't have the time or incentive to give real-time updates. Why…
> Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed.
It has.
> Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates?
The escalation team inside PS now drafts customer messaging within ~5 minutes of the impact being identified (usually about 5 minutes into an event) and if the impact is significant enough to post to the public dashboard, than may take another 5 minutes. Depending on the type of impact, affected customers will be notified via the personal health dashboard.
PS owns the tooling that does this, and is responsible for driving the process, but the service org's (e.g. EC2, S3 etc.) representative often makes the call on whether to post to the public status page or not (depending on the scale of the impact, e.g. 20% API failure rate for 5% customers probably won't make the status page, but affected customers will get notices). TT is almost out ... but the PS tooling supports it and its replacement, and provides easy access and summaries for internal teams (so you don't need to refresh TT or subscribe to the ticket just to see what the status is).
Re: Google Cloud networking issues in us-east1
#295Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.
Seems like if it continues to be a problem that more multi-cloud solutions will present themselves (Terraform does that sort of thing, right?).
Terraform does not handle data locality. Since compute generally sits next to data for latency and cost reasons one should first think about how to ensure that their (perhaps considerable) data set is stored and synchronised elsewhere before worrying about which infrastructure manifest tool to use.
Re: Google Cloud networking issues in us-east1
#296Earlier quoted context omitted.
It can be hard. Getting redundant separated paths under/over railroad tracks, for example, might require political power that not everyone has. Google, of course, has plenty.
> Getting redundant separated paths under/over railroad tracks, for example, might require political power that not everyone has. Google, of course, has plenty. But Google's vendors might have less. One would hope that Google is auditing claims of independence from vendors at least somewhat, but at some level they have to rely on vendor representation and SLAs if they aren't going to do it all themselves.
The companies who operate the cross-country backbone fibers have independently verified fibre maps and you can also audit them with their cooperation. And those who operate last-mile metro networks are usually highly reputed ISPs (at least in India where there is decent competition in this space) who have a lot to lose if their reputation is damaged. Also, the community of their customers is small and they all talk to each other. So it is hard to make fake claims and get away with it. Usually, when cable cuts happen, it is more a question of whose traffic is rerouted on the available paths and whose traffic is dropped. If you are high-paying customer with strong SLAs then your traffic is usually safe and will displace a lower SLA customer's traffic. You will notice latency spikes due to rerouting and maybe temporary glitches w.r.t link stabilization. Since you see this so often, your BGP timers etc are all tuned to be patient and avoid cascading failures.
Re: Google Cloud networking issues in us-east1
#297Earlier quoted context omitted.
I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operati…
Curious how you would imagine handling something like GDPR or SOX compliance in this alternative world you’re proposing. You can’t magically foresee new requirements and new implied complexity for all future time.
To many European companies GDPR did not really change the operational requirements - only the penalties for not meeting them.
Re: Google Cloud networking issues in us-east1
#298Earlier quoted context omitted.
I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operati…
I got an email yesterday that told me the boring old HPUX server (which was racked before my intern was even born) barfed all over its boring old 50-pin SCSI drive and ops went scrambling to find one in storage so the boring old Oracle DB that was responsible for production lines running could be recovered. Took us around an hour, cost us a boring 5 figures. Luckily our sysadmin knows how to hide “unused parts” for d…
Sounds like you were bitten by the technical debt that was created by placing an important database behind a single point of failure - a single physical server. Of course something would break sooner or later, PA-RISC and Itanium servers were great, but still not magic.
Important things go onto clusters, or at least have a (hot or cold) standby server.
Re: Google Cloud networking issues in us-east1
#299Earlier quoted context omitted.
I don’t think it’s happened (yet) although some of the earlier outages when AWS was younger were pretty far reaching. I think all of S3 has gone down a time or two.
All of S3 has, but that’s because S3 had a single choke point in a single region for a long time.
> All of S3 has, but that’s because S3 had a single choke point in a single region for a long time.
The only S3 event here was limited to us-east-1: https://aws.amazon.com/premiumsupport/technology/pes/
Some APIs were impacted, because they are global by nature (e.g create-bucket). But S3 was working fine in all other regions, for existing buckets.
However, many websites were affected, because they didn't use any of the existing S3 features that allow for regional redundancy, simply because S3 had been so reliable they didn't know/think they needed to have critical assets in a bucket in a 2nd region that they could fail over to.
Admittedly, even the AWS status page was impacted, because it also relied on S3 in us-east-1.
S3 has done a lot of work to improve matters since, and mechanisms have been put in place to ensure that all AWS services don't have inter-region dependencies for "static" operation.
However, it is still incorrect to claim that it was all of S3. Many customers who use S3 only in other regions were totally unaffected.
Re: Google Cloud networking issues in us-east1
#300Earlier quoted context omitted.
Seems like if it continues to be a problem that more multi-cloud solutions will present themselves (Terraform does that sort of thing, right?).
What do people do to mitigate DNS services from going down? Is it possible to have multiple services for that? And CDN's too as per our recent CloudFlare issues.
DNS caching combined with multiple servers makes it one of the most reliable services by design.
For small hobby projects I simply use a 3rd party 2ndary DNS service.