Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io outage – resolved

status.flyio.net

Post reply on HN
Print view View original

221–230 of 287 posts

Re: Fly.io outage – resolved

#221

Post by pier25 »

infecto wrote: post #204
k__ wrote: post #168

Earlier quoted context omitted.

It's basically the new Heroku with less lock-in, because it works with Docker. You get edge computing, autoscaling, and load balancing without additional configuration. Not as flexible as AWS, but also much easier to setup and maintain. But the reliability issues suck now and then.

I have asked this multiple times but is anyone really using edge compute and getting value out of it? I am certain there are cases but I have not seen any of them written up before.

We have an embeddable audio player served globally with very low latency. This wouldn't be possible without edge compute/data.

Re: Fly.io outage – resolved

#222

Post by [deleted] »

cryptos wrote: post #162

Fly.io seems to be a bit of a mixed bag: https://news.ycombinator.com/item?id=41917436 https://news.ycombinator.com/item?id=35044516 https://news.ycombinator.com/item?id=34742946 https://news.ycombinator.com/item?id=34229751 If a cloud platform doesn't really provide reliability, I'd say it's probably not worth it. You could better just rent a (virtual) server and save the cloud tax.

[deleted]

Re: Fly.io outage – resolved

#223

Post by hansvm »

huijzer wrote: post #170
cryptos wrote: post #162

Fly.io seems to be a bit of a mixed bag: https://news.ycombinator.com/item?id=41917436 https://news.ycombinator.com/item?id=35044516 https://news.ycombinator.com/item?id=34742946 https://news.ycombinator.com/item?id=34229751 If a cloud platform doesn't really provide reliability, I'd say it's probably not worth it. You could better just rent a (virtual) server and save the cloud tax.

For experiments and hobby projects the value proposition is amazing. Where else can you spin up an independent instance for $1.94 per month?* *Note this is for an instance with only 256MB RAM ( https://fly.io/docs/about/pricing/ ), but it's definitely possible to run non-trivial projects on that. Rust-based web servers like Rocket require only about 10MB RAM. Basic PHP servers should also fit from what I can find.

I used to use Racknerd for that sort of thing, and the costs were around there -- maybe $1.90/mo for a 512MB instance. It was easy to squeeze several hobby projects onto the machine.

Re: Fly.io outage – resolved

#224

Post by treesknees »

TiredOfLife wrote: post #185
huijzer wrote: post #170

Earlier quoted context omitted.

For experiments and hobby projects the value proposition is amazing. Where else can you spin up an independent instance for $1.94 per month?* *Note this is for an instance with only 256MB RAM ( https://fly.io/docs/about/pricing/ ), but it's definitely possible to run non-trivial projects on that. Rust-based web servers like Rocket require only about 10MB RAM. Basic PHP servers should also fit from what I can find.

Oracle free is one 4 core 24gb ram vps + 2 dualcore amd vps.

And actually, it's the resources that are free (CPU, memory, network) and you're allowed to split them up into multiple VMs if you want to.

One of my VMs had an uptime of more than 1050 days before the infrastructure rebooted it, so in terms of availability they've certainly surprised me.

The only downside I've come across with Oracle Free is that the 'best' regions are typically full. I ended up provisioning my free VMs in another region/country and it works fine.

I suppose another downside (if you want to view it this way) is they will delete idle unused free VMs after a certain time period. You have to add a credit card to your account to "upgrade" your account and run free resource indefinitely. While you're not charged for anything, it makes me nervous forking over a CC number to Oracle.

Re: Fly.io outage – resolved

#225

Post by SteveNuts »

latch wrote: post #99
duxup wrote: post #13

Earlier quoted context omitted.

When I worked for a company who worked with big banks / financial institutions we used to run disaster recovery tests. Effectively a simulated outage where the company would try to run off their backup sites. They ran everything from those sites, it was impressive. Once in a while we'd have a real outage that matched the test we ran as recently as the weekend before. I was helping a bank switch over to the DR site(s)…

In most BIG banks, "Vice President" is almost an entry-level title. Easily have 1000s of them. For example, this article points out that Goldman Sachs had ~12K VPs out of more than 30K employees: https://web.archive.org/web/20150311012855/https://www.wsj.c...

Just like all Sales folks have heavily inflated titles, no customer wants to think they're dealing with a junior salesperson/loan officer when you're about to hand over your money.

It seems like every vendor sales team I work with is an "executive" or "director of sales" even though in reality they're just regular old salespeople.

Re: Fly.io outage – resolved

#226

Post by bdcravens »

ignoramous wrote: post #122
jart wrote: post #92

fly.io publishes their post-mortems here: https://fly.io/infra-log/ The last post-mortem they wrote is very interesting and full of details. Basically back in 2016 the heart or keystone component of fly.io production infrastructure was called consul, which is a highly secure TLS server that tracks shared state and it requires that both the server certificate and the client certificate be authenticated. Since it was c…

On that Consul outage, Fly Infra concludes, "The moral of the story is, no more half-measures." On their careers page [1], the Fly team goes, "We're not big believers in tech debt." As an outsider, reads like a cacophony of contradictions? [1] https://fly.io/docs/hiring/working/#we-re-ruthless-about-doi...

"full measures" aren't the same thing as tech debt. Complexity isn't even the same thing as tech debt.

Re: Fly.io outage – resolved

#227

Post by willsmith72 »

ploxiln wrote: post #107
xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

I think you can't avoid the fact that these holiday weeks are different from regular weeks. If you "change freeze" then you also freeze out the little fixes and perf tuning that usually happens across these systems, because they're not "critical". And then inevitably it turns out that there's a special marketing/product push, with special pricing logic that needs new code, and new UI widgets, causing a huge traffic/l…

Any big tech company with large peak periods disagrees with you. It's absolutely worth freezing non-critical changes.

Urgent business change needs to go through? Sure, be prepared to defend to a vp/exec why it needs to go in now.

Urgent security fix? Yep same vp will approve it.

It's a no-brainer to stop your typical changes which aren't needed for a couple of weeks. By the way, it doesn't mean your whole pipeline needs to stop. You can still have stuff ready to go to prod or pre prod after the freeze

Re: Fly.io outage – resolved

#228

Post by ignoramous »

mattgreenrocks wrote: post #220
Aeolun wrote: post #160

Earlier quoted context omitted.

Two contradictory statements do not read like a 'cacophony' of anything to me xD I think you need a whole lot more than two to do that word justice.

The comment section doing what it does best!

For brevity I chose to put up only the conclusion from a postmortem (of which I've read plenty by now) and another point from their otherwise comparatively shorter careers page, which imo capture the inherent tension between building out fast & building out right. This is not something I've started complaining about today or yesterday. I've used Fly in prod for 4 years and spilled much ink on this topic on their forums already. Even if I critique, I remain optimistic about Fly despite the seemingly endless list of failure modes building such complex systems entail: https://community.fly.io/t/fly-down/10224/15

(personally speaking, I'm humble enough because I can hardly build a toy side-project right!)

Re: Fly.io outage – resolved

#229

Post by TacticalCoder »

davidgl wrote: post #169
benhoyt wrote: post #8

My fly.io-hosted website went down for 5 minutes (6 hours ago), but then came right back up, and has been up ever since. I use a free monitoring service that checks it every 5 minutes, so it's possible it missed another short bit of downtime. But fly.io has been pretty reliable overall for me!

Same for us, down for ~5 mins, back up and fine, error was 501

Someone said 16 minutes: so it's not even 5 nines service.

Re: Fly.io outage – resolved

#230

Post by sofixa »

encom wrote: post #215
pc86 wrote: post #212

Earlier quoted context omitted.

Now just to worry about GCP getting shut down with a few days' notice. /s But in all seriousness the gall to raise prices before actually fixing the reliability problems is pretty shocking. I understand it's a bit of a chicken-and-egg thing where you maybe are tight on resources but there's no scenario where it's acceptable to have a product with these kinds of problems and then raise prices on existing customers who…

No /s is needed. Relying on any Google product long term is crazy.

Google's b2b products are relatively stable (relative to their b2c free services). You generally get somewhere like a year of notice if they shut it down.
Post reply on HN