Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reliability: It’s not great

community.fly.io

Post reply on HN
Print view View original

231–240 of 476 posts

Re: Reliability: It’s not great

#231

Post by otterley »

tptacek wrote: post #166
pyentropy wrote: post #152

Almost half of the issues are caused by their use of HashiCorp products. As someone that has started tons of Consul clusters, analyzed tons of Terraform states, developed providers and wrote a HCL parser, I must say this: HashiCorp built a brand of consistent design & docs, security, strict configuration, distributed-algos-made-approachable... but at its core, it's a very fragile ecosystem. The only benefit of HashiC…

We are asking to HashiCorp products to do things they were not designed to do, in configurations that they don't expect to be deployed in. Take a step back, and the idea of a single global namespace bound up with Raft consistency for a fleet deployed in dozens of regions, providing near-real-time state propagation, is just not at all reasonable. Our state propagation needs are much closer to those of a routing protoc…

Well, why did you do that? If you’d asked them whether this was a supported configuration or intended purpose, they’d have said no; and anyone who had experience deploying Consul at large scale would have told you the same.

There is truly no compression algorithm for experience.

Re: Reliability: It’s not great

#232

Post by mike_d »

trilobyte wrote: post #30
samwillis wrote: post #18

Fundamentally I think some of the problems come down to the difference between what Fly set out to build and what the market currently want. Fly (to my understanding) at its core is about edge compute. That is where they started and what the team are most excited about developing. It's a brilliant idea, they have the skills and expertise. They are going to be successful at it. However, at the same time the market is…

> generous free level of service, This is likely the biggest culprit for a lot of these companies. Too many of us have grown up in the culture of getting hosting and platform for "free", but at some point the companies providing it still have to pay the bills. There has to be a better pricing model that let's someone deploy their relatively small, low-traffic app for $10s/month or even $200 - $300 / year for the basi…

All of the forum posts and howtos on spinning up an AdGuard or similar service on Fly.io instead of a local Raspberry Pi probably didn't help things. While it does drive user growth, it isn't the type of users you want.

One of the hardest lessons every business needs to learn is how to say no to the users they don't need.

Re: Reliability: It’s not great

#233

Post by preseinger »

plasma wrote: post #222

Thanks for sharing! Would it help to replace Corrosion with a simpler "Here's my local known state" blob that is POST'd to blob storage (for example) on a major cloud provider, and have another service read that at intervals? Just to make it really simple. There will be a better way than that, but my thought is if you can make it simpler (known state is always just pushed, so missing updates auto-recovers and avoids…

+1 — service discovery _feels_ like something that gossip can solve, but that's only true at global/open/untrusted scale, in a well-scoped and authoritative domain like a company it's really not, it's really just another typical, expected-to-be-consistent state problem

Re: Reliability: It’s not great

#234

Post by aeyes »

chronark wrote: post #200
chime wrote: post #197

Earlier quoted context omitted.

They kind of offer that with their Redis (via Upstash). But for our use-case, we needed it to be managed PG and Redis. Going out of the LAN introduces too much latency.

Upstash Redis for Fly runs on Fly infrastructure and we observe latencies in the low single digit milliseconds.

Maybe that is what Fly had in mind: That some company like Upstash would bring "managed" serverless Postgres to their platform.

Unless somebody natively implements clustering in Postgres I don't see that happening anytime soon, all existing tools require way too many moving parts.

I don't know why they'd even want Postgres for the type of service they are offering, KV or maybe SQLite seem like a better fit.

Re: Reliability: It’s not great

#235

Post by mike_d »

jrochkind1 wrote: post #35

I remain kind of amazed about how heroku managed to pull off what they pulled off, in the first case. Also: > The Heroku exodus broke our assumptions. Pre-Heroku, most of the apps we were running were spread across regions. And: we were growing about 15% per month. But post-Heroku, we got a huge influx of apps in just a few hot spots — and at 30% per month. I hadn't before seen anyone with a big picture view confirm…

> I remain kind of amazed about how heroku managed to pull off what they pulled off

Heroku was built on top of AWS. They didn't have to handle many of the hard problems Fly does.

Re: Reliability: It’s not great

#237

Post by gizmo »

dadrian wrote: post #180
gizmo wrote: post #118

Earlier quoted context omitted.

It's strategic messaging. It can't be genuine, because of what it is. The benefit they get is publicity and damage control, and as you can tell by the many responses here, it buys them time because many developers are willing to give them the benefit of the doubt. Companies that engage in this kind of candor are careful not to disclose those things that would really hurt their business. Those things are still kept se…

Sorry, what? Do you expect that no company can think about what to write before they post it, or that any post about anything internal must cover all internal issues? Posts must be either all roses or a no-thought laundry list of everything bad?

Of course not. The point is that the reader should recognize corporate communication for what it is: fundamentally self-serving. Corporate communication should therefore be met with thoughtful skepticism, and not with naivety or cynicism.

Re: Reliability: It’s not great

#238

Post by notpushkin »

arjvik wrote: post #211
notpushkin wrote: post #184

Earlier quoted context omitted.

Dokku is also nice and battle-tested: https://dokku.com/ And may I also plug Lunni, a self-hosted Docker Swarm-based PaaS I'm working on right now: https://lunni.dev/ Both work pretty well on $5 servers.

Lunni has got an interesting concept — and I can actually see some good uses for it! Is the actual "production" workflow still pasting a Docker Compose file in? I would much rather have an automated deployment process that doesn't require human input, that way it can be scripted as part of CI/CD, etc. Personally, I fell in love with `git push production` (naming a git remote `production`) to trigger a deploy. Ironica…

I'll start with the Swarm since it's a major point actually: Docker's Swarm mode is comparable to Kubernetes or Nomad: you can launch a cluster of servers and run your application there.

Unlike Kubernetes or Nomad though, it uses mostly the same concepts Docker Compose does, to the point that your development docker-compose.yml file will likely just work there (with some minimal tweaks). I love this website that talks more about it: https://dockerswarm.rocks/

Edit: As opposed to `docker compose up`, when running on a single server: not much. It will restart on server reboot by default, and allow you to run multiple replicas of a service (deprecated in Docker Compose), but that's it. Most important though, it would allow you to add more nodes later on, and it will then scale your services across the whole swarm – so you can start with just one server and scale to hundreds if needed.

> I would much rather have an automated deployment process that doesn't require human input, that way it can be scripted as part of CI/CD, etc.

This is almost doable with Lunni. This guide will walk through setting up a CI for a typical webapp that packages it in a Docker image and pushes to a registry: https://lunni.dev/docs/deploy/from-git/ (currently for GitLab CI and GitHub Actions only)

As for the continuous delivery, we're gonna have a webhook that you can call when your CI pipeline is finished. It's not exposed in the UI yet but I'll try to prioritize it (now that I remember I wanted to do it :')

`git push production` feels a bit easier, but I'm a bit concerned about bloat: for this to work, we'll have to bundle some sort of CI and container registry with Lunni itself. I think sticking with third-party CI is a more elegant approach here. What do you think?

Re: Reliability: It’s not great

#239

Post by nomilk »

What an earnest post, and how damn refreshing it is to see such concern for users, accountability, honesty and openness (quite a contrast to another PaaS)!

I moved one app successfully from heroku to fly and attempted to move a few others. These are my experiences (both good and bad):

Great:

- The load time on the pages is insanely faster on fly than heroku. Sometimes I thought I was on the localhost version of the app, it was that snappy.

- Love that it uses a Dockerfile

- Love paying for what I use (compared to Heroku's rigid minimum of $16/month for hobby dyno w/ postgres for baby apps, or $34/month just to get a second web dyno for toddler apps). The same apps are Not great:

- I find the fly.toml file hard to understand and use, and the cycle time slow to fix or tinker with it. It's partly (entirely?) a 'me' problem because I haven't spent a huge amount of time reading the documentation.

- I found scheduling a rake task in a rails app time consuming (~days) the first time, but very easy (15 minutes) the second and subsequent times, once I knew a way that worked (cron didn't work; had to use a tool I hadn't used before 'supercronic').

- Deploys sometimes time out with `Error failed to fetch an image or build from source: error rendering push status stream: EOF`. Most layers copied, but randomly, some layers wouldn't. All I could do is keep trying until it worked, which it did, 2 hours later. Not the end of the world, but an annoying complication when you're already trying to solve complex problems.

- I followed a youtube video on how to move a rails app from heroku to fly, and it worked on a modern app, but I couldn't quite get fly happy when moving the older app - something to do with postgres versions, and I didn't want to spend all day figuring it out. I'm not hugely experienced with docker, it could have been an easy fix for someone more experienced.

On reflection, 3 of the 4 negatives above are solvable by me reading the docs more thoroughly and getting more proficient with docker.

I look forward to continuing using and exploring fly, and can't be happier with the directness, transparency and care from fly staff. A platform with huge potential.

Re: Reliability: It’s not great

#240

Post by machiaweliczny »

ChrisMarshallNY wrote: post #94

Well, I feel for them. Scaling up is a bitch. I've been lucky, in the past, but a lot of that, is because I have "overengineered," and the tools/frameworks have advanced to meet the new demand. I am in the middle of a complete, bottom-to-top rewrite of the app we've been developing for the last couple of years. It's going great, but making this leap was a fraught decision. It's mainly, so I wouldn't have to write a p…

Don’t know about expected usage but 10K rows seems like low number to me. If you mean 10K req/s then perf starts to matter but usually it’s SQL queries that fail first if you have 100K+ rows. In general good caching solves most of stuff + read/write separation. That’s all from my poor experience.

What I mean is that these scaling problems don’t have much to do with app logic but having nice core is good so :thumbsup:

Post reply on HN