Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What software engineers can learn from the rapid collapse of Fast

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

391–400 of 482 posts

Re: What software engineers can learn from the rapid collapse of Fast

#391

Post by namdnay »

pyb wrote: post #218
maybetoofast wrote: post #207

Earlier quoted context omitted.

I joined Fast towards the end of last year. I didn't get as much exposure to the rest of the company as parent comment, but I'll say I saw a weird air of complacency and assumed success. I first ignored my instincts that something was wrong, but as I saw more of how the company operated, I became convinced that the single largest problem the company had was the culture. On the surface, I agree with the parent comment…

Good point, I am always surprise when well funded startups hire former big-tech engineers, rather than experienced startup engineers.

> I am always surprise when well funded startups hire former big-tech engineers, rather than experienced startup engineers

i'd say that if you plan on scaling, you need a bit of both. a lot of tricky scale/process/architecture questions have effectively been "solved" in big orgs, it's useful to have people who can bring this experience

to take a completely random examples: setting up a performance improvement plan, performing incident post-mortems, organising oncall rotas, branching strategies when you have a mix of small customers on standard product and big accounts with customisations etc

Re: What software engineers can learn from the rapid collapse of Fast

#392

Post by JumpCrisscross »

b20000 wrote: post #312
JumpCrisscross wrote: post #303

Earlier quoted context omitted.

> How on Earth did the company raise so much money with so few milestones of any kind being achieved? Fast pitched itself as taking the prudent path compared with e.g. Bolt. The latter went for big clients first. Fast started with small businesses. There were clearly other issues. But the closing era of cheap capital favoured the bold, and Bolt’s seizure of the fertile low grounds cut off Fast from its growth. Despit…

this doesn't answer the question at all, it's just a bunch of bullshit words strung together. we never raised any money, and we had more to show than Fast although we were in a completely different industry. this goes to show that all that matters is knowing people, it doesn't actually matter whether you have a real business and have achieved anything at all.

> we never raised any money

Who? (Also, given an open market, as was the case when the Amazon patent expired in 2017, the company fundraises to scale will beat its more timid competitor. This is basic strategy.)

If you were in a different industry…great execution in the wrong industry is as good as a Ferrari in a swamp.

> all that matters is knowing people

Sounds like you’ve been through some shit. I’m sorry for that.

The point I was making, summarily, is that Bolt ate Fast’s lunch. It went straight for the big customers. Fast wanted to grow towards them incrementally. In this story, if anyone had a fundraising connections advantage, it was Fast—they were Stripe backed. Bolt outmanoeuvred them (and was lucky).

This is unorthodox strategy. Fast played it conservatively. Bolt swung for the fences. Curiously, Bolt landed the home run.

Re: What software engineers can learn from the rapid collapse of Fast

#393

Post by kragen »

zozbot234 wrote: post #132
lolsoftware wrote: post #33

Earlier quoted context omitted.

The TailScale folks caught a bunch of flak for their "do things that don't scale" approach to databases. But, honestly, most startups would be better off following that approach than what Fast did. Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had.

Tailscale's "database from 2022" is neither here nor there. What you should do in this situation is use the most boring tech, such as PostgreSQL with a simple HA setup. Thus avoiding the "play with nifty toys" trap and maximizing the chance of making appropriate choices.

Your server has 128 gibibytes of RAM, and has for five years now. In fact, you can buy off-the-shelf servers with a tebibyte of RAM now. As long as you have some way to recover from a power failure, if there's something you could do with Postgres without sharding fifteen years ago, you can do it in RAM on a single server now. But, if it was disk-bound, you can do it about a million times faster than you could do it then.

It's reasonable to argue that shelling out the US$800 for a server with 128 gibibytes of RAM (or renting one from a cloud provider for US$150 a month) amounts to buying nifty toys for the engineering department. But if you're paying each engineer US$200k per year in salary and a similar amount in benefits, that's US$190 an hour, so each engineer's workday buys you almost four such servers. Your RAM has a minimum transaction time of about 100 ns, while an SSD is more like 10000 ns. If you have 100+ GiB of data to query, the alternative to buying nifty toys is paying the engineering department to do optimization work that wouldn't be necessary if your data was stored in memory that was 100x faster.

The rough equivalency here is that adding one more engineer to your team costs as much as adding 16 dedicated Xeon servers at Hetzner with 128 GiB of RAM each.

Sometimes the best answer is still a relational database, of course! It's often easier to write your queries in relational terms than by looping over hash tables, though probably not if you're using an ORM, and Postgres is delighted to cache all your data in RAM.

At some point, you may grow beyond being able to run your production environment on a single server, at which point splitting things into tiers is worthwhile. (In some cases you start at that point, for whatever reason, but a lot of web services can serve a million users on a single server. It sure sounds like Fast's service would have worked fine on a single server.)

Re: What software engineers can learn from the rapid collapse of Fast

#394

Post by nicksiscoe »

I don't think it's fair to assign blame to "positivity" and "optimism" as others in the comments have.

It takes an incredible amount of positivity to start or join a small startup. You have to believe that you/your coworkers will accomplish something that very few people have done successfully, and that requires a lot of confidence when things look gloomy.

As a total outsider here, it sounds like most ex-Fast employees are very talented and positive people. Perhaps the problem was not overall "company positivity" so much as it was executive leadership being naive and unwilling to A) identify when serious pivots/changes needed to be made and B) actually make those pivots.

Positivity is good, but not when it translates to unchecked naiveté. If you believe too strongly in your ability to succeed, you might start rejecting any signs of failure as they crop up. When something goes wrong or a mistake is made, you might find a scapegoat or otherwise discount the severity of the problem.

The other "reason for failure" might simply be that... well... startups are hard. Sometimes it just doesn't work. There might not be anyone to blame.

Re: What software engineers can learn from the rapid collapse of Fast

#395

Post by zozbot234 »

kragen wrote: post #393
zozbot234 wrote: post #132

Earlier quoted context omitted.

Tailscale's "database from 2022" is neither here nor there. What you should do in this situation is use the most boring tech, such as PostgreSQL with a simple HA setup. Thus avoiding the "play with nifty toys" trap and maximizing the chance of making appropriate choices.

Your server has 128 gibibytes of RAM, and has for five years now. In fact, you can buy off-the-shelf servers with a tebibyte of RAM now. As long as you have some way to recover from a power failure, if there's something you could do with Postgres without sharding fifteen years ago, you can do it in RAM on a single server now. But, if it was disk-bound, you can do it about a million times faster than you could do it t…

The point of a RDBMS has never been to save on RAM, raw files have undesirable properties for lots of data management scenarios. You can have a very simple schema if you like, e.g. a RDBMS makes for a damn good key-value store.

Re: What software engineers can learn from the rapid collapse of Fast

#396

Post by [deleted] »

rockostrich wrote: post #74
mwcampbell wrote: post #20

I think the most applicable warning for engineers when it comes to the actual work, as opposed to whether one should join a particular startup, is this: > Engineers calculated the load Fast had in needing to serve their traffic. The Fast button was rendered less than 500,000 times per day - rarely needed to ever serve more than a few requests per second. > One of the few warning signs engineers noticed is how Fast sp…

This is pretty hilarious to read. I did an interview screen with them at the end of last year for a data engineering position (former coworker was high up on the design side of things and he gave a referral). The first bit of the technical question was just "write a SQL query" and boiled down to doing a group by, count, and limit. The second was something like "what if we needed to return this query from a distribute…

[deleted]

Re: What software engineers can learn from the rapid collapse of Fast

#397

Post by zild3d »

l2silver wrote: post #229

Am I crazy, or is the answer "Nothing"? The product was built, they just couldn't sell it fast enough. Engineering culture had nothing to do with this failure. And if they had landed one of those big clients, they would be kicking ass. You never know how it is going to go. Sometimes you go after the small sales and die. Sometimes you go after the big ones and live.

> Engineering culture had nothing to do with this failure.

Why keep hiring so many engineers when they aren't needed yet? Sounds like a cultural issue to continue hiring so aggressively when it's only resulting in way higher burn than you can support

Re: What software engineers can learn from the rapid collapse of Fast

#398

Post by hef19898 »

namdnay wrote: post #391
pyb wrote: post #218

Earlier quoted context omitted.

Good point, I am always surprise when well funded startups hire former big-tech engineers, rather than experienced startup engineers.

> I am always surprise when well funded startups hire former big-tech engineers, rather than experienced startup engineers i'd say that if you plan on scaling, you need a bit of both. a lot of tricky scale/process/architecture questions have effectively been "solved" in big orgs, it's useful to have people who can bring this experience to take a completely random examples: setting up a performance improvement plan, p…

The risk, and I see that currently in non-software functions, is hiring people from big corps that know how to manage the people that solved those scale/... problems instead of the problems themselves. With the result that those senior people default back to building an organization that looks like the one they are used to. Which doesn't solve a single one of those solved problems.

Quite likely to happen of the founders / existing hiring managers have no idea themselves how big corp solved these problems.

Re: What software engineers can learn from the rapid collapse of Fast

#399

Post by matwood »

pyb wrote: post #218
maybetoofast wrote: post #207

Earlier quoted context omitted.

I joined Fast towards the end of last year. I didn't get as much exposure to the rest of the company as parent comment, but I'll say I saw a weird air of complacency and assumed success. I first ignored my instincts that something was wrong, but as I saw more of how the company operated, I became convinced that the single largest problem the company had was the culture. On the surface, I agree with the parent comment…

Good point, I am always surprise when well funded startups hire former big-tech engineers, rather than experienced startup engineers.

Yeah...it's often a different type of person who goes from 0->1 than from 1->100. Neither type is better than the other, just different.
Post reply on HN