Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building for the 99% Developers

future.a16z.com

Post reply on HN
Print view View original

71–80 of 310 posts

Re: Building for the 99% Developers

#71

Post by JimmyRuska »

scarface74 wrote: post #61
JimmyRuska wrote: post #48

Earlier quoted context omitted.

In a past life, we were reading from twitter firehose and the offshore implemented a solution that spawns a large spark cluster to read from twitter, dumps it to kinesis, then has a cloud function that writes it to dynamo, then merged it into redshift. It worked poorly, took months to build, and lost data. I implemented a python script that read from twitter firehose and wrote directly to redshift. It was about 50 li…

You wrote individual insert statements to RedShift ?

It would buffer the data and flush to s3 every few minutes, then run redshift s3 imports.

Re: Building for the 99% Developers

#73

Post by travisgriggs »

oxplot wrote: post #62

Here are my rules for picking framework/languages: - Avoid if specifically mentions big companies as users: Facebook, Twitter, Google, these have as many devs as needed to throw at the most trivial tasks, and then some. I am in a team of 3 that need to get shit done, not chase package manager and compiler throw-ups. Tell me your language is used by a one person show to serve millions of people and then I'll listen. -…

Would you be willing to share some of the languages/frameworks that have made the cut or washed out for you based on the rubric you’ve given. I’m honestly curious.

Re: Building for the 99% Developers

#74

Post by urthor »

latch wrote: post #52
adra wrote: post #46

Earlier quoted context omitted.

Scale vertically and you're still a single point of failure away from a big, potentially fatal outage. The coding and architecture can often be trivial though. Definitely good for internal systems and more green programmers. Scale horizontally and you're often constraining and or complicating your design/architecture, but you can handle outages far better, if not seamlessly. It's much harder to get right, so it's usu…

It's wrong to mix horizontal scaling with availability. They are two different things that solve two different problems and solving one doesn't necessarily mean you've solved the other. You can scale horizontally in a way that decreases your total availability, e.g. by sharding.

Essentially.

Limited horizontal replication is not the same as PAXOS.

Re: Building for the 99% Developers

#75

Post by deathanatos »

alberth wrote: post #54
deathanatos wrote: post #51

Earlier quoted context omitted.

You're not the only commenter saying this, so I gotta ask: whatever VM that SQLite DB is on, is your business cool with a business disruption, dataloss, or both when that VM goes down, when that AZ goes down, or that disk fails? Or am I missing something? If you have some sort of fail-over-to-last-backup plan, is that not just a distributed database with more steps, and why not something like RDS, or CockroachDB? (Fr…

Mentioning RDS is an apples to oranges comparison. RDS is a fully managed service by a 3rd party. Nevertheless, checkout https://litestream.io - given that SQLite is just a file, it’s incredibly easy to replicate. Failover is coming in the next release, which is already beta. Also checkout this Expensify blog post on how they use SQLite. It’s scales amazingly well. https://blog.expensify.com/2018/01/08/scaling-sqlite…

I mention RDS because it mostly solves the failover/continuity problem. While failovers on RDS are typically outages, they're also typically quite short lived, no more than a few minutes. That can work for many people/business's requirements. That it is managed means you (in theory) can rely on AWS to manage the underlying infrastructure. You just worry about the SQL. It is a "boring technology" choice, as the article that HN likes to post every now and then says.

Litestream: What are the durability guarantees? (Are there any?) what would make me pick that over RDS or Cockroach? "given that SQLite is just a file, it’s incredibly easy to replicate" — sorry, that just doesn't click with me. What about a file is "inherently easy to replicate"? I would not know how to implement something that wouldn't be O(n) with the filesize. Ideally you'd just sync up changes, but how to get those? Normal DBs with streaming WALs (RDS/PG/MySQL) or Raft-based replication (Cockroach) have good answers there.

Expensify: I skimmed the article, but AFAICT, the article talks about scaling QPS to a single SQLite file, but nothing about replication/durability.

Re: Building for the 99% Developers

#76

Post by djbusby »

muglug wrote: post #31

Article mentions a tweet which narrows in on a prime culprit: https://twitter.com/copyconstruct/status/1456129831821709315 > a lot of what most “developer influencers” say is fairly aspirational. Their own companies don’t necessarily do things as smoothly as they preach to others. There's a pride element there: developers don't want to admit that they work in less-than-stellar conditions. Some developer talks also do…

Easy to automate with Hylafax (2002), probably some SaaS API (2022) (there are at least a dozen Fax/SaaS things now)

Re: Building for the 99% Developers

#77

Post by oxplot »

travisgriggs wrote: post #73
oxplot wrote: post #62

Here are my rules for picking framework/languages: - Avoid if specifically mentions big companies as users: Facebook, Twitter, Google, these have as many devs as needed to throw at the most trivial tasks, and then some. I am in a team of 3 that need to get shit done, not chase package manager and compiler throw-ups. Tell me your language is used by a one person show to serve millions of people and then I'll listen. -…

Would you be willing to share some of the languages/frameworks that have made the cut or washed out for you based on the rubric you’ve given. I’m honestly curious.

I specifically avoided mentioning examples as that'll stir up a lang-war.

Re: Building for the 99% Developers

#79

Post by alphabettsy »

oleg_antonyan wrote: post #34
pcstl wrote: post #21

Earlier quoted context omitted.

Out of sincere curiosity: Why do you think GraphQL is so bad? I've been using GraphQL for a little while now and I've had nothing but good experiences (although my use case might not be the most common) so I'm interested in knowing what makes you think so poorly of it.

Not the one you asked, but I'm also stuck with supporting graphql in relatively small project. Zoomers invented SOAP. It's a perfect example of adopting what FAANGS do just for the sake of it. It's probably great when you have dozens of consumers with very different needs and usage patterns. But for the single frontend it adds way too much complexity on the server-side. Debugging, testing, writing tons of boilerate c…

Zoomers invented SOAP? Are we thinking of the same Zoomers? Same SOAP?

Re: Building for the 99% Developers

#80

Post by coryrc »

deathanatos wrote: post #51
alberth wrote: post #10

We way over complicate things. SQLite can be used for 95% of real world use cases. And whenever WAL2 + BEGIN CONCURRENT get merged into main, it will be able to handle 99% use cases. And don’t fool yourself into believing you’re not in that 1%.

You're not the only commenter saying this, so I gotta ask: whatever VM that SQLite DB is on, is your business cool with a business disruption, dataloss, or both when that VM goes down, when that AZ goes down, or that disk fails? Or am I missing something? If you have some sort of fail-over-to-last-backup plan, is that not just a distributed database with more steps, and why not something like RDS, or CockroachDB? (Fr…

> is your business cool with a business disruption

Every few years in Seattle, there's a big snowstorm, and everything just shuts down. We live.

If a service goes down once in a blue moon, I could catch up my email, spend some time morale building with my team, or, god forbid, just go home because work isn't that important.

Post reply on HN