Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Good system design

seangoedecke.com

Post reply on HN
Print view View original

211–220 of 400 posts

Re: Good system design

#211

Post by dondraper36 »

stavros wrote: post #208
dondraper36 wrote: post #184

Earlier quoted context omitted.

There's another reason for that. Deep in my heart, I would love to be part of a team that works on truly data-intensive applications (as Martin Kleppmann would call them) where all the complexity is justified. For example, I am more of the "All you need is Postgres" kind of software engineer. But reading all those fancy blog posts on how some team at Discord works with 1 trillion messages with Cassandra and ScyllaDB…

I feel like the phrase "all you need is Postgres" has the (often unspoken) continuation of "until you actually get to a trillion messages". In other words, the developers you're envious of didn't start with Cassandra and ScyllaDB, they started with the problem of too many messages. That's not an architectural choice, that's product success.

Absolutely. To put it differently, unfortunately not everyone has a chance to be part of a product's organic evolution from "all we need is Postgres" to "holy crap, we're a success, what is Cassandra by the way?"

Re: Good system design

#212

Post by mupuff1234 »

alixanderwang wrote: post #131

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…

I think you might be missing the point.

Your answers are completely valid but you have to communicate to the interviewer that you considered the possibilities and the tradeoffs.

If the interviewer needs to "forcefully" extract from you the logic behind your design choices than a lot of times that's enough to fail you.

Re: Good system design

#213

Post by DanielHB »

dondraper36 wrote: post #184
UK-AL wrote: post #165

Earlier quoted context omitted.

These are the places that actually pay well.

There's another reason for that. Deep in my heart, I would love to be part of a team that works on truly data-intensive applications (as Martin Kleppmann would call them) where all the complexity is justified. For example, I am more of the "All you need is Postgres" kind of software engineer. But reading all those fancy blog posts on how some team at Discord works with 1 trillion messages with Cassandra and ScyllaDB…

Only places that are making good money can afford to have overengineering.

Overengineering is more prevalent the more money a company makes and companies who overengineers will pay good money to keep the overengineering working.

Re: Good system design

#214

Post by stavros »

santiagobasulto wrote: post #170
alixanderwang wrote: post #131

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…

I’ve been in software for 20 years and it’s the first time I hear “back pressure”. Am I too old already?

You've just never played Factorio.

Re: Good system design

#215

Post by pphysch »

jillesvangurp wrote: post #16
bravesoul2 wrote: post #3

He doesnt seem to mention Conway or team topology which is an important part of system design too.

You should adapt your team to the architecture, not the other way around. My former Ph.D. supervisor who moonlights as a consultant on this topic uses a nice acronym to capture this: BAPO. Business, Architecture, Process, and Organization. The idea is to end up with optimal business, an optimal architecture & design for that business, the minimum of manual processes that are necessitated by that architecture, and an…

> My former Ph.D. supervisor who moonlights as a consultant on this topic uses a nice acronym to capture this: BAPO. Business, Architecture, Process, and Organization.

It's funny and not surprising that this came from an academic/consultant combo.

Business/mission/product should come first, absolutely.

But prematurely designing your system architecture and automations before your processes emerge is a sure-fire way to overengineer and waste everyone's time, except the consultant you are paying by the hour.

Re: Good system design

#216

Post by firesteelrain »

As a system architect, software engineer and systems engineer, I see these posts and what is called system design seems to intermix systems design with software design (being that software described herein is a lower level component of the overall system)

Re: Good system design

#217

Post by marcosdumay »

Yokohiii wrote: post #121
tossandthrow wrote: post #113

Earlier quoted context omitted.

It is a granluarity tradeoff. With SQL you need to explicitly test all queries where the shape granularity is down to field level. When you map data onto an object model (in the dto sense, not oop sense) you have bigger building blocks. This gives a simpler application that is more reliable. Obviously you need to pick a performant orm - and it seems a lot of people in these threads have been traumatized. Personally,…

In my ears that's just neglect? You assume your ORM does the basic data mapping right and don't verify it?

> You assume your ORM does the basic data mapping right

You know, it should. There's no good reason for an ORM to ever fail at runtime due to mapping problems instead of compile time or start time. (Except, of course if you change it during the software's execution.)

Re: Good system design

#218

Post by nostrademons »

_fat_santa wrote: post #169
alixanderwang wrote: post #131

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…

This goes back to "interviews go both ways". All those answers you gave are very reasonable and if I was your interviewer I'd pass you with flying colors. On the other hand if you're interviewing at a place that doesn't pass you with flying colors for those responses, that really says more about them than it does about you and may not be a great place to work. But to your point, many times one interviews for a job th…

If I were the interviewer, I'd try to adjust the problem statement with some hypotheticals to tease out their depth of knowledge:

> "That's not really worth considering for this amount of QPS"

"What if Michael Jackson dies and your (search|news|celebrity gossip) service gets a spike in traffic way beyond the design parameters? How would you anticipate and mitigate such an event?"

(Extra points if the answer is not necessarily backpressure but they start talking about DDoS mitigation, outlier detection, caching or serving static results from extremely-common queries, spinning up new capacity to adjust to traffic spikes, blackholing traffic to protect the overall service, etc.)

> Interviewer: "Why wouldn't you use a queue here instead of a cron job?" "I don't think it's necessary for what this app is, but here's the tradeoffs."

"What if you have a subset of customers that demand faster responses than a cron job can provide?"

(And then that can become a discussion about splitting off traffic based on requirements, whether it's even worth adding the logic to split traffic vs. just using a queue for everyone, perhaps making direct API requests without either a queue or cron job for requests from just those customers, relying on the fact that they are not numerous or these requests are infrequent to trade capacity for latency, etc.)

> How would you choose between sql and nosql db?"

I would've expected the candidate to at least be able to talk about indexing, tradeoffs of joining in the DB vs. in the application, schema migrations and upgrades, creating separation between data-at-rest vs. data-in-flight, etc. If they can't do that and just handwave away as "whatever the team is most comfortable with", that's a legit hole in their knowledge. Usually you ask system design interviews of senior candidates that will be deciding on architecture and, if not hiring out the team directly, providing input to senior managers who will be hiring, so you can swap out the team nearly as easily as swapping out the architecture.

Re: Good system design

#219

Post by [deleted] »

torginus wrote: post #63
bambax wrote: post #14

> When querying the database, query the database. It’s almost always more efficient to get the database to do the work than to do it yourself. For instance, if you need data from multiple tables, JOIN them instead of making separate queries and stitching them together in-memory. Oh yes! Never do a join in the application code! But also: use views! (and stored procedures if you can). A view is an abstraction about the…

Are you sure about this? Let's say you run a webshop and have two tables, one for orders with 5 fields, one for customers, with 20 fields. Let's say you have 10k customers, and 1m orders. A query performing a full join on this and getting all the data would result in 25 million fields transmitted, while 2 separate queries and a client side manual join would be just 5m for orders, and 200k for customers.

[deleted]

Re: Good system design

#220

Post by willio58 »

atomicnumber3 wrote: post #185
_fat_santa wrote: post #169

Earlier quoted context omitted.

This goes back to "interviews go both ways". All those answers you gave are very reasonable and if I was your interviewer I'd pass you with flying colors. On the other hand if you're interviewing at a place that doesn't pass you with flying colors for those responses, that really says more about them than it does about you and may not be a great place to work. But to your point, many times one interviews for a job th…

If I was your interviewer, I would: respect your answers a lot, not be able to check off anything on my rubric, try to explain this in the debrief, get told we have to stick to the rubric to counter bias, and then watch while they pass on you for someone who decided to play architecture jenga instead. I would potentially even consider emailing you to apologize later, then not do it because I'd probably get in trouble…

I’ve interviewed dozens of people and while I rarely do system design questions and our process isn’t nearly as check-all-the-boxes, it’s funny how accurate your comment still is. Near the later stages especially, politics starts coming in.
Post reply on HN