Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Good system design

seangoedecke.com

Post reply on HN
Print view View original

261–270 of 400 posts

Re: Good system design

#261

Post by rekabis »

> But in most cases replication lag can be worked around with simple tricks: for instance, when you update a record but need to use it right after, you can fill in the updated details in-memory instead of immediately re-reading after a write.

I found myself truly confused by this one - does this actually need stating? Do people actually re-read immediately after a write? Provided you got confirmation that a write was successful and the data doesn’t have anything that an SQL trigger would change, what would be the point of an immediate read instead of just using the DB “successfully written” response as a go-ahead to just update the in-memory data?

Re: Good system design

#262

Post by setr »

maxbond wrote: post #25
setr wrote: post #17

Earlier quoted context omitted.

If you take the statement at face value — essentially storing booleans in the db ever is a bad smell - then he’s correct. Although I’m not even sure it’s broadly a good principle, even in the on_at case; if you actually care about this kind of thing, you should be storing it properly in some kind of audit table. Switching bool to timestamp is more of a weird lazy hack that probably won’t be all that useful in practic…

Audit tables are a big ask both in terms of programming effort to design and support them, and in terms of performance hit due to write amplification (all inserts and updates cause an additional write to an audit table). Whereas making a bool into a timestamp is free. Including timestamps on rows (including created_at and updated_at) are real bacon savers when you've deployed a bug and corrupted some rows and need to…

> Including timestamps on rows (including created_at and updated_at) are real bacon savers when you've deployed a bug and corrupted some rows and need to eg refund orders created in a certain window.

But that’s my point. You’re making an active decision to record timestamps on important events (and no bool was being converted here); bool —> timestamp everywhere is not the same thing — the bool data type is not a useful signal for whether this change needs to be timestamp-tracked.

Either think and choose to track these particular changes or not, and drop in the appropriate tracking. The mindless Bool—>timestamp change is only ever suggested because it’s a “why not”, not because it’s a good practice that often leads to good things.

The audit table is of course just deciding “every change ever is important”

Re: Good system design

#263

Post by grey-area »

ZYbCRq22HbJ2y7 wrote: post #36
grey-area wrote: post #32

Earlier quoted context omitted.

I’d see the booleans as a bad thing in almost all cases, instead of a boolean you can have a timestamp or an integer field (which can expand later). In the is_a case almost always a type or kind is better as you’ll rarely just have bears even if you only start with bears, just as you rarely have just two states for a status field (say on or off), often these expand in use to include things like suspended, deleted and…

Yeah, I mean, an integer can definitely hold more data than a boolean. If your data was simple enough, you could have an integer hold the entire meaning of a table's row, if every client understood how it was interpreted. You could do bitwise manipulations, encodings, and so on. Sometimes it is nice to understand what the data means in the schema alone. You can do that with enums, etc. ate_an_apple_in_may_2024 saw_an…

I’ve never seen anything remotely like that in a schema and it seems inappropriate anyway - those should IMO be timestamps like saw_eclipse_at not booleans. You should not encode business rules in the schema (like certain magic dates) because those business rules always change over time.

Re: Good system design

#264

Post by __turbobrew__ »

hn8726 wrote: post #256

What's the best resource to learn those things in practice? Other than _just trying_ yourself? I'm a senior dev in another area who wants to get into the backend development, but unsurprisingly has limited time to spend on learning a completely new thing,

There are some books out there like “Distributed Systems” by Tanenbaum. Then there are the various company publications like “The SRE Book” by google and eng blogs.

You could also contribute to an open source project like kubernetes or postgres to get your feet wet.

Like all things, the best way to get better is to do it.

Re: Good system design

#265

Post by no_wizard »

DanielHB wrote: post #213
dondraper36 wrote: post #184

Earlier quoted context omitted.

There's another reason for that. Deep in my heart, I would love to be part of a team that works on truly data-intensive applications (as Martin Kleppmann would call them) where all the complexity is justified. For example, I am more of the "All you need is Postgres" kind of software engineer. But reading all those fancy blog posts on how some team at Discord works with 1 trillion messages with Cassandra and ScyllaDB…

Only places that are making good money can afford to have overengineering. Overengineering is more prevalent the more money a company makes and companies who overengineers will pay good money to keep the overengineering working.

Something about my old CTO and VP of Eng I respected is they were still technical enough to call out this kind of thing. For as big as that company was they really held down complexity and overengineering to a real minimum.

Unfortunately the rest of the executive has leaned on them so hard about AI boosting productivity they aren’t able to avoid thst becoming a mess

Re: Good system design

#266

Post by nlawalker »

renewiltord wrote: post #251
nlawalker wrote: post #200

Earlier quoted context omitted.

> I want you to start with these answers then we can layer on complexity if you’ve solved the problem and there’s time left to go into navel gazing mode. Do you tell people this explicitly? If so, good on you; if not, please start! I think one of the biggest problems with interviews these days is misaligned expectations, particularly interviewees coming in assuming that what's desired is immediate evidence that they'…

At the level where this matters, the skill to figure it out from context is important. You aren’t the guy converting spec to code. You’re the spec maker.

I agree, but I think my point is that the interview context and expectations can differ radically different from the role context, depending on the interviewer. If the expectation of the interviewer is that the interviewee should be asking questions to determine scale needs, then they should be explicit about that. For all the interviewee knows, you're going to ding them and ultimately fail them for asking too many questions and not exhibiting knowledge and experience.

Re: Good system design

#267

Post by no_wizard »

Swizec wrote: post #204
nlawalker wrote: post #200

Earlier quoted context omitted.

> I want you to start with these answers then we can layer on complexity if you’ve solved the problem and there’s time left to go into navel gazing mode. Do you tell people this explicitly? If so, good on you; if not, please start! I think one of the biggest problems with interviews these days is misaligned expectations, particularly interviewees coming in assuming that what's desired is immediate evidence that they'…

> Do you tell people this explicitly? Yes and no. I give them rough scale numbers to design for. Part of the interview is knowing why I’m telling you this.

Or asking to get there, I find that to also be acceptable

Re: Good system design

#268

Post by bambax »

torginus wrote: post #63
bambax wrote: post #14

> When querying the database, query the database. It’s almost always more efficient to get the database to do the work than to do it yourself. For instance, if you need data from multiple tables, JOIN them instead of making separate queries and stitching them together in-memory. Oh yes! Never do a join in the application code! But also: use views! (and stored procedures if you can). A view is an abstraction about the…

Are you sure about this? Let's say you run a webshop and have two tables, one for orders with 5 fields, one for customers, with 20 fields. Let's say you have 10k customers, and 1m orders. A query performing a full join on this and getting all the data would result in 25 million fields transmitted, while 2 separate queries and a client side manual join would be just 5m for orders, and 200k for customers.

One way to think about 1-to-many relationships is to think in the other way, "many-to-one". You don't join the orders to the customers, you join the customers to the orders (enrich the orders with customer information).

It's very natural to want customer information when querying an order, and if you have a view like orders_with_customer_info, you get that with zero effort when querying that view by order id.

You also get consolidated data (orders by customer) by doing

  select count(*), sum(amount) from orders_with_customer_info group by customer_id
which I think is pretty straightforward.

Re: Good system design

#269

Post by no_wizard »

dondraper36 wrote: post #199
Aurornis wrote: post #196

Earlier quoted context omitted.

Identifying candidates who repeat buzzwords without understanding tradeoffs is easy. It’s part of the questioning process to understand the tradeoffs. The problem with the comment above is that it’s not discussing tradeoffs at all. It’s just jumping to conclusions and dodging any discussion of tradeoffs. If you answer questions like that, it’s impossible to tell if the candidate is being wise or if they’re simply BSi…

Yes, I would probably phrase it like this. "Under the current load, I would go super simple and use X, which can work fine long enough until it doesn't. And then we can think about horizontal scaling and use Y and Z". Then proceed with a deeper discussion of Y and Z, probably. After all, interviewing and understanding what your interviewer expects to hear is also a valuable skill (same as with your boss or client).

Even better would be to clarify under the current load and if reasonably expected future load is similar, I would use X for Y reasons.

Sometimes the “trick” is in todays load is not tomorrows

Re: Good system design

#270

Post by no_wizard »

santiagobasulto wrote: post #170
alixanderwang wrote: post #131

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…

I’ve been in software for 20 years and it’s the first time I hear “back pressure”. Am I too old already?

Somewhere surprising but if you never dealt with scaling issues of a certain nature it may have never came up.

Though you might be familiar with other terms that effectively mean the same thing, like counter pressure

Post reply on HN