Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Good system design

seangoedecke.com

Post reply on HN
Print view View original

351–360 of 400 posts

Re: Good system design

#351

Post by 0xDEAFBEAD »

bencornia wrote: post #225
bambax wrote: post #14

> When querying the database, query the database. It’s almost always more efficient to get the database to do the work than to do it yourself. For instance, if you need data from multiple tables, JOIN them instead of making separate queries and stitching them together in-memory. Oh yes! Never do a join in the application code! But also: use views! (and stored procedures if you can). A view is an abstraction about the…

Views are great. Stored procedures are cursed.

>Stored procedures are cursed.

Elaborate?

Re: Good system design

#352

Post by danhite »

santiagobasulto wrote: post #170
alixanderwang wrote: post #131

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…

I’ve been in software for 20 years and it’s the first time I hear “back pressure”. Am I too old already?

> I’ve been in software for 20 years and it’s the first time I hear “back pressure”. Am I too old already?

I first wrote code 50 years ago (I am 63yo) so yes, imo we are too old, but ...

It is worth noting that systems concepts/techniques often have analogues aka different names and histories in different fields and subfields.

If I were to "explain" back pressure to an ordinary person I might model my analogy to the logic of this ~classic joke:

Bob: Let's go to Trendio(TM) for dinner tonight!? Carol: Oh, nobody goes there anymore, it's too crowded!

Also, often a modern take-this-for-granted concept may be seen as an outgrowth of previous problems or solutions.

For example back pressure is conceptually adjacent to the clever~hack/design of random backoff in Ethernet.

Or if talking to a math geek or traffic planner you might relate it to ~modern understanding of congestion including oddities like possibly removing roads/routes to ~paradoxically improve traffic flow.

We are deep in the Information Age barreling towards Singularities, so none of us, young or old, see and understand but a tiny fraction of where we've been, are, or might be going.

Cue Calvin & Hobbes cartoon of us racing downhill in a fragile box.

Perhaps, as others have essentially suggested, merging your mind with an ~AI will help (albeit temporarily, imo). I prefer to think of us/greybeards as potentially Wise, yet, paradoxically, clueless.

Beginner's Mind, with likely no time/future for Mastery, is still potentially pleasant, and I would argue useful for Debugging.

Obviously this modern AI tsunami is phase shifting us all into debug~mode anyway, eh?

Re: Good system design

#353

Post by elcdodedocle »

Good article describing a boilerplate framework & techniques for (web) backend systems architecture implemented with off-the-shelf components. But like every systems architect I have ever met, it does not even put a thought into security or data governance (a priori).

Re: Good system design

#354

Post by brainzap »

motorest wrote: post #92

What a great article. It's always a treat to read this sort of take. I have some remarks though. Taken from the article: > Avoid having five different services all write to the same table. Instead, have four of them send API requests (or emit events) to the first service, and keep the writing logic in that one service. This is not so cut-and-dry. The trade offs are far from obvious or acceptable. If the five services…

Airflow 2 used database to coordinate, airflow 3 switched to API.

Re: Good system design

#355

Post by vbezhenar »

dkarl wrote: post #173
motorest wrote: post #92

What a great article. It's always a treat to read this sort of take. I have some remarks though. Taken from the article: > Avoid having five different services all write to the same table. Instead, have four of them send API requests (or emit events) to the first service, and keep the writing logic in that one service. This is not so cut-and-dry. The trade offs are far from obvious or acceptable. If the five services…

> And what exactly do you buy yourself? APIs can be evolved much more easily than shared database schemas. Having worked with many instances of each kind of system, I think this outweighs all of the other considerations, and I don't think I'll ever again design a system with multiple services accessing the same database schema. It was maybe a good idea if you were a small company in the early 2000s, when databases we…

I've implement an interesting service 15 years ago. Recently I've heard of it.

So this service was basically an "universal integration service". Company wanted to share some data, and they wanted to implement it in an universal way. So basically I've implemented SOAP web service which received request with SQL text and responded with list of rows. This service was surprisingly popular and used a lot.

I was smart enough, so I built a limited SQL syntax parser and UI, so administrator could just set up tables and columns they wanted to share for this specific client. SQL query was limited in a sense that it worked only with one table, simple set of columns and some limited conditions (that I bothered to implement).

The reason I've heard about it few months ago is that they shared with me, that they caught a malicious guy, who worked at some company integrating with this system and he tried to do SQL attack. They noticed errors in the logs and caught him.

Their database is pretty much done and frozen, regarding to schema. They hardly evolve it. So this service turned out ot be pretty backwards-compatible. And simple changes, of course, could be supported with view, if necessary.

Re: Good system design

#356

Post by porridgeraisin »

hk1337 wrote: post #233
mattmanser wrote: post #56

Earlier quoted context omitted.

I'm not sure if this is what you mean, but I think a big thing missing from the article is how you should isolate you business logic. A great software design will separate all business logic into its own layer. That might be a distinct project, module, or namespace, depending on what your language supports. Keep business logic out of SQL and out of web server code (controllers, web helpers, middleware, etc.). Then yo…

Pretty much yeah. I think you need to stay somewhat vigilant because it can creep in under the guise of validating data. Triggers is something I was thinking about that is grossly misused.

I like using triggers for validation important for maintaining data integrity.

Example: let's say you have a parent table P whose rows are soft deleted (update P set _deleted=1 where id=?) and a child table C (foreign key on P.id) On delete cascade isn't gonna work here obviously. So I use a trigger to emulate on delete cascade.

Also, I've seen triggers to cancel transactions raise abort(..) if they violate some property that can't be retrofitted into a proper db constraint due to team politics at a previous workplace.

Re: Good system design

#357

Post by AbstractH24 »

alixanderwang wrote: post #131

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…

Tools that reduce the barrier to entry to creating things make it easier to solve problems with less scale to pay for the overhead. Generative AI is among these tools, but so are low code platforms, so is React, so is AWS, heck, so is the power grid. But in recent times generative AI is a big leap forward.

We’re at the start of another cycle of a lot of niche products followed by the rise of big Acme megacorps who conquer them all economies of scale that compete on margin. It comes just as we’re at the tail end of this cycle with tech as we knew it for the last 50 or so years.

Re: Good system design

#358

Post by [deleted] »

rekabis wrote: post #261

> But in most cases replication lag can be worked around with simple tricks: for instance, when you update a record but need to use it right after, you can fill in the updated details in-memory instead of immediately re-reading after a write. I found myself truly confused by this one - does this actually need stating? Do people actually re-read immediately after a write? Provided you got confirmation that a write was…

[deleted]

Re: Good system design

#359

Post by porridgeraisin »

rekabis wrote: post #261

> But in most cases replication lag can be worked around with simple tricks: for instance, when you update a record but need to use it right after, you can fill in the updated details in-memory instead of immediately re-reading after a write. I found myself truly confused by this one - does this actually need stating? Do people actually re-read immediately after a write? Provided you got confirmation that a write was…

Here's why it happens:

A lot of the time the datastructure you pass into writeAPI(obj) is different from the datastructure that is returned from readAPI(obj) -- even if the information contained is the same!

No one wants to do that data structure transformation and potentially miss an edge case/break some implicit assumption about the data structure some fuckall downstream consumer has.

However it is already done in the readAPI() function. So latency and throughput be damned, let us do:

  writeAPI(objects)
  objects = readAPI(objects)
To be clear, I'm talking about the typical bloated data structure we all know and love: 20+ fields, different fields redundant for different services, sometimes they are empty in which case we have to fall back to calculate that field differently. And it is this hacky due to a quick bug fix during a sev1 last year that was never revisited to be fixed "correctly". There is a ticket hanging around somewhere to do this, but the assignee has left the company.

Re: Good system design

#360

Post by bccdee »

alixanderwang wrote: post #131

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…

I think the right tack, if you're going to dismiss something as needlessly complex, is to call out the circumstances that would make it necessary and then describe what you'd do under those conditions.

"Backpressure? I don't think you'll have enough traffic to make backpressure necessary. The mode of failure here is that you run out of queue space and start dropping messages, and it's not a big deal if some messages get dropped here. But if we do decide that dropped messages are causing problems, and if it starts becoming a regular occurrence (we'll set up observability), here's how the producer can poll the queue size and return an error to the user under heavy load.

Post reply on HN