Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Good system design

seangoedecke.com

Post reply on HN
Print view View original

281–290 of 400 posts

Re: Good system design

#281

Post by radlad »

tossandthrow wrote: post #133
radlad wrote: post #117

Earlier quoted context omitted.

The issue with GraphQL tends to be unoptimized joins instead. Is your GraphQL API available for public consumers? How do you manage them issuing inefficient queries? I've most often seen this countered through data loaders (batched queries that are merged in code) instead of joins, or query whitelists.

While this api in particular is not publicly exposed, that would not be a concern. The key is to hold the same schema on the database as on the graphql and use tooling that can translate a gql query into a single query.

The issue I've seen with GraphQL isn't necessarily the count of queries run, but rather the performance or said queries (i.e. most SQL queries are not performant without proper indexes for the specific use case, but GraphQL allows lots of flexibility in what queries users can run.)

Re: Good system design

#282

Post by firesteelrain »

StevenWaterman wrote: post #278
firesteelrain wrote: post #216

As a system architect, software engineer and systems engineer, I see these posts and what is called system design seems to intermix systems design with software design (being that software described herein is a lower level component of the overall system)

I'm glad I'm not the only one thinking that. These are such minutiae. Where's the discussion about humans? They're probably the most important part of your system, and the most chaotic, and the part that needs the most careful design. It's hinted at a little bit in the OP, with: > What does good system design look like? I’ve written before that it looks underwhelming This is because there are humans in your system! O…

Exactly. Systems design at the SE level is socio-technical: the humans and their feedback loops matter as much as the tech. Many “good” designs fall apart because they forgot that the system boundary doesn’t stop at the codebase.

Re: Good system design

#283

Post by zbentley »

CuriouslyC wrote: post #224
dkarl wrote: post #173

Earlier quoted context omitted.

> And what exactly do you buy yourself? APIs can be evolved much more easily than shared database schemas. Having worked with many instances of each kind of system, I think this outweighs all of the other considerations, and I don't think I'll ever again design a system with multiple services accessing the same database schema. It was maybe a good idea if you were a small company in the early 2000s, when databases we…

Service specific views, my guy.

And when the underlying tables have to change, what then?

Views are good, and help with this situation. But if the data is complicated, big, and even somewhat frequently changes shape (DDL), views only help a little.

That said, I think that API update coordination is often much harder than schema change coordination (due to API behavior having many more dimensions along which it can change than database query behavior), so I am generally open to multiple services sharing a database—so long as it’s within reason and understood to pose risks/be appropriately justified and used responsibly.

Re: Good system design

#284

Post by zbentley »

marcosdumay wrote: post #197
paffdragon wrote: post #104

Earlier quoted context omitted.

> the interface being consumed is the database, which you do not need to design or implement You absolutely should design and implement it, exactly because it is now your interface. In fact, it will add more constraints to your design, because now you have different consumers and potentially writers all competing for the same resource with potentially different access patterns. Plus the maintenance overhead that migr…

> Plus the maintenance overhead that migrations of such shared tables come with. Moving your data types from SQL into another language solves exactly 0 migration problems. Every migration you can hide with that abstraction language you can also hide in SQL. Databases can express exactly the same behaviors as your application code.

I’m generally pro SQL-as-interface, but this is just wrong.

Not only are there all sorts of bizarre constraints imposed by databases on migration behavior that application code can’t express (for example, how can I implement a transaction-plus-double-write pattern to migrate to use a new table because the locks taken to add an index to the old table require unacceptably long downtime? There are probably some SQL engines out there that can do this with views, but most people solve it on the client side for good reason), but there are plenty of changes that you just plain can’t do without a uniform service layer in front of your database. Note that “uniform service layer” doesn’t necessarily mean “networked service layer”, this can be in-process if you can prevent people from bypassing your querying functions and going directly to the DB.

Re: Good system design

#285

Post by zbentley »

paffdragon wrote: post #151
wahnfrieden wrote: post #140

Earlier quoted context omitted.

A reused code library for DB use is an alternative there

That moves your API layer to the client library you need to distribute and build for your customers in programming languages they support. There are some cases where a thick client makes sense, but usually easier to do it server side and let customers consume the API from their env, it is easier to patch the server than to ship library updates to all users.

I think most of the discussion in this thread assumes that “customers” of the interface are other groups in the same organization using the database for a shared overarching business/goal, not external end user customers.

For external end users, absolutely provide an API, no argument here. The internal service interactions behind that API are a less simple answer, though.

Re: Good system design

#286

Post by philjohn »

mkozlows wrote: post #254
alixanderwang wrote: post #131

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…

(For context, I've conducted hundreds of system design interviews and trained a dozen other people on how to do them at my company. Other interviewers may do things differently or care about other things, but I think what I'm saying here isn't too far off normal.) I think three things about what you're saying: 1. The answers you're giving don't provide a lot of signal (the queue one being the exception). The question…

+1 on the signal. A great candidate won't need you to pry further than asking about backpressure, they'll explain WHY it's not necessary for the qps, what qps it would start becoming necessary, and how they would build it into their design down the line if the service takes off.

One of the things I tell people preparing for system design interviews is the more senior you are, the more you need to drive the interview yourself, knowing when to go deep, what to go deep on, and how to give the most signal to the interviewer.

Re: Good system design

#287

Post by UncleFullstack »

It can be kind of horrifying at times. A couple of years ago I was interviewing for work and ended up talking to a big liquor distributor - their challenge was dealing with a bunch of text files over FTP, and it was comically bad, they had a $300k annual spend on AWS, Kubernetes, the works. And they could have done the whole thing on a single EC2 instance with a couple of shell scripts. Needless to say I was laughed out of the room.

Re: Good system design

#288

Post by maxbond »

setr wrote: post #262
maxbond wrote: post #25

Earlier quoted context omitted.

Audit tables are a big ask both in terms of programming effort to design and support them, and in terms of performance hit due to write amplification (all inserts and updates cause an additional write to an audit table). Whereas making a bool into a timestamp is free. Including timestamps on rows (including created_at and updated_at) are real bacon savers when you've deployed a bug and corrupted some rows and need to…

> Including timestamps on rows (including created_at and updated_at) are real bacon savers when you've deployed a bug and corrupted some rows and need to eg refund orders created in a certain window. But that’s my point. You’re making an active decision to record timestamps on important events (and no bool was being converted here); bool —> timestamp everywhere is not the same thing — the bool data type is not a usef…

Setting a boolean is, quite often, an important event you want to keep track of. (Specifically, when it is a flag indicating an event took place, eg, once it's been set it is rarely if ever unset.) Of course it isn't universally applicable to every boolean column; nothing is. Of course you need to understand your schema and the problem you're solving. That doesn't make it mindless or make it any less valuable. It is a good pattern which does lead to good things. If it is mindless, your suggestion of tracking every change in an audit table, and the created_at and updated_at fields you approved of, are equally mindless.

As it stands, I feel I've articulated straightforward and realistic advantages of using timestamps as booleans. But I haven't heard from you why I shouldn't.

Re: Good system design

#289

Post by zbentley »

tremon wrote: post #174
Too wrote: post #141

Earlier quoted context omitted.

With an ORM your application code is your views. You can write reusable plain functions as abstractions, returning QuerySets that allow further filters being chained onto the query, before the actual SQL is materialized and sent to the database. The result of this doesn’t have to match the original object models you defined, it’s still possible to be flexible with group bys resulting in dictionaries.

But converting a SQL relation to a set of dictionaries already carries a lot of overhead: every cell in the resultset must be converted to a key-value pair. And the normal mechanics of vertical "slicing" a set of dictionaries is much more expensive than doing the same in a 2d relation array. So while you might want to offer a dictionary-like interface for the result set, please don't use a dictionary-like data struct…

There are valid reasons to avoid complex ORM/query result representations, but this isn’t one of them.

I have very rarely seen or even heard of the result representation data structure for an SQL query being a bottleneck. The additional time and space needed to represent a raw tabular result in a more useful way on the client are nearly always rounding errors compared by the time needed to RPC the query itself and the space taken up by the raw bytes returned. Given that, and the engineering time wasted working with (and fixing inevitable bugs in) fully tabular result data structures (arrays, bytes), this is bad advice.

Re: Good system design

#290

Post by cygn »

santiagobasulto wrote: post #170
alixanderwang wrote: post #131

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…

I’ve been in software for 20 years and it’s the first time I hear “back pressure”. Am I too old already?

https://medium.com/@jayphelps/backpressure-explained-the-flo...
Post reply on HN