Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Good system design

seangoedecke.com

Post reply on HN
Print view View original

301–310 of 400 posts

Re: Good system design

#301

Post by alphazard »

An entire post about "good system design" that completely fixates on the solution domain, and doesn't talk about the problem domain at all. The hardest part of system design is the interface that the system presents to users. That determines how they will use it and what they can use it for.

A software system trades problems for different problems. e.g. We will manage your TODO list, provide consistency, durability, security, better than you could do yourself. But in order to get these benefits you have to understand our model, we have TODOs, users, lists, permissions, etc.

Decisions about the interface (what problems the system presents to the users) are the most consequential, and the most costly to get wrong. If you aren't spending most of your time arguing about the interface, then you are wasting your time arguing about things that are comparatively easier to change later. Literally everything else about the system can be changed without bothering the users.

Re: Good system design

#302

Post by ndriscoll »

yojo wrote: post #246
atomicnumber3 wrote: post #185

Earlier quoted context omitted.

If I was your interviewer, I would: respect your answers a lot, not be able to check off anything on my rubric, try to explain this in the debrief, get told we have to stick to the rubric to counter bias, and then watch while they pass on you for someone who decided to play architecture jenga instead. I would potentially even consider emailing you to apologize later, then not do it because I'd probably get in trouble…

If a candidate doesn’t ask clarifying questions that lead them to an understanding of QPS, storage requirements, and throughput considerations, that’s a mark against. At that point, if you want to see them design a distributed system with all the bells and whistles, you should stop them, tell them the kind of traffic they need to handle, then let them go again. If they persist in designing a system that cannot handle…

The problem with this is people seem to have mismatched understandings of what a single system can handle. e.g. my 8 year old quad core i5 desktop with a bit of batching optimization can handle 5 digit requests per second with 15 ms p99 with some nontrivial application logic doing several joins. I don't think I've tried that same benchmark on a modern minipc, but I expect it should be similar. That's well above what most companies will ever need to handle. Visa advertises they can process ~70k tps worldwide.

Last time I interviewed I was asked about designing a system to handle 10s of thousands of events per minute, and if you thought about the problem a little you'd realize most of them didn't require real work to be done. I answered something along the lines of "you don't need to do anything special. Just normal postgres/mysql usage can handle more than that on a laptop". After I got hired I learned the rubric had some expected answers about queues (e.g. Kafka) in it. No idea why still.

Re: Good system design

#303

Post by SatvikBeri »

nvarsj wrote: post #80

> Paradoxically, good design is self-effacing: bad design is often more impressive than good. Rings very true. Engineers are rated based on the "complexity" of the work they do. This system seems to encourage over-engineered solutions to all problems. I don't think there is enough appreciation for KISS - which I first learned about as an undergrad 20 years ago.

Every now and then, I try to go through our codebase and write up the parts that we rarely think about – these are usually the cases where we made good decisions early on.

Re: Good system design

#304

Post by SatvikBeri »

dondraper36 wrote: post #211
stavros wrote: post #208

Earlier quoted context omitted.

I feel like the phrase "all you need is Postgres" has the (often unspoken) continuation of "until you actually get to a trillion messages". In other words, the developers you're envious of didn't start with Cassandra and ScyllaDB, they started with the problem of too many messages. That's not an architectural choice, that's product success.

Absolutely. To put it differently, unfortunately not everyone has a chance to be part of a product's organic evolution from "all we need is Postgres" to "holy crap, we're a success, what is Cassandra by the way?"

As a data point, I've been at two data-intensive startups where they eventually needed to pull (some) of their table-like data out of postgres, and for both that was past a $100MM valuation.

This varies by domain of course, but non-postgres solutions are generally built for very specific problems – they're worse than postgres at everything except one or two cases.

Re: Good system design

#305

Post by paffdragon »

hamandcheese wrote: post #295
paffdragon wrote: post #104

Earlier quoted context omitted.

> the interface being consumed is the database, which you do not need to design or implement You absolutely should design and implement it, exactly because it is now your interface. In fact, it will add more constraints to your design, because now you have different consumers and potentially writers all competing for the same resource with potentially different access patterns. Plus the maintenance overhead that migr…

> In fact, it will add more constraints to your design, because now you have different consumers and potentially writers all competing for the same resource with potentially different access patterns. Plus the maintenance overhead that migrations of such shared tables come with. And eventually you might have data in this table that are only needed for some of the services, so you now need to implement views and acces…

It's not that it is not possible, but whether it's a good idea.

The usual problem is that some team exposes one of their internal tables and they don't have control over what type of queries are run against it that could impact their service when the access patterns differ. Or when the external team is asking for extra fields that do not make sense for the owning team's model. Or adding some externally sourced information. Or the team moving from PostgreSQL to S3 or DynamoDB. And this is not an exhaustive list. An API layer is more flexible and can remain stable over a longer time than exposing internal implementation depending on a particular technology implemented in a particular way at the time they agreed on sharing.

This is, of course, not a concern inside the same team or very closely working teams. They can handle the necessary coordination. So, there are always exceptions and simple use cases where DB access works just fine. Especially, if you don't already have an API, which could be a bigger investment to set up for something simple if it's not even known yet the idea will work etc.

Re: Good system design

#306

Post by gmm1990 »

fogzen wrote: post #280
gmm1990 wrote: post #232

I seem to gravitate towards nosql type databases, defining tables in a ddl and then again in the code seems repetitive, and slows down changes. But the idea would be that the code is what defines the table. It'd be nice though to hear some of the drawbacks of this. Maybe for very relational things it makes sense to be able to write join queries so data is completely repeated, but my understanding would be that most d…

I think there's a tension between databases as programs to store and retrieve data quickly, efficiently, and reliably vs. programs to enforce business rules and domain modeling. I am firmly in the former camp. In my opinion databases should be for storing and retrieving data as quickly and efficiently as possible. But the consensus in the database world seems to be that databases are primarily for enforcing business…

That makes sense. Maybe it’s easier in an organization/ some people’s mental model to put guards around changing database because it’s separate from the code, standard across many organizations and in my opinion just harder to change.

Re: Good system design

#307

Post by dkarl »

zbentley wrote: post #283
CuriouslyC wrote: post #224

Earlier quoted context omitted.

Service specific views, my guy.

And when the underlying tables have to change, what then? Views are good, and help with this situation. But if the data is complicated, big, and even somewhat frequently changes shape (DDL), views only help a little. That said, I think that API update coordination is often much harder than schema change coordination (due to API behavior having many more dimensions along which it can change than database query behavio…

100%. Views don't even cover all the use cases of schema evolution, unless you're willing to duplicate business logic between stored procedures and services, but schema evolution is only the start of it. API versioning gives you a lot more flexibility to evolve how data is stored and accessed. Some parts of your data might get shifted into other data stores, some functionality might get outsourced to third party APIs, you might have to start supporting third-party integrations, etc. Try doing that from a view -- or rather, please don't!

Re: Good system design

#308

Post by setr »

maxbond wrote: post #288
setr wrote: post #262

Earlier quoted context omitted.

> Including timestamps on rows (including created_at and updated_at) are real bacon savers when you've deployed a bug and corrupted some rows and need to eg refund orders created in a certain window. But that’s my point. You’re making an active decision to record timestamps on important events (and no bool was being converted here); bool —> timestamp everywhere is not the same thing — the bool data type is not a usef…

Setting a boolean is, quite often, an important event you want to keep track of. (Specifically, when it is a flag indicating an event took place, eg, once it's been set it is rarely if ever unset.) Of course it isn't universally applicable to every boolean column; nothing is. Of course you need to understand your schema and the problem you're solving. That doesn't make it mindless or make it any less valuable. It is…

The original premise, from TFA, is

    Another is the Twitter-optimized “you’re a terrible engineer if you ever store booleans in a database” clever trick
That is, universal application. My point is: doing that, for the most part, will not the track the things you actually want, because bools are not the driving decision maker on value of tracking (and many things that need to be tracked will not be bools). It may accidentally capture useful tracking, but it’s more by accident than anything else. Thus, universal application is clearly wrong, and TFA is right to apply it conditionally.

You appear to agree on all of these points. I’m not arguing that, should you need to timetrack a bool, bool—>timestamp is a bad way to do it. As I’ve said, the universal application of it is only stated because it’s free to do. But being free doesn’t make it useful, and since you agree on that as well, I don’t know if there’s any argument actually occurring here.

Re: Good system design

#309

Post by try_the_bass »

__turbobrew__ wrote: post #260
ozgrakkurt wrote: post #177

Earlier quoted context omitted.

You don’t want to work at a company like this anyway.

If you want to get top dollar at a FAANG you will need to go through these type of system design interviews. You could say you shouldn’t work for a FAANG which is fair, but FAANG pays top dollar.

So if you're after the FAANG money, you have to play the FAANG interview game. If you're unwilling to play the FAANG interview game, then maybe you shouldn't be pursuing FAANG money.

Re: Good system design

#310

Post by ndriscoll »

zbentley wrote: post #284
marcosdumay wrote: post #197

Earlier quoted context omitted.

> Plus the maintenance overhead that migrations of such shared tables come with. Moving your data types from SQL into another language solves exactly 0 migration problems. Every migration you can hide with that abstraction language you can also hide in SQL. Databases can express exactly the same behaviors as your application code.

I’m generally pro SQL-as-interface, but this is just wrong. Not only are there all sorts of bizarre constraints imposed by databases on migration behavior that application code can’t express (for example, how can I implement a transaction-plus-double-write pattern to migrate to use a new table because the locks taken to add an index to the old table require unacceptably long downtime? There are probably some SQL engi…

Either triggers or create index concurrently? Do most people solve that on the client side? Doesn't e.g. percona use triggers?
Post reply on HN