Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Good system design

seangoedecke.com

Post reply on HN
Print view View original

291–300 of 400 posts

Re: Good system design

#291

Post by zbentley »

quibono wrote: post #70
KronisLV wrote: post #26

> Schema design should be flexible, because once you have thousands or millions of records, it can be an enormous pain to change the schema. However, if you make it too flexible (e.g. by sticking everything in a “value” JSON column, or using “keys” and “values” tables to track arbitrary data) you load a ton of complexity into the application code (and likely buy some very awkward performance constraints). Drawing the…

> storing audit data in the same DB and it inevitably having functionality written against it, your audit data becoming a part of the business logic What's the "proper" way to do this? Separate DB? Separate data store?

I was also onboard with GP’s comment until I got to this part.

Audit data in the same DB is great, because it can be written transactionally for relatively cheap (multi table updates, triggers, actual transactions with multiple writes, etc).

After that, sure, ship it elsewhere and prune the audit tables if you like. But having the audit writes go directly to Kafka or whatnot is a pain because it requires your client logic to a) have a distributed publish-event transaction (which can work in this case more easily than distributed transactions in general with careful use of idempotency keys, read back, or transactional outboxes, but it’s complicated and requires everyone writing to auditable tables to play along), and b) reduces your reliability because now the audit store or its message queue needs to be online for every write as well as your database.

And there’s plenty of good reasons for business logic to use (only for reads) audit data. What else would business logic do if an audit table existed and there was a business need to e.g. show customers a change history for something? Build another redundant audit system instead?

Re: Good system design

#292

Post by sgarland »

mdavid626 wrote: post #49
bambax wrote: post #14

> When querying the database, query the database. It’s almost always more efficient to get the database to do the work than to do it yourself. For instance, if you need data from multiple tables, JOIN them instead of making separate queries and stitching them together in-memory. Oh yes! Never do a join in the application code! But also: use views! (and stored procedures if you can). A view is an abstraction about the…

I disagree. In modern highly scalable architectures I’d prefer doing joins in the layer front of the database (backend). The “backend” scales much easier than the database. Loading data by simple indexes, eg. user_id, and joining it on the backend, keeps the db fast. Spinning up another backend instance is easy - unlike db instance. If you think, your joins must happen in db, because data too big to be loaded to memo…

Unless all your tables have the same width - or you’re doing weird things with constants in your SELECTs - you can’t UNION the various queries, so they’re sequential. You could parallelize those I suppose, but now you’re adding more complexity.

If you want a KV store, use a KV store. If you want an RDBMS, then use its features. They haven’t changed much in the last 50 years for a reason.

Re: Good system design

#293

Post by pragmatic »

alixanderwang wrote: post #131

> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…

Yes, and then you get the job there and regret it bc they’ll have either have an over engineered Rube Goldberg contraption or they have system envy bc they’ve read about this architecture in blogs and THINK they need K8s and they still fix all their problems.

Re: Good system design

#294

Post by axpy906 »

It’s under appreciated how many of us fall for over engineering. I’ve been there. Just the other day I had coworker suggest that our startup not use cloud blob storage because it’s unreliable and we should build our own.

Maybe it’s just harder to design with something simple that is possible to extend and build on. Maybe I am missing something. That said I agree with the author from my decade of experience.

Re: Good system design

#295

Post by hamandcheese »

paffdragon wrote: post #104
motorest wrote: post #92

What a great article. It's always a treat to read this sort of take. I have some remarks though. Taken from the article: > Avoid having five different services all write to the same table. Instead, have four of them send API requests (or emit events) to the first service, and keep the writing logic in that one service. This is not so cut-and-dry. The trade offs are far from obvious or acceptable. If the five services…

> the interface being consumed is the database, which you do not need to design or implement You absolutely should design and implement it, exactly because it is now your interface. In fact, it will add more constraints to your design, because now you have different consumers and potentially writers all competing for the same resource with potentially different access patterns. Plus the maintenance overhead that migr…

> In fact, it will add more constraints to your design, because now you have different consumers and potentially writers all competing for the same resource with potentially different access patterns. Plus the maintenance overhead that migrations of such shared tables come with. And eventually you might have data in this table that are only needed for some of the services, so you now need to implement views and access controls at the DB level.

PostgreSQL, to name one example, can handle every one of these challenges.

Re: Good system design

#296

Post by digitalPhonix »

aembleton wrote: post #146
digitalPhonix wrote: post #71

Earlier quoted context omitted.

What sort of application is regularly doing a query for “all data”?

Client report generation.

Even then, DB side aggregation and joins would be beneficial.

I’m asking specifically about “all data” unfiltered, un-aggregated, un-joined as the parent’s analysis was on.

Re: Good system design

#297

Post by YZF »

It seems that there is no appreciation of a good architecture/system design any more. Nobody cares.

Leadership can't tell the difference. If anything the worse designs seem more impressive. Engineers often enjoy a bad design because it creates more work and job security. When there's a lot of work it seems like you're getting things done. Managers enjoy a bad design because it helps empire building, now that there is more work we need to hire more people and do more manager-y things. There are also a lot of inexperienced engineers in the work force who have never seen a well designed system.

In an organization running these badly designed system it's a political suicide to argue the design is bad. If the business is successful even more so because everyone will assume that a successful business means well designed software. A successful business will directly reward a bad design.

Re: Good system design

#298

Post by rawgabbit »

I wonder why the author views CQRS negatively and then later gives this classic CQRS advice:

      >What this means in practice is having one service that knows about the state - i.e. it talks to a database - and other services that do stateless things. Avoid having five different services all write to the same table. Instead, have four of them send API requests (or emit events) to the first service, and keep the writing logic in that one service.

Re: Good system design

#299

Post by gopher_space »

yojo wrote: post #246
atomicnumber3 wrote: post #185

Earlier quoted context omitted.

If I was your interviewer, I would: respect your answers a lot, not be able to check off anything on my rubric, try to explain this in the debrief, get told we have to stick to the rubric to counter bias, and then watch while they pass on you for someone who decided to play architecture jenga instead. I would potentially even consider emailing you to apologize later, then not do it because I'd probably get in trouble…

If a candidate doesn’t ask clarifying questions that lead them to an understanding of QPS, storage requirements, and throughput considerations, that’s a mark against. At that point, if you want to see them design a distributed system with all the bells and whistles, you should stop them, tell them the kind of traffic they need to handle, then let them go again. If they persist in designing a system that cannot handle…

I’m also going to need a dollar value on your data and a list of consequences. We will spend our allotted time together in Excel.

Re: Good system design

#300

Post by paffdragon »

zbentley wrote: post #285
paffdragon wrote: post #151

Earlier quoted context omitted.

That moves your API layer to the client library you need to distribute and build for your customers in programming languages they support. There are some cases where a thick client makes sense, but usually easier to do it server side and let customers consume the API from their env, it is easier to patch the server than to ship library updates to all users.

I think most of the discussion in this thread assumes that “customers” of the interface are other groups in the same organization using the database for a shared overarching business/goal, not external end user customers. For external end users, absolutely provide an API, no argument here. The internal service interactions behind that API are a less simple answer, though.

It's definitely worse for external customers, of course. But it's still not that easy even for internal customers. The main problem is that usually the tables exposed are not meant to be public interfaces, so the team takes on an external dependency to their internal schema. And that other team could have completely different goals and priorities, speed and size, management and end users with different requirements. At some point the other team might start to ask the first team for adding some innocent looking fields to their internal table for them. Also first team might need to make changes to support their own service that might not be compatible with the other team. The other team making queries that are not in control of the team owning the DB, which could impact performance. If possible, it is better to agree on an API and avoid depending on internal implementations directly even for internal customers. There are always some exceptions, e.g. very close or subteams under same management and same customers could be fine. Or if the table in question was explicitly designed as a public interface, it is rare, but possible.
Post reply on HN