This is nonsense masquerading as advice. "Add indexes ... but don't add too many" is a perfect example. It's 100% correct ... and also 100% something no one can actually change their actions based on ... which means it's also 100% worthless advice.
Good system design
181–190 of 400 posts
Re: Good system design
#182Earlier quoted context omitted.
What sort of application is regularly doing a query for “all data”?
Client report generation.
1) As soon as reporting requirements get serious, you build a data warehouse. Because odds are, the client will want to combine data from multiple systems in their reports anyway. If not today, then they will tomorrow.
2) such reports never need all the data, it's mostly about top N volume queries or month-over-month performance data. When a reporting application does query all the data, it's because it's building its own data warehouse so the query usually happens only once per day, at a specific time, which means the load is entirely predictable.
Re: Good system design
#183> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…
If you know that those are not the answers they are looking for, you can reasonably pass by modifying the answer only slightly, while still getting your point across. If you can't, you might be getting interviewed by people you do not what to work with and you should want to know that.
Except these are the people in your way of getting that job that could be potentially life/career changing for you financially or otherwise. In this market or depending on your situation that would be hard to ignore.
Re: Good system design
#184Earlier quoted context omitted.
Do you _want_ to work in these places? In my experience, if they expect you to run kube using kube in the interview, thats exactly what they do in their ststems as well.
These are the places that actually pay well.
There's another reason for that. Deep in my heart, I would love to be part of a team that works on truly data-intensive applications (as Martin Kleppmann would call them) where all the complexity is justified.
For example, I am more of the "All you need is Postgres" kind of software engineer. But reading all those fancy blog posts on how some team at Discord works with 1 trillion messages with Cassandra and ScyllaDB makes me envious.
Also, it seems that to be hired by such employers you need to prove that you already have such experience, which is a bit of a catch-22 situation.
Re: Good system design
#185> I’m often alone on this. Engineers look at complex systems with many interesting parts and think “wow, a lot of system design is happening here!” In fact, a complex system usually reflects an absence of good design. For any job-hunters, it's important you forget this during interviews. In the past I've made the mistake of trying to convey this in system design interviews. Some hypothetical startup app > Interviewer…
This goes back to "interviews go both ways". All those answers you gave are very reasonable and if I was your interviewer I'd pass you with flying colors. On the other hand if you're interviewing at a place that doesn't pass you with flying colors for those responses, that really says more about them than it does about you and may not be a great place to work. But to your point, many times one interviews for a job th…
If I was your interviewer, I would: respect your answers a lot, not be able to check off anything on my rubric, try to explain this in the debrief, get told we have to stick to the rubric to counter bias, and then watch while they pass on you for someone who decided to play architecture jenga instead. I would potentially even consider emailing you to apologize later, then not do it because I'd probably get in trouble for exposing us to liability or something because apologizing can be construed as admission of guilt.
Re: Good system design
#186> Schema design should be flexible, because once you have thousands or millions of records, it can be an enormous pain to change the schema. However, if you make it too flexible (e.g. by sticking everything in a “value” JSON column, or using “keys” and “values” tables to track arbitrary data) you load a ton of complexity into the application code (and likely buy some very awkward performance constraints). Drawing the…
> storing audit data in the same DB and it inevitably having functionality written against it, your audit data becoming a part of the business logic What's the "proper" way to do this? Separate DB? Separate data store?
Separate schema, no read permissions for the application identity is sufficient. It's not like "separate db" makes it magically unqueryable.
Re: Good system design
#187Never write an article about good system design. In all seriousness, this is an extraordinary subtle and complex area, and there are few rules. For example, "if you need data from multiple tables, JOIN them instead of making separate queries and stitching them together in-memory" may be useful in certain circumstances. For highly scalable consumer systems, the rule of "avoid joins as much as possible" can work a lot…
And by "system" we mainly meant "transactional website."
And this is the point. You need to narrow the scope to make something like this useful. Writing a paper on "Good transportation design" is kind of meaningless. Do you mean cars, trucks, boats, planes, spacecraft, scooters, fighters, tanks ? Do you mean roadways that can accommodate some subset ?
If you mean "transactional websites", and assuming you mean something like product catalogs and being able to purchase, that narrows it down quite a lot.
Or does it ?
For the majority of use cases, Craigs list, ebay, Amazon are the best fit.
Next in number of use cases are Wix/Square/etc where you design your UI.
Then comes all in one systems with UI/ORM based on Python/Ruby/etc where you need to design your own DB schema and UI, but the "design" is already done for you.
The next step is custom designed systems like the one the article talks about, where complete off the shelf is not suitable
And then there are the highly scalable systems
The article is perfectly fine if we are discussing custom designed but not necessarily the highest in scalability.
Re: Good system design
#188Also be careful not to reach for system design when you only need software design: https://lukerissacher.com/blog/optimizing_your_web_app
Re: Good system design
#189Excellent article. In this vein, are there any books, articles, or other media that we can learn more of these sorts of principles from?
Re: Good system design
#190Earlier quoted context omitted.
Audit tables are a big ask both in terms of programming effort to design and support them, and in terms of performance hit due to write amplification (all inserts and updates cause an additional write to an audit table). Whereas making a bool into a timestamp is free. Including timestamps on rows (including created_at and updated_at) are real bacon savers when you've deployed a bug and corrupted some rows and need to…
Audit tables are a dumb concept because they imply bolting on an actual source of truth in addition to the regular not so source of truth tables , and only if the programmer gets around to it (like documentation or logging or whatever else falls along the wayside).
This doesn't make sense to me-- if the regular tables don't capture the true state, then the audit tables based on them will not magically become a source of truth either.