Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In defense of simple architectures

danluu.com

Post reply on HN
Print view View original

111–120 of 196 posts

Re: In defense of simple architectures

#111

Post by hinkley »

lifefeed wrote: post #61

I was interviewing for software jobs recently, and while I was studying up on the "system design" portion I kept circling around the same insight that Dan Luu writes about so well here. I would sit down at an interview and try to create these "proper" system designs with boxes and arrows and failovers and caches and well tuned databases. But in the back of my mind I kept thinking, "didn't Facebook scale to a billion…

Before 'eventual consistency' was coined as a phrase, there was an old, powerful and deeply unsexy form of eventual consistency called "batch processing".

For small batches you do an interval at a convenient time, such as a time of day where the hardware is undersubscribed for some other task. As the batches grow then you have an online system that continues to work until you get rather far down the list of consequences in queuing theory. Once you get 24 hours and 1 minute of tasks per day you never catch up (it never ceases to amaze me how often I can find someone who will fight me on this point), and you must be aware that substantially before that breaking point, you can experience rather long average queuing delays.

But if your workload is spiky, you can smear 250 minutes of peak traffic out over 6-18 hours with no problems at all. You need a safe place to stash the queue and a little sophistication around recovering from failures/upgrades. Those aren't necessarily Simple tools, but if that's the most complex part of your system you're doing pretty okay.

Re: In defense of simple architectures

#112

Post by danielvaughn »

Beltalowda wrote: post #82
danielvaughn wrote: post #52

Earlier quoted context omitted.

Ok I feel very validated now - I'm not used to microservices so didn't know what was typical. It felt crazy, so good to know based on this comment's responses that it is indeed crazy. For example, in order to sign up a user...the client hits the /signup endpoint, which first lands on the server-gateway service. Then that is passed along to an account-service which creates the user. Then the accounts-service hits a NA…

Even the most fanatical microservices proponent will tell you that's just bonkers. At my very first programming job many years ago I was given a bunch of code written by a string of "previous guys" (mostly interns) over a period of 10 years or more and was told "good luck with it". I was the only developer, with no real technical oversight. It was my first "real" programming job, but I had been programming for many y…

Ha, yeah I created my fair share of complex stuff when I first started.

For some reason, even today, my personal projects tend to get very complex. But I think that’s just because I’m working on hard problems since they’re passion projects.

Re: In defense of simple architectures

#113

Post by quirino »

lifefeed wrote: post #61

I was interviewing for software jobs recently, and while I was studying up on the "system design" portion I kept circling around the same insight that Dan Luu writes about so well here. I would sit down at an interview and try to create these "proper" system designs with boxes and arrows and failovers and caches and well tuned databases. But in the back of my mind I kept thinking, "didn't Facebook scale to a billion…

I imagine this is one of the posts the comment is talking about https://rachelbythebay.com/w/2020/05/07/serv/

Re: In defense of simple architectures

#114

Post by stevenhuang »

dmoy wrote: post #90
endorphine wrote: post #86

Earlier quoted context omitted.

Doesn't the fact that its opened in append only mode (Linux) mitigate data races with regards to writes?

Here is my list of numbers: 1,Here is my list of letters: a,b,2,3,d

Although not a POSIX requirement, in practice for unix-like systems, file writes are atomic across concurrent writers.

You maybe thinking of stdio buffering, where calls to printf etc get split into multiple write calls. Then in those cases, it's possible to get errant interleaved writes.

Re: In defense of simple architectures

#115

Post by jameshart »

This doesn't sound very simple at all. It's a single codebase that handles all your mobile API interactions, authentication, account management, presumably usage tracking and notifications, and all your offline processing, all interacting with a single database and queue infrastructure? And that same codebase marshalls all that through a GraphQL API and implements a custom data protocol?

And you're calling that simple?

I've worked on monolithic codebases, and the one thing none of them have ever been is simple. They have complex interdependencies (oh hey, like database transaction scopes); they have that 'one weird way of doing things' that affects every part of the system (like, 'everything has to be available over GraphQL')...

Re: In defense of simple architectures

#116

Post by derefr »

pavlov wrote: post #34

There are some web apps still in production that I wrote almost a decade ago in Node+Express in the simplest, dumbest style imaginable. The only dependencies are Express and some third-party API connectors. The database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM. I figured these toys would be replaced pretty…

How do you feel about SQLite? Because when I read this architecture description, it mostly vibes with me, until I think about what happens to the data in the event of a power cut.

Is there logic in your app to potentially throw the last line away (incl. truncating it from the file) if it's invalid due to being the result of a non-atomic write? If so, seems a bit Not-Invented-Here compared to just using a (runtime-embedded) library that does that for you :)

Re: In defense of simple architectures

#117

Post by brundolf »

Beltalowda wrote: post #62
dmw_ng wrote: post #41

Earlier quoted context omitted.

> database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM Apps like this tend to perform like an absolute whippet too (or if they dont, getting them to perform well is often a 5 line change). It's really freeing to be able to write scans and filters with simple loops that still return results faster than a networ…

Another issue with "just a JSON file" as a database is that you need to be a bit careful to avoid race conditions and the like, e.g. if two web pages try to write the same database at the same time. It's not an issue for all applications, and not that hard to get right, but does require some effort. This is a huge reason I prefer SQLite for simple file storage needs.

A normal Express app (assuming it's one process per JSON file) shouldn't have that problem, because JavaScript is single-threaded

Re: In defense of simple architectures

#118

Post by mpyne »

Beltalowda wrote: post #78
gherkinnn wrote: post #47

At the risk of making an ad-hominem attack, I found this website unreadable. Minimalism is fine. But there comes a point when there's so little, it is nothing. danluu.com is a bucket of sand facing an overbuilt cathedral.

I set a user style in Stylus for danluu.com: body { font: 16px/1.6em sans-serif; max-width: 50em; margin: auto; } Can even add it manually in the inspector if you want.

I do something similar, except it's the CSS from whatever of the 'Better Fucking Website' hits come up for me on search each time.

I do wish I didn't have to do that though.

Re: In defense of simple architectures

#119

Post by hu3 »

chubot wrote: post #97
gherkinnn wrote: post #47

At the risk of making an ad-hominem attack, I found this website unreadable. Minimalism is fine. But there comes a point when there's so little, it is nothing. danluu.com is a bucket of sand facing an overbuilt cathedral.

Reader mode in your browser goes a long way, I think they all have it now

Thanks. In Chrome I had to enable reader mode in chrome://flags

Re: In defense of simple architectures

#120

Post by justinpombrio »

derefr wrote: post #116
pavlov wrote: post #34

There are some web apps still in production that I wrote almost a decade ago in Node+Express in the simplest, dumbest style imaginable. The only dependencies are Express and some third-party API connectors. The database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM. I figured these toys would be replaced pretty…

How do you feel about SQLite? Because when I read this architecture description, it mostly vibes with me, until I think about what happens to the data in the event of a power cut. Is there logic in your app to potentially throw the last line away (incl. truncating it from the file) if it's invalid due to being the result of a non-atomic write? If so, seems a bit Not-Invented-Here compared to just using a (runtime-emb…

If you're not writing data very often, this isn't a concern. For example, if you assume 5 microseconds to write a line, and 1 line written per hour on average, then the chance that the power goes out while you're writing a line is 10^-9, i.e. will never happen.
Post reply on HN