In defense of simple architectures • Hacker News Board

Re: In defense of simple architectures

#121

Post by weaksauce » Thu, Apr 07, 2022, 1:10 AM UTC

uuyi wrote: post #88

pavlov wrote: post #34

There are some web apps still in production that I wrote almost a decade ago in Node+Express in the simplest, dumbest style imaginable. The only dependencies are Express and some third-party API connectors. The database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM. I figured these toys would be replaced pretty…

I would love to see more stuff like that. An application I have written recently for personal use is a double entry accounting system after GNUcash hosed itself and gave me a headache. This is based on Go and SQLite. The entire thing is one file (go embed rocks) and serves a simple http interface with a few JS functions like it is 2002 again. The back end is a proper relational model that is stored in one .db file. I…

sqlite really does very well with reads but not as much for locking writes. not saying it couldn't scale to many users but I think the other person is a bit optimistic on a double entry accounting app being only reads. I would imagine it could certainly easily serve a few hundred though if not a few thousand.

Re: In defense of simple architectures

#122

Post by pavlov » Thu, Apr 07, 2022, 1:12 AM UTC

derefr wrote: post #116

pavlov wrote: post #34

There are some web apps still in production that I wrote almost a decade ago in Node+Express in the simplest, dumbest style imaginable. The only dependencies are Express and some third-party API connectors. The database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM. I figured these toys would be replaced pretty…

How do you feel about SQLite? Because when I read this architecture description, it mostly vibes with me, until I think about what happens to the data in the event of a power cut. Is there logic in your app to potentially throw the last line away (incl. truncating it from the file) if it's invalid due to being the result of a non-atomic write? If so, seems a bit Not-Invented-Here compared to just using a (runtime-emb…

The AOF reader will discard and emit a warning about lines that can’t be parsed, but that’s the extent of it.

These apps are on Digital Ocean, and I don’t remember ever having unplanned downtime with them. They do sometimes migrate instances with advance notice, but that’s a clean shutdown.

I’m sure SQLite is a better choice for almost any app. My reason for not using it was to try avoiding dependencies out of curiosity, and also that I honestly really don’t like writing SQL — it just feels boring and error-prone. (Like eating celery, I know objectively it’s good for me.)

Re: In defense of simple architectures

#123

Post by weaksauce » Thu, Apr 07, 2022, 1:15 AM UTC

derefr wrote: post #116

pavlov wrote: post #34

There are some web apps still in production that I wrote almost a decade ago in Node+Express in the simplest, dumbest style imaginable. The only dependencies are Express and some third-party API connectors. The database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM. I figured these toys would be replaced pretty…

How do you feel about SQLite? Because when I read this architecture description, it mostly vibes with me, until I think about what happens to the data in the event of a power cut. Is there logic in your app to potentially throw the last line away (incl. truncating it from the file) if it's invalid due to being the result of a non-atomic write? If so, seems a bit Not-Invented-Here compared to just using a (runtime-emb…

> SQLite responds gracefully to memory allocation failures and disk I/O errors. Transactions are ACID even if interrupted by system crashes or power failures. All of this is verified by the automated tests using special test harnesses which simulate system failures.

from the sqlite about page. it's one of the most bullet proof and hardened pieces of software out there I think. it was made basically for exactly the ops use case of a file on disk. but who knows what the use case is for them. maybe writes to the db are few and far between so it's a fairly moot point.

Re: In defense of simple architectures

#124

Post by throwaquestion5 » Thu, Apr 07, 2022, 1:25 AM UTC

Beltalowda wrote: post #78

gherkinnn wrote: post #47

At the risk of making an ad-hominem attack, I found this website unreadable. Minimalism is fine. But there comes a point when there's so little, it is nothing. danluu.com is a bucket of sand facing an overbuilt cathedral.

I set a user style in Stylus for danluu.com: body { font: 16px/1.6em sans-serif; max-width: 50em; margin: auto; } Can even add it manually in the inspector if you want.

Thanks for sharing. Personally I also add background-color: #edd1b0; for any site I plan on read more than five minutes. For me is more pleasant to read compared with than a white background.

Re: In defense of simple architectures

#125

Post by djbusby » Thu, Apr 07, 2022, 1:26 AM UTC

brundolf wrote: post #117

Beltalowda wrote: post #62

Earlier quoted context omitted.

Another issue with "just a JSON file" as a database is that you need to be a bit careful to avoid race conditions and the like, e.g. if two web pages try to write the same database at the same time. It's not an issue for all applications, and not that hard to get right, but does require some effort. This is a huge reason I prefer SQLite for simple file storage needs.

A normal Express app (assuming it's one process per JSON file) shouldn't have that problem, because JavaScript is single-threaded

Is it possible a write is interrupted on it's turn in the event-loop, and crossed with another?

Re: In defense of simple architectures

#126

Post by brundolf » Thu, Apr 07, 2022, 1:32 AM UTC

djbusby wrote: post #125

brundolf wrote: post #117

Earlier quoted context omitted.

A normal Express app (assuming it's one process per JSON file) shouldn't have that problem, because JavaScript is single-threaded

Is it possible a write is interrupted on it's turn in the event-loop, and crossed with another?

Hmm. I wouldn't think so, but I don't actually know

Still, given the strategy at hand, the in-memory JS object (exclusively single-threaded) is the source of truth, and just gets mirrored in the file system (and doesn't get read again until the next startup). So you should have an eventual-consistency situation in the worst case (any racing issue between file-writes would just put the file in a stale state, and the next file-write would bring it back up to consistency)

Re: In defense of simple architectures

#127

Post by rr808 » Thu, Apr 07, 2022, 1:36 AM UTC

lifefeed wrote: post #61

I was interviewing for software jobs recently, and while I was studying up on the "system design" portion I kept circling around the same insight that Dan Luu writes about so well here. I would sit down at an interview and try to create these "proper" system designs with boxes and arrows and failovers and caches and well tuned databases. But in the back of my mind I kept thinking, "didn't Facebook scale to a billion…

Yeah I just did an interview where my design was a database, a few lambdas and a webserver and after I was thinking they must think I dont know much, I should have beefed it up a bit.

Re: In defense of simple architectures

#128

Post by eternityforest » Thu, Apr 07, 2022, 1:48 AM UTC

derefr wrote: post #116

pavlov wrote: post #34

There are some web apps still in production that I wrote almost a decade ago in Node+Express in the simplest, dumbest style imaginable. The only dependencies are Express and some third-party API connectors. The database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM. I figured these toys would be replaced pretty…

How do you feel about SQLite? Because when I read this architecture description, it mostly vibes with me, until I think about what happens to the data in the event of a power cut. Is there logic in your app to potentially throw the last line away (incl. truncating it from the file) if it's invalid due to being the result of a non-atomic write? If so, seems a bit Not-Invented-Here compared to just using a (runtime-emb…

SQLite doesn't do "lines" like a text file, or truncate anything(Or, it probably does internally, but that's not how users think of it).

It's a real SQL DB with real records and transactions, and it is one of the most trusted and reliable pieces of software ever made. Like, check out the change log to get a sense of how they do stuff.

Re: In defense of simple architectures

#129

Post by spurgu » Thu, Apr 07, 2022, 1:49 AM UTC

pavlov wrote: post #34

There are some web apps still in production that I wrote almost a decade ago in Node+Express in the simplest, dumbest style imaginable. The only dependencies are Express and some third-party API connectors. The database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM. I figured these toys would be replaced pretty…

I have literal dozens of these kinds (Node+Express under PM2) of small apps running around everywhere in production (almost 8 years = almost a decade ;). Using SQLite (when you need an actual DB) makes things a lot easier in this regard as well.

I've tried doing some thing in Python (my initial programming love) over the years but I keep going insane because every time I'm forced to read up on the state of ecosystem (choosing versions and package managers and whatnot) and it drives me insane. I just install Node+Express and can get to work immediately (and finish quickly).

Re: In defense of simple architectures

#130

Post by barbazoo » Thu, Apr 07, 2022, 1:50 AM UTC

rr808 wrote: post #127

lifefeed wrote: post #61

I was interviewing for software jobs recently, and while I was studying up on the "system design" portion I kept circling around the same insight that Dan Luu writes about so well here. I would sit down at an interview and try to create these "proper" system designs with boxes and arrows and failovers and caches and well tuned databases. But in the back of my mind I kept thinking, "didn't Facebook scale to a billion…

Yeah I just did an interview where my design was a database, a few lambdas and a webserver and after I was thinking they must think I dont know much, I should have beefed it up a bit.

It's important to justify the design you come up with. Explain why the design is simple, pros and cons and when you'd opt for a more complicated one to solve which particular issue.