Live data from Hacker News

Json-Base – Database built as JSON files

github.com

181–189 of 189 posts

Re: Json-Base – Database built as JSON files

#181

Earlier quoted context omitted.

Being ignorant didn't make you a prima donna tho, as above says. Also, they have to have some clue about the domain, because the domain is their own problem and they're writing a solution for it. So I don't think we can really just someone as not having any clue about their own engineering challenges.... especially if they're working solutions to them.... Antirez said literally he didn't know about existing solutions…

> they have to have some clue about the domain, because the domain is their own problem They can be lifelong experts on their problem, yet have no clue about writing a database engine and low-level programming in general. > Antirez said literally he didn't know about existing solutions when he went to write redis Nobody is born with knowledge. The difference is that Antirez studied previous solutions, studied how to…

> They can be lifelong experts on their problem, yet have no clue about writing a database engine and low-level programming in general.

Why be so superior and judgemental except to compensate some insecurity? People are smart. They can learn and discover.

> The difference is that Antirez studied previous solutions, studied how to do it, and then applied that knowledge right.

Not really. From the man himself

In order to make them scalable we had to invent many new concepts, that were already known in the field most of the times, but we didn’t know, nor we cared to check. Problem? Let’s figure out a solution. We wanted to solve problems but we wanted, even more, to have fun. This was the playful environment where Redis was born.

http://antirez.com/news/133

He reinvented things that already experience that he didn't know about, and he didn't care, and it was fun. This is okay. And it's awesome.

> Instead, that person did the equivalent of building a bridge disregarding everything humans learnt about it since the Roman empire. It will not be a surprise if the bridge ends up collapsing.

It will be a surprise. People are not stupid. The guy is not incapable of solving problems. He's already solved them. If he faces more, why assume he can't solve them, too?

Software is really not like a bridge that's hard to keep patching. It's something you can keep reworking and improving. And along the way, who are you to judge that he won't come up with something even better than before?

If you have that attitude, you might have dismissed Antirez, and his nascent Redis in their very first stages, and poopooed his enthusiasm.

Everyone is learning. If your own child displays enthusiasm building something that from you perspective your can see many problems in, would you poopoo his enthusiasm? I think you'd find it inspiring and I think that's the response that works.

So there's no need to be so sour about people making new things. BTW if you're such a database expert mind linking to a database you've written yourself? Not because I think you having done that will make your criticisms valid, as I say it's not the point, but I'm just wondering if you can talk like an expert about something you haven't done.

You can start from a place of not knowing and end up creating something better than what exists. That's awesome and possible. if you don't believe that it's just your own pessimism, but that's not about anyone else, so don't pretend it applies to them, because this invention is a thing that really happens.

There's no barrier to entry to creating new stuff. Nobody has to prove to you they've passed some test to start working on whatever they want.

Now, even if they don't create something better, who cares? Even if what they create is the same or worse, by some measure, everything has tradeoffs, if they make it and use it, it must be better for some reason for them.

Their effort to build something does not make them bad, and saying you can use it in production is not doing something bad, because it's your responsibility what you choose to use, you're engineering is not controlled by what they say so you can't blame them for your tech choices. So you can't pretend that them creating and promoting their new work is somehow bad.

It's a good thing. Maybe you don't care for it, but that's not about them... it's just about you.

If you don't like it, you can have that view that you don't like it. That's okay. But it doesn't make it bad.

But I think you should be careful to not try to discourage people. Even you have that view... maybe just write it down yourself, or talk to people outside. Because I think poopooing creators and there new stuff is actually harmful.

Just say you were you 10 years ago, and somehow your comment combined with others to hurt Antirez feelings enough that he gave up working on redis and used something else. Then you'd be responsible for preventing redis getting into the world.

Now, maybe that's why you're in this game of poopooing. Because that is a LOT of power, to think your can hurt someone enough to get them to stop working on something, and then you get to change the future. Of course, someone like me, I think it's better if you change the future by creating your own stuff, rather than trying to destroy other people's stuff and hurt them, but I understand if that possibility is what attracts you to this. But do you really want this? And do you really want that karma?

You talk about responsibility as in his recommending is harmful, when you are actually the one responsible. I mean that, his recommending his work isn't harmful, because he's not trying to hurt anyone's feeling by doing that, and it's not his responsibility what others chose to use. But you're expressed discouraging and critical attitude, risks hurting something's feelings, and changing the outcome. So in this situation, you actually bear big responsibility for harm, but you're talking about someone else as responsible. So I think you either don't understand responsibility, or you're trying to distract, or to pretend to justify your own harming, or both. I just think you need to be careful doing this, because it could have unintended affects, and even if the affects are really what you intend because of the power... do you really want that karma?

Now maybe none of this is what you intend. But hopefully this has opened your eyes to why invention is good, and to some context in which your actions exist.

It seems I have a different view to you. I'm okay if you have the view you do, just be careful with other people's feelings and your own karma in where and how you express that.

Re: Json-Base – Database built as JSON files

#182

Earlier quoted context omitted.

> Next, his problem was that the database would get corrupt sometimes when something bad happened in the middle of writing the file. I'm not sure i understand how this can happen... unless you try to update JSON in-place (which is a very bad idea for any text-based format), what you do is encode/write the entire JSON from scratch. So either the file is written properly or it isn't written. Honestly from the entire me…

1. A cosmic ray storm turned all ASCII charactees into ECBDIC 2. Lightning struck the 12 V feed and upped the voltage to 10 MV, turning all 0 and 1’s into 6’s 3. Someone spilled a New England Pale Ale on the server 4. The process was assinated by the mysterious killer only known from his modus operandi of leaving OOM written in blood across the syslog 5. Birds nested within the server and fed all the SATA cables to t…

Disk writes aren't atomic but you know if they failed or not (unless your OS is lying to you but that is a problem with the OS, not your code).

Re: Json-Base – Database built as JSON files

#183

Earlier quoted context omitted.

> Next, his problem was that the database would get corrupt sometimes when something bad happened in the middle of writing the file. I'm not sure i understand how this can happen... unless you try to update JSON in-place (which is a very bad idea for any text-based format), what you do is encode/write the entire JSON from scratch. So either the file is written properly or it isn't written. Honestly from the entire me…

Here, have a read. Warning: if you haven’t done low-level development, you might walk away wondering if your entire life has been a lie. https://danluu.com/file-consistency/

I've read that at the past but this article has an issue:

> if there's a crash during the write

It never brings up how can there be a crash in the first place (also the entire article is too filesystem specific).

Re: Json-Base – Database built as JSON files

#184

Earlier quoted context omitted.

They are if you perform the correct sequence of fsync operations on both the file and the directories, and use a file system which is correctly implemented.

Renames aren’t atomic on crash .

What crash?

The article linked above never explains that part, it only assumes that it will happen. From the code it sounds as if the crash can happen in the OS itself (but then the entire kernel will crash). At that point things are completely outside your control and you might as well running on broken hardware.

Re: Json-Base – Database built as JSON files

#185
post #16

Someone at my old company basically did this and put it into production. The first problem he encountered was that multiple connections couldn't both be using the database at a time without clobbering each other. "No problem," he thought, this is a good use case for micro services. A service sitting on top would ensure that there was only one operation being performed at a time. Next, his problem was that the databas…

I used sets of flat JSON files as our "database" in the Wunderlist iOS and macOS clients.

Worked like a charm, never had a problem with it.

It was actually put in as a placeholder until we had time to think about a real storage solution, but it turned out we never needed anything more sophisticated, and were actually the fastest and most reliable clients we had. In fact, every time I encountered a performance problem I was hopeful that I would finally have a good reason to do that real implementation, but it invariably turned out to be a simple bug.

- Cocoa has -writeToFile:atomically:, which writes a new file and then renames, so no write-corruption

- We were lucky that lists had just the right granularity for a single file to be read/written atomically

- We likely wrote (quite) a bit more data than absolutely necessary, but I/O tends to have large fixed overheads so medium files tend to take around the same time as small files

- We did not do anything with the data on disk except read it, so not a DB

- We really did use files, not JSON strings inside SQLite

- We flushed to disk asynchronously, but as quickly as possible

Re: Json-Base – Database built as JSON files

#186
post #137

Earlier quoted context omitted.

File renames are atomic. This is a solved problem: 1. Write your updates to a copy of the file. 2. Do an atomic rename of that copy to the original.

Renames aren't atomic on crash. https://danluu.com/file-consistency/

Hmm...unlike the rest of the post, he just asserts this without support.

His sources seem to disagree, certainly with that kind of blanket statement. For example:

Our study takes a pessimistic view of file-system behavior; for example, we even consider the case where renames are not atomic on a system crash.

So this is clearly considered an outlier/unusual.

(e.g., a single 512-byte write or file rename operation are guaranteed to be atomic by many current file systems when running on a hard-disk drive)

[https://www.usenix.org/system/files/conference/osdi14/osdi14...]

I remember reading quite a bit about the (performance reducing) lengths filesystems go to in order to ensure consistency of directory entries even in case of a crash, and for example how "soft updates" were introduced to accomplish the same consistency with less of a performance degradation.

Looking at it from another angle, if you are running on top of a filesystem that cannot keep itself consistent, then you are SOL, there really isn't anything you can do to mitigate.

Just like we can't guarantee that we will be able to persist data that's in memory to disk if the OS is free to kill us at any time. "Best effort" it is, which means getting the data to disk as quickly as possible and not corrupting what is there.

Re: Json-Base – Database built as JSON files

#187
post #79
post #66

Earlier quoted context omitted.

> but more importantly avoid these hard-learned patterns later. Depends, I’ve known people who have gone through similar experiences and still poo-poo all those “unnecessarily bloated” solutions like a proper database.

I make crappy thrown together frontends and will probably forever poo-poo the 'unnecessarily bloated' js frameworks

Then you are a bad dev with no critical thoughts

Re: Json-Base – Database built as JSON files

#188

Earlier quoted context omitted.

Here, have a read. Warning: if you haven’t done low-level development, you might walk away wondering if your entire life has been a lie. https://danluu.com/file-consistency/

I've read that at the past but this article has an issue: > if there's a crash during the write It never brings up how can there be a crash in the first place (also the entire article is too filesystem specific).

Any crash. Power failure, hardware failure, kernel panic while working for another process, software bugs, etc.

It's impossible to make crash-free systems. If your goal is to maximize the chance that your data remains valid, you have to plan on that.

Re: Json-Base – Database built as JSON files

#189

I went further, to log the logical changes of state as json files. (aka command sourcing) To reduce the IO overhead, I batch multiple json values into a larger file. I don't need random access because I'll replay all the changes when the server start. Going to open source the library soon.

I'm also working on a similar system - a JSON datastore with a event stream so there is a full history of changes. However we are using Postgres as a backend. Our code is already technically Open Source, but it's not relatively stable and not doc'd yet, so I won't link.

I can see why people prefer sql database over plain files, for better edge cases handing. But that requires more configurations and resources.
Post reply on HN