Json-Base – Database built as JSON files • Hacker News Board

Re: Json-Base – Database built as JSON files

#131

Post by waheoo » Fri, Jul 03, 2020, 12:48 AM UTC

btown wrote: post #77

bvinc wrote: post #16

Someone at my old company basically did this and put it into production. The first problem he encountered was that multiple connections couldn't both be using the database at a time without clobbering each other. "No problem," he thought, this is a good use case for micro services. A service sitting on top would ensure that there was only one operation being performed at a time. Next, his problem was that the databas…

> So the next problem was that you couldn't query quickly for things that weren't the primary key. So new columns had to be added called "opt_key1" and "opt_key2" where certain rows could put key values, and indexes could be added on those columns, so you could quickly query by it's first optional key, or it's second optional key. It's all fun and games until you realize DynamoDB works more or less the same way: http…

"webscale"

Re: Json-Base – Database built as JSON files

#132

Post by wafn » Fri, Jul 03, 2020, 12:56 AM UTC

First thing I looked for was JSON.stringify and yep there it is. Gonna be limited to databases that are V8's maximum string length, which changes every once in awhile and I believe is currently 2^58 (bytes). You may say that no JSON-based database should be that large and might be right, but still.

Re: Json-Base – Database built as JSON files

#133

Post by mbreese » Fri, Jul 03, 2020, 1:30 AM UTC

adamredwoods wrote: post #124

chmod775 wrote: post #114

You definitely shouldn't use this in production. Looking at the code there's: - race conditions everywhere. - bad and inconsistent formatting, which doesn't help with the - huge if-else monstrosities. - Also uses synchronous IO and asynchronous IO randomly. - Uses try-catch liberally, doesn't check the caught errors, and just re-tries blindly forever in some cases. If you do any parallel updates/inserts/removals with…

They're using readFileSync? Doesn't this lock up the thread?

It’s been awhile since I’ve looked at JavaScript like this, but I would guess that the thread lock would only exist so long as the read() call is running. That returns and then the write is initiated. It doesn’t look like there is a lock that exists throughout the update callback. If that’s missing, then if you had two threads/processes operating on the same table, you’d all but guarantee one of those update calls would be lost (in the best case scenario).

Or maybe there a different locking mechanism in place that my cursory look missed?

Re: Json-Base – Database built as JSON files

#134

Post by boring_twenties » Fri, Jul 03, 2020, 1:34 AM UTC

smt88 wrote: post #93

This is both more effort and a worse outcome than just using Postgres with JSONB data types.

I had the same thought, especially nowadays when you can just type `docker run --name mydb postgres` and wait like 3 minutes.

Re: Json-Base – Database built as JSON files

#135

Post by mbreese » Fri, Jul 03, 2020, 1:43 AM UTC

jarym wrote: post #31

What ever you try to do with JSON has probably been tried before with XML. Including XML databases. Now I’ll accept that XML databases has their use (especially if it involved storing and transforming third-party XML) but I can’t think of any good use for this when there’s SO many better options.

Having lived through XML databases and build systems, it’s sad/funny/interesting(?) to see this all play out again.

This has all happened before, and it will all happen again.

Re: Json-Base – Database built as JSON files

#136

Post by louis8799 » Fri, Jul 03, 2020, 1:44 AM UTC

bvinc wrote: post #16

Someone at my old company basically did this and put it into production. The first problem he encountered was that multiple connections couldn't both be using the database at a time without clobbering each other. "No problem," he thought, this is a good use case for micro services. A service sitting on top would ensure that there was only one operation being performed at a time. Next, his problem was that the databas…

Oh yeah, so basically had he pushed further, he would have realized to avoid corruption, he would need to implement "write ahead log (WAL)". An for the implementation of WAL and other performance concern, he would have realized that storing the JSON as string is not the way to go, he'd need to implement other binary data structure. Then he'd have realized that he had just invented another NoSQL DB.

Had he pushed further.....

Had he pushed further, he'd have raised funding for the newly invented NoSQL DB, and built a startup company on top of it.

Re: Json-Base – Database built as JSON files

#137

Post by Zarel » Fri, Jul 03, 2020, 1:53 AM UTC

lalaland1125 wrote: post #125

oxymoron wrote: post #112

Earlier quoted context omitted.

1. A cosmic ray storm turned all ASCII charactees into ECBDIC 2. Lightning struck the 12 V feed and upped the voltage to 10 MV, turning all 0 and 1’s into 6’s 3. Someone spilled a New England Pale Ale on the server 4. The process was assinated by the mysterious killer only known from his modus operandi of leaving OOM written in blood across the syslog 5. Birds nested within the server and fed all the SATA cables to t…

File renames are atomic. This is a solved problem: 1. Write your updates to a copy of the file. 2. Do an atomic rename of that copy to the original.

Renames aren't atomic on crash.

https://danluu.com/file-consistency/

Re: Json-Base – Database built as JSON files

#138

Post by aabbcc1241 » Fri, Jul 03, 2020, 1:54 AM UTC

I went further, to log the logical changes of state as json files. (aka command sourcing)

To reduce the IO overhead, I batch multiple json values into a larger file.

I don't need random access because I'll replay all the changes when the server start.

Going to open source the library soon.

Re: Json-Base – Database built as JSON files

#139

Post by methodin » Fri, Jul 03, 2020, 2:09 AM UTC

btown wrote: post #77

bvinc wrote: post #16

Someone at my old company basically did this and put it into production. The first problem he encountered was that multiple connections couldn't both be using the database at a time without clobbering each other. "No problem," he thought, this is a good use case for micro services. A service sitting on top would ensure that there was only one operation being performed at a time. Next, his problem was that the databas…

> So the next problem was that you couldn't query quickly for things that weren't the primary key. So new columns had to be added called "opt_key1" and "opt_key2" where certain rows could put key values, and indexes could be added on those columns, so you could quickly query by it's first optional key, or it's second optional key. It's all fun and games until you realize DynamoDB works more or less the same way: http…

A lot of things are hard to do in DynamoDB but for the scenarios it excels at there are not many peers.

Re: Json-Base – Database built as JSON files

#140

Post by marci » Fri, Jul 03, 2020, 2:25 AM UTC

russnewcomer wrote: post #83

marci wrote: post #71

Earlier quoted context omitted.

something that will work in 5 years on ancient windows7 and ie7? couchdb there's nothing to manage. On Windows, you install couchdb.msi or whatever, installed as a windows service, it automatically boots at startup time. Start IE7 go to localhost:5984/_utils, you get the DB's UI. At that point, all you did was installation. One click later, you created the first db called 'somedb', a click later, you created the firs…

That is a little more complex than, "Here's a .exe, it will save files into your My Documents. Click Export to make a zip, import to read someone else's zip." Plus generating instructions on how to do that would have been tough, and this way should be much easier to spread the app around. Basically, you don't have a bad idea, and if I were a couch expert or were not on the other side of the world, I might have chosen…

I understand, it was mostly or the-in-the-future-maybe-have-a-web-function-to-sync.

I also suggested Couch because outside of western countries, you seldom find laptops or desktops (outside of cities), but smartphones with a recent browser are ubiquitous, so if it worked on IE7 it would run anywhere, even in the most remote area with no/crappy network. And the first time I used couch, my programming "knowledge" was very basic HTML (no JS).

> if we went with the browser I would have to support mobile phones and I don't want to support mobile phones for this use case for a lot of other reasons.

Yeah... all in all I completely misinterpreted the requirement of your use-case.