Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store

github.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store

#21

Post by jakewins »

nodesocket wrote: post #5
CJefferson wrote: post #3

If you are going to invoke the sacred name of 'sqlite', then I expect to see rock-solid code and quality testing. Instead, there are two tiny tests, and I found a couple of issues in 30 seconds of looking at the code. EDIT: I should prove my point. 1) Race condition on calculating uuid (obviously won't be too serious for a good uuid implementation) 2) No check that file is written successfully. 3) No check when json…

Good point on 2 and 3. Should just involve checking the previous commands return code before echoing the uuid. I'll get that patched up and ship a new version in a bit. Not sure what you mean exactly by 1 and 4 though.

Many operating systems do not handle tens of thousands of file handles open simultaneously very well.

For instance, if you run this database on Linux, the RAM overhead for each document is 1K just for the file handle, see the comment at the bottom here http://lxr.free-electrons.com/source/fs/file_table.c

Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store

#22

Post by CJefferson »

nodesocket wrote: post #5
CJefferson wrote: post #3

If you are going to invoke the sacred name of 'sqlite', then I expect to see rock-solid code and quality testing. Instead, there are two tiny tests, and I found a couple of issues in 30 seconds of looking at the code. EDIT: I should prove my point. 1) Race condition on calculating uuid (obviously won't be too serious for a good uuid implementation) 2) No check that file is written successfully. 3) No check when json…

Good point on 2 and 3. Should just involve checking the previous commands return code before echoing the uuid. I'll get that patched up and ship a new version in a bit. Not sure what you mean exactly by 1 and 4 though.

With 1, you are trusting the directory returned by uuid will be unique --it should be, but it might not be, in particular if it is based on time and I run in two threads.

With 4, I find once a directory has about 100,000 files in it, things get bad, from the simple (ls * won't work) to the nastier (git starts using huge amounts of space, you'll hit github's size limit even though all your files are quite small).

Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store

#23

Post by tyingq »

nodesocket wrote: post #5
CJefferson wrote: post #3

If you are going to invoke the sacred name of 'sqlite', then I expect to see rock-solid code and quality testing. Instead, there are two tiny tests, and I found a couple of issues in 30 seconds of looking at the code. EDIT: I should prove my point. 1) Race condition on calculating uuid (obviously won't be too serious for a good uuid implementation) 2) No check that file is written successfully. 3) No check when json…

Good point on 2 and 3. Should just involve checking the previous commands return code before echoing the uuid. I'll get that patched up and ship a new version in a bit. Not sure what you mean exactly by 1 and 4 though.

On 4, the point might be that thousands of files in a single directory can create performance issues with readdir().

The usual workaround to that is to create subdirectories based on the first few characters of the filename.

Since you're using bash, you could do something like:

  $ echo 1df8be33-4392-471a-99af-3df967b87cb6 |sed -e 's/^\(.\)\(.\)/\1\/\2\/\1\2/'
  1/d/1df8be33-4392-471a-99af-3df967b87cb6
Edit: You may want more than 2 levels of directories, or directories with 2 character names instead of 1, etc. Apache's mod_cache defaults to 2 levels of 2 character names, but the filenames are base 64, so more possibilities than hex.

Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store

#24

Post by tyingq »

TheGreatPotatoe wrote: post #20
noonespecial wrote: post #19

Hint: If you really want to use your filesystem as a database (and don't mind the haters, there's a long and storied history of doing just this), make sure you break it up into many subdirs usually based on the first few characters of the uuid. Example: ./jasondir/aa/bb/cc/aabbccdd You won't like what happens when you put 100k files in one directory.

There was an old document management system called Keyfile that did just that.

Apache's mod_cache does this as well.

Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store

#25

Post by tyingq »

edraferi wrote: post #11
mtolan wrote: post #7

I think your title might not be the best; * Serverless typically implies something accessible from more than the host you're on * It's not zero configuration, there's at least 1 configurable parameter (albeit with a sensible default) * I'm actually not even clear why this is a document store limited to JSON, other than you're piping it though a JSON python module. Having done some things like this in the past, as you…

I agree that "serverless" now mainly means "Function as a Service." The goal here is convey that there is no server component of the database, similar to SQLite and unlike PostgreSQL. What is the correct terminology to highlight this architectural distinction?

"daemonless" makes the most sense to me, though I doubt that would catch on.

Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store

#27

Post by throwanem »

nodesocket wrote: post #4
CJefferson wrote: post #3

If you are going to invoke the sacred name of 'sqlite', then I expect to see rock-solid code and quality testing. Instead, there are two tiny tests, and I found a couple of issues in 30 seconds of looking at the code. EDIT: I should prove my point. 1) Race condition on calculating uuid (obviously won't be too serious for a good uuid implementation) 2) No check that file is written successfully. 3) No check when json…

Create some issues and pull requests and help out. :-)

Show me why your project is worth some of my finite time and effort, and I might. This looks like a toy project to me, and while there's nothing wrong with a toy project, I'm not seeing why it's more fun for me to play with this toy than some other of the myriad from which I have to choose.

Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store

#29

Post by deif »

edraferi wrote: post #11
mtolan wrote: post #7

I think your title might not be the best; * Serverless typically implies something accessible from more than the host you're on * It's not zero configuration, there's at least 1 configurable parameter (albeit with a sensible default) * I'm actually not even clear why this is a document store limited to JSON, other than you're piping it though a JSON python module. Having done some things like this in the past, as you…

I agree that "serverless" now mainly means "Function as a Service." The goal here is convey that there is no server component of the database, similar to SQLite and unlike PostgreSQL. What is the correct terminology to highlight this architectural distinction?

What's wrong with 'local'? Serverless is a terrible word to describe this.

Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store

#30

Post by Waterluvian »

I think there is a lot of feedback, mostly constructive criticisms. I would encourage you to embrace a lot of what everyone is saying and not to take it personally. You've done far better than most developers: you authored something and shared it with the world. To this day I'm still too afraid to show HN or Reddit my work.
Post reply on HN