Cool! Can you add an append function?
Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store
41–50 of 66 posts
Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store
#42Earlier quoted context omitted.
Good point on 2 and 3. Should just involve checking the previous commands return code before echoing the uuid. I'll get that patched up and ship a new version in a bit. Not sure what you mean exactly by 1 and 4 though.
With 1, you are trusting the directory returned by uuid will be unique --it should be, but it might not be, in particular if it is based on time and I run in two threads. With 4, I find once a directory has about 100,000 files in it, things get bad, from the simple (ls * won't work) to the nastier (git starts using huge amounts of space, you'll hit github's size limit even though all your files are quite small).
I would not let anyone close to the code I'm working on if he wanted to verify whether a properly generated UUID clashes with some previously generated UUID.
You could just as do stuff like "int x=1; assert(x == 1);" and so on. And it would be just as futile.
Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store
#43Hint: If you really want to use your filesystem as a database (and don't mind the haters, there's a long and storied history of doing just this), make sure you break it up into many subdirs usually based on the first few characters of the uuid. Example: ./jasondir/aa/bb/cc/aabbccdd You won't like what happens when you put 100k files in one directory.
There was an old document management system called Keyfile that did just that.
Git does it too.
Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store
#44Earlier quoted context omitted.
I agree that "serverless" now mainly means "Function as a Service." The goal here is convey that there is no server component of the database, similar to SQLite and unlike PostgreSQL. What is the correct terminology to highlight this architectural distinction?
What's wrong with 'local'? Serverless is a terrible word to describe this.
I always thought they were "embedded" databases.
Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store
#45Earlier quoted context omitted.
There is already an issue about supporting sharding https://github.com/nodesocket/jsonlite/issues/15 .
I thought it was "serverless"? what does "sharding" have to do with anything? I'm sorry but it seems like you throw around some vocabulary that doesn't describe your project properly, to make it look bigger than it really is. That's nice marketing but it does not feel like there is much effort put into it, when I looked at your bash script.
"The word shard means a small part of a whole."
The idea is breakup uuid keys into sub directories based on the first couple of letters to prevent file system performance issues. Seems like the right usage of the word from a dictionary perspective.
Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store
#46Serverless means anything really, a cli app is now serverless. I didn't know I was writing serverless applications for 20 years then. Serverless was already a bad buzz word, it gets more and more meaningless by the hour. By the way sqlite just got support for JSON : https://www.sqlite.org/json1.html off-topic : I'd like to see more books on database implementation for beginners. Of all the crap load of CS books that…
No, serverless database is not a recent buzz word, but a term frequently used to describe DB that doesn't need a separate server running to which clients connect: compare MySQL, PostgreSQL against BDB, LMDB, SQLite.
In my communities we called the library-style dbs "embedded". I've seen serverless used in this way but it didn't seem common.
Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store
#47Earlier quoted context omitted.
With 1, you are trusting the directory returned by uuid will be unique --it should be, but it might not be, in particular if it is based on time and I run in two threads. With 4, I find once a directory has about 100,000 files in it, things get bad, from the simple (ls * won't work) to the nastier (git starts using huge amounts of space, you'll hit github's size limit even though all your files are quite small).
I would not let anyone close to the code I'm working on if he wanted to verify whether a properly generated UUID clashes with some previously generated UUID. You could just as do stuff like "int x=1; assert(x == 1);" and so on. And it would be just as futile.
In this case, you are relying on what installed program or function is called when you invoke uuid in the shell. It seems reasonable to me to check if it is having sensibly, if the reliability of your program counts on it. I have no idea what dodgy uuid programs you might find installed.
Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store
#48Earlier quoted context omitted.
hasn't the file system improved to the point where this is less of a problem?
Even if it's running on a filesystem that deals with it well, standard utilities like 'ls' don't deal with it well, as the default behavior is to sort. You end up having to look up obscure options, like 'ls -U' to disable sorting. You can also run into issues like "Argument list too long". ARG_MAX is larger on linux than it used to be, but it's pretty short on older kernels. I assume similar issues might exist on oth…
I'm not sure that's so much of a problem, I just tried:
mkdir uuid; cd uuid
uuid -v 4 -n 1000000 |\
while read uuid; do
touch $uuid;
done
time ls uuid|wc -l
425621
real 0m1.796s
user 0m1.552s
sys 0m0.240s
This is in a hyper-v vm on a Surface 4 pro/i5.
The fact that the "uuid" program can quickly generate uuids make me wonder if maybe one approach would be to generate uuids to (a) fifo(s), and then let db thread/processes read uuids from the other end?
Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store
#49Why not just use SQLite? Its JSON1 extension is pretty good, and then you get all the rest of SQLite's goodness as part of the package. https://sqlite.org/json1.html
Re: Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store
#50Earlier quoted context omitted.
No, serverless database is not a recent buzz word, but a term frequently used to describe DB that doesn't need a separate server running to which clients connect: compare MySQL, PostgreSQL against BDB, LMDB, SQLite.
In my communities we called the library-style dbs "embedded". I've seen serverless used in this way but it didn't seem common.
Not every serverless database is embedded, but most of them - those that are also called "database library" are. Some examples of DBs that are both embedded and call themselves serverless:
- SQLite: https://sqlite.org/serverless.html
- UnQlite: https://www.unqlite.org/features.html#serverless
Serverless is used to highlight this specific property of them. The OP's database is NOT embedded, but it is serverless.