Previous discussion from back in August: https://news.ycombinator.com/item?id=12211754
Noms – A versioned, forkable, syncable database
11–20 of 107 posts
Re: Noms – A versioned, forkable, syncable database
#12Earlier quoted context omitted.
If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!
No not really. CVS and git are basically the same thing but git is a lot better in many ways. I don't know if noms is really the git for data but those other tools are more like CVS in this analogy: clunky and slow.
They are so different it's hard to know what you mean by "basically the same thing"
Re: Noms – A versioned, forkable, syncable database
#13Earlier quoted context omitted.
If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!
Which ones are you thinking of? The nice thing about git is that it doesn't really require much of the server. I run my own "git hosting" with linode, apt-get, and ssh. Most of the solutions for immutable big data don't have that level of convenience, as far as I know. Anything which requires a lot of sysadmin and ops work will eventually become a commercial cloud service... git on the other hand is useful by itself,…
Most binary formats don't diff well.
Re: Noms – A versioned, forkable, syncable database
#14Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?
If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!
Can you point me at some? Because I've tried a few immutable data stores and been disappointed every time. Given about 10 GB of JSON structures, I keep finding things that can't outperform the boring combo of:
* Convert the versioned data to tab-separated values
* COPY it into Postgres every time
* Hope Postgres can act immutable enough even though it wasn't designed to be
The closest I've come to improving this situation was Kyoto Cabinet (unusable license) and rolling my own damn hashtable (it worked okay but adding new kinds of indexes was just unmaintainable, there's a reason databases should be made by experts).
Re: Noms – A versioned, forkable, syncable database
#15Looks farther along than http://dat-data.com/, another commendable distributed VCS for data. One distinction is that dat provides additional utilities for querying and compositing the data structures represented in any csv, json, and yaml files that stores.
Re: Noms – A versioned, forkable, syncable database
#16Earlier quoted context omitted.
If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!
Can you point me at some? Because I've tried a few immutable data stores and been disappointed every time. Given about 10 GB of JSON structures, I keep finding things that can't outperform the boring combo of: * Convert the versioned data to tab-separated values * COPY it into Postgres every time * Hope Postgres can act immutable enough even though it wasn't designed to be The closest I've come to improving this situ…
I was curious about Camlistore and Tahoe-LAFS at one point. I didn't investigate them very thoroughly, however. How do they compare?
Re: Noms – A versioned, forkable, syncable database
#17I've been wanting to use something like this.
But...
* It's a big jump from relational or noSQL DB's, so there aren't (m)any adapters that I can see for it for JPA, ActiveRecord, etc.
* I'd really like to see a benchmark for each noms implementation compared to postgres, mysql, oracle, and mssql server, if there is a way to do apples-to-apples.
* "noms" is unfortunately is really bad for SEO because noms is a common word in French. If it could be nomsdb or nomnomnoms or something less exactly French, that'd be better. It's going to be tough to find support online easily otherwise.
* SQL compatibility.
* Fault tolerance (how easily does it corrupt), HA, mirroring, full/partial replication, sharding, archival, partial history truncation, etc.
It seems a little like a dolphin jumping into a pool of hungry sharks. It might be more evolved and more capable in some ways, but it's going to get its ass handed to it on speed and lack of features.
Still- I can't wait to try it.
Re: Noms – A versioned, forkable, syncable database
#18Looks farther along than http://dat-data.com/ , another commendable distributed VCS for data. One distinction is that dat provides additional utilities for querying and compositing the data structures represented in any csv, json, and yaml files that stores.
One of the other design goals of Dat is to support continually-divergent forking, which they perceive as being useful for communities of analysts processing common datasets but to ultimately different ends. Of course, you never have to merge forks in git, but in it's current form they (dat devs) say that it's not really ideal.
Re: Noms – A versioned, forkable, syncable database
#19How is it that you have 2 reference implementations, written in 2 different cross platform environments, yet there is no support for Windows?
Why would I use this if I can't use it everywhere?
Re: Noms – A versioned, forkable, syncable database
#20Earlier quoted context omitted.
No not really. CVS and git are basically the same thing but git is a lot better in many ways. I don't know if noms is really the git for data but those other tools are more like CVS in this analogy: clunky and slow.
They are so different it's hard to know what you mean by "basically the same thing"
Yes there are significant differences in terms of feature, implementation, etc but they have the same goal: manage source code versions.