101–107 of 107 posts
Earlier quoted context omitted.
Good devs use all three platforms. Or at least two.
Use, because they have to. This being said they usually develop more in one than the others. I have not met until now anyone who was equally proficient in developping software across all platforms/environments.
...in your very uninformed opinion...
Hi Hacker News. I'm one of the founders of the Noms project and Attic Labs, the company behind it. Happy to answer any questions. In the meantime, as long as I've got your attention, here's a few new stuffs we've been working on since last time Noms was discussed here in August: - A prototype query language, and a demo of how to create indexes in Noms: https://www.youtube.com/watch?v=fv6_T5yaWns - Support for merging…
Looks really interesting! This may sound like a stupid question, but do you have any publication I can cite by any chance?
Weird choice for the uri: http://localhost:8000::dbname Why not http://localhost:8000/dbname ?
Wow, I missed that. That violates the URI spec, assuming the author(s) were intending to use a URI. https://tools.ietf.org/html/rfc3986 (see "3.3. Path").
data Database = InMemory | LevelDB Path | ViaHTTP URL
newtype DataSet = DS Text
newtype Hash = Hash Text
data Accessor = AccessDS DataSet | AccessValue (Either DataSet Hash) Path
type DBAccessor = (Database, Accessor)
Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?
This seems like an interesting project that tackles some of the data versioning stuff. However, I believe that, at least in data science, we need data versioning closely tied to the analyses themselves for complete reproducibility.
That is, we need the versioning tied to the inputs/outputs of data pipeline stages, such that we can reproduce pipeline runs at any time and incrementally improve and run pipelines based on diffs in data.
As mentioned elsewhere in the comments, Pachyderm (http://pachyderm.io/) does exactly this. Working both as git for data, but also enabling data pipelining and analyses with the data versioning.
I really like this, I've always thought that git needed to support diff modes different from textline-based because even if this is fit for most programming languages what you really what is to see differences between ASTs (take into account those absurd change counts when just changing the indentation or imagine a normal diff of LISP source). Maybe there's some way of replacing git with noms to get there(even if it…
Python code, if interested: https://github.com/WorldMaker/tokdiff
Earlier quoted context omitted.
Wow, I missed that. That violates the URI spec, assuming the author(s) were intending to use a URI. https://tools.ietf.org/html/rfc3986 (see "3.3. Path").
Right, it intentionally violates the URI spec by appending something to the end of it. The data structure they're storing has a natural pair-of-structures at its top level: data Database = InMemory | LevelDB Path | ViaHTTP URL newtype DataSet = DS Text newtype Hash = Hash Text data Accessor = AccessDS DataSet | AccessValue (Either DataSet Hash) Path type DBAccessor = (Database, Accessor) They elected to basically enc…
Earlier quoted context omitted.
Can you point me at some? Because I've tried a few immutable data stores and been disappointed every time. Given about 10 GB of JSON structures, I keep finding things that can't outperform the boring combo of: * Convert the versioned data to tab-separated values * COPY it into Postgres every time * Hope Postgres can act immutable enough even though it wasn't designed to be The closest I've come to improving this situ…
(Disclaimer - I work at pachyderm) http://pachyderm.io Pachyderm is git for data. We work hard to make sure we can store data of different types (binary, text, json) efficiently. We also work hard to give you good mechanisms to read the data in a distributed way. I'd be curious how this suits your purposes.
While I can see how a git-based filesystem can help with some use cases, does it do any kind of indexing at all? I see that the FAQ recommends exporting the data from Pachyderm into PostgreSQL, which leaves me where I am now.