81–90 of 180 posts
Very cool. I would love to have something like this production ready. Some day... Anyone who finds this interesting may also be intrigued by Irmin [0] - a library for applications to persist data in a git-compatible format. [0] - https://github.com/mirage/irmin
Not only has Irmin been around longer (with full JS support thanks to js_of_ocaml) it also has a pretty big deployment under its belt.
I haven't had a chance to dig into the code yet, but I notice that you say two replicas of the same database can be disconnected, altered, and then merged. Could you explain how Noms takes care of that, particularly in the case of collisions?
Earlier quoted context omitted.
But isn't ` / ` more or less similar to ` :: `? The only difference is the choice of a delimiter to disambiguate between a database and a dataset. For me, the first scheme is much more familiar.
Say we did just do / . What does the path " http://demo.noms.io/cli-tour/sf-fire-inspections/raw" refer to? Is the database " http://demo.noms.io" and the dataset "cli-tour/sf-fire-inspections/raw"? Is the database " http://demo.noms.io/cli-tour/sf-fire-inspections" and the dataset "raw"? In our sample data (see https://github.com/attic-labs/noms/blob/master/doc/cli-tour.... for example) we actually have this exact p…
- I remember the Rails community trying to use `;` which broke Mongrel 1. Mongrel's parser was generated from the RFC. There was a huge flame war about that back in the day. The Rails core team at the time thought that Mongrel should make an exception to a reserved character. (And after all was said and done, it got changed back to `/` for that particular use-case).
- When working on IPv6 support about 3 years ago, one of the things I added to an open source Ruby project was IPv6 literals into the URL. This was a case of using `:`. Even though this was defined in the RFC specifying the literal, I found out at that time the Ruby standard library was written in a way that assumes you would never have `:` in the URL other than to delimit the port. I ended up having to do some workarounds for that.
That's with Ruby. I wouldn't be surprised if many other extant libraries parsing URLs that might break -- at least not without escaping those characters.
See: https://perishablepress.com/stop-using-unsafe-characters-in-...
You don't NEED ":". You NEED some sort of delimiter that can clearly distinguish between database and dataset; you happen to pick ':' to satisfy that. There might be a different delimiter that works better.
The other option is to not pretend that is a URL and call that something else.
Post-script: I think this project is a great idea. I'm looking forward to see how it turns out.
My queston is on scalability. You say "large datasets" on the website. What is large? 1x/10x/100x Terabytes? 1x/10x/100x Petabytes? What kind of access rates? Etc. Very general answers are okay -- I'm trying to wrap my head around whether this is even in the right ballpark for my world. Distinguishing current proof-of-concept vs. design-goal scale is okay too. Thanks!
Honest answer is: we don't know yet -- we're working our way up from the bottom. But we (cautiously) don't see any reason why the basic design shouldn't scale to very large (e.g. petabyte) datasets, and that is our eventual goal. That said, we do think there are a lot (even maybe the majority) of use cases in the GB-TB range.
I think the git-like features (history, merging) are very helpful for internal work, but when the dataset must be published, I think in most cases only the newest snapshot should be made available. But then the question is what format should it have...?
Earlier quoted context omitted.
But isn't ` / ` more or less similar to ` :: `? The only difference is the choice of a delimiter to disambiguate between a database and a dataset. For me, the first scheme is much more familiar.
Say we did just do / . What does the path " http://demo.noms.io/cli-tour/sf-fire-inspections/raw" refer to? Is the database " http://demo.noms.io" and the dataset "cli-tour/sf-fire-inspections/raw"? Is the database " http://demo.noms.io/cli-tour/sf-fire-inspections" and the dataset "raw"? In our sample data (see https://github.com/attic-labs/noms/blob/master/doc/cli-tour.... for example) we actually have this exact p…
You could easily say everything before either ? or ; always refers to a database, and use a query parameter or a semicolon to delineate a dataset. Or you resource paths:
Address a dataset:
http://demo.noms.io/?dataset=cli-tour/sf-fire-inspections/raw
http://demo.noms.io/;cli-tour/sf-fire-inspections/raw
http://demo.noms.io/dataset/cli-tour/sf-fire-inspections/raw
http://demo.noms.io/database/cli-tour/sf-fire-inspections
http://demo.noms.io/catalog/cli-tour/sf-fire-inspections
http://demo.noms.io/database/cli-tour/sf-fire-inspections;raw
http://demo.noms.io/database/cli-tour/sf-fire-inspections?dataset=rawEarlier quoted context omitted.
To play devil's advocate, Git "reinvented the wheel," but it was a much nicer wheel. Not saying this is to databases what Git was to versioning, but there's a reason to strive for that.
Git's author felt the alternative (gratis) systems were lacking. Noms's author, on the contrary, praises Git but doesn't build on it. He chooses to implement the same technology himself, and from the docs it's not clear to me why that is.
However, 1 question I have is:
Couldn't you just put a CSV/JSON file(s) behind VCS?
Eg. Drop my CSV/JSON file(s) onto github.com and then it will be version-controlled ?
Earlier quoted context omitted.
Say we did just do / . What does the path " http://demo.noms.io/cli-tour/sf-fire-inspections/raw" refer to? Is the database " http://demo.noms.io" and the dataset "cli-tour/sf-fire-inspections/raw"? Is the database " http://demo.noms.io/cli-tour/sf-fire-inspections" and the dataset "raw"? In our sample data (see https://github.com/attic-labs/noms/blob/master/doc/cli-tour.... for example) we actually have this exact p…
There are issues with using `:` in an URL, if you plan on using the URL in a way that's compatible with the extant software out there. I remember: - I remember the Rails community trying to use `;` which broke Mongrel 1. Mongrel's parser was generated from the RFC. There was a huge flame war about that back in the day. The Rails core team at the time thought that Mongrel should make an exception to a reserved charact…
So:
http://localhost:8000::dataset
may break code that tries to discern the host name. However:
http://localhost:8000/::dataset
Might not. Further, you could also reserve `_` in your scheme to refer to the default database:
http://localhost:8000/_::dataset
But as I mentioned in my previous reply, there may be unintended consequences. If this is something you guys want to do (and have HTTP/HTTPS URL compatibility) to check it out on different language/platform and see if your scheme breaks things. (And definitely see if Windows library assumes this; Windows file paths uses `:` as a reserved character)