Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Noms – A new decentralized database based on ideas from Git

medium.com

Post reply on HN
Print view View original

151–160 of 180 posts

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#151

Post by adamhepner »

aboodman wrote: post #64
archgoon wrote: post #62

Earlier quoted context omitted.

This is actually extremely common. For example, if you browse the UC Irvine ML datasets https://archive.ics.uci.edu/ml/index.html You'll find that many are in csv format. If you do a search on data.gov http://catalog.data.gov/dataset#sec-res_format You'll see that it's about as popular as JSON. Also, the World Health Organization http://www.who.int/tb/country/data/download/en/ Also, many of the datasets at kaggle are…

Right. A shocking amount of public data is distributed this way. Also, we routinely talk to developers who complain about the difficulty of consuming data snapshots from partners, parsing it, trying to understand how it has changed since last time, etc. With high value datasets, people frequently build an API to combat these problems. But it's hard to design a good API, and even if you succeed, it has to be secured,…

Not only public data. At my main project I'm testing systems that generally crunch data from various sources, and yes, most of them are in CSV format, and then we process them only slightly (some filtering, aggregation, translation), and spit other CSVs out. I was amazed that the company had not bothered creating more... civilized (?) solution for internal data processing - but I guess that since it works, there's no drive to change it on a whim.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#152

Post by rejschaap »

nartz wrote: post #102

So, i realize this project is early, but it would be EXTREMELY helpful to walk through someone's use case - like, who is the target here? A business analyst who iterates on cleaning / analyzing small excel csvs? Or someone else? After watching the screencast, all I saw was a bunch of commands explained (could have read the docs for that), instead, I'd like to walk through a use-case where this solves someones problem…

This is clearly a technology-driven project. Which is fine, maybe business/use cases will emerge.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#153

Post by IanCal »

adamhepner wrote: post #151
aboodman wrote: post #64

Earlier quoted context omitted.

Right. A shocking amount of public data is distributed this way. Also, we routinely talk to developers who complain about the difficulty of consuming data snapshots from partners, parsing it, trying to understand how it has changed since last time, etc. With high value datasets, people frequently build an API to combat these problems. But it's hard to design a good API, and even if you succeed, it has to be secured,…

Not only public data. At my main project I'm testing systems that generally crunch data from various sources, and yes, most of them are in CSV format, and then we process them only slightly (some filtering, aggregation, translation), and spit other CSVs out. I was amazed that the company had not bothered creating more... civilized (?) solution for internal data processing - but I guess that since it works, there's no…

CSV for shoving files around (or TSV or whatever similar thing) is great because it generally just works. I can throw it into virtually any language or system, open and read it myself, grep it, check it with any platform. I can often get away with just looking at the files and absolutely nothing else, though a data dictionary is hugely appreciated.

I don't need to make sure I've got postgres 9.5 setup with a particular user account & set configs for the password, start ES (but not version Y because of a feature change) on port Z, etc. I don't need to manage making sure the two branches I'm looking at don't overlap or try to write to the same database. Keeping multiple results and comparing their output can be easily done as they're just files to be moved. Small tasks that read a file and spit out another can be checkpointed just by making them look to see if the file they expect to create already exists.

I'm hugely in favour of CSV for external data too. Sure, provide other options as well, but I love that the "get all the data" command can be as simple as a curl command. I don't want to read your API docs and build something custom that tries to grab everything, I don't want to iterate over 2M pages, I don't want to deal with timeouts, rate limits, etc. Just give me a URL with a compressed CSV file.

All the problems that come along with it, for me, are related to poor data management which I doubt a format change would fix.

Maybe CSV isn't the best internally, but for a vast amount of cases it's nearly the best and gives you a lot of flexibility. My general advice would be to start with CSV unless you've got a good reason not to, and then try and move to a different line based thing (jsonl, messagepack?). It is highly unlikely to be the biggest problem you have with your data, and the time spent putting it into a more "sane" format is often (in my experience) better spent on QA and analysis of the data itself.

I'd say the current problem is that lots of data is available only either in excel files, pdfs, and APIs pointing to a possibly constantly changing data store.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#154

Post by rejschaap »

Interesting project, would just like to say that the Git workflow isn't that great and CVS isn't that bad.

The Git workflow is quite complicated and will probably not appeal to people who typically just use Excel for everything.

It is true that CVS is messy, but its strength is that it is really simple, and it can easily be fixed.

Also, CVS can be versioned with Git quite well in many cases.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#155

Post by msane »

bkalman wrote: post #74
zphds wrote: post #70

Earlier quoted context omitted.

But isn't ` / ` more or less similar to ` :: `? The only difference is the choice of a delimiter to disambiguate between a database and a dataset. For me, the first scheme is much more familiar.

Say we did just do / . What does the path " http://demo.noms.io/cli-tour/sf-fire-inspections/raw" refer to? Is the database " http://demo.noms.io" and the dataset "cli-tour/sf-fire-inspections/raw"? Is the database " http://demo.noms.io/cli-tour/sf-fire-inspections" and the dataset "raw"? In our sample data (see https://github.com/attic-labs/noms/blob/master/doc/cli-tour.... for example) we actually have this exact p…

:: breaks the url for clients / is not supported in the URL specs. Use the fragment or query.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#156

Post by adekok »

josephg wrote: post #110
rafaelweinstein wrote: post #106

Earlier quoted context omitted.

This is a question that we've gotten quite a bit. It's our view that there's no magic solution to conflicts. There are logical conflicts in the real world that must be arbitrated. That said, it's a surprisingly basic thing, but just knowing what changed from party (a) and party (b)'s perspective (relative to their most recently agreed-upon state) is somewhat rare or ad-hoc in existing systems. In noms, you can direct…

I've played with this problem on and off over the last few years. ShareDB[1] is powered by JSON OT[2], in which each change describes the meaning behind what you're trying to do. (For example, 'increment counter' is different from 'change counter from 2 to 3'. They look the same, but behave differently in the case of conflicts). Just knowing what changed often isn't enough to do proper resolution. I've spent years on…

> Just knowing what changed often isn't enough to do proper resolution.

This is why my git commit logs are sometimes:

     perl -p -i -e 's/FOO/BAR/g' $(find src -name "*.[ch]" -print)
Having a "high level" description of what changed makes manual merges easier.

I wish git had conflict resolution like this. Treating data as an arbitrary sequence of bits is general and correct. Treating data as having a particular format can be useful, too.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#157

Post by Lazare »

im_down_w_otp wrote: post #135
Lazare wrote: post #133

Earlier quoted context omitted.

> How is, "whoever hits the queue most often" a useful deterministic resolution strategy? It's a deterministic resolution strategy, and is thus useful. > I guess it's functionally no worse than wall-clock time or something, Wall-clock time is not deterministic; therefore it's far worse. When dealing with distributed systems, deterministic processes are critical. Multiple systems all being right is awesome, but multip…

Is it deterministic in a way that's useful? From the perspective of the end-user its going to appear random because they don't control the system environment where "highest revision number" can mean something useful to them. In fact, the CouchDB guide even alludes to this when they talk about not relying on this scheme for complex conflict resolution it seems. Two nodes split. A and B. Say there are 100 updates to A…

> Is it deterministic in a way that's useful?

Yes, because it allows to maintain a consistent state across distributed nodes.

> From the perspective of the end-user its going to appear random

...but consistent. If every node picks a random revision on conflict, then when multiple clients try to continue editing, they'll end up increasing the conflicts.

> The split heals, the system picks B because 500 > 100, but the write you actually want to dominate is A. The user can't control which replica gets hit more often, or when a split happens, so while this might be deterministic inside the DB it is semantically random from the user's perspective.

Yeah, but what happens if B picks B, and A picks A? Now the write you're looking for is either there or not there, depending on on which node you're talking to.

> how would this look any different to the user?

Everything is going to look random to the user, no?

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#158

Post by espadrine »

im_down_w_otp wrote: post #123
espadrine wrote: post #120

Earlier quoted context omitted.

Google's Spanner, for instance, relies on their TrueTime design, which requires having a GPS clock and an atomic clock on each datacenter, I believe. Most designs simply rely on NTP or a similar time synchronization system. Another approach is to maintain total order of writes. Assuming some form of consensus protocol to determine write order, the unicity of the order ensures synchronization. That design, however, te…

Time isn't a reliable resource in this context.

They mention striving to support many contexts. In the demo, they showcase offline editing of a single CSV entry. If the granularity is the atomic types, then a conflict only occurs when the very same field in the very same row is concurrently edited.

Then, the system can show the conflict and offer a default that keeps the operation with the highest timestamp, or if the timestamps are identical, the one with the highest hash.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#159

Post by penetrarthur »

nartz wrote: post #102

So, i realize this project is early, but it would be EXTREMELY helpful to walk through someone's use case - like, who is the target here? A business analyst who iterates on cleaning / analyzing small excel csvs? Or someone else? After watching the screencast, all I saw was a bunch of commands explained (could have read the docs for that), instead, I'd like to walk through a use-case where this solves someones problem…

CMS

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#160

Post by penetrarthur »

tlb wrote: post #34

Strawman marketing alert: "The most common way to share data today is to post CSV files on a website". Maybe there are a bunch of people that still do that somewhere, but if so, they ain't early adopters of decentralized database technology and so not your target customers. It's always better to talk about what your most likely customers are doing now.

And Dropbox is just an FTP with SVN.
Post reply on HN