Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dolt is Git for data

dolthub.com

Post reply on HN
Print view View original

81–90 of 196 posts

Re: Dolt is Git for data

#81

Post by visarga »

jdoliner wrote: post #39
timsehn wrote: post #25

Earlier quoted context omitted.

Here is an earlier blog we published on comparison's to Pachyderm: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-... We got a blog on the storage system coming on Wednesday. It's a mashup of a Merkle DAG and a B-tree called a Prolly Tree. It comes from an open source package called Noms ( https://github.com/attic-labs/noms ). I'm not familiar with CRDT. Will read up on that.

Weighing in as Pachyderm founder. The post Tim links here is a very apt description of what Pachyderm does. We're designed for version controlling data pipelines, as well as the data they input and output. Pachyderm's filesystem, pfs, is the component that's most similar to dolt. Pfs is a filesystem, rather than a database, so it tends to be used for bigger data formats like videos, genomics files, sometimes database…

How does Pachyderm deal with GDPR requests. Is it possible to remove a file not just from the present but also from the history? It would be no use to delete a file on GPDR request from the current version while still keeping it around in past commits.

Re: Dolt is Git for data

#82

Post by irrational »

timdorr wrote: post #7

Any reason or history behind the name? It means "a stupid person", which seems like a bad choice IMHO: https://www.merriam-webster.com/dictionary/dolt

Git - (Chiefly British Slang) An unpleasant, contemptible, or frustratingly obtuse person.

https://www.ahdictionary.com/word/search.html?q=git

Dolt - A stupid person; a dunce.

https://www.ahdictionary.com/word/search.html?q=dolt

Re: Dolt is Git for data

#83

Post by staflow »

peteforde wrote: post #36

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…

1. Develop a solution with catchy name.

2. ONLY and only after it’s finished, start looking for problem that fits that solution.

3. Realize there is none.

4. ?????

5. Profit!

Gee I wonder why our economy is so ineffective..

Re: Dolt is Git for data

#84

Post by irrational »

dnautics wrote: post #19
StavrosK wrote: post #10

Earlier quoted context omitted.

What does git mean?

honestly, maybe this reflects my americanness but I presumed it was derived from the (western film/culture) word, a corruption of 'get'. Today I learned that it means something else in british.

I learned about the British meaning from reading the Harry Potter books.

Re: Dolt is Git for data

#85

Post by waynesonfire »

philipov wrote: post #51
peteforde wrote: post #36

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…

Git succeeded because it was free, and then business models were able to be built up around the open-source ecosystem after a market evolved naturally. There is a need, but if you go into it trying to build a business from scratch, you're going to have a bad time.

being free is a huge plus; but more equally or even importantly, it's a better product.

Re: Dolt is Git for data

#86

Post by visarga »

aantix wrote: post #61

Slightly related - how does ML track new data input and ensure that the data hasn't introduced a regression? I would assume there's an automated test suite, but also some way of diffing large amounts of input data and visualizing those input additions relative to model classifications? What are the common tools for this?

You generally can't analyse the accuracy of an ML system by each individual piece of data in the training set. Each batch of examples slightly changes the model making their updates interact and combine during the training process, so it becomes extremely difficult to assign the contribution of individual examples. Of course you could retrain the model leaving one example out, but that would be exceedingly slow and the result would be inconclusive from a single run because the stochastic noise of the training process is larger than the effect of removing or adding one example.

Related areas are confidence calibration, active learning and hard example detection during training. Another approach is to synthesise a new, much smaller dataset that would train a neural net to the same accuracy of the original larger dataset.

Re: Dolt is Git for data

#87

Post by chmaynard »

staflow wrote: post #83
peteforde wrote: post #36

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…

1. Develop a solution with catchy name. 2. ONLY and only after it’s finished, start looking for problem that fits that solution. 3. Realize there is none. 4. ????? 5. Profit! Gee I wonder why our economy is so ineffective..

Step 4 is where you create a snazzy website with lots of seductive images, clever animations, and "Free Trial" buttons.

Re: Dolt is Git for data

#88

Post by peteforde »

drewmol wrote: post #67
peteforde wrote: post #36

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…

Do you consider your effort a mistake looking back? Were you able to pay yourself fairly or was it a losing investment?

The answer to that can only be nuanced.

On one hand, startups are as exciting as bank heists. You put together an amazing team, do your homework as best you can, then get killed trying to execute your perfect plan. I'm proud of what we built and we all learned a lot the hard way.

On the down-side, building a company is emotionally, physically, mentally exhausting. It wasn't really a matter of whether I could pay myself fairly; I drew a typical programmer salary along with everyone else.

However, the important detail is that you're spending someone else's money and every penny of it represents someone you respect putting their trust in you, and you feel the weight of that every day.

Ultimately, I don't exactly regret it but I certainly wish that we weren't so convincing that we convinced ourselves of a market opportunity that we couldn't access or didn't exist at all. There was so much heat for "data" in 2011 it really seemed like we just had to show up with an amazing product.

We were wrong.

Re: Dolt is Git for data

#89

Post by dnautics »

hinkley wrote: post #34
dnautics wrote: post #19

Earlier quoted context omitted.

honestly, maybe this reflects my americanness but I presumed it was derived from the (western film/culture) word, a corruption of 'get'. Today I learned that it means something else in british.

You can hardly watch more than a few hours of british comedy without someone being called a 'stupid git' at some point.

Sadly my consumption of British comedy has been -it crowd (of course), -monty python, and -mr bean (no words)

It's such a short word I'd probably have missed it or misheard it as 'cad' or something.

Re: Dolt is Git for data

#90

Post by kidintech »

peteforde wrote: post #36

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…

Palantir is successful as well, but no clue what the hateboner on HN for them is, since it never even gets mentioned in these threads.
Post reply on HN