Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

github.com

Post reply on HN
Print view View original

171–180 of 185 posts

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#171

Post by zomglings »

StreamBright wrote: post #169
zomglings wrote: post #162

Earlier quoted context omitted.

That would be really great. Let me know if there's any way we can help. Maybe if we released a small version of the dataset for testing/benchmarking and then I could take care of running the final processing on the full dataset?

That would be amazing. I get back my internet tomorrow and i can play with the dataset see how much we could optimize.

Hi StreamBright - just published the development version of the dataset also to Kaggle: https://www.kaggle.com/simiotic/github-code-snippets-develop...

Compressed, it's 471 MB. Uncompressed, just a little more than 3 GB.

If you want to get in touch with me in a better way than HN comments two good options:

1. My email is in my profile

2. You can direct message me (@zomglings) on the Bugout community Slack: https://join.slack.com/t/bugout-dev/shared_invite/zt-fhepyt8...

Looking forward to collaborating with you. :)

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#172

Post by Klwohu »

edgyquant wrote: post #154
Klwohu wrote: post #131

Earlier quoted context omitted.

This is the issue with names. Even though the project is called doit the DoIt stylizing makes it look problematic. It's a non-starter, hopefully the author makes a big change. Just choosing lower case for the project would be enough.

It’s not though it is dolt, meaning a stupid person, like git (stupid simple version control.)

Oh wow, that's awful! I thought this was just an innocent mistake.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#174

Post by st1ck »

zomglings wrote: post #76

If anyone from the dolt team is reading this, I'd like to make an enquiry: At bugout.dev, we have an ongoing crawl of public GitHub. We just created a dataset of code snippets crawled from popular GitHub repositories, listed by language, license, github repo, and commit hash and are looking to release it publicly and keep it up-to-date with our GitHub crawl. The dataset for a single crawl comes in at about 60GB. We u…

You can just try converting it into Parquet with ZSTD compression (and set high enough level). You can even fiddle with dictionary settings: https://arrow.apache.org/docs/python/generated/pyarrow.parqu...

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#175

Post by st1ck »

zomglings wrote: post #76

If anyone from the dolt team is reading this, I'd like to make an enquiry: At bugout.dev, we have an ongoing crawl of public GitHub. We just created a dataset of code snippets crawled from popular GitHub repositories, listed by language, license, github repo, and commit hash and are looking to release it publicly and keep it up-to-date with our GitHub crawl. The dataset for a single crawl comes in at about 60GB. We u…

Dataset similar to yours (github data) which you can query using Clickhouse: https://gh.clickhouse.tech/explorer/

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#176

Post by valtism »

I've looked into Dolt perviously for powering our product that is Git for financials.

The trouble is that most of our complexity lies in our data's relationships. Merging is very tricky when some data that has been added in one branch has not had a specific change in properties when other data in a master branch has been modified.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#177

Post by StreamBright »

zomglings wrote: post #171
StreamBright wrote: post #169

Earlier quoted context omitted.

That would be amazing. I get back my internet tomorrow and i can play with the dataset see how much we could optimize.

Hi StreamBright - just published the development version of the dataset also to Kaggle: https://www.kaggle.com/simiotic/github-code-snippets-develop... Compressed, it's 471 MB. Uncompressed, just a little more than 3 GB. If you want to get in touch with me in a better way than HN comments two good options: 1. My email is in my profile 2. You can direct message me (@zomglings) on the Bugout community Slack: https://jo…

I have messaged you on Slack.

The preliminary tests show significant reduction in space usage when using Parquet over Sqlite3. This is not unexpected at all. Parquet is much better for analytical use.

- 2.9G Mar 8 08:37 snippets-dev.db

- 427M Mar 8 14:05 test1.parquet

(venv) snippets-dev sqlite3 -header -csv -readonly snippets-dev.db 'SELECT COUNT() FROM snippets;' COUNT() 4850000

(venv) snippets-dev python test.py test1.parquet 4850000

I can share the 3 lines of Python and 1 line for SQL I used to convert the dataset.

There was only weird thing that I encountered, you somehow have a new line character in the commit_hash field for every value.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#178

Post by LukeEF »

sixdimensional wrote: post #81
crazygringo wrote: post #60

This is absolutely fascinating, conceptually. However, I'm struggling to figure out a real-world use case for this. I'd love if anyone here can enlighten me. I don't see how it can be for production databases involving lots of users, because while it seems appealing as a way to upgrade and then roll back, you'd lose all the new data inserted in the meantime. When you roll back, you generally want to roll back changes…

I am not associated to Dolt, but I really like the idea of Dolt personally. I do see use cases, but not without challenges. One of the main use cases you can see them targeting, and that I think makes a ton of sense, is providing tools for collecting, maintaining and publishing reference data sets using crowd sourcing. For example, they are doing this with hospital charge codes (a.k.a. chargemaster data). Hospitals i…

Your final example is very like the data mesh ideas coming out of thought works and elsewhere (https://martinfowler.com/articles/data-mesh-principles.html). Data products being owned by data producers and common data models ONLY when required. It is as much organizational model as technology, but I don't really think this maps to SQL tables. You probably want to look at a versioned knowledge graph of some kind. Downside is no SQL, upside is flexibility and speed. (disclaimer - I work over at TerminusDB, the graph version of Dolt)

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#179

Post by LukeEF »

dang wrote: post #83

Some related past threads: Dolt is Git for data - https://news.ycombinator.com/item?id=22731928 - March 2020 (191 comments) Git for Data – A TerminusDB Technical Paper [pdf] - https://news.ycombinator.com/item?id=22045801 - Jan 2020 (5 comments) Ask HN: Would you use a “git for data”? - https://news.ycombinator.com/item?id=11537934 - April 2016 (10 comments)

thanks for the mention of TerminusDB (https://github.com/terminusdb/terminusdb) -> we are the graph cousins of Dolt. Great to see so much energy in the version control database world!

We are currently focused on data mesh use cases. Rather than trying to be GitHub for Data, we're trying to be YOUR GitHub for Data. Get all that good git lineage, pull, push, clone etc. and have data producers in your org own their data.

We see lots of organizations with big 'shadow data' problems and data being centrally managed rather than curated by domain experts.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#180

Post by LukeEF »

justincormack wrote: post #9

I collected all the git for data open source projects I could find a few months back, there have been a bunch of interesting approaches https://docs.google.com/spreadsheets/d/1jGQY_wjj7dYVne6toyzm...

great list - i talked through it and what each tool does in a MLOps Community meetup: https://www.youtube.com/watch?v=r5uxntl_hWg
Post reply on HN