Any info on how this differs from Dat?
Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
91–100 of 185 posts
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#92Since you've hosted it on GitHub, would you also consider providing a JavaScript client library for read access?
Could pave the way for single page web applications (hosted via GitHub Pages) working with Dolt as their database of choice.
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#93This is absolutely fascinating, conceptually. However, I'm struggling to figure out a real-world use case for this. I'd love if anyone here can enlighten me. I don't see how it can be for production databases involving lots of users, because while it seems appealing as a way to upgrade and then roll back, you'd lose all the new data inserted in the meantime. When you roll back, you generally want to roll back changes…
I might look at it for work. Compliance requires us to keep track of who made what change when, and who it was approved by in case regulators need it. Right now, this often means making an MR on Git with your snippet of SQL, getting it approved, and then manually executing it. This would let us bring the apply stage in, as well as avoid "that SQL query didn't do exactly what I expected" issues. It's possible to do it…
That's a great use case, let us know how we can help. We have a discord if you want to chat about it:
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#94Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#95Naive question here. Aside from it being mysql, what is different here than just using git + sqlite.
Update: When I posted, I'd forgotten that SQLite db file is a binary. Not sure what I was thinking.
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#96Naive question here. Aside from it being mysql, what is different here than just using git + sqlite. Update: When I posted, I'd forgotten that SQLite db file is a binary. Not sure what I was thinking.
Merging a SQLite database is challenging.
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#97If anyone from the dolt team is reading this, I'd like to make an enquiry: At bugout.dev, we have an ongoing crawl of public GitHub. We just created a dataset of code snippets crawled from popular GitHub repositories, listed by language, license, github repo, and commit hash and are looking to release it publicly and keep it up-to-date with our GitHub crawl. The dataset for a single crawl comes in at about 60GB. We u…
This is the dataset on Kaggle - https://www.kaggle.com/simiotic/github-code-snippets
Yeah, that sized database is likely to be a challenge unless the computer system it's running on has scads of memory.
One of my projects (DBHub.io) is putting effort towards working through the problem with larger sized SQLite databases (~10GB), and that's mainly through using bare metal hosts with lots of memory. eg 64GB, 128GB, etc.
Putting the same data into PostgreSQL, or even MySQL, would likely be much more efficient memory wise. :)
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#98I collected all the git for data open source projects I could find a few months back, there have been a bunch of interesting approaches https://docs.google.com/spreadsheets/d/1jGQY_wjj7dYVne6toyzm...
This one seems to be missing: https://projectnessie.org/
It's also missing DBHub.io. ;)
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#99I love the idea of this project so much. Being able to more easily share comprehensible data is an interest of mine.
It is not the first time I have seen immutable B-trees being used as a method for being able to query a dataset on a different point in time.
Spanner (and its derivatives) uses a similar technique to ensure backup consistency. Solutions such as CockroachDB also allows you to query data in the past [1], and then uses a garbage collector to delete older unused data. The Time-to-live of history data is configurable.
[1] https://www.cockroachlabs.com/docs/stable/as-of-system-time....
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#100This is absolutely fascinating, conceptually. However, I'm struggling to figure out a real-world use case for this. I'd love if anyone here can enlighten me. I don't see how it can be for production databases involving lots of users, because while it seems appealing as a way to upgrade and then roll back, you'd lose all the new data inserted in the meantime. When you roll back, you generally want to roll back changes…
Here's one thing I'd have used it for: Video game assets.
Say you have a tabletop game engine for designing starships. Different settings have different lists of parts. Some settings are run by a game's DM, some are collaborative efforts. I ended up saving the lists of parts in huge JSON files and dumping those into git. However, for much the same reason that data science is often done in a REPL or notebook type interface, it turned out that by far the most efficient way for people to iterate on assets was to boot up the game, fiddle with the parts in-engine until things looked right, then replicate their changes back into the JSON. With this, we could just save the asset database directly.
The same reasoning should hold for effectively any dataset which a) can be factored into encapsulated parts b) isn't natively linear c) needs multiple developers. Game assets are one example, as I described above. Other datasets that hold: ML training/testing sets, dictionaries, spreadsheets, catalogs, datasets for bio papers.