I like this. A while ago, I was asking my co-founder the question "Why isn't there Git for data like there is for code?" while working on a database migrations engine that aims to provide automatic database migrations (data & schema migrations). After all, code is data.
Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
161–170 of 185 posts
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#162If anyone from the dolt team is reading this, I'd like to make an enquiry: At bugout.dev, we have an ongoing crawl of public GitHub. We just created a dataset of code snippets crawled from popular GitHub repositories, listed by language, license, github repo, and commit hash and are looking to release it publicly and keep it up-to-date with our GitHub crawl. The dataset for a single crawl comes in at about 60GB. We u…
You have other options too. If I have time i can try to reduce the size with a columnar format that is designed for this use case (repeated values, static dataset).
That would be really great. Let me know if there's any way we can help. Maybe if we released a small version of the dataset for testing/benchmarking and then I could take care of running the final processing on the full dataset?
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#163Earlier quoted context omitted.
This is the dataset on Kaggle - https://www.kaggle.com/simiotic/github-code-snippets
Yeah, that sized database is likely to be a challenge unless the computer system it's running on has scads of memory. One of my projects (DBHub.io) is putting effort towards working through the problem with larger sized SQLite databases (~10GB), and that's mainly through using bare metal hosts with lots of memory. eg 64GB, 128GB, etc. Putting the same data into PostgreSQL, or even MySQL, would likely be much more eff…
Uh, personal question here. Where does your ~10G number come from? I pretty much run my life on the Apple Notes app. My Notes database is about 12G and now I’m scared.
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#164This is exactly what I was looking for!
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#165I've been working on https://github.com/projectnessie/nessie for about a year now. Its similar to Dolt in spirit but aimed at big data/data lakes. Would welcome feedback from the community.
Its very exciting to see this field picking up speed. Tons of interesting problems to be solved :-)
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#166Earlier quoted context omitted.
Git is too complicated. It's barely usable for daily tasks. Look at how many people have to Google for basic things like uncommitting a commit, or cleaning your local repo to mirror a remote one. Complexity is a liability. Mercurial has a nicer interface. And now I see the real simplicity of non-distributed source control systems. I have never actually needed to work in a distributed manner, just client-server. I hav…
> Git is too complicated. It's barely usable for daily tasks. Look at how many people have to Google for basic things like uncommitting a commit, or cleaning your local repo to mirror a remote one. Cars are too complicated. They are barely usable for daily tasks. Look at how many people have to Google for basic things like changing a fan belt, or fixing cylinder head gasket. You can fill in almost anything here. Most…
To me it sounds like you're proving the exact opposite. I'd assume most car owners never need to change a fan belt themselves, while everyone who uses git daily needed at some point to revert a commit. "How to turn right" isn't huge on stackoverflow last time I checked...
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#167This is absolutely fascinating, conceptually. However, I'm struggling to figure out a real-world use case for this. I'd love if anyone here can enlighten me. I don't see how it can be for production databases involving lots of users, because while it seems appealing as a way to upgrade and then roll back, you'd lose all the new data inserted in the meantime. When you roll back, you generally want to roll back changes…
The application backing use case is best suited for when you have parts of your database that get updated periodically and need human review. So you have a production database that you serve to your customers. Then you have a branch / fork of that (dev) that your development team adds batches of products to. Once a week you do a data release: submit a PR from dev -> prod, have somebody review all the new copy, and me…
[deleted]
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#168Is something like "dolt diff master..somebranch -- mytable" possible?
Same question: Are in-between branch comparisons possible in dolt? Which "diff" subcommands to you plan to support in the future?
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#169Earlier quoted context omitted.
You have other options too. If I have time i can try to reduce the size with a columnar format that is designed for this use case (repeated values, static dataset).
That would be really great. Let me know if there's any way we can help. Maybe if we released a small version of the dataset for testing/benchmarking and then I could take care of running the final processing on the full dataset?
That would be amazing. I get back my internet tomorrow and i can play with the dataset see how much we could optimize.
Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge
#170Wordpress would have benefited from this. What a lot of webmasters want is, test the site locally, then merge it back. A lot of people turned to Jekyll or Hugo for the very reason that it can be checked into git, and git is reliable. A static website can’t get hacked, whereas anyone who has been burnt with Wordpress security fail knows they’d prefer a static site. And even more: People would like to pass the new webs…
At the root of this lies the problem that content, configuration and code is stored in one blurp (a single db). Never clearly bounded nor contained. Config is spread over tables. Usergenerated content (from comments to orders) mixed up with redactional content: often even in tables, sometimes even in the same column. Drupal suffers the same. What is needed, is a clear line between configuration (and logic), redaction…
Very well summarised. I can see clearly you too have tried to apply normal software lifecycle principles and come out the other end with a rather dismal impression of WordPress.