Same name as my Phoenix inspired framework for go: https://codeberg.org/lixgo/lix
Lix – universal version control system for binary files
51–59 of 59 posts
Re: Lix – universal version control system for binary files
#52Lix is also a soft fork of the official Nix package manager implementation: https://lix.systems/
Lix is also the name of a computer science and mathematics laboratory: https://www.lix.polytechnique.fr/
Re: Lix – universal version control system for binary files
#53Git can display diff between binary files using custom diff drivers: > Put the following line in your .gitattributes file: *.docx diff=word > This tells Git that any file that matches this pattern (.docx) should use the “word” filter when you try to view a diff that contains changes. What is the “word” filter? You have to set it up [in .gitconfig]. https://git-scm.com/book/en/v2/Customizing-Git-Git-Attribute...
Would be interesting to see some tooling built around being a custom diff driver for a bunch of different standard formats!
I had some interesting luck with the generic approach to unzip the DOCX/XLSX/ODT/etc, then to the contents recursively apply other filters like XML and JSON formatters/prettifiers.
(My work [1] in this space predated git so it wasn't written as a git diff filter, instead it automated source control. But the same principles could be used in the other direction.)
Not the highest level diffs you could possibly get, but at least for a programmer even ugly XML and JSON diffs were still nice to have over binary diffs.
Re: Lix – universal version control system for binary files
#54Earlier quoted context omitted.
> But then the first thing it talks about is diffing files. Which honestly shouldn’t even be a feature of VCS. That’s just a separate layer. There is nuance between git line by line diffing and what lix does. For text diffing it holds true that diffing is a separate layer. Text files are small in size which allows on the fly diffing (that's what git does) by comparing two docs. On the fly diffing doesn't work for str…
> On the fly diffing doesn't work for structured file formats like xlsx, fig, dwg etc. It's too expensive. Both in terms of materializing two files at specific commits, and then diffing these two files. I don’t think that’s actually true? How often are binary files being diffed? How long does it take to materialize? How long to run a diff algorithm? I’ve worked with some tools that can diff images. Works great. Not a…
I think our goals indeed differ.
> How often are binary files being diffed? How long does it take to materialize? How long to run a diff algorithm?
If version control is embedded in an app, constantly.
Imagine a cell in a spreadsheet. An application wants to display a "blame" for a cell C43 i.e. how did the cell change over time?
The lix way is this SQL query
SELECT * from state_history WHERE file_id AND schema_key "excel_cell" AND entity_id C43;
Diffing on the fly is not possible. The information on what changed needs to be available without diffing. Otherwise, diffing an entire spreadsheet file for every commit on how cell C43 changed takes ages.
Re: Lix – universal version control system for binary files
#55Earlier quoted context omitted.
Some feedback about the primary use case. Your Lix doc (LLM written but with typos?) is sort of weird, handwaving how Lix does version control over, say, Excel, to say it's about working with SQL databases: How does Lix work? Lix adds a version control system on top of SQL databases that let's you query virtual tables like file, file_history, etc. via plain SQL. These table's are version controlled. Then it gets weir…
Does the following make more sense to you in respect to SQL? Lix uses SQL databases as storage and query engine. Aka you get a filesystem on top of your SQL database that is version controlled. Or, the analogy to git: Git uses the computers filesystem as storage layer. Lix uses SQL database (with the nice benefit of making everything queryable via SQL). > Lix doesn't reinvent databases — durability, ACID, and corrupt…
Tracking versions is easy. Controlling versions is hard. Knowing what the actual semantic deltas are in binary files (what your doc examples claimed) is hard.
Now you say you're not tackling that problem, so the docs are doubly weird.
Would also imagine less of this is a SQL-shaped problem, per se, so plenty tech is better for tracking and controlling changes than SQL. The shape of the problem seems less theory of querying sets, more branching journal with proof chain, or hash tree.
Re: Lix – universal version control system for binary files
#56Earlier quoted context omitted.
Does the following make more sense to you in respect to SQL? Lix uses SQL databases as storage and query engine. Aka you get a filesystem on top of your SQL database that is version controlled. Or, the analogy to git: Git uses the computers filesystem as storage layer. Lix uses SQL database (with the nice benefit of making everything queryable via SQL). > Lix doesn't reinvent databases — durability, ACID, and corrupt…
Tracking versions is easy. Controlling versions is hard. Knowing what the actual semantic deltas are in binary files (what your doc examples claimed) is hard. Now you say you're not tackling that problem, so the docs are doubly weird. Would also imagine less of this is a SQL-shaped problem, per se, so plenty tech is better for tracking and controlling changes than SQL. The shape of the problem seems less theory of qu…
> Knowing what the actual semantic deltas are in binary files (what your doc examples claimed) is hard.
Hm that is what lix provides?
SQL is just the interface to query the deltas.
Anyhow, lesson learned. The primary use case for lix is embedding. Positioning lix for binary files leading to existing systems. None of those support the embedded use case. Thus, don't position lix for binary files :)
Re: Lix – universal version control system for binary files
#57Looks cool, but seems kind of weird that it only works through an sdk. Should there be a cli or something? Edit: Oh I see. Seems like their use case is embedding version control into another application.
Correct. Lix has been developed with the embedded use-case in mind. Someone can write a CLI for it. Though, the primary use case is not code version control but embedding into applications
Thanks for the clarification. We have a bunch of SSIS/SSRS packages in version control that are essentially just big balls of XML. Git has a really hard time dealing with them. I was thinking Lix may possibly be a good option.
Re: Lix – universal version control system for binary files
#58Earlier quoted context omitted.
Correct. Lix has been developed with the embedded use-case in mind. Someone can write a CLI for it. Though, the primary use case is not code version control but embedding into applications
Thanks for the clarification. We have a bunch of SSIS/SSRS packages in version control that are essentially just big balls of XML. Git has a really hard time dealing with them. I was thinking Lix may possibly be a good option.
The SSIS use case is a good one for lix.
Write an SSIS plugin (essentially just parse the XML) and lix can track the xml props in the SSIS package. Opposed to line by line stuff that git does.
Re: Lix – universal version control system for binary files
#59It was initially hard for me to understand how this could work but it looks like there is a plugin system?
Yes. The tracking works via plugins to keep it generic. Here is a rough illustration: File change -> Plugin (detects changes) -> Lix It works surprisingly well because most standard file formats have off the shelf parsers. Parse a file format, and et voila, it is trivial to diff. Then pass on a standard schema for changes to lix and you end up with a generic API to query changes.
Writing the plugin seems to be a hard problem. It’s not clear from the link if there are non trivial plugins written for real formats, not toy not POC but a for reals plugins