SnowFS – a fast, scalable version control file storage for graphic files • Hacker News Board

SnowFS – a fast, scalable version control file storage for graphic files

#1

Post by karateka » Sun, Feb 21, 2021, 7:16 AM UTC

SnowFS – a fast, scalable version control file storage for graphic files
github.com

Re: SnowFS – a fast, scalable version control file storage for graphic files

#2

Post by pjmlp » Sun, Feb 21, 2021, 1:48 PM UTC

JavaScript[0] isn't what I would call a fast storage repository, but I guess it works out for prototyping.

On the context of porting to C and C++, or make it execute faster, I can see two options with minor rewrites.

Use AssemblyScript and generate native code via WebAssembly AOT compilers.

Try to adapt TypeScript to C++ compiler from Microsoft's MakeCode project.

Implement your own C++ code generator.

It would be much easier than keep using multiple code bases in parallel, plus any memory corruption issues would be most likely bugs on the code generator.

[0] - Yes I know the source code is Typescript.

Re: SnowFS – a fast, scalable version control file storage for graphic files

#3

Post by kevlar1818 » Sun, Feb 21, 2021, 2:22 PM UTC

Very interesting. I'd like to learn more about how it works. How does this compare to DVC[1], for instance?

I'll throw in a shameless plug for my tool in this area, Dud[2]. Dud is to DVC what Flask is to Django.

Are the mentioned benchmarks published somewhere?

[1]: https://dvc.org [2]: https://github.com/kevin-hanselman/dud

Re: SnowFS – a fast, scalable version control file storage for graphic files

#4

Post by sebastian_io » Sun, Feb 21, 2021, 6:45 PM UTC

kevlar1818 wrote: post #3

Very interesting. I'd like to learn more about how it works. How does this compare to DVC[1], for instance? I'll throw in a shameless plug for my tool in this area, Dud[2]. Dud is to DVC what Flask is to Django. Are the mentioned benchmarks published somewhere? [1]: https://dvc.org [2]: https://github.com/kevin-hanselman/dud

SnowFS is still at the beginnings, so it will shape over time. For the benchmarks, these are the results on a MacBook-2019 with APFS for a 4GB file:

[git add texture.psd: 20164ms] [snow add texture.psd: 4596ms] [git rm texture.psd: 575ms] [snow rm texture.psd: 111ms] [git checkout HEAD~1: 9739ms] [snow checkout HEAD~1: 1ms]

You might get slightly slower speeds on NTFS for 'add' and 'checkout' but still very performant

Re: SnowFS – a fast, scalable version control file storage for graphic files

#5

Post by sebastian_io » Sun, Feb 21, 2021, 6:47 PM UTC

pjmlp wrote: post #2

JavaScript[0] isn't what I would call a fast storage repository, but I guess it works out for prototyping. On the context of porting to C and C++, or make it execute faster, I can see two options with minor rewrites. Use AssemblyScript and generate native code via WebAssembly AOT compilers. Try to adapt TypeScript to C++ compiler from Microsoft's MakeCode project. Implement your own C++ code generator. It would be mu…

Indeed, TS/JS is great for its quick turnaround times for prototyping. But the I/O performance is executed by the underlying C/C++ layer. For the rest TS/JS is fast enough. But a full C++ backport is still on the horizon

Re: SnowFS – a fast, scalable version control file storage for graphic files

#6

Post by jarym » Sun, Feb 21, 2021, 7:40 PM UTC

I need to look into this more but unless you already know C/C++ really well then Rust may be a better choice.

Re: SnowFS – a fast, scalable version control file storage for graphic files

#7

Post by sebastian_io » Sun, Feb 21, 2021, 7:45 PM UTC

jarym wrote: post #6

I need to look into this more but unless you already know C/C++ really well then Rust may be a better choice.

Same here, the aimed goal is a C++ backport, bc I am a C++ dev. But Rust is on my radar and happy to learn

Re: SnowFS – a fast, scalable version control file storage for graphic files

#8

Post by amelius » Sun, Feb 21, 2021, 8:52 PM UTC

But what if I have software and images (e.g. as in game software)? Which version control system do I use?

Honestly, I think the effort was better spent on an improved version of Git.

Re: SnowFS – a fast, scalable version control file storage for graphic files

#9

Post by sebastian_io » Sun, Feb 21, 2021, 8:57 PM UTC

amelius wrote: post #8

But what if I have software and images (e.g. as in game software)? Which version control system do I use? Honestly, I think the effort was better spent on an improved version of Git.

Programmers, game makers and 2D/3D artists are very different target audiences with very different needs. To name one example, the commit hash integrity which is a foundation of Git is a must-have for software projects, but might be not useful in environments like VFX or CG productions

Re: SnowFS – a fast, scalable version control file storage for graphic files

#10

Post by mbreese » Sun, Feb 21, 2021, 9:25 PM UTC

sebastian_io wrote: post #9

amelius wrote: post #8

But what if I have software and images (e.g. as in game software)? Which version control system do I use? Honestly, I think the effort was better spent on an improved version of Git.

Programmers, game makers and 2D/3D artists are very different target audiences with very different needs. To name one example, the commit hash integrity which is a foundation of Git is a must-have for software projects, but might be not useful in environments like VFX or CG productions

Also, the project is very clear that it is for individuals with no collaboration tools built in. While git does have utility for a solo developer, having a separate system for a solo artist (that isn’t working with assets for developers), starts make more sense.