Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lore – Open source version control system designed for scalability

lore.org

Post reply on HN
Print view View original

231–240 of 717 posts

Re: Lore – Open source version control system designed for scalability

#231

Post by danudey »

regnerba wrote: post #37
rootlocus wrote: post #33

Earlier quoted context omitted.

Git LFS has file locking, and no VCS can provide you with the tools for diffing binary assets. I don't see any meaningful difference between Perforce, Diversion or Lore and git + LFS + file locking. Unless there's a meaningful performance impact for large projects (I only work on small / medium projects), the capabilities are the same. However, I get excellent git support for code in any editor, as opposed to Diversi…

Git LFS for example does not support file chunking. So a single byte change on a large (100s of gigs) file means downloading the whole file again. Lore does chunking of binary files which means faster downloads and better de-dupping on the backend.

Also, Lore seems to support checking out only the assets you actually need (on-demand hydration and sparse checkouts), meaning that a level designer can check out just the level that they're working on without having to manually configure a git sparse-checkout (and then not being able to see any of the non-checked-out files).

If this supports dynamic hydration of files, either as they're accessed (like Dropbox with offline files) or by somehow knowing which files need which other files (building a dependency graph) then it could be a massive win both for speed and efficiency of downloads but also for conserving disk space on developer machines.

And since it has API bindings, it's possible that's something that could be built into IDE plugins, so that your editor (Godot, Unity, etc) can know which assets need which other assets and automatically trigger hydration, including when you e.g. try to use a new model/texture/etc in a scene that hasn't used it yet.

Re: Lore – Open source version control system designed for scalability

#232

Post by kps »

asveikau wrote: post #197
ur-whale wrote: post #129

Earlier quoted context omitted.

Git is certainly not great with binary assets, but calling perforce SOTA ... ouch. If perforce is the best there is out there for large binary asset management, then there is a blue ocean worth of potential improvement for git. Perforce is a piece of crap, a relic of the 20th century that must die in a fiery inferno.

15 years ago, both Google and Microsoft were on perforce. (The latter through a fork with a different name.)

Google still uses Piper, which started as a Perforce clone, though many people use it through a frontend like `fig` (try pronouncing ‘Piper HG’) or `jj`.

Re: Lore – Open source version control system designed for scalability

#234

Post by frollogaston »

wrwills wrote: post #227
ksec wrote: post #202

Turns out it is not really new but only open sourced it now. From the FQA. >Lore, formerly called Unreal Revision Control, is the built-in version control system for UEFN (Unreal Editor for Fortnite), where creators have been using it to version their islands. It is also seeing progressive adoption by internal Epic teams, and is being implemented as the backing store for UEFN’s cook pipeline, where it replaces tradit…

I suspect the use of Rust rather than C++ might have something to do with the fact that Simon Peyton Jones and Lennart Augustsson (both of Haskell fame) both work at Epic and there would have been a strong internal push to do this in a language with some functional programming features. Rust rather than Verse because that would probably not be the right tool for the job (even if Simon works on it). Rust rather than H…

I don't see how Rust is more functional programming oriented than C++

Re: Lore – Open source version control system designed for scalability

#235

Post by gritzko »

niek_pas wrote: post #22

Just today as I pushed some changes to Github, I was thinking how user-unfriendly Git's UI is: Enumerating objects: 5, done. Counting objects: 100% (5/5), done. Delta compression using up to 10 threads Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 290 bytes | 290.00 KiB/s, done. Total 3 (delta 2), reused 0 (delta 0), pack-reused 0 remote: Resolving deltas: 100% (2/2), completed with 2 local obje…

Every Beagle command:

    gritzko@spot ~/beagle $ be get
     19:07  get ?#0ac49e6a
     16:58  post ?0ac49e6a#POST-018 put:/post: banner on stdout
     19:07  new beagle/test/be-post-put-banner.sh
     19:07  upd dog/INDEX.md
     ...more stuff...
     19:07  del test/post/01-bare-msg/01.put.err.txt
     19:07  del test/post/01-bare-msg/02.post.err.txt
     19:07  get abc?4222dfab

Re: Lore – Open source version control system designed for scalability

#236

Post by danudey »

zipy124 wrote: post #70
rootlocus wrote: post #33

Earlier quoted context omitted.

Git LFS has file locking, and no VCS can provide you with the tools for diffing binary assets. I don't see any meaningful difference between Perforce, Diversion or Lore and git + LFS + file locking. Unless there's a meaningful performance impact for large projects (I only work on small / medium projects), the capabilities are the same. However, I get excellent git support for code in any editor, as opposed to Diversi…

Git LFS breaks so often that it can't be seen as a serious professional tool tbh. I've had nothing but trouble with it. If it wants to be serious it needs to be built into Git, not added as some after thought.

> Git LFS breaks so often that it can't be seen as a serious professional tool tbh. I've had nothing but trouble with it. If it wants to be serious it needs to be built into Git, not added as some after thought.

At Fortinet we migrated our SVN repositories to git and ran into a ton of issues; developers over the past ten years had done tons of little mistakes that added up, like accidentally checking an entire Windows virtual machine into the repo. In SVN they deleted it and no one ended up caring, but in Git of course it became part of the repo history.

I did a huge amount of work for the migration, 99% of which was analyzing each repo to find out what files/file extensions were overly large, and then either:

1. Filtering them out of the git history completely during import

2. Converting them to LFS objects after the import

The LFS process was certainly better than the other alternatives, which were 'check everything into the git history' or 'remove all the un-diffable binary files and hope that they weren't needed for anything', but it was still not ideal.

Every developer (out of thousands, across multiple countries, timezones, and native languages) had to set their system up properly; if you missed a command, or if you reinstalled your OS and forgot to set up one of the aliases or hooks, then you would end up checking binary blobs into git rather than LFS, or checking out LFS idents rather than the actual files they needed.

We also had the issue of developers fetching code over SSH but LFS files over HTTPS, which would be fine except that we wanted to prevent access to HTTPS from most subnets, so while the developers could use SSH to clone or pull using their 2FA token their client would then make an HTTP request that wouldn't work unless they were on the version control VPN, which.... blah blah blah.

So yeah, it worked better than the alternative, but it did not work _well_ a lot of the time.

Re: Lore – Open source version control system designed for scalability

#237

Post by VibrantClarity »

superkuh wrote: post #63
jacobgold wrote: post #21

I'd trust this project more if it was named Data.

"Lore" is appropriate. Epic games is a very unethical company that steals from people. Myself in paticular. I bought Rocket League the game for linux from Psyonix. Epic bought Psyonix and immediately removed the game clients for linux and mac os. I can no longer play. They stole from me and many others. It'd be one thing if they just shut down the game entirely, but stealing it from only some people while keeping it…

I haven't played it since the buyout either, but I'm told it works perfectly with Proton.

Re: Lore – Open source version control system designed for scalability

#238

Post by gavinhoward »

frollogaston wrote: post #185
gavinhoward wrote: post #179

Earlier quoted context omitted.

When it comes to large assets, wasting large chunks of space is a problem. If your chunks are 64 kib average (from the Lore document), but changes only average 1 kib (which could be a high estimate), then you will still run out of space 64 times faster and need to read 64 times more data off of the disk for certain operations. It also makes diffing hard, as well as diff viewing.

What do you do instead of chunking your snapshots? Storing diffs is usually the other approach.

The best answer I have is for you to read the "History Model" section of that design doc through the "Implementing CRDTs" subsection.

Re: Lore – Open source version control system designed for scalability

#239

Post by swiftcoder »

I don't see a workflow for locking assets while they are under modification. This is kind of important for assets? Since we don't really have great merge workflows for meshes/animations/sounds/etc.

I also don't see any sort of GUI client? So the whole art team is going to have to get up close and personal with the CLI

Re: Lore – Open source version control system designed for scalability

#240

Post by danudey »

lentil_soup wrote: post #50
rootlocus wrote: post #33

Earlier quoted context omitted.

Git LFS has file locking, and no VCS can provide you with the tools for diffing binary assets. I don't see any meaningful difference between Perforce, Diversion or Lore and git + LFS + file locking. Unless there's a meaningful performance impact for large projects (I only work on small / medium projects), the capabilities are the same. However, I get excellent git support for code in any editor, as opposed to Diversi…

don't think it supports branches it's also tough when you have 1TB of data, over 1mm files and you might want to lock hundreds files in one go

I mean, Git LFS 'supports branches' in that the LFS content identifiers are checked into git as files and Git operated normally; LFS is just a way to replace those content identifiers with the actual content, and then vice-versa when you commit.

I think branching is the one thing that didn't get more complicated with LFS.

Post reply on HN