Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What is in that .git directory?

blog.meain.io

Post reply on HN
Print view View original

21–30 of 47 posts

Re: What is in that .git directory?

#21

Post by tharos47 »

charles_f wrote: post #5
js2 wrote: post #4

Earlier quoted context omitted.

Unpack the files (git-unpack). Maybe it was one large file that someone added, then deleted in a later commit. You'd have to rewrite history to get rid of it entirely. Alternately it might be a bunch of medium sized files that were added and removed. It may take a little while to track down, but I'd start by unpacking. This stack-overflow looks like it contains a reasonable description about how to rewrite history to…

Thanks! Yeah I plan to get to the bottom of it. I will probably propose to just keep a branch with full history somewhere (we need to keep history for auditability) and reset the main branch from a recent state.

I can recommend git-filter-repo instead, it's relatively recent and there is a lot of outdated info on the internet about cleaning git repos. The --analyse flag will generate a report about files in your repo even if they were deleted. I used it to cleanup a number of repo and it helped in detecting large files commited by mistake 10 years ago. The history rewrite removed the files and we didn't need to create a new repo (old history still works fine).

Re: What is in that .git directory?

#23

Post by rollcat »

It's fairly easy to grab info from .git for your own purposes. For example, the program that generates my PS1 peeks there (without wasting precious cycles on shelling out to the git command) to find the current branch we're on:

https://github.com/rollcat/etc/blob/b2fd739/cmd/prompter/mai...

Re: What is in that .git directory?

#24

Post by charcircuit »

theamk wrote: post #19
charcircuit wrote: post #14

Earlier quoted context omitted.

Did you get that backwards? sqlite is a proper database that actually tests its resilience. Just because you can't sync a sqlite file, that doesn't mean it isn't resilient it just means you need to back it up via pushing to another repo or using the backup command. Syncing by just copying files over while a disk is still being used is fragile in general.

have you seen sqlite official documentation on corruption resistance? https://www.sqlite.org/howtocorrupt.html supported failure modes, tested and handled: "application crash, or an operating-system crash, or even a power failure" - so basically proper atomic renames. git does this well. unsupported failure modes: "Backup or restore while a transaction is active" - when you backup your machine, do you really treat ea…

>when you backup your machine, do you really treat each sqlite specially?

Yes, but only for servers where the database is being used. On my desktop if the database isn't being used it is safe to copy so I don't worry about it. Backing up a git repo while git is writing to the repo isn't safe either.

>or, you know, downloading database file and forgetting to grab journal at the same time

You should not be downloading an actively used sqlite database anyways. If you backup the sqlite database before downloading it there won't be a journal file.

>did you ever hardlink or bind-mounted a database file? prepare for corruption...

You just have to link both the database and WAL file. This is somewhat challenging since the WAL file will be deleted by default if all processes close the database. It's better to link or mount the directory that contains the database file. If you link only some of the files from .git, then git won't work properly either.

Re: What is in that .git directory?

#25

Post by masklinn »

charles_f wrote: post #5
js2 wrote: post #4

Earlier quoted context omitted.

Unpack the files (git-unpack). Maybe it was one large file that someone added, then deleted in a later commit. You'd have to rewrite history to get rid of it entirely. Alternately it might be a bunch of medium sized files that were added and removed. It may take a little while to track down, but I'd start by unpacking. This stack-overflow looks like it contains a reasonable description about how to rewrite history to…

Thanks! Yeah I plan to get to the bottom of it. I will probably propose to just keep a branch with full history somewhere (we need to keep history for auditability) and reset the main branch from a recent state.

Have you already tried a "gc --aggressive"? It's not exactly fast or cheap, but some repositories are very badly packed and only a full reset will fix them.

An other useful high-level option is git-sizer (https://github.com/github/git-sizer) which tries to expose a few useful trouble spots, there's not much that can be done if the repository is just big (long history of wide working copies with lots of changes), but sometimes it's just that there are a bunch of large binary assets.

This may be more likely if the repository was converted from a centralised VCS where storing large assets or files is less of an issue, likewise the bad compression. Though obviously removing such large assets from the core repository still requires rewriting the entire thing.

Re: What is in that .git directory?

#28

Post by conceptme »

charles_f wrote: post #3

By random chance I ended up in the git internals doc^1 today, also lovely refered to as plumbing and porcelain. It's a fantastic read, very well explained. I wish all doc was written with such explicit care to be understood. It reads like a good friend is trying to explain you something. What got me into that was a 51Gb ".pack" file that I wanted to understand. If you wonder about that, they're pack files, and what t…

You could try bfg https://rtyley.github.io/bfg-repo-cleaner/
Post reply on HN