Live data from Hacker News

Facebook's git repo is 54GB

twitter.com

221–230 of 245 posts

Re: Facebook's git repo is 54GB

#221
post #78
post #63

Earlier quoted context omitted.

Yes, but the new intern would be able to read all the source and "secret sauces". I doubt that an intern on Google would've access to the search codebase. I'd wager that only a handful of trusted employees have access to that codebase.

All Google code is stored in one master repository, for all products. An intern can look at the search code.

Not so. Search is specifically sequestered.

Re: Facebook's git repo is 54GB

#222

Earlier quoted context omitted.

With Android, ant makes naive assumptions. For example, most open source library projects that you include in a project don't change from build to build, but ant dutifully recompiles them each time instead of caching the output until the files in that project are changed or I manually clean the build output.

That surely means Ant is make, only worse.

I'm confused. That doesn't sound like Make's behavior at all.

Make, when used properly, is still an pretty smart tool.

Re: Facebook's git repo is 54GB

#224

Earlier quoted context omitted.

Its not bad, is really nice, but Git has one problem, when you codebase is big, the process takes a long time, imagine git scanning those 8GB every time you do a commit, that is why Facebook was looking to port all their code to another VCS

I think it's worth making a distinction between the Git plumbing and the Git porcelain when talking about performance. The core functionality (the plumbing) is very fast regardless of repository size. The slowdowns people describe are almost always related to the porcelain commands, which are poorly optimized. Almost every porcelain-level command will cause Git to lstat() every file in your tree, as well as check for…

There was some testing early last year, but I think it's about time for someone to post another reminder

http://git.661346.n2.nabble.com/inotify-to-minimize-stat-cal...

Re: Facebook's git repo is 54GB

#225
post #124

Earlier quoted context omitted.

I don't see how this is an Ant-specific issue. Ant is just calling into javac with a classpath parameter. The actual execution time spent in Ant should be minimal.

With Android, ant makes naive assumptions. For example, most open source library projects that you include in a project don't change from build to build, but ant dutifully recompiles them each time instead of caching the output until the files in that project are changed or I manually clean the build output.

Doesn't and use prebuild jars for libraries? If not then maybe you should think about switching to maven.

Re: Facebook's git repo is 54GB

#226
post #21

Earlier quoted context omitted.

The big .git directory is probably binary revisions? Is there any good way around that in git?

you're not going to merge binary files, so git isn't the right tool. the standard way is to use maven. git handles your sources, and anything binary (libs, resources etc) goes on the nexus (where it is versioned centrally) and is referenced in your poms: simple and powerful

git is "the stupid content tracker", not "the stupid merge tool". Even for things you have no intention of branching or merging, it still gives you control over versioning... and there's a huge gap between the level of control a hash tree gives you versus trusting some remote server to consistently give you the same file when you give it the same request.

Re: Facebook's git repo is 54GB

#227
post #205

Earlier quoted context omitted.

SOA isn't a magic bullet. What if multiple services are utilizing a shared library? For each service to be independent in the way I think you are advocating for, you would need multiple copies of that shared library (either via separate copies in separate repos or a shared copy via something like subrepos). Multiple copies leads to copies getting out of sync. You (likely) lose the ability to perform a single atomic c…

> "What if multiple services are utilizing a shared library? For each service to be independent in the way I think you are advocating for, you would need multiple copies of that shared library (either via separate copies in separate repos or a shared copy via something like subrepos)." No, you have a notion of packages in your build system and deployment system. You want to use FooWizz framework for your new service…

I think you are missing the point. Versioning and package management problems can largely go away when your entire code base is derived from a single repo. After all, library versioning and packaging are indirections to better solve common deployment and distribution requirements. These problems don't have to exist when you control all the endpoints. If you could build and distribute a 1 GB self-contained, statically linked binary, library versioning and packages become largely irrelevant.

Re: Facebook's git repo is 54GB

#228
post #83

Although this is large for a company that deals mostly in web-based projects, it's nothing compared to repository sizes in game development. Usually game assets are in one repository (including compiled binaries) and code in another. The repository containing the game itself can grow to hundreds of gigabytes in size due to tracking revision history on art assets (models, movies, textures, animation data, etc). I woul…

I wonder what the biggest code base in the world is. Like you say, games include all sorts of binary assets. Any idea how much actual code is there? The Linux kernel is only 175MB https://git.wiki.kernel.org/index.php/GitBenchmarks#Estimate... The F22 has some 1.7 million LOC http://en.wikipedia.org/wiki/Lockheed_Martin_F-22_Raptor#cit... This graph shows some pretty big things http://dailyinfographic.com/wp-content/…

looking at the plot, i wonder what the heck is up with healthcare.gov... what on earth does the site do that really requires 5e8 loc?

Re: Facebook's git repo is 54GB

#229

  NAFV_P@DEC-PDP9000:~$ python
  Python 2.7.3 (default, Feb 27 2014, 19:58:35)
  [GCC 4.6.3] on linux2
  Type "help", "copyright", "credits" or "license" for more information
  >>> t=54*2**30
  >>> t
  57982058496
  # let's assume a char is 2mm wide, 500 chars per meter
  >>> t/500.0
  115964116.992 #meters of code
  # assume 80 chars per line, a char is 5mm high, 200 lines per meter
  >>> u=80*200.0
  >>> v=t/u
  >>> v
  3623878.656 # height of code in meters
  # 1000 meters per km
  >>> v/1000.0
  3623.878656 # km of code, it's about 385,000 km from the Earth to the Moon
  >>> from sys import stdout
  >>> stdout.write("that's a hella lotta code\n")

Re: Facebook's git repo is 54GB

#230
post #141
post #59

Earlier quoted context omitted.

Facebook tends to throw engineer time at the problem, though. I know one Facebook DevCon I went to they presented how they completely wrote their own build system because Ant was too slow for them.

At my company, I wrote our own build system, because "make", "waf", "shake" and various others do not give any useful guarantee, and we've debugged cryptic under-specified dependencies way too many times. Make clean on a large repo and lack of automatic work sharing hurt too. Also, auto detecting inputs rather than being forced to specify them is nice. Especially as virtually all input specs in Makefiles are wrong or…

What guarantees did you find lacking in shake?
Post reply on HN