Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Counting Objects

githubengineering.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Counting Objects

#31

Post by spb »

alagappanr wrote: post #20

I wonder why GitHub has a separate domain githubengineering.com for this blog instead of a subdomain like engineering.github.com. I notice that there is an inactive user account called engineering. If at all an User Page is created by that account, it would be available at engineering.github.io.

GitHub Pages used to be served under github.com subdomains (they were switched after GitHub realized this was a same-origin security hole), and said username.github.com subdomains still redirect to github.io for backward compatibility.

It's not unheard of for GitHub to rename inactive accounts, though: most likely, they gave this its own domain for something like SEO purposes (as it's content marketing).

Re: Counting Objects

#32

Post by scott_karana »

klodolph wrote: post #19
scott_karana wrote: post #15

Earlier quoted context omitted.

Ruby MRI and YARV are both written in C, so to be a true Ruby hacker you need some C chops. :-)

Interesting. I've heard that thrown around before, but I've never heard someone say the same thing about Python, even though the main Python implementation is C.

Really? Lots of the most famous Python modules are partially implemented in C(++) for performance reasons too, it's not exclusive to Ruby.

For example, Numpy is 56% C. https://github.com/numpy/numpy

Re: Counting Objects

#33

Post by Anderkent »

rakoo wrote: post #26

Not specific to the post, but: > we're sending few objects, all from the tip of the repository, and these objects will usually be delta'ed against older objects that won't be sent. Therefore, Git tries to find new delta bases for these objects. Why is this the case ? git can send thin packs if the receiver already has the objects, why does it still need to find a full base to diff against ? (Not counting when initial…

Their initial test case was a full clone, in which case you can't really send a shallow pack.

The problem, as I understand it, was that when you requested a full clone of fork1/repo.git, they'd find all objects reachable from refs in that repo, but git would by default generate deltas for those objects that referred to objects from other forks. When it noticed those objects were not going to be sent to the client, nor did client know about them, git recovered by doing expensive matching across the objects that it was sending, and without having traversed the graph before its heuristics weren't working properly so this took forever.

Re: Counting Objects

#34

Post by jmount »

"When you fork a repository on GitHub, we create a shallow copy of it. This copy has no objects of its own, but it has access to all the objects of an alternate ..."

So does this mean one could attach a GitHub repository by having a lot of shill accounts cline it and add random objects (possibly having a performance impact on the original)? I understand the engineering need for the use of alternates, but wonder about the lowered degree of isolation.

Re: Counting Objects

#35

Post by steveklabnik »

klodolph wrote: post #19
scott_karana wrote: post #15

Earlier quoted context omitted.

Ruby MRI and YARV are both written in C, so to be a true Ruby hacker you need some C chops. :-)

Interesting. I've heard that thrown around before, but I've never heard someone say the same thing about Python, even though the main Python implementation is C.

I have a talk at a Ruby conference last week and even put a small amount of assembly on the screen... While many Rubyists aren't expert C programmers, they know enough to look at Ruby's source every once in a while to figure out what's going on.

Re: Counting Objects

#36

Post by politician »

The Azure team responsible for the Local Git implementation needs to read this fantastic article.

I've been putting up with 10-minute deploys due to precisely this issue of counting objects. It's slow because we don't use Local Git as our source-of-record repository (because commits initiate a deployment step), so every deploy involves a clean fetch into a new tmpdir.

At least now I know why our deploys are getting slower and slower.

Re: Counting Objects

#38

Post by mikekchar »

omouse wrote: post #18
brian_cloutier wrote: post #14

This is technical recruiting done right, a lucid walkthrough of a hard problem complete with links to the implementation. It must have taken at least a few weeks of engineer-time to write, Github is awesome for making this public. I wish they had talked a little more about the tradeoff they made. They mentioned that splitting packfiles by fork was space-prohibitive, but ended up with a solution which must take more s…

This is an actual interesting problem that takes skill and knowledge to solve and it actually produces immense business value too (at least for Github). This is the kind of stuff more software engineers should be working on. I'm dying over here working on a small React JS component and some CSS layout tinkering, my brain's got graph traversal algorithms and bitmap indexes on its mind and can't do anything about it.

Every large problem is a series of small problems. I've worked on large complex systems and on small toy problems. It really isn't much different except for one thing. With toy problems you can get away with making a lot of mistakes because the interactions are simple. As the system becomes more and more complex, your execution as a programmer becomes more and more important. You have to think very deeply about the "why" and you have to be able to test your assumptions well.

If you have ever played the game of go, when you first start out the board is empty. You have to place your stone somewhere, but actually it doesn't really matter where. Over time, as you place more and more stones, your choices become more and more important (and ironically, you have more and more constraints). The weird thing is that, even though it doesn't matter where you place those first stones, it becomes very important where the stones were originally placed as the situation unfolds.

Programming is similar. When you first start a project, it really doesn't matter what you do. Almost everything will work to one degree or another. But the original decisions gain more and more weight as the project becomes more and more complex (and you are faced with more and more compromises). Eventually those original decisions can make or break your project, even though it didn't matter at first what they were (of course, this is why refactoring is so important --- but that's a different discussion).

My point (finally) is, that even though you may be making simple changes on simple systems, it doesn't have to stop you from understanding the implications of your work should the project become larger. Take the opportunity to polish your skills and make your "opening game" as perfect as you can make it.

I agree that at some point every programmer must start working on complex systems in order to grow. If you are at that point and your employer does not offer complex problems, then maybe it is time to move. However, don't neglect your "opening game". It is very, very important because, as I said at the beginning, every large problem is a series of small problems.

Re: Counting Objects

#40

Post by omouse »

mikekchar wrote: post #38
omouse wrote: post #18

Earlier quoted context omitted.

This is an actual interesting problem that takes skill and knowledge to solve and it actually produces immense business value too (at least for Github). This is the kind of stuff more software engineers should be working on. I'm dying over here working on a small React JS component and some CSS layout tinkering, my brain's got graph traversal algorithms and bitmap indexes on its mind and can't do anything about it.

Every large problem is a series of small problems. I've worked on large complex systems and on small toy problems. It really isn't much different except for one thing. With toy problems you can get away with making a lot of mistakes because the interactions are simple. As the system becomes more and more complex, your execution as a programmer becomes more and more important. You have to think very deeply about the "…

I only recently started playing Go and the similarity that you point is fantastic.

Take the opportunity to polish your skills and make your "opening game" as perfect as you can make it

Yep, exactly what I've been doing.

If you are at that point and your employer does not offer complex problems, then maybe it is time to move.

Yep ;-)

Post reply on HN