Live data from Hacker News

Congratulations on creating the one billionth repository on GitHub

github.com

71–80 of 146 posts

Re: Congratulations on creating the one billionth repository on GitHub

#71
Probably created via a script that just repeatedly checked https://api.github.com/repositories/999999999 until it showed up, and then created a new repository. Since repositories can be modified, could have even given it some buffer and created a bunch of repos, just delete the ones that don't get the right number. [append] Looking at the author's other repo created yesterday, I'm betting "yep" was supposed to be the magic number, and "shit" was an admission of missing the mark.

Does anyone remember D666666 from Facebook? It was a massive codemod; the author used a technique similar to this one to get that particular number.

Re: Congratulations on creating the one billionth repository on GitHub

#72

Earlier quoted context omitted.

The company where I did my stint as CTO I turned up, noticed they were using 32-bit integers as primary keys on one of their key tables that already had 1.3 billion rows and, at the rate they were adding them, would overflow on primary key values within months… so we ran a fairly urgent project to upgrade the IDs to 64-bit to avoid the total meltdown that would have ensued otherwise.

What are the challenges of such projects? How many people are usually involved? Does it incur downtimes or significant technical challenges for either the infrastructure or the codebase?

Changing the type of the column is no big deal per se, except on a massive table it’s a non-trivial operation, BUT you also have to change the type in everything that touches it, everywhere it’s assigned or copied, everywhere it’s sent over the wire and deserialized where assumptions might be made, any tests, and on, and on. And god help you if you’ve got stuff like int.MaxValue having a special meaning (we didn’t in this context, fortunately).

Our hosting environment at that time was a data centre so we were limited on storage, which complicated matters a bit. Like ideally you’d create a copy of the table but with a wider PK column and write to both tables, then migrate your reads, etc., but we couldn’t do that because the table was massive and we didn’t have enough space. Procuring more drives was possible but took sometimes weeks - no just dragging a slider in your cloud portal. And then of course you’d have to schedule a maintenance window for somebody to plug it in. It was absolutely archaic, especially when you consider this was late 2017/early 2018.

You need multiple environments so you can do thorough testing, which we barely had at that point, and because every major system component was impacted, we had to redeploy our entire platform. Also, because it was the PK column affected, we couldn’t do any kind of staged migration or rollback without the project becoming much more complex and taking a lot longer - time we didn’t have due to the rate at which we were consuming 32-bit integer values.

In the end it went off without a hitch, but pushing it live was still a bit of a white knuckle moment.

Re: Congratulations on creating the one billionth repository on GitHub

#75
post #67
post #47

Reminds me of the 100 millionth OpenStreetMap changeset (commit). A few people, myself included, were casually trying for it but in the end it went to someone who wasn't trying and just busy mapping Africa! Much more wholesome, seeing it with hindsight. This person was also previously nominated for an OSM award. I guess it helps that openstreetmap doesn't really allow for creating crap, because it's all live in produ…

Your kind of comment is exactly why HN still rules. What a fun story. Thanks for sharing

Aww, thanks! I wasn't sure if I should go off-topic this much so I'm happy to hear this!

Re: Congratulations on creating the one billionth repository on GitHub

#77
post #31

Earlier quoted context omitted.

The rate of creation is like meh, but being able to enumerate all of the repos might be problematic, following new repos and scanning them for leaked credentials could be a negative... but github may have a feed of new repos anyway? Also, having a sequence implies at least a global lock on that sequence during repo creation. Repo creation could otherwise be a scoped lock. OTOH, it's not necessarily handled that way -…

>following new repos and scanning them for leaked credentials could be a negative People do this. GitHub started doing it too so now you get a nice email from them first instead of another kind of surprise.

Email, bleh, I'm sure I'm not the only one who basically /dev/null's emails from github about pearl-clutching "security" but I wanted to point out that for quite a few providers they actually have an integration to revoke them if found in a public repo, which I think is way more handy

https://docs.github.com/en/code-security/secret-scanning/sec...

and the list is way bigger than I recalled: https://docs.github.com/en/code-security/secret-scanning/int...

Re: Congratulations on creating the one billionth repository on GitHub

#78
I wish I were still at Apple. Probably most people here know that Apple uses an internal tool called "Radar" since, forever. Each "Radar" has an ID (bug #) associated with it.

Radars that were bug #1,000,000, etc. were kind of special. Unless someone screwed up (and let down the whole team) they were usually faux-Radars with lots of inside jokes, etc.

Pulling up one was enough since the Radar could reference other Radars ... and generally you would go down the rabbit hole at that point enjoying the ride.

I was a dumbass not to capture (heck, even print) a few of those when I had the opportunity.

Re: Congratulations on creating the one billionth repository on GitHub

#79

I wish I were still at Apple. Probably most people here know that Apple uses an internal tool called "Radar" since, forever. Each "Radar" has an ID (bug #) associated with it. Radars that were bug #1,000,000, etc. were kind of special. Unless someone screwed up (and let down the whole team) they were usually faux-Radars with lots of inside jokes, etc. Pulling up one was enough since the Radar could reference other Ra…

> I was a dumbass not to capture (heck, even print) a few of those when I had the opportunity.

On the other hand, given how Apple deals with confidential data, you probably wouldn't want to be caught exfiltrating internal documents however benign they are.

Re: Congratulations on creating the one billionth repository on GitHub

#80
post #35

Makes me wonder how many repositories exist in general, from all the local Forgejo and Gitlab servers. Heck, include Subversion and Mercurial and git's other friends (and foes!) Did anyone make a search engine for these yet, so we'd be able to get an estimate by searching for the word "a" or so? (This always seemed like the big upside of centralised GitHub to me: people can actually find your code. I've been thinking…

Well, based on the API enumeration mentioned in sibling comments, surely one doesn't have to estimate

https://docs.gitlab.com/api/projects/#list-all-projects (for dumb reasons it seems GL calls them Projects, not Repositories)

https://codeberg.org/api/swagger#/repository/repoGetByID (that was linked to by the Forgejo.org site, so presumably it's the same for it and Codeberg) and its friend https://gitea.com/api/swagger#/repository/repoGetByID

Heptapod is a "friendly fork" of GitLab CE so its API works the same: https://heptapod.net/pages/faq#api-hgrc

and then I'd guess one would need to index the per-project GitLab instances: Gnome, GNU (if they ever open theirs back up), whatever's going on with Savannah, probably Sourceforge, maybe sourcehut (assuming he doesn't have some political reason to block you), etc

If I won the lottery, I'd probably bankroll a sourcegraph instance (from back when they were Apache) across everything I could get my hands upon, and donate snapshots of it to the Internet Archive

Post reply on HN