Does anyone remember D666666 from Facebook? It was a massive codemod; the author used a technique similar to this one to get that particular number.
Congratulations on creating the one billionth repository on GitHub
71–80 of 146 posts
Re: Congratulations on creating the one billionth repository on GitHub
#72Earlier quoted context omitted.
The company where I did my stint as CTO I turned up, noticed they were using 32-bit integers as primary keys on one of their key tables that already had 1.3 billion rows and, at the rate they were adding them, would overflow on primary key values within months… so we ran a fairly urgent project to upgrade the IDs to 64-bit to avoid the total meltdown that would have ensued otherwise.
What are the challenges of such projects? How many people are usually involved? Does it incur downtimes or significant technical challenges for either the infrastructure or the codebase?
Our hosting environment at that time was a data centre so we were limited on storage, which complicated matters a bit. Like ideally you’d create a copy of the table but with a wider PK column and write to both tables, then migrate your reads, etc., but we couldn’t do that because the table was massive and we didn’t have enough space. Procuring more drives was possible but took sometimes weeks - no just dragging a slider in your cloud portal. And then of course you’d have to schedule a maintenance window for somebody to plug it in. It was absolutely archaic, especially when you consider this was late 2017/early 2018.
You need multiple environments so you can do thorough testing, which we barely had at that point, and because every major system component was impacted, we had to redeploy our entire platform. Also, because it was the PK column affected, we couldn’t do any kind of staged migration or rollback without the project becoming much more complex and taking a lot longer - time we didn’t have due to the rate at which we were consuming 32-bit integer values.
In the end it went off without a hitch, but pushing it live was still a bit of a white knuckle moment.
Re: Congratulations on creating the one billionth repository on GitHub
#73[flagged]
Re: Congratulations on creating the one billionth repository on GitHub
#74Re: Congratulations on creating the one billionth repository on GitHub
#75Reminds me of the 100 millionth OpenStreetMap changeset (commit). A few people, myself included, were casually trying for it but in the end it went to someone who wasn't trying and just busy mapping Africa! Much more wholesome, seeing it with hindsight. This person was also previously nominated for an OSM award. I guess it helps that openstreetmap doesn't really allow for creating crap, because it's all live in produ…
Your kind of comment is exactly why HN still rules. What a fun story. Thanks for sharing
Re: Congratulations on creating the one billionth repository on GitHub
#76Re: Congratulations on creating the one billionth repository on GitHub
#77Earlier quoted context omitted.
The rate of creation is like meh, but being able to enumerate all of the repos might be problematic, following new repos and scanning them for leaked credentials could be a negative... but github may have a feed of new repos anyway? Also, having a sequence implies at least a global lock on that sequence during repo creation. Repo creation could otherwise be a scoped lock. OTOH, it's not necessarily handled that way -…
>following new repos and scanning them for leaked credentials could be a negative People do this. GitHub started doing it too so now you get a nice email from them first instead of another kind of surprise.
https://docs.github.com/en/code-security/secret-scanning/sec...
and the list is way bigger than I recalled: https://docs.github.com/en/code-security/secret-scanning/int...
Re: Congratulations on creating the one billionth repository on GitHub
#78Radars that were bug #1,000,000, etc. were kind of special. Unless someone screwed up (and let down the whole team) they were usually faux-Radars with lots of inside jokes, etc.
Pulling up one was enough since the Radar could reference other Radars ... and generally you would go down the rabbit hole at that point enjoying the ride.
I was a dumbass not to capture (heck, even print) a few of those when I had the opportunity.
Re: Congratulations on creating the one billionth repository on GitHub
#79I wish I were still at Apple. Probably most people here know that Apple uses an internal tool called "Radar" since, forever. Each "Radar" has an ID (bug #) associated with it. Radars that were bug #1,000,000, etc. were kind of special. Unless someone screwed up (and let down the whole team) they were usually faux-Radars with lots of inside jokes, etc. Pulling up one was enough since the Radar could reference other Ra…
On the other hand, given how Apple deals with confidential data, you probably wouldn't want to be caught exfiltrating internal documents however benign they are.
Re: Congratulations on creating the one billionth repository on GitHub
#80Makes me wonder how many repositories exist in general, from all the local Forgejo and Gitlab servers. Heck, include Subversion and Mercurial and git's other friends (and foes!) Did anyone make a search engine for these yet, so we'd be able to get an estimate by searching for the word "a" or so? (This always seemed like the big upside of centralised GitHub to me: people can actually find your code. I've been thinking…
https://docs.gitlab.com/api/projects/#list-all-projects (for dumb reasons it seems GL calls them Projects, not Repositories)
https://codeberg.org/api/swagger#/repository/repoGetByID (that was linked to by the Forgejo.org site, so presumably it's the same for it and Codeberg) and its friend https://gitea.com/api/swagger#/repository/repoGetByID
Heptapod is a "friendly fork" of GitLab CE so its API works the same: https://heptapod.net/pages/faq#api-hgrc
and then I'd guess one would need to index the per-project GitLab instances: Gnome, GNU (if they ever open theirs back up), whatever's going on with Savannah, probably Sourceforge, maybe sourcehut (assuming he doesn't have some political reason to block you), etc
If I won the lottery, I'd probably bankroll a sourcegraph instance (from back when they were Apache) across everything I could get my hands upon, and donate snapshots of it to the Internet Archive