Respect to GitHub for committing to the bit
Congratulations on creating the one billionth repository on GitHub
61–70 of 146 posts
Re: Congratulations on creating the one billionth repository on GitHub
#62Awesome! Only a little over a billion more to go before GitHub’s very own OpenAPI Spec can start overflowing int32 on repositories too, just like it already does for workflows run IDs! https://github.com/github/rest-api-description/issues/4511
The company where I did my stint as CTO I turned up, noticed they were using 32-bit integers as primary keys on one of their key tables that already had 1.3 billion rows and, at the rate they were adding them, would overflow on primary key values within months… so we ran a fairly urgent project to upgrade the IDs to 64-bit to avoid the total meltdown that would have ensued otherwise.
What are the challenges of such projects? How many people are usually involved? Does it incur downtimes or significant technical challenges for either the infrastructure or the codebase?
Re: Congratulations on creating the one billionth repository on GitHub
#63Re: Congratulations on creating the one billionth repository on GitHub
#64Earlier quoted context omitted.
The company where I did my stint as CTO I turned up, noticed they were using 32-bit integers as primary keys on one of their key tables that already had 1.3 billion rows and, at the rate they were adding them, would overflow on primary key values within months… so we ran a fairly urgent project to upgrade the IDs to 64-bit to avoid the total meltdown that would have ensued otherwise.
What are the challenges of such projects? How many people are usually involved? Does it incur downtimes or significant technical challenges for either the infrastructure or the codebase?
Not the original commenter, but I've read through half a dozen post-mortems about this kind of thing. The answer is: yes. There's challenges and sometimes downtime and/or breaking changes are inevitable.
For one, if your IDs are approaching the 2^31 signed integer limit, then by definition, you have nearly two billion rows, which is a very big DB table! There are only a handful of systems that can handle any kind of change to that volume of data quickly. Everything you do to it will either need hours of downtime or careful orchestration of incremental/rolling changes. This issue tends to manifest first on the "biggest" and hence most important table in the business such as "sales entries" or "user comments". It's never some peripheral thing that nobody cares about.
Second, if you're using small integer IDs, that decision was probably motivated in part because you're using those integers as foreign keys and for making your secondary indexes more efficient. GUIDs are "simpler" in some ways but need 4x the data storage (assuming you're using a clustered database like MySQL or SQL Server). Even just the change from 32-bits to 64-bits doubles the size of the storage in a lot of places. For 2 billion rows, this is 8 GB more data minimum, but is almost certainly north of 100 GB across all tables and indexes.
Third, many database engines will refuse to establish foreign key constraints if the types don't match. This can force big-bang changes or very complex duplication of data during the migration phase.
Fourth, this is a breaking change to all of your APIs, both internal and external. Every ORM, REST endpoint, etc... will have to be updated with a new major version. There's a chance that all of your analytics, ETL jobs, etc... will also need to be touched.
Fun times.
Re: Congratulations on creating the one billionth repository on GitHub
#65I'm wondering if AasishPokhrel created this repo for the purpose of being the billionth.
I think he’s in university for software development in Nepal, and it’s really touching that a milestone could go so deeply into the world. Hopefully he has a big spot for this on his resume and can find a great career in development!
Re: Congratulations on creating the one billionth repository on GitHub
#66Re: Congratulations on creating the one billionth repository on GitHub
#67Reminds me of the 100 millionth OpenStreetMap changeset (commit). A few people, myself included, were casually trying for it but in the end it went to someone who wasn't trying and just busy mapping Africa! Much more wholesome, seeing it with hindsight. This person was also previously nominated for an OSM award. I guess it helps that openstreetmap doesn't really allow for creating crap, because it's all live in produ…
Your kind of comment is exactly why HN still rules. What a fun story. Thanks for sharing
Re: Congratulations on creating the one billionth repository on GitHub
#68Earlier quoted context omitted.
What are the challenges of such projects? How many people are usually involved? Does it incur downtimes or significant technical challenges for either the infrastructure or the codebase?
Not the original commenter, but I've read through half a dozen post-mortems about this kind of thing. The answer is: yes. There's challenges and sometimes downtime and/or breaking changes are inevitable. For one, if your IDs are approaching the 2^31 signed integer limit, then by definition, you have nearly two billion rows , which is a very big DB table! There are only a handful of systems that can handle any kind of…
> For one, if your IDs are approaching the 2^31 signed integer limit, then by definition, you have nearly two billion rows
Just wanted to nitpick this; this is not actually definitively true. A failed insert in some systems will increment the counter and deleting rows usually does not allow the deleted ID to be re-used (new inserts use the current counter). Of course, that is beside the point: the typical case of a table approaching this limit is a very large table.
Re: Congratulations on creating the one billionth repository on GitHub
#69[flagged]
Re: Congratulations on creating the one billionth repository on GitHub
#70I'm wondering if AasishPokhrel created this repo for the purpose of being the billionth.
I think he’s in university for software development in Nepal, and it’s really touching that a milestone could go so deeply into the world. Hopefully he has a big spot for this on his resume and can find a great career in development!
I don't get why this needs a big spot in his resume, and why it should lead to a great career. A company/hiring manager that thinks being lucky to hit a magic number on some system has any relevance to work, I'd rate as very insane...