Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every GitHub object has two IDs

greptile.com

Post reply on HN
Print view View original

31–40 of 76 posts

Re: Every GitHub object has two IDs

#31

Post by innoying »

The newer global node IDs (which can be forced via the 'X-Github-Next-Global-ID' header [1]) have a prefix indicating the "type" of object delimited by an underscore, then a base64 encoded msgpack payload. For most objects it contains just a version (starting at 0) followed by the numeric "databaseId" field, but some are more complex.

For example, my GitHub user [2] has the node ID "U_kgDOAAhEkg". Users are "U_" and then the remaining data decodes to: [0, 541842] which matches the numeric ID for my user accepted by the REST API [3].

You shouldn't rely on any of this implementation of course, instead just directly query the "databaseId" field from the GraphQL API where you need interoperability. And in the other direction the REST API returns the "node_id" field for the GraphQL API.

For folks who finds this interesting, you might also like [4] which details GitHub's ETag implementation for the REST API.

[1] https://docs.github.com/en/graphql/guides/migrating-graphql-... [2] https://api.github.com/user/541842 [3] https://gchq.github.io/CyberChef/#recipe=Find_/_Replace(%7B'... [4] https://github.com/bored-engineer/github-conditional-http-tr...

Re: Every GitHub object has two IDs

#32

Post by vlovich123 »

krisoft wrote: post #11
agwa wrote: post #4

> GitHub's migration guide tells developers to treat the new IDs as opaque strings and treat them as references. However it was clear that there was some underlying structure to these IDs as we just saw with the bitmasking Great, so now GitHub can't change the structure of their IDs without breaking this person's code. The lesson is that if you're designing an API and want an ID to be opaque you have to literally enc…

> Great, so now GitHub can't change the structure of their IDs without breaking this person's code. And that is all the fault of the person who treated a documented opaque value as if it has some specific structure. > The lesson is that if you're designing an API and want an ID to be opaque you have to literally encrypt it. The lesson is that you should stop caring about breaking people’s code who go against the docu…

Sounds like you’ve maybe never actually run a service or API library at scale. There’s so many factors that go into a decision like that at a company that it’s never so simple. Is the person impacted influential? You’ve got a reputation hit if they negatively blog about how you screwed them after something was working for years. Is a customer who’s worth 10% of your annual revenue impacted? Bet your ass your management chain won’t let you do a breaking change / revert any you made by declaring an incident.

Even in OSS land, you risk alienating the community you’ve built if they’re meaningfully impact. You only do this if the impact is minimal or you don’t care about alienating anyone using your software.

Re: Every GitHub object has two IDs

#33

Post by vlovich123 »

agwa wrote: post #4

> GitHub's migration guide tells developers to treat the new IDs as opaque strings and treat them as references. However it was clear that there was some underlying structure to these IDs as we just saw with the bitmasking Great, so now GitHub can't change the structure of their IDs without breaking this person's code. The lesson is that if you're designing an API and want an ID to be opaque you have to literally enc…

Literally how I designed all the public facing R2 tokens like multipart uploads. It’s also a security barrier because forging and stealing of said tokens is harder and any vulnerability has to be done with cooperation of your servers and can be quickly shut down if needed.

Re: Every GitHub object has two IDs

#34

Post by vlovich123 »

lelandfe wrote: post #17
maxbond wrote: post #6

Earlier quoted context omitted.

You could also say, if I tell you something is an opaque identifier, and you introspect it, it's your problem if your code breaks. I told you not to do that.

Once "you" becomes a big enough "them" it becomes a problem again.

Exactly. When you owe the bank $10M it’s a you problem. When you owe the bank $100B it’s a them problem.

Re: Every GitHub object has two IDs

#35

Post by whateveracct »

agwa wrote: post #4

> GitHub's migration guide tells developers to treat the new IDs as opaque strings and treat them as references. However it was clear that there was some underlying structure to these IDs as we just saw with the bitmasking Great, so now GitHub can't change the structure of their IDs without breaking this person's code. The lesson is that if you're designing an API and want an ID to be opaque you have to literally enc…

Who cares if their code is broken in this case? Stupid games stupid prizes.

Re: Every GitHub object has two IDs

#36

Post by zzo38computer »

I had seen GitHub node IDs, although I had not used them or tried to decode them (although I could see they seem to be base64), since I only used the REST API, which reports node IDs but does not use them as input.

It looks like a good explanation of the node IDs, though. However, like another comment says, you should not rely on the format of node IDs.

Re: Every GitHub object has two IDs

#37

Post by halestock »

krisoft wrote: post #11
agwa wrote: post #4

> GitHub's migration guide tells developers to treat the new IDs as opaque strings and treat them as references. However it was clear that there was some underlying structure to these IDs as we just saw with the bitmasking Great, so now GitHub can't change the structure of their IDs without breaking this person's code. The lesson is that if you're designing an API and want an ID to be opaque you have to literally enc…

> Great, so now GitHub can't change the structure of their IDs without breaking this person's code. And that is all the fault of the person who treated a documented opaque value as if it has some specific structure. > The lesson is that if you're designing an API and want an ID to be opaque you have to literally encrypt it. The lesson is that you should stop caring about breaking people’s code who go against the docu…

> The lesson is that you should stop caring about breaking people’s code who go against the documentation this way. When it breaks you shrug. Their code was always buggy and it just happened to be working for them until then. You are not their dad. You are not responsible for their misfortune.

Sure, but good luck running a business with that mindset.

Re: Every GitHub object has two IDs

#38

Post by irjustin »

vlovich123 wrote: post #32
krisoft wrote: post #11

Earlier quoted context omitted.

> Great, so now GitHub can't change the structure of their IDs without breaking this person's code. And that is all the fault of the person who treated a documented opaque value as if it has some specific structure. > The lesson is that if you're designing an API and want an ID to be opaque you have to literally encrypt it. The lesson is that you should stop caring about breaking people’s code who go against the docu…

Sounds like you’ve maybe never actually run a service or API library at scale. There’s so many factors that go into a decision like that at a company that it’s never so simple. Is the person impacted influential? You’ve got a reputation hit if they negatively blog about how you screwed them after something was working for years. Is a customer who’s worth 10% of your annual revenue impacted? Bet your ass your manageme…

> Sounds like you’ve maybe never actually run a service or API library at scale.

What was the saying? When your scale is big enough, even your bugs have users.

Re: Every GitHub object has two IDs

#39

Post by bastawhiz »

1. The list of "scopes" are the object hierarchy that owns the resource. That lets you figure out which shard a resource should be in. You want all the resources for the same repository on the same shard, otherwise if you simply hash the id, one shard going down takes down much of your service since everything is spread more or less uniformly across shards.

2. The object identifier is at the end. That should be strictly increasing, so all the resources for the same scope are ordered in the DB. This is one of the benefits of uuid7.

3. The first element is almost certainly a version. If you do a migration like this, you don't want to rule out doing it again. If you're packing bits, it's nearly impossible to know what's in the data without an identifier, so without the version you might not be able know whether the id is new or old.

Another commenter mentioned that you should encrypt this data. Hard pass! Decrypting each id is decidedly slower than b64 decode. Moreover, if you're picking apart IDs, you're relying on an interface that was never made for you. There's nothing sensitive in there: you're just setting yourself up for a possible (probable?) world of pain in the future. GitHub doesn't have to stop you from shooting your foot off.

Moreover, encrypting the contents of the ID makes them sort randomly. This is to be avoided: it means similar/related objects are not stored near each other, and you can't do simple range scans over your data.

You could decrypt the ids on the way in and store both the unencrypted and encrypted versions in the DB, but why? That's a lot of complexity, effort, and resources to stop randos on the Internet from relying on an internal, non-sensitive data format.

As for the old IDs that are still appearing, they are almost certainly:

1. Sharded by their own id (i.e., users are sharded by user id, not repo id), so you don't need additional information. Use something like rendezvous hashing to choose the shard.

2. Got sharded before the new id format was developed, and it's just not worth the trouble to change

Re: Every GitHub object has two IDs

#40

Post by raincole »

irjustin wrote: post #38
vlovich123 wrote: post #32

Earlier quoted context omitted.

Sounds like you’ve maybe never actually run a service or API library at scale. There’s so many factors that go into a decision like that at a company that it’s never so simple. Is the person impacted influential? You’ve got a reputation hit if they negatively blog about how you screwed them after something was working for years. Is a customer who’s worth 10% of your annual revenue impacted? Bet your ass your manageme…

> Sounds like you’ve maybe never actually run a service or API library at scale. What was the saying? When your scale is big enough, even your bugs have users.

Yeah, but when you are big enough you can afford to not care individual users.

VScode once broke a very popular extension that used a private API. Microsoft (righteously) didn't bother to ask if the private API had users.

Post reply on HN