Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every GitHub object has two IDs

greptile.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Every GitHub object has two IDs

#51

Post by Kwpolska »

Some commenters mentioned that the GraphQL API exposes the database IDs and a URL to the object directly in the schema. But the author did not know that and instead reverse-engineered the ID generation logic.

This is one of many reasons why GraphQL sucks. Developers will do anything to avoid reading docs. In the REST API, the developerId and url fields would be easily discoverable by looking at the API response. But in GraphQL, there is no way to get all fields. You need to get a list of fields from the docs and request them explicitly.

Re: Every GitHub object has two IDs

#52

Post by catlifeonmars »

siralonso wrote: post #26

I wouldn't decode them like this, it's fragile, and global node IDs are supposed to be opaque in GraphQL. I see that GitHub exposes a `databaseId` field on many of their types (like PullRequest) - is that what you're looking for? [1] Most GraphQL APIs that serve objects that implement the Node interface just base-64-encode the type name and the database ID, but I definitely wouldn't rely on that always being the case…

If you want to store metadata in identifiers, an easy fix to preventing users from depending on arbitrary characteristics is to encrypt the data. You see this a lot with pagination tokens.

Re: Every GitHub object has two IDs

#53

Post by catlifeonmars »

zzo38computer wrote: post #36

I had seen GitHub node IDs, although I had not used them or tried to decode them (although I could see they seem to be base64), since I only used the REST API, which reports node IDs but does not use them as input. It looks like a good explanation of the node IDs, though. However, like another comment says, you should not rely on the format of node IDs.

GitHub should have encrypted their node ids. Now they risk breaking a ton of users if they decide they want to change anything about the id generation (for example to scale the db horizontally by removing sequential ids).

Several companies I’ve worked for have had policies outright blocking the use of nonrandom identifiers in production code.

Re: Every GitHub object has two IDs

#54

Post by ldjb »

Kwpolska wrote: post #51

Some commenters mentioned that the GraphQL API exposes the database IDs and a URL to the object directly in the schema. But the author did not know that and instead reverse-engineered the ID generation logic. This is one of many reasons why GraphQL sucks. Developers will do anything to avoid reading docs. In the REST API, the developerId and url fields would be easily discoverable by looking at the API response. But…

The author may well have been aware of this. However, since the author didn't retrieve those database IDs or URLs in the first place, they would have had to make further requests to retrieve them, which they wanted to avoid doing.

"I was looking at either backfilling millions of records or migrating our entire database, and neither sounded fun."

Re: Every GitHub object has two IDs

#55

Post by blurayfin »

Most of what the author discovered is real and technically correct, but it is also undocumented, unsupported, and risky to rely on.

GitHub has changed node ID internals before, quietly. If they add a field to the MessagePack array, switch encodings, encrypt payloads, introduce UUID-backed IDs..

every system relying on this will break instantly.

Re: Every GitHub object has two IDs

#56

Post by misiek08 »

catlifeonmars wrote: post #53
zzo38computer wrote: post #36

I had seen GitHub node IDs, although I had not used them or tried to decode them (although I could see they seem to be base64), since I only used the REST API, which reports node IDs but does not use them as input. It looks like a good explanation of the node IDs, though. However, like another comment says, you should not rely on the format of node IDs.

GitHub should have encrypted their node ids. Now they risk breaking a ton of users if they decide they want to change anything about the id generation (for example to scale the db horizontally by removing sequential ids). Several companies I’ve worked for have had policies outright blocking the use of nonrandom identifiers in production code.

Just out of curiosity, because I saw way too many i++ implementations - were things like UUIDv7 allowed or because of timestamp they are not random enough? While having such conversations I assume it’s already good enough, but maybe I’ll learn something here!

Re: Every GitHub object has two IDs

#57

Post by catlifeonmars »

misiek08 wrote: post #56
catlifeonmars wrote: post #53

Earlier quoted context omitted.

GitHub should have encrypted their node ids. Now they risk breaking a ton of users if they decide they want to change anything about the id generation (for example to scale the db horizontally by removing sequential ids). Several companies I’ve worked for have had policies outright blocking the use of nonrandom identifiers in production code.

Just out of curiosity, because I saw way too many i++ implementations - were things like UUIDv7 allowed or because of timestamp they are not random enough? While having such conversations I assume it’s already good enough, but maybe I’ll learn something here!

UUIDv7 wasn’t as ubiquitous at the time so it did not come up.

Here’s how I would think about it: do I want users to depend on the ordering of UUIDv7? Now I’m tied to that implementation and probably have to start worrying about clock skew.

If it’s not a feature you want to support, don’t expose it. Otherwise you’re on the hook for it. If you do explicitly want to provide time ordering as a feature, then UUIDv7 is a great choice and preferable to rolling your own format.

Re: Every GitHub object has two IDs

#58

Post by bob1029 »

The only GitHub identifier Ive ever bothered to store explicitly (I.e., in its own dedicated column) is an immutable URL key like issue/pr # or commit hash. I've stored comment ids but I've never thought about it. They just get sucked up with the rest of the JSON blob.

Not everything has to be forced through some normalizing layer. You can maintain coarse rows at the grain of each issue/PR and keep everything else in the blob. JSON is super fast. Unless you're making crosscutting queries along comment dimensions, I don't think this would ever show up on a profiler.

Re: Every GitHub object has two IDs

#59

Post by progval »

bob1029 wrote: post #58

The only GitHub identifier Ive ever bothered to store explicitly (I.e., in its own dedicated column) is an immutable URL key like issue/pr # or commit hash. I've stored comment ids but I've never thought about it. They just get sucked up with the rest of the JSON blob. Not everything has to be forced through some normalizing layer. You can maintain coarse rows at the grain of each issue/PR and keep everything else in…

> immutable URL key like issue/pr

they are not immutable because repositories can change URLs (renamed or moved to a different org).

Re: Every GitHub object has two IDs

#60

Post by _cenw »

I remember a time when the v3 API didn't even have IDs and whenever someone in your org changed their username or renamed a repo you'd be left guessing who it is.

This is also the reason I wrote our current user onboarding / repo management code from scratch, because the terraform provider sucks and without any management you'll have a wave of "x got offboarded but they were the only admin on this repo" requests. Every repo is owned by a team. Access is only ever per-team.

Post reply on HN