Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

groups.google.com

Post reply on HN
Print view View original

41–48 of 48 posts

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#41

Post by riffraff »

Retric wrote: post #20
jonknee wrote: post #13

Earlier quoted context omitted.

64 bits is enough to have everyone on Earth send over a billion Tweets and still have enough room to find a new solution. That sounds like more than enough to me.

The problem is they added a timestamp and people assumed they would not need the full 64 bit ID. IMO, it's not a bad idea on their part. A 32 bit UNIX timestamp * 2 ^ 32 + a 32 bit sequential id let's them track up to 4.2 billion tweets a second and should work just find up to the year 2106. Edit: As to why it's a good idea, you can have different systems handing out ID's without stepping on each other’s toes or even…

this is exactly what they have done in their new ids and the snowflake system. They have a timestamp, a sequence number and a system identifer, plus a few neat properties.

I don't think they can be blamed for using a trivial incremental key when they had 10 users, I am sure they were not expecting to have 200M :)

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#42

Post by pedrocr »

prodigal_erik wrote: post #38
sedachv wrote: post #26

I'm going to take the opposite position of most commenters here and decry the fact that almost all programming languages get arithmetic wrong. Adding two positive numbers should never result in a negative number. Adding a one to an integer should result in the next largest integer. There are new languages created all the time that don't have built-in support for arbitrary precision integers or rational numbers. They…

Arithmetic is a relatively self-contained part of a language. The problem isn't that JavaScript's arithmetic is lame, it's that they didn't leave any way for the programmer to fix it—even if you make rational bignums out of arrays of doubles (or whatever), you can't make them drop-in replacements by overloading operators. It's a strange oversight for a language which lets instances override methods.

>It's a strange oversight for a language which lets instances override methods.

That's only because everything is an instance in Javascript. There are no classes, it's prototype OO.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#44

Post by wahnfrieden »

mayanklahiri wrote: post #31
drawkbox wrote: post #18

Earlier quoted context omitted.

Still surprised people use incrementing ints for ids over uuids for web apis that may become immense. http://www.ietf.org/rfc/rfc4122.txt uuids are made just for this purpose, string based, never bigger than 40 characters (with dashes and curly braces). Most products use uuids or Microsoft's name for them guids.

You don't even need to go that far. For non-critical applications (i.e., your web app), you can randomly generate a small string, say 12 bytes, using base-62 characters (A-Za-z0-9) to serve as a probably unique user ID (with VERY high probability).

Heard of GUIDs? Make part of it a timestamp, and the chance of collisions goes down astronomically further.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#45

Post by wladimir »

sedachv wrote: post #26

I'm going to take the opposite position of most commenters here and decry the fact that almost all programming languages get arithmetic wrong. Adding two positive numbers should never result in a negative number. Adding a one to an integer should result in the next largest integer. There are new languages created all the time that don't have built-in support for arbitrary precision integers or rational numbers. They…

A lot of languages also get arithmetic right, for example Python. Most languages that get it 'wrong' are low-level languages (such as C), which have a good reason, as CPUs have limited width registers. However, I agree that there is no excuse for Javascript to not have arbitrary length integers. Then again, that language is broken and ugly in a lot of ways.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#46

Post by morgo »

beagle3 wrote: post #21
morgo wrote: post #14

Earlier quoted context omitted.

-1. With hindsight it's easy to make a comment like yours, but many clients rely on the status_id to be monotonic. You would be changing behavior in a way that breaks apps. Twitter doesn't want to do that.

-1 all you like. It's engineering, not hindsight -- when building a system, I always ask myself "how does this scale" which usually translates to "on what attribute does this shard". Look at e.g. YouTube and many other sites around the same time. They knew what they were doing; Twitter didn't. I _have_ actually designed such a system in 1999, that used 48 bits, and it worked perfectly well. (Only had 28 bits for the…

I don't see how your comparison is remotely fair to Twitter. Youtube is just a website, if they wanted to change how they allocate video numbers, nobody except for rogue bots will notice.

Twitter is an API. Even if they have the knowledge of how to fix past mistakes, they need to ask for feedback, give plenty of notice, and set a deadline of when the old version is cut off.

Maybe they didn't do everything perfect day one, but I don't think there are any APIs the size of Twitter. Cut them some slack, they're not morons.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#47

Post by beagle3 »

morgo wrote: post #46
beagle3 wrote: post #21

Earlier quoted context omitted.

-1 all you like. It's engineering, not hindsight -- when building a system, I always ask myself "how does this scale" which usually translates to "on what attribute does this shard". Look at e.g. YouTube and many other sites around the same time. They knew what they were doing; Twitter didn't. I _have_ actually designed such a system in 1999, that used 48 bits, and it worked perfectly well. (Only had 28 bits for the…

I don't see how your comparison is remotely fair to Twitter. Youtube is just a website, if they wanted to change how they allocate video numbers, nobody except for rogue bots will notice. Twitter is an API. Even if they have the knowledge of how to fix past mistakes, they need to ask for feedback, give plenty of notice, and set a deadline of when the old version is cut off. Maybe they didn't do everything perfect day…

The point was exactly that YouTube, which had a simpler problem to solve at around the same time (or earlier) did it properly.

There do exist independent youtube clients (though not as many as Twitters's), but using the encoding they did, youtube has made it so that it is never going to be an issue, whereas for twitter it has already been a significant issue twice (that I'm aware of).

It's very easy to dismiss sound engineering in retrospect as luck or as "how could anyone have known".

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#48

Post by mayanklahiri »

wahnfrieden wrote: post #44
mayanklahiri wrote: post #31

Earlier quoted context omitted.

You don't even need to go that far. For non-critical applications (i.e., your web app), you can randomly generate a small string, say 12 bytes, using base-62 characters (A-Za-z0-9) to serve as a probably unique user ID (with VERY high probability).

Heard of GUIDs? Make part of it a timestamp, and the chance of collisions goes down astronomically further.

In some cases, including a timestamp in an ID can be giving away information considered private. Sure, you could just hash the resultant ID, but then you're getting back to random digits anyway.
Post reply on HN