Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

groups.google.com

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#21

Post by beagle3 »

morgo wrote: post #14
beagle3 wrote: post #12

That's truly, truly dumb on twitter's behalf. They should have started with using the whole numeric range and/or switched to string ids long ago. As I have suggested in a post 9 months ago, and as I would have designed this system at any date since 2002 or so, the twit ID would be composed of userid bits + time bits. In the post below I suggested 32+32, but other divisions are acceptable depending on your "bot user"…

-1. With hindsight it's easy to make a comment like yours, but many clients rely on the status_id to be monotonic. You would be changing behavior in a way that breaks apps. Twitter doesn't want to do that.

-1 all you like. It's engineering, not hindsight -- when building a system, I always ask myself "how does this scale" which usually translates to "on what attribute does this shard".

Look at e.g. YouTube and many other sites around the same time. They knew what they were doing; Twitter didn't.

I _have_ actually designed such a system in 1999, that used 48 bits, and it worked perfectly well. (Only had 28 bits for the user-id, which would have been broken at the 250M users -- alas the system never had more than than 5M; This was in the years 1999-2003).

The only way you can shard absolutely monotonic is (effectively) randomly, which is an option however you assign ids; but other assignments let you build a much cheaper, much more robust system.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#23

Post by jonhohle »

drawkbox wrote: post #18
simonw wrote: post #8

My rule of thumb is to use strings for all IDs that belong to external systems. A bunch of people got burned in the same way a few years ago when Flickr photo IDs rolled past 32 bits.

Still surprised people use incrementing ints for ids over uuids for web apis that may become immense. http://www.ietf.org/rfc/rfc4122.txt uuids are made just for this purpose, string based, never bigger than 40 characters (with dashes and curly braces). Most products use uuids or Microsoft's name for them guids.

I prefer my ID's universally unique. I worry that globally unique ID's won't scale as we begin to colonize other planets. (yes, its a joke.)

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#24

Post by jmcnevin »

Being in that majority of people who haven't built Twitter-scale systems in the past, could someone explain to me why Twitter is moving to a new form of twitter ID in the first place? What is wrong with their current system?

Also, it seems strange that they would include the new ID in string AND integer form in their JSON. I realize that they don't want to break existing javascript apps, but isn't there a significant bandwidth cost in adding that sort of kludge to the API when you're serving a quadrillion of these api requests every day?

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#25

Post by dangrossman »

jmcnevin wrote: post #24

Being in that majority of people who haven't built Twitter-scale systems in the past, could someone explain to me why Twitter is moving to a new form of twitter ID in the first place? What is wrong with their current system? Also, it seems strange that they would include the new ID in string AND integer form in their JSON. I realize that they don't want to break existing javascript apps, but isn't there a significant…

How do you generate sequential IDs a quadrillion times a day if they're coming from 100 different computers? Lots of very fast coordination.

In the new system, they don't have to coordinate every server just to make sure IDs are sequential. They just use the time stamp and some machine-specific information.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#26

Post by sedachv »

I'm going to take the opposite position of most commenters here and decry the fact that almost all programming languages get arithmetic wrong.

Adding two positive numbers should never result in a negative number. Adding a one to an integer should result in the next largest integer.

There are new languages created all the time that don't have built-in support for arbitrary precision integers or rational numbers. They might have some neat ideas, but if your language can't even get arithmetic right, it's garbage.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#27

Post by seiji »

siculars wrote: post #17
icco wrote: post #15

Earlier quoted context omitted.

I imagine that they did think of this, but a good string hash takes time. So switching to 64-bits takes much less time, and it would work until they finished snowflake.

The 64bit id is snowflake.

Technically, snowflake ids are 63 bits. Java doesn't have unsigned longs, so they don't touch the sign bit.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#28

Post by ryansking »

drawkbox wrote: post #18
simonw wrote: post #8

My rule of thumb is to use strings for all IDs that belong to external systems. A bunch of people got burned in the same way a few years ago when Flickr photo IDs rolled past 32 bits.

Still surprised people use incrementing ints for ids over uuids for web apis that may become immense. http://www.ietf.org/rfc/rfc4122.txt uuids are made just for this purpose, string based, never bigger than 40 characters (with dashes and curly braces). Most products use uuids or Microsoft's name for them guids.

we don't use incrementing ids, but needed ids that increase over time (so that you can sort tweets by them). hence, snowflake: http://github.com/twitter/snowflake

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#29

Post by ryansking »

simonw wrote: post #8

My rule of thumb is to use strings for all IDs that belong to external systems. A bunch of people got burned in the same way a few years ago when Flickr photo IDs rolled past 32 bits.

The twitter api includes string representations of the ids already and we encourage everyone to use those rather than the numeric ones.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#30

Post by ryansking »

icco wrote: post #15
BigZaphod wrote: post #2

When the first 32-bit troubles occurred, they should have just switched to a string-based ID, IMO.

I imagine that they did think of this, but a good string hash takes time. So switching to 64-bits takes much less time, and it would work until they finished snowflake.

we've been running snowflake for weeks already
Post reply on HN