Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

groups.google.com

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#31

Post by mayanklahiri »

drawkbox wrote: post #18
simonw wrote: post #8

My rule of thumb is to use strings for all IDs that belong to external systems. A bunch of people got burned in the same way a few years ago when Flickr photo IDs rolled past 32 bits.

Still surprised people use incrementing ints for ids over uuids for web apis that may become immense. http://www.ietf.org/rfc/rfc4122.txt uuids are made just for this purpose, string based, never bigger than 40 characters (with dashes and curly braces). Most products use uuids or Microsoft's name for them guids.

You don't even need to go that far. For non-critical applications (i.e., your web app), you can randomly generate a small string, say 12 bytes, using base-62 characters (A-Za-z0-9) to serve as a probably unique user ID (with VERY high probability).

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#32

Post by ryansking »

beagle3 wrote: post #12

That's truly, truly dumb on twitter's behalf. They should have started with using the whole numeric range and/or switched to string ids long ago. As I have suggested in a post 9 months ago, and as I would have designed this system at any date since 2002 or so, the twit ID would be composed of userid bits + time bits. In the post below I suggested 32+32, but other divisions are acceptable depending on your "bot user"…

This wouldn't work for us. We need the ids to sort by time across users, which this wouldn't do.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#33

Post by lurchpop »

simonw wrote: post #8

My rule of thumb is to use strings for all IDs that belong to external systems. A bunch of people got burned in the same way a few years ago when Flickr photo IDs rolled past 32 bits.

yeah. don't get why everyone didn't just convert to string after the last twitpocalypse

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#34

Post by schammy »

simonw wrote: post #8

My rule of thumb is to use strings for all IDs that belong to external systems. A bunch of people got burned in the same way a few years ago when Flickr photo IDs rolled past 32 bits.

Yeah, only problem is that is very inefficient from a storage perspective. It depends on the situation of course. If it's only a few thousand per day for example, not a big deal, but otherwise that could end up biting you in the ass hard after a while.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#35

Post by flyosity »

riffraff wrote: post #10
flyosity wrote: post #9

Earlier quoted context omitted.

Agreed. And also, there's not a lot of manipulation/math to be done on these IDs once they come back to the client. They're only used again when an operation happens on a tweet (favoriting it, replying to it) and in that case you're just taking the ID and sending it back to Twitter.

not really. You (can) use since_id and max_id in many places, which means you need them to be ordered (and trivially so). This is of course doable with strings too, once you decide what the ordering is.

Tweets come back ordered newest to oldest so there's nothing to order if you don't want to. And accessing since_id or max_id just to send it back to Twitter is exactly what I was referring to: no math to be done, just storing & accessing a value so it doesn't have to be in number format.

In a more practical application, a timeline comes back and the top-most and bottom-most IDs are stored. When a user gets to the bottom an API call is made to load more so you look at the bottom-most ID. If they want to load new tweets you look at the top-most ID. No math needed, just looking up values. They could've been strings all along and it wouldn't have mattered much.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#36

Post by tjpick »

jonhohle wrote: post #23
drawkbox wrote: post #18

Earlier quoted context omitted.

Still surprised people use incrementing ints for ids over uuids for web apis that may become immense. http://www.ietf.org/rfc/rfc4122.txt uuids are made just for this purpose, string based, never bigger than 40 characters (with dashes and curly braces). Most products use uuids or Microsoft's name for them guids.

I prefer my ID's universally unique. I worry that globally unique ID's won't scale as we begin to colonize other planets. (yes, its a joke.)

Stating it is a joke kind of ruined the joke.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#37

Post by nsfmc »

beagle3 wrote: post #21
morgo wrote: post #14

Earlier quoted context omitted.

-1. With hindsight it's easy to make a comment like yours, but many clients rely on the status_id to be monotonic. You would be changing behavior in a way that breaks apps. Twitter doesn't want to do that.

-1 all you like. It's engineering, not hindsight -- when building a system, I always ask myself "how does this scale" which usually translates to "on what attribute does this shard". Look at e.g. YouTube and many other sites around the same time. They knew what they were doing; Twitter didn't. I _have_ actually designed such a system in 1999, that used 48 bits, and it worked perfectly well. (Only had 28 bits for the…

I am wiser than any god or scientist, for I have squared the circle and cubed Earth's sphere, thus I have created 4 simultaneous separate 24 hour days within a 4-corner (as in a 4-corner classroom) rotation of Earth. See for yourself the absolute proof!

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#38

Post by prodigal_erik »

sedachv wrote: post #26

I'm going to take the opposite position of most commenters here and decry the fact that almost all programming languages get arithmetic wrong. Adding two positive numbers should never result in a negative number. Adding a one to an integer should result in the next largest integer. There are new languages created all the time that don't have built-in support for arbitrary precision integers or rational numbers. They…

Arithmetic is a relatively self-contained part of a language. The problem isn't that JavaScript's arithmetic is lame, it's that they didn't leave any way for the programmer to fix it—even if you make rational bignums out of arrays of doubles (or whatever), you can't make them drop-in replacements by overloading operators. It's a strange oversight for a language which lets instances override methods.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#39

Post by riffraff »

flyosity wrote: post #35
riffraff wrote: post #10

Earlier quoted context omitted.

not really. You (can) use since_id and max_id in many places, which means you need them to be ordered (and trivially so). This is of course doable with strings too, once you decide what the ordering is.

Tweets come back ordered newest to oldest so there's nothing to order if you don't want to. And accessing since_id or max_id just to send it back to Twitter is exactly what I was referring to: no math to be done, just storing & accessing a value so it doesn't have to be in number format. In a more practical application, a timeline comes back and the top-most and bottom-most IDs are stored. When a user gets to the bot…

I said that there are more operations than favouring and replying by id.

I don't understand how saying you're gonna access the timeline without looking at the values denies that.

Moreover, I did not object to using strings, I said you only want them to be orderable.

Re: Twitter IDs to roll past 53 bits in a couple days (may break javascript apps)

#40

Post by mbreese »

Retric wrote: post #20
jonknee wrote: post #13

Earlier quoted context omitted.

64 bits is enough to have everyone on Earth send over a billion Tweets and still have enough room to find a new solution. That sounds like more than enough to me.

The problem is they added a timestamp and people assumed they would not need the full 64 bit ID. IMO, it's not a bad idea on their part. A 32 bit UNIX timestamp * 2 ^ 32 + a 32 bit sequential id let's them track up to 4.2 billion tweets a second and should work just find up to the year 2106. Edit: As to why it's a good idea, you can have different systems handing out ID's without stepping on each other’s toes or even…

Yeah, I guess this is why I didn't understand how they could use 64 bit numbers to begin with... I couldn't see how they were going to be able to generate them all without leaving huge gaps in the number-space. If you used 64bit ints you'd be unable to have one machine do the generating, so you'd have to have some sort of offset for the worker who generated the ID at the very least.

And once you've done that, why not just go all out and use UUIDs?

Post reply on HN