Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Time Sync Service

aws.amazon.com

Post reply on HN
Print view View original

31–40 of 94 posts

Re: Amazon Time Sync Service

#31

Post by sokoloff »

dekhn wrote: post #27
skj wrote: post #11

Earlier quoted context omitted.

Google uses map reduce extensively... where it's appropriate. True time helps with things like spanner transactions. It's just a totally different use case.

The tech lead of the Google MapReduce team (which no longer exists) just received their award for turning down mapreduce. IIRC it was officially done 5 years ago. However I believe the code to delete MR was never checked in and I'm not sure if there are still users. MapReduce was used at Google for highly inappropriate things. For example, the machine learning system I worked on, Sibyl https://www.datanami.com/2014/0…

It seems to me “there is no other technical system in the company capable to perform the task” is a valid technical justification.

Re: Amazon Time Sync Service

#32

Post by Groxx »

londons_explore wrote: post #6

It's really hard to use these API's correctly. Remember... your CPU can halt at any time for any number of milliseconds. That means simple things like: upperBound, lowerBound = readTime() if (upperBound Are incorrect... There is no guarantee that the 'if' statement didn't take many milliseconds, and that the stuff didn't end up happening after the deadline. It's also very easy to write code that works, but is theoret…

Yeah, I was thinking about this too. Without a real-time OS on separate hardware, and/or some kind of guaranteed "will not context-switch out of your process for more than X time" + strict coding habits, this seems like at best a practical (>?)99% improvement and at worst snake-oil. Certainly not reliable in any case.

edit: well, after reading Spivak's comment[1] a bit, I guess it does provide strict upper bounds. May be useful to reduce how often you need to use fallback behaviors / get more byzantine. Though I'm not yet sure how to turn that into something useful. No doubt there are some though.

[1] https://news.ycombinator.com/item?id=29103093

Re: Amazon Time Sync Service

#33

Post by rp1 »

mrep wrote: post #16
drewda wrote: post #13

Earlier quoted context omitted.

Yes, which is why it's amusing in hindsight that for a decade everyone* outside Google was forcing all* their distributed data tasks into the MapReduce paradigm, without considering alternative approaches like the one used by Spanner. * slight exaggerations, I know

I'm not sure how you think a distributed data processing technology would "fake-out" other companies when building/choosing database technology. They are totally different problem sets. MapReduce does not have a set in stone data source/sink and can use multiple things like bigtable and spanner so they are complementary technologies.

I think the parent commenter might be referring to systems like Hive or HBase built on top of Hadoop and do have a lot of overlap with a large scale database system.

Re: Amazon Time Sync Service

#35

Post by dheera »

londons_explore wrote: post #6

It's really hard to use these API's correctly. Remember... your CPU can halt at any time for any number of milliseconds. That means simple things like: upperBound, lowerBound = readTime() if (upperBound Are incorrect... There is no guarantee that the 'if' statement didn't take many milliseconds, and that the stuff didn't end up happening after the deadline. It's also very easy to write code that works, but is theoret…

How about just

    syncTime()
And they take care of the mess

Re: Amazon Time Sync Service

#36

Post by Spivak »

londons_explore wrote: post #6

It's really hard to use these API's correctly. Remember... your CPU can halt at any time for any number of milliseconds. That means simple things like: upperBound, lowerBound = readTime() if (upperBound Are incorrect... There is no guarantee that the 'if' statement didn't take many milliseconds, and that the stuff didn't end up happening after the deadline. It's also very easy to write code that works, but is theoret…

I mean it’s not that hard even on non-rt preemptive schedulers. You can’t avoid false negatives because you could be preempted after func completes but before the time is fetched but if you get a result it will be valid.

    def must_complete_before(func, deadline):
        result = func()
        lower, upper = time.now()
        if upper

Re: Amazon Time Sync Service

#38

Post by [deleted] »

Spivak wrote: post #36
londons_explore wrote: post #6

It's really hard to use these API's correctly. Remember... your CPU can halt at any time for any number of milliseconds. That means simple things like: upperBound, lowerBound = readTime() if (upperBound Are incorrect... There is no guarantee that the 'if' statement didn't take many milliseconds, and that the stuff didn't end up happening after the deadline. It's also very easy to write code that works, but is theoret…

I mean it’s not that hard even on non-rt preemptive schedulers. You can’t avoid false negatives because you could be preempted after func completes but before the time is fetched but if you get a result it will be valid. def must_complete_before(func, deadline): result = func() lower, upper = time.now() if upper

[deleted]

Re: Amazon Time Sync Service

#39

Post by loxias »

Two observations/questions, the first probably naive:

* It's not that hard to get your own "world class" time server, for under a thousand. A Rb standard slaved to a GPSDO is gonna be so accurate and stable, and use that to drive a SBC that supports IEEE1588, where you run your NTP and PTP server. Oh, but I guess that box, while inexpensive, isn't in Amazons DC, so doesn't help you.

* PTP's absence in the Amazon Time Sync Service article is quite conspicuous!

Re: Amazon Time Sync Service

#40

Post by [deleted] »

drewda wrote: post #9

Perhaps the biggest "fake-out" in 21st century computing: Google publicly released its MapReduce paper -- directing most of the rest of the industry toward loosely coupled, overly complex distributed data processing systems like Hadoop for the following decade -- but internally they just bought a bunch of atomic clocks and built a distributed RDBMS. I know this is a somewhat simplified story, but it does make me chuc…

[deleted]
Post reply on HN