Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Time Sync Service

aws.amazon.com

Post reply on HN
Print view View original

81–90 of 94 posts

Re: Amazon Time Sync Service

#81

Post by dekhn »

oblio wrote: post #70
dekhn wrote: post #27

Earlier quoted context omitted.

The tech lead of the Google MapReduce team (which no longer exists) just received their award for turning down mapreduce. IIRC it was officially done 5 years ago. However I believe the code to delete MR was never checked in and I'm not sure if there are still users. MapReduce was used at Google for highly inappropriate things. For example, the machine learning system I worked on, Sibyl https://www.datanami.com/2014/0…

> handle the constant failures endemic to Google's internal systems This sounds bad.

yeah it's crazy, I don't understand why Google's data center machines shit themselves so often. Probably cheap ram.

Re: Amazon Time Sync Service

#82

Post by jeffbee »

Psyonic wrote: post #45
Spivak wrote: post #36

Earlier quoted context omitted.

I mean it’s not that hard even on non-rt preemptive schedulers. You can’t avoid false negatives because you could be preempted after func completes but before the time is fetched but if you get a result it will be valid. def must_complete_before(func, deadline): result = func() lower, upper = time.now() if upper

Assuming func() doesn’t have side effects, sure. But if the point was to gate func() to only run before deadline this doesn’t really help.

Yeah but that doesn’t have anything to do with this API. If you want to use it that way, obviously the reasoning is defective.

This API is to determine if something that happened at a given instant is definitely in the past. That’s it!

Re: Amazon Time Sync Service

#83

Post by VirusNewbie »

drewda wrote: post #47
VirusNewbie wrote: post #23

Earlier quoted context omitted.

It's rare to see a comment on HN that misunderstands basic distributed systems concepts. MapReduce the paper has nothing to do with a database. You're likely conflating the fact that to achieve fault tolerant distributed computation, hadoop and hadoop like systems use a database like filesystem. However, no one is looking at Map Reduce type jobs as a replacement for a database and vice versa. That's like saying "wow…

> You're likely conflating the fact that to achieve fault tolerant distributed computation, hadoop and hadoop like systems use a database like filesystem. Yes, I specifically mentioned non-Google users adopting Hadoop, since it encompassed both a MapReduce implementation and supporting infrastructure. Once on the bandwagon inspired by the MapReduce paper, many orgs didn't just use MapReduce itself for parallelized ba…

The reason spanner is a big deal is because it provides relational semantics including cross table transactions. That is why spanner is interesting.

No one, not even the most ambitious and clueless market folks at Cloudera, has ever sold Hadoop and MR type technologies as a replacement for RDBMs. Spanner can be used as a replacement. These two technologies don't solve the same problem.

You still need MR to do analytics at scale on spanner!

Re: Amazon Time Sync Service

#84

Post by dekhn »

jpgvm wrote: post #42
sokoloff wrote: post #31

Earlier quoted context omitted.

It seems to me “there is no other technical system in the company capable to perform the task” is a valid technical justification.

i.e this is a crappy tool but it's best tool we have come up with for this problem thus far. There are lots of such tools which are used begrudgingly by people that have an intuition for the fact it can be done better but not the concrete idea and/or time to implement it.

it wasn't a crappy tool (mapreduce was amazing) but it definitely was an impedence mismatch for this particular job. Later, we tried to get SIbyl to move to the underlying compute engine that Flume was built on top of it but it turned out to be more profitable to just let it die slowly.

Re: Amazon Time Sync Service

#85

Post by froh »

NavinF wrote: post #77
froh wrote: post #71

Earlier quoted context omitted.

That's when you move from NTP to PTP, which in a LAN provides sub microsecond accuracy. https://en.m.wikipedia.org/wiki/Precision_Time_Protocol With that underneath you need just one gps clock reciever for the whole LAN, and PTP capable Ethernet adapter and switches throughout, of course. Which most are, these days.

Do you have experience with this? Playing around with PTP has been on my TODO list for quite a while, but it seems to be quite a hassle. At home, I already have a GPS module with its PPS pin connected to an Pi running an NTP server. In my data center, I use clock.sjc.he.net and that's about as good as NTP gets. Certainly good enough for one-way latency measurements. Are there any SBCs with hardware timestamping and a…

It _is_ a hassle, and you should have a need for that level of synchronicity. We did it in our team for an in-vehicle network, with the vehicle GPS reciever as the network PTP time master and we figured how to access the ethernet PTP register both from userspace and from the kernel. Kernel for the system clock, user space to avoid latencies from kernel Userland interaction in some high precision real time task. It was a major effort and it required low level hackery way above my personal head.

Re: Amazon Time Sync Service

#86

Post by foota »

bigsparky wrote: post #78
foota wrote: post #66

Earlier quoted context omitted.

The combination of a time and guaranteed error bound can be used as a primitive in distributed systems. You can get timestamps from requests coming from different systems and relate them in a meaningful way. At least, in a false negative fashion. E.g., if A occurs at T + 100ms with 1ms uncertainty and B occurs at T + 105ms with 2ms uncertainty you can conclude that A happened before B. This does not allow you to alwa…

You must know some pretty smart five year olds!

That doesn't say L5? /s

I missed that part :-)

Re: Amazon Time Sync Service

#87

Post by gnu8 »

foota wrote: post #66
strimp099 wrote: post #34

Can someone please explain to me what this is all about like I’m five years old?

The combination of a time and guaranteed error bound can be used as a primitive in distributed systems. You can get timestamps from requests coming from different systems and relate them in a meaningful way. At least, in a false negative fashion. E.g., if A occurs at T + 100ms with 1ms uncertainty and B occurs at T + 105ms with 2ms uncertainty you can conclude that A happened before B. This does not allow you to alwa…

None of this has to do with Amazon though. I am very uncomfortable with Amazon (or the others) pretending that they invented precision timekeeping and that it is a special feature only they have because it seems like a lot of people believe it and build systems that rely on an API that is proprietary to a cloud vendor rather than just installing ntp.

Re: Amazon Time Sync Service

#88

Post by loxias »

staticassertion wrote: post #64
travisgriggs wrote: post #53

Earlier quoted context omitted.

When Microsoft did that kind of thing we called it “embrace and extend.”

That sounds different. Embrace and extend means implementing a protocol and then adding specific non-spec features. For example, postgres 'embraces' SQL and 'extends' it with specific features - in the future postgres could 'extinguish' SQL by breaking from the spec, forcing an ecosystem split.

Thank you. I was in the process of writing this exact comment but kept tripping over my own words and eventually gave up.

Re: Amazon Time Sync Service

#89

Post by loxias »

bogomipz wrote: post #48
Groxx wrote: post #37

Earlier quoted context omitted.

Time is hard. By asking both super-microscopic stuff and stuff way out in space, you can now find out what time it is!

One is cesium? Whats the other stuff, way out in space?

You think that's magic, check out optical lattice clocks. Blows my mind every time I re-read about them and accept "yes, this actually works".

Re: Amazon Time Sync Service

#90

Post by tatersolid »

bogomipz wrote: post #48
Groxx wrote: post #37

Earlier quoted context omitted.

Time is hard. By asking both super-microscopic stuff and stuff way out in space, you can now find out what time it is!

One is cesium? Whats the other stuff, way out in space?

Some Pulsars have extremely stable frequencies https://www.nytimes.com/1983/02/09/us/pulsar-termed-most-acc...
Post reply on HN