Introducing Atlas: Netflix's Primary Telemetry Platform • Hacker News Board

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#41

Post by CrankyFool » Sat, Dec 13, 2014, 6:39 AM UTC

copperlight wrote: post #40

lifeisstillgood wrote: post #31

Earlier quoted context omitted.

What kind of ratio of metadata traffic (telemetry) to total traffic did you see? How does this divide between "system level" and "application level"? My client is lookin at these telemetry problems now, is there possibility of commercial high-level consultancy coming out of Netflix / colleagues ? Ping me on details in my profile if you can help?

Telemetry traffic is a small fraction of the the total traffic running through a region, partially due to the use of the Smile data format (binary JSON) for delivering metrics from the client to the Atlas backend. When you give developers tools for creating and aggregating highly dimensional metrics, they tend to create lots of metrics so that they can answer interesting business questions about the use of their appl…

And this, of course, doesn't account for cases where a minor developer error results in code that, say, creates a new metric for every source IP address from which we see a request. Dynamic metric names FTW.

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#42

Post by CrankyFool » Sat, Dec 13, 2014, 6:42 AM UTC

lifeisstillgood wrote: post #32

misframer wrote: post #30

Earlier quoted context omitted.

1,000 metrics per server is quite reasonable. I work for a performance management company and we handle thousands of time series metrics per monitored server at one-second resolution.

What's the rough ratio for system level, process level and app level (ie total MB, MB / process and "a customer just signed up")? How much traffic does that add up to? It seems a lot.

above, copperlight quotes about 3%-5% of metrics being system-level. A pretty small number would be process-level, I'm guessing, with the vast majority being app-level.

At Netflix-sized, the answer to pretty much any question is "a lot." :)

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#43

Post by lifeisstillgood » Sat, Dec 13, 2014, 11:56 AM UTC

CrankyFool wrote: post #41

copperlight wrote: post #40

Earlier quoted context omitted.

Telemetry traffic is a small fraction of the the total traffic running through a region, partially due to the use of the Smile data format (binary JSON) for delivering metrics from the client to the Atlas backend. When you give developers tools for creating and aggregating highly dimensional metrics, they tend to create lots of metrics so that they can answer interesting business questions about the use of their appl…

And this, of course, doesn't account for cases where a minor developer error results in code that, say, creates a new metric for every source IP address from which we see a request. Dynamic metric names FTW.

Thank you both for your insights

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#44

Post by Rapzid » Sat, Dec 13, 2014, 1:49 PM UTC

Very interesting.. I'm guessing you can do less than a minute on granularity? That's a lifetime to me.

I'm working on my own little system based on Riak for storage and their new search functionality to index the streams(what I call a unique metric). I have a goal of graphite API compatibility though so it can be a drop-in replacement for me. Worked out a schema for mapping graphite metric names to the stream dimensions, wrote a graphite function parser/lexar, etc. Atlas should definitely be worth a look for some fresh ideas :)

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#45

Post by CrankyFool » Sat, Dec 13, 2014, 3:02 PM UTC

Rapzid wrote: post #44

Very interesting.. I'm guessing you can do less than a minute on granularity? That's a lifetime to me. I'm working on my own little system based on Riak for storage and their new search functionality to index the streams(what I call a unique metric). I have a goal of graphite API compatibility though so it can be a drop-in replacement for me. Worked out a schema for mapping graphite metric names to the stream dimensi…

We CAN do (We haven't opensourced much of the pipeline ... yet)