Introducing Atlas: Netflix's Primary Telemetry Platform • Hacker News Board

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#31

Post by lifeisstillgood » Sat, Dec 13, 2014, 1:11 AM UTC

copperlight wrote: post #27

crescentfresh wrote: post #18

Earlier quoted context omitted.

What are the sources for data for Atlas vs for Suro at Netflix? (Suro: http://techblog.netflix.com/2013/12/announcing-suro-backbone... ). Suro was/is used to collect "more than 1.5 million events per second during peak hours, or around 80 billion events per day" from ec2 instances.

There are several different data sources for Atlas: * There is a poller cluster that gathers SNMP and HTTP healthcheck metrics and forwards them to the Atlas backend. * There are on-instance log parsers written in Perl and Python that count events in Apache HTTPd and Tomcat logs and send data to the Atlas backend. * The Servo library [0] is used to instrument Java code with counters, timers and gauges. There is a sep…

What kind of ratio of metadata traffic (telemetry) to total traffic did you see? How does this divide between "system level" and "application level"?

My client is lookin at these telemetry problems now, is there possibility of commercial high-level consultancy coming out of Netflix / colleagues ? Ping me on details in my profile if you can help?

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#32

Post by lifeisstillgood » Sat, Dec 13, 2014, 1:13 AM UTC

misframer wrote: post #30

lifeisstillgood wrote: post #29

20 million different time series. I mean that is a lot. If you have say, 20,000 servers running that is still 1,000 different time series per server. Memory, CPU, logins, logouts, customer selections, I mean I struggle to get to those numbers.

1,000 metrics per server is quite reasonable. I work for a performance management company and we handle thousands of time series metrics per monitored server at one-second resolution.

What's the rough ratio for system level, process level and app level (ie total MB, MB / process and "a customer just signed up")?

How much traffic does that add up to? It seems a lot.

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#33

Post by CrankyFool » Sat, Dec 13, 2014, 1:15 AM UTC

misframer wrote: post #28

I see "anomaly detection" is listed under real-time analytics. Do they use Holt-Winters for that or something else?

"Anomaly detection" is one of those vague terms that can mean anything from "it's gone above the pre-set limit, and that's anomalous" to "the system has studied the signal to learn what the accepted limits should be, and it's exceeded these limits." We mostly mean the latter for anomaly detection.

The Insight Engineering team at Netflix is largely composed of four kinds of engineers: Platform/back-end engineers, UI engineers, Site Reliability Engineers, and Real-Time Analytics (RTA) engineers; it's the latter group of engineers who are looking into ways to quickly (and efficienlty) detect anomalies in a truly-absurd amount of data.

The RTA group is now about 6 months old or thereabouts; I have high hopes that we'll see some public presentations from them soon that will be helpful to other people outside Netflix.

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#34

Post by pasbesoin » Sat, Dec 13, 2014, 1:49 AM UTC

pasbesoin wrote: post #17

brendangregg wrote: post #8

I work at Netflix and use Atlas every day. It's our go-to performance monitoring tool, and has solved countless performance and reliability issues. It's exciting to have it open source! I summarized it in a talk recently at Surge 2014, where I showed its role for a performance investigation, and how it is central to everything: http://youtu.be/H-E0MQTID0g?t=22m http://www.slideshare.net/brendangregg/netflix-from-clou…

OT, but I'm using your app on a stock Nexus 7 (at 4.4.4), and I find a couple of controls very difficult to trigger, while others respond just fine at the first touch. The "back up 30 seconds" button can be very fussy and difficult to trigger. Sometimes it takes many touches before it will trigger. Also, when in a series the end titles of an episode display, it can be quite difficult to get a touch to register so tha…

I'll take the downvotes if that's necessary.

Netflix had months-long problems with the audio on some of their programs, including programs that others here mentioned watching.

It was only after I made some comments regarding these audio problems, here and on Reddit, that they were fixed.

(I'd reported them via Netflix's problem reporting mechanisms, months earlier when I discovered them, to no effect.)

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#35

Post by bones6 » Sat, Dec 13, 2014, 1:56 AM UTC

brendangregg wrote: post #8

I work at Netflix and use Atlas every day. It's our go-to performance monitoring tool, and has solved countless performance and reliability issues. It's exciting to have it open source! I summarized it in a talk recently at Surge 2014, where I showed its role for a performance investigation, and how it is central to everything: http://youtu.be/H-E0MQTID0g?t=22m http://www.slideshare.net/brendangregg/netflix-from-clou…

I really enjoyed your overall Linux Performance Tool talk: http://youtu.be/SN7Z0eCn0VY

Thanks for sharing your expertise with performance monitoring. Releasing Atlas is adding tons of value to what I've already seen.

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#36

Post by malingo » Sat, Dec 13, 2014, 2:55 AM UTC

malingo wrote: post #15

I really enjoy learning about real-world actor-model systems like this. Having access to the source code is even better. Are there other similar examples?

Can't edit my earlier post so I'll reply.

I found this page on other companies & projects using Akka: http://doc.akka.io/docs/akka/2.0.1/additional/companies-usin...

I just learned about Akka recently (by way of Akka.net), and I'm encouraged by these frameworks that look like legitimate alternatives to Erlang. Erlang has always seemed like an extreme measure for implementing a distributed concurrent system, so it's good to know there are Java/Scala/C#/F# options.

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#37

Post by dkhenry » Sat, Dec 13, 2014, 3:30 AM UTC

Thaxll wrote: post #21

dkhenry wrote: post #9

Sounds like a great platform. I wonder how many instances of everything they need for peak time. I have a system that is rated for 5M Time series data points per second and it takes about 150 physical servers so I am curious about what a Netflix sized ( 20M time series data points per second ) would look like.

In the video they say that they're close to 1B/min. http://youtu.be/tHrT6kQR7vw?t=36m

They list a peak of 1.2 billion / minute which is 20M/ second

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#38

Post by CrankyFool » Sat, Dec 13, 2014, 3:56 AM UTC

lifeisstillgood wrote: post #29

20 million different time series. I mean that is a lot. If you have say, 20,000 servers running that is still 1,000 different time series per server. Memory, CPU, logins, logouts, customer selections, I mean I struggle to get to those numbers.

It's actually more like close to 1.2 billion different time series -- we report most metrics on one minute granularity, but they're not all reporting at the same second (thank God), so on average we're getting up to 20M time series per minute.

But this of course just makes the question more reasonable -- 1.2B different time series? Really?

Yup. We get a bunch of system telemetry, and a bunch of default application telemetry, without even getting traffic hitting the box, but that's a relatively small percentage of the overall volume. Developers LOVE metrics.

So imagine you want to measure requests to our API, and these are some tags you want to keep track of: request type: 5 different types result: 2 possible values (success, failure) originating country: 50 countries originating device type: 200 devices

And let's say you've got a 1000 instances reporting this data.

Suddenly you've got 5 * 2 * 50 * 200 * 1000

Oh look. Here's 100M different metrics.

And that's a relatively trivial example.

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#39

Post by copperlight » Sat, Dec 13, 2014, 4:34 AM UTC

paulasmuth wrote: post #26

Sounds similar to FnordMetric ( http://fnordmetric.io/chartsql ) which also supports dimensional timeseries data. Major differences between Atlas and FnordMetric on first sight: - SQL based query and charting frontend (ChartSQL), so you don't have to learn yet another DSL - ships with a a wire compatible StatsD API - supports pluggable backends - renders charts to SVG - will probably not scale to petabytes of data ou…

Interesting - I had not heard of Fnordmetrics before. There are not many monitoring systems that implement dimensionality for metrics tagging. It looks like Fnordmetrics goes about it slightly differently, but it seems to achieve the same end goal of arbitrary grouping by like characteristics.

In the Atlas eco-system, Servo and Atlas client eliminate the need for StatsD. The combination of these products allow for code-level instrumentation and delivery of metrics. The Prana sidecar is available for non-Java applications to deliver metrics to the Atlas backend as formatted JSON objects.

Atlas supports multiple backend storage systems, although this is not easily pluggable just yet. Earlier iterations of Atlas had support for MongoDB and Cassandra as storage backends, but there were issues obtaining enough IOPS to satisfy the read and write performance requirements at scale, so storage was switched to in-memory.

Atlas can return data in JSON format (?format=json) which is suitable for JS or SVG based rendering systems such as Highcharts. There is also a streaming API that trades response time for increased data payloads.

It takes some time to learn the Atlas Stack Language, but the fact that it is URL-based means that the browser is the interactive query editor. Using tools like Chrome's Edit URL to help handle long strings, you can make small changes to queries iteratively and see the results in less than 2 seconds in many cases. Average PNG render time is typically less than 10 seconds; slow rendering can take around 30 seconds.

Re: Introducing Atlas: Netflix's Primary Telemetry Platform

#40

Post by copperlight » Sat, Dec 13, 2014, 5:23 AM UTC

lifeisstillgood wrote: post #31

copperlight wrote: post #27

Earlier quoted context omitted.

There are several different data sources for Atlas: * There is a poller cluster that gathers SNMP and HTTP healthcheck metrics and forwards them to the Atlas backend. * There are on-instance log parsers written in Perl and Python that count events in Apache HTTPd and Tomcat logs and send data to the Atlas backend. * The Servo library [0] is used to instrument Java code with counters, timers and gauges. There is a sep…

What kind of ratio of metadata traffic (telemetry) to total traffic did you see? How does this divide between "system level" and "application level"? My client is lookin at these telemetry problems now, is there possibility of commercial high-level consultancy coming out of Netflix / colleagues ? Ping me on details in my profile if you can help?

Telemetry traffic is a small fraction of the the total traffic running through a region, partially due to the use of the Smile data format (binary JSON) for delivering metrics from the client to the Atlas backend.

When you give developers tools for creating and aggregating highly dimensional metrics, they tend to create lots of metrics so that they can answer interesting business questions about the use of their applications. We have some developers who have written code that produces up to 150,000 metrics per instance and the vast majority of these metrics are application-level. We typically see 3-5% of the metrics delivered from an instance are system-level performance metrics.