21–30 of 45 posts
Sounds like a great platform. I wonder how many instances of everything they need for peak time. I have a system that is rated for 5M Time series data points per second and it takes about 150 physical servers so I am curious about what a Netflix sized ( 20M time series data points per second ) would look like.
Wait, didn't Hashicorp release Atlas a couple of days ago?
Sounds like a great platform. I wonder how many instances of everything they need for peak time. I have a system that is rated for 5M Time series data points per second and it takes about 150 physical servers so I am curious about what a Netflix sized ( 20M time series data points per second ) would look like.
As the announcement notes, we have multiple tiers holding different data horizons. The most active, and large, tier is the one holding the last six hours of data. That tier, being the most critical one (we try to train our engineers to only need 6 hours of data to understand how their system is working in the worst case), is mirrored. Right now, each of those mirrors is about[0] 756 r3.2xl instances.
[0] For a very exact definition of "about," though that exact number could change in the next 5 minutes, or 5 hours, or 5 days, or not until we see another metrics increase.
Sounds like a great platform. I wonder how many instances of everything they need for peak time. I have a system that is rated for 5M Time series data points per second and it takes about 150 physical servers so I am curious about what a Netflix sized ( 20M time series data points per second ) would look like.
In the video they say that they're close to 1B/min. http://youtu.be/tHrT6kQR7vw?t=36m
(The presenter in the video)
- SQL based query and charting frontend (ChartSQL), so you don't have to learn yet another DSL
- ships with a a wire compatible StatsD API
- supports pluggable backends
- renders charts to SVG
- will probably not scale to petabytes of data out-of-the box
- single c++ binary, deploy it in 5 minutes using docker
- includes an interactive query editor
I work at Netflix and use Atlas every day. It's our go-to performance monitoring tool, and has solved countless performance and reliability issues. It's exciting to have it open source! I summarized it in a talk recently at Surge 2014, where I showed its role for a performance investigation, and how it is central to everything: http://youtu.be/H-E0MQTID0g?t=22m http://www.slideshare.net/brendangregg/netflix-from-clou…
What are the sources for data for Atlas vs for Suro at Netflix? (Suro: http://techblog.netflix.com/2013/12/announcing-suro-backbone... ). Suro was/is used to collect "more than 1.5 million events per second during peak hours, or around 80 billion events per day" from ec2 instances.
* There is a poller cluster that gathers SNMP and HTTP healthcheck metrics and forwards them to the Atlas backend.
* There are on-instance log parsers written in Perl and Python that count events in Apache HTTPd and Tomcat logs and send data to the Atlas backend.
* The Servo library [0] is used to instrument Java code with counters, timers and gauges. There is a separate client implementation that handles forwarding metrics to the Atlas backend. The client also polls and reports JMX metrics from the JVM that it runs inside. Spectator [1] is a new library that provides cleaner abstractions of Servo concepts.
* The Prana sidecar [2] was extended to provide REST endpoints for Servo and the client, so that metrics can be delivered from non-Java code.
[0] https://github.com/Netflix/servo
If you have say, 20,000 servers running that is still 1,000 different time series per server. Memory, CPU, logins, logouts, customer selections, I mean I struggle to get to those numbers.
20 million different time series. I mean that is a lot. If you have say, 20,000 servers running that is still 1,000 different time series per server. Memory, CPU, logins, logouts, customer selections, I mean I struggle to get to those numbers.