Why Not to Build a Time-Series Database
outlyer.com
Why Not to Build a Time-Series Database
1–10 of 128 posts
Re: Why Not to Build a Time-Series Database
#2Nice article.
>its not uncommon for some of our customers to send us millions of metrics every minute
What kind of customers/services generate millions of points a minute?
Re: Why Not to Build a Time-Series Database
#3Nice article. >its not uncommon for some of our customers to send us millions of metrics every minute What kind of customers/services generate millions of points a minute?
Customers using a large-scale analytics database...
Re: Why Not to Build a Time-Series Database
#4Nice article. >its not uncommon for some of our customers to send us millions of metrics every minute What kind of customers/services generate millions of points a minute?
Some companies record as much as they can, e.g. "here's all of the user's mouse-movements, their full User Agent, etc." or "here's everywhere the user tapped on the app and which of the notifications we sent them that they responded to, and more!"
The example from the article was "one team at one of our customers decided to dump 30 million metrics on us, send all of their mobile product metrics into Outlyer"
Re: Why Not to Build a Time-Series Database
#5Nice article. >its not uncommon for some of our customers to send us millions of metrics every minute What kind of customers/services generate millions of points a minute?
1 mil is only 16K per second.
Re: Why Not to Build a Time-Series Database
#6Nice article. >its not uncommon for some of our customers to send us millions of metrics every minute What kind of customers/services generate millions of points a minute?
Transportation companies--truck fleet data, airplanes, etc.
Re: Why Not to Build a Time-Series Database
#7Nice article. >its not uncommon for some of our customers to send us millions of metrics every minute What kind of customers/services generate millions of points a minute?
At one employer most customers would opt-in to upload their system logs, and company would analyze to anticipate problems for preventive maintenance.
The result was a tens of terabytes a day Niagara of data.
When I left they were in early stages of Hadoop because ordinary parse/analyze was starting to fall behind.
Re: Why Not to Build a Time-Series Database
#8Nice article. >its not uncommon for some of our customers to send us millions of metrics every minute What kind of customers/services generate millions of points a minute?
I use to work at a Fortune 50 retailer on the cloud platform (a lot of tooling around CI/CD for the teams that manage the website). We had a large problem with keeping the metrics pipeline current. A major issue is that be default, Spring Boot publishes about 500 different metrics on a 10 second slice. Allowing every application to pump out that many default metrics, most of which are never used, means that it takes only 334 instances to get one million metrics per minute (1,000,000 / 6 / 500 = 333 1/3). I would guess we had a couple thousand instances in production on a normal day. In a couple weeks, they have Black Friday. Any team that hasn't been able to fix performance problems are given the go ahead to just throw money at it and scale horizontally in obscene ways.
Of course, our metrics were all handled in house. From talking to the teams that handled the metrics pipeline, the vendors were great for smaller companies, but there was no off the shelf solution for a companies that large with that volume. But I did very little with that myself, other than look into the fact that Spring Boot published way too many default metrics. Who needs P50, P70, P75, P80, P85, P90 - P99 on all web requests?! Just set a default that is small and worthwhile and let the developers adjust as needed.
Re: Why Not to Build a Time-Series Database
#9Nice article. >its not uncommon for some of our customers to send us millions of metrics every minute What kind of customers/services generate millions of points a minute?
I used to work at Meraki. There, we would, every 5 minutes or so, record byte and packet counts classified by traffic class and remote end of the connection for every single client connected to an SMB internal or customer-facing network.
(Meraki actually did implement its own time-series database, and after I left published a paper describing its design and implementation. https://meraki.cisco.com/lib/pdf/trust/lt-paper.pdf. Good quote on the motivation: "As discussed in Section 2.3.3, customers have a nearly insatiable demand for high-resolution historical data, even though they mostly query data from the recent past."
Re: Why Not to Build a Time-Series Database
#10It's pretty ridiculous that "Time-Series Database" has come to mean ingesting massive amounts of streaming data. They've been around a long time and have many use cases.
They're a great way to store data efficiently, accessing specific data if you know the time range you are looking for is very fast and simple, and you can roll your own in a few dozen lines of C if that's what you want to do. If that's all you need, why not?