Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

M3DB, a distributed timeseries database

m3db.io

Post reply on HN
Print view View original

111–120 of 138 posts

Re: M3DB, a distributed timeseries database

#111

Post by iamleppert »

jjeaff wrote: post #96
carlisle_ wrote: post #39

Earlier quoted context omitted.

>It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case. Former Uber engineer here. I can assure you that while our engineering team was massive, there was anything but too little work. If anything most engineers were massively overtaxed.…

There seems to be a pervasive misconception by the very employees at Uber, that they built their own chat platform. When in reality, and someone please correct me if I am wrong, uChat was a white labeled Mattermost. I have heard that the team that put it together actually tried to hide that fact from the company (for the glory, I guess). But that could be apocryphal.

Why not just use IRC?? It scaled for the entire internet.

Re: M3DB, a distributed timeseries database

#112

Post by closeparen »

rixed wrote: post #110
bostik wrote: post #48

Earlier quoted context omitted.

TSDBs are a special case of databases. And oh boy, time-series is hard . Your regular RDBMS is going to be either write-heavy or read-heavy. You can pretty easily[ß] optimise the database for one of these utilisation patterns. But a TSDB basically combines the worst of both worlds: telemetry at any scale is important, and monitoring reliability in an always-online system is not optional. TSDBs are written to very fre…

> But because they are also used for system-wide monitoring, they are read from all the time. And this is the billion dollars mistake of the current devop culture. I believe we are doing monitoring wrong. Real time monitoring need no persistent storage. Troubleshooting does need persistent storage, but not monitoring, and unless your infrastructure is broken all the time then querying past data must occur only rarely…

When you have very high cardinality series, it's decently common to slot in a "rollup" or "reducer" query between collection and persistence.

This is similar to the insight you mentioned: I don't actually need to store a separate series per hostname in perpetuity just to know when the worst one is out of control. I can ask it to persist the top five individually, and then an average.

But when responding to a PagerDuty alert, the first thing I want to see is a plot of the alerting series. The next thing I want to see is a plot of every series about that service (bonus points if you can find the ones that have discontinuities with similar timing). So you're really not getting out of the "fast serving" business, just reducing the load on it.

Re: M3DB, a distributed timeseries database

#113

Post by winrid »

synack wrote: post #57

I setup a lot of Uber's early metrics infrastructure, so I can speak to how they got to the place where building a custom solution was the right answer. In the beginning, we didn't really have metrics, we had logs. Lots of logs. We tried to use Splunk to get some insight from those. It kinda worked and their sales team initially quoted a high-but-reasonable price for licensing. When we were ready to move forward, the…

Awesome story, thanks.

Re: M3DB, a distributed timeseries database

#114

Post by StreamBright »

richieartoul wrote: post #4
missosoup wrote: post #2

Uber has started many projects that ended up getting open sourced. And many of them are now either abandoned or on life support. H3 comes to mind as something we almost ended up using but luckily avoided. These open-sourcings seem a bit like PR pieces with no guarantees of any support or evolution after being published.

Chronosphere, a startup founded by two of the early M3 engineers, just raised 11 million dollars to build a monitoring platform based around M3DB: https://techcrunch.com/2019/11/05/chronosphere-launches-with... Uber also uses M3DB extensively internally and the project is nowhere near being abandoned or on life support: https://github.com/m3db/m3/commits/master

>> While the founders, CEO Martin Mao and CTO Rob Skillington, were working at Uber, they recognized a gap in the monitoring industry, particularly around cloud-native technologies like containers and microservices

What is the actual gap that is not addressed with one or all of the following?

- Prometheus / Grafana [1]

- Datadog [2]

- Cloudwatch [3]

1. https://docs.docker.com/config/thirdparty/prometheus/

2. https://www.datadoghq.com/blog/introducing-live-container-mo...

3. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Re: M3DB, a distributed timeseries database

#115

Post by rixed »

closeparen wrote: post #112
rixed wrote: post #110

Earlier quoted context omitted.

> But because they are also used for system-wide monitoring, they are read from all the time. And this is the billion dollars mistake of the current devop culture. I believe we are doing monitoring wrong. Real time monitoring need no persistent storage. Troubleshooting does need persistent storage, but not monitoring, and unless your infrastructure is broken all the time then querying past data must occur only rarely…

When you have very high cardinality series, it's decently common to slot in a "rollup" or "reducer" query between collection and persistence. This is similar to the insight you mentioned: I don't actually need to store a separate series per hostname in perpetuity just to know when the worst one is out of control. I can ask it to persist the top five individually, and then an average. But when responding to a PagerDut…

> But when responding to a PagerDuty alert, the first thing I want to see is a plot of the alerting series.

When you do receive a page, you can then hit a database. This happens rarely enough (hopefully!) so that your database can be optimised for writes only, not for writes and reads, as $parent suggested.

I would even argue that this is a similar use case than dashboarding: when paged you want to see the last 3 days or so of accurate data + some longer term averages/trends/baseline. All of this could fit in RAM. At least in the current prototype I mentioned above the ambition is to be able to serve recent data for such dashboards out of RAM without bothering with a DB, and have the DB completely out of the way for the monitoring use case (while still having a persistent store for long term capacity planning / business analyses).

Assuming doubles are compressed and a collection rate of 1/10s, 3 days of data is <100KiB per timeseries, ie 0.5GiB for 10k source timeseries.

Re: M3DB, a distributed timeseries database

#116

Post by 4d617832 »

synack wrote: post #57

I setup a lot of Uber's early metrics infrastructure, so I can speak to how they got to the place where building a custom solution was the right answer. In the beginning, we didn't really have metrics, we had logs. Lots of logs. We tried to use Splunk to get some insight from those. It kinda worked and their sales team initially quoted a high-but-reasonable price for licensing. When we were ready to move forward, the…

Really interesting read for me. I am currently not so far from your SSD point but our setup still works fine most of the time. It’s just 100k/m though. I am trying to use more of the Go implementations of the graphite stack which did improve load. I will consider m3db probably to get some benchmarks. All the other ones would require some more people as you said.

Re: M3DB, a distributed timeseries database

#117

Post by sagichmal »

cube2222 wrote: post #74
hagen1778 wrote: post #55

Earlier quoted context omitted.

Since you've done your research, would you mind to post a short list of alternatives and reasons why have they been rejected? Thanks!

This was as of november. Raw Prometheus: Isn't able to hold my data. Thanos: I liked the project, it's architecture and ease of deployment, but after spending a non-trivial amount of time with it I wasn't able to setup any long-term caching. Thanos uses the prometheus storage format. So whenever I was querying one metric, it was downloading all metrics which were in the same block (all metrics basically afaik), this…

> Victoria Metrics: As far as I know it's very well engineered and performs great

As someone who has auditioned it, briefly, let me assure you that it is certainly not the former, and only appears to be the latter due to a lot of cut corners and spec-violating implementations.

Re: M3DB, a distributed timeseries database

#118

Post by idjango »

rixed wrote: post #107
idjango wrote: post #73

Earlier quoted context omitted.

I also confirm that. Several companies have successfully transitioned their monitoring stack from graphite initial python implementation to a clickhouse based backend.

Not to bad-mouth Clickhouse but the original python implementation of graphite + carbon was setting the bar very low, though, and transitioning from there to anything would have increasing performances by orders of magnitude.

You should read the link below [1]. Even if it's not uber scale I suspect yandex to use something similar.

I agree that python implementation of graphite was not particularly fast but there was faster implementation in C that companies used first to significantly increase performance. Then coordination of storage backend becomes complex when you try to scale the initial design. This is where clickhouse really shine. It provide out of the box distributed storage with compaction, rollup and fast querying. The other layers are stateless, which means that they'll scale with your computing ressources.

M3DB is roughly doing the same thing as clickhouse but clickhouse is much more advanced database that has proven records of running at petabyte scale without a sweat. For example they now have tiered storage which means that you can store recent event in nvme and rollup to standard HDD...

[1]https://medium.com/avitotech/metrics-storage-how-we-migrated...

Re: M3DB, a distributed timeseries database

#119

Post by cube2222 »

sciurus wrote: post #100
cube2222 wrote: post #82

Earlier quoted context omitted.

Here you have the specifics: https://m3db.github.io/m3/m3db/architecture/engine/ I admit I’ve exaggerated a bit as Prometheus doesn’t support downsampling, in m3db I only keep 2 weeks of data at full resolution, 2 months at lower, and 5 years at even lower.

Downsampling might not save as much space as you think depending on how m3db works. https://github.com/thanos-io/thanos/issues/813 goes in to why for a similar project.

I know about the Thanos problem, and it's one of the things I didn't really like about it using the prometheus storage format.

It does work in m3db.

Following are the on-disk sizes of one replica:

2 weeks at 15s res: 90G

2 months at 1m res: 160G

3 months at 5m res: 60G

EDIT: The only thing is that m3db doesn't really downsample. You just create one namespace (table) for each resolution, and set the m3coordinator up so that it writes to each at the wanted interval, then set up a different retention for it. (this way you have duplicates in recent data)

M3 namespaces aren't set up for a specific resolution. The writer decides and you can write various series in different resolutions to one namespace in theory.

Re: M3DB, a distributed timeseries database

#120

Post by cube2222 »

cube2222 wrote: post #74
hagen1778 wrote: post #55

Earlier quoted context omitted.

Since you've done your research, would you mind to post a short list of alternatives and reasons why have they been rejected? Thanks!

This was as of november. Raw Prometheus: Isn't able to hold my data. Thanos: I liked the project, it's architecture and ease of deployment, but after spending a non-trivial amount of time with it I wasn't able to setup any long-term caching. Thanos uses the prometheus storage format. So whenever I was querying one metric, it was downloading all metrics which were in the same block (all metrics basically afaik), this…

Another comment on m3db:

m3query did have some inconsistencies compared to prometheus in how queries using intervals evaluated. (and sometimes didn't return any data because of it).

Having stepped through the code I don't remember the reason why that was, but I ended up using prometheus instances using remote_read from the m3coordinator (gateway), working like a charm.

Post reply on HN