Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

M3DB, a distributed timeseries database

m3db.io

Post reply on HN
Print view View original

81–90 of 138 posts

Re: M3DB, a distributed timeseries database

#81

Post by roskilli »

mbell wrote: post #72
roskilli wrote: post #70

Earlier quoted context omitted.

While this is true, for a metrics workload it does not work great I have both seen and heard from others, mainly due to the fact it does not have an inverted index - so finding a small subset of metrics in a dataset of billions of metrics ends up taking significant time due to the scan required to find the timeseries matching the arbitrary number of dimensions specified to find the timeseries you're looking for. If y…

Most practical applications using Clickhouse for metrics data store the metric index separately. What index you want really depends on the metric system, e.g. with graphite data you don't want an inverted index, you want a trie.

Yes I've seen that also work, it's a lot of stitching together things yourself and we had to put a lot of caching in front of the inverted index we were using, however definitely plausible. ClickHouse doesn't do any streaming of data between nodes as you scale up and down which was a big thing for us since we had large datasets and needed to rebalance when cluster expanded/shrunk.

With regards to trie vs inverted index for Graphite data, I'd actually still be inclined to say inverted index is better based on the amount of queries I saw at Uber with Graphite where people did `servers.*.disk.bytes-used` type queries which is way faster to do using an inverted index since you have a postings list for each part of the dot-separated metric name, rather than traversing a trie with thousands to tens of thousands of entries in index 1 host part of the Graphite name. This is what M3DB does[0].

[0]: https://github.com/m3db/m3/blob/b2f5b55e8313eb48f023e08f6d53...

Re: M3DB, a distributed timeseries database

#82

Post by cube2222 »

hagen1778 wrote: post #79
cube2222 wrote: post #74

Earlier quoted context omitted.

This was as of november. Raw Prometheus: Isn't able to hold my data. Thanos: I liked the project, it's architecture and ease of deployment, but after spending a non-trivial amount of time with it I wasn't able to setup any long-term caching. Thanos uses the prometheus storage format. So whenever I was querying one metric, it was downloading all metrics which were in the same block (all metrics basically afaik), this…

Thanks for details! Really appreciate it. > It also uses a magnitude of storage space less than raw prometheus AFAIK, Prometheus compression is about 1.2-3 bytes per datapoint. A magnitude less is 0.12-0.3 bytes - are these numbers correct?

Here you have the specifics: https://m3db.github.io/m3/m3db/architecture/engine/

I admit I’ve exaggerated a bit as Prometheus doesn’t support downsampling, in m3db I only keep 2 weeks of data at full resolution, 2 months at lower, and 5 years at even lower.

Re: M3DB, a distributed timeseries database

#83

Post by iblaine »

missosoup wrote: post #2

Uber has started many projects that ended up getting open sourced. And many of them are now either abandoned or on life support. H3 comes to mind as something we almost ended up using but luckily avoided. These open-sourcings seem a bit like PR pieces with no guarantees of any support or evolution after being published.

Open sourcing projects is the new merit badge for engineers. But at least there’s more good than bad from it. Hudi is at least one Uber project I can point to off the top of my head that is a great idea.

Re: M3DB, a distributed timeseries database

#84

Post by parentheses »

missosoup wrote: post #2

Uber has started many projects that ended up getting open sourced. And many of them are now either abandoned or on life support. H3 comes to mind as something we almost ended up using but luckily avoided. These open-sourcings seem a bit like PR pieces with no guarantees of any support or evolution after being published.

The issue here is not the company but the fact that the owners of the original library did not figure out how to “disown” the library.

Open sourcing something is naturally more expensive than not. It’s seldom the case that impact to the community triggers contributions that outweigh that cost.

The fallacy we hold is that companies will prop up software that is open source for everyone to use despite the lacking community contribution.

We as engineers should push ourselves to contribute when we find issues - rather than simply create tickets that represent work were want to have done for free. This is how open source software dies.

There is a minority that does this.

Re: M3DB, a distributed timeseries database

#85

Post by remote_phone »

pas wrote: post #24
hkarthik wrote: post #20

Earlier quoted context omitted.

I can give you an ex-insiders view on this. Uber made an early strategic decision to invest in on-premise infrastructure due to fears that either Amazon or Google would enter the on-demand market as competitors and bring their cloud infrastructure to bear and potentially squeeze us for costs. Azure wasn’t much of an option during this time. This decision limited our adoption of cloud native solutions like SpannerDB a…

Netflix loves Cassandra, right? [0][1] So could someone describe why it wasn't a great fit for Uber? How come it was easier to invent the wheel in Go compared to cobbling together something with Cassandra/ES/Kafka (or other Java gadgets from the Hadoop ecosystem)? [0]: https://netflixtechblog.com/scaling-time-series-data-storage... [1]: https://www.datastax.com/resources/video/cassandra-netflix-a...

It was an epic failure because you need a team to support and guide Cassandra use properly but no one wanted to do the grunt work. The VP of infrastructure MM openly called it “toil vs talent”, meaning those that did the grunt work would be held in high esteem and get yearly bonuses, but the promotions would go to those with “talent”, ie creating new things.

When people are openly and stupidly incentivized like this, expect those people to behave in a predictable way. People started building new services to get promotions instead of “toiling” at supporting their fellow engineers.

It affected most of engineering but especially in teams like Cassandra, where you needed guidance and support to properly use it effectively, it was a disaster. There should have been open office hours to help people with questions and to ensure that teams were using it properly but there wasn’t. Instead people were left to do what they wanted with no structure or guidance and Cassandra was completely misused. Productions problems ensued, people left the team because they didn’t want to be oncall fixing fires all the time, and eventually it came to the point where they decided to stop supporting it altogether. It was a complete disaster caused by very poor engineering management.

We all knew that Netflix and Facebook use it without issues, but because of stupid management, it failed at Uber.

Re: M3DB, a distributed timeseries database

#87

Post by throwaway5752 »

missosoup wrote: post #2

Uber has started many projects that ended up getting open sourced. And many of them are now either abandoned or on life support. H3 comes to mind as something we almost ended up using but luckily avoided. These open-sourcings seem a bit like PR pieces with no guarantees of any support or evolution after being published.

Lots of good open source projects fail. But not every company is willing to open source code like this, though, and I'm very happy that Uber did so in this case.

I get your frustration, but everyone should remember there are never any promises of support with open source software, regardless of how well supported it is at a particular time.

Re: M3DB, a distributed timeseries database

#88

Post by 1996 »

buro9 wrote: post #12
tnolet wrote: post #10

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.

It's a database for a metric platform. Think of OpenTSDB and Prometheus. Or for a better comparison think of Thanos https://thanos.io/ As to whether they could fulfil Uber's needs, the thing about scale (real massive scale - I work at Cloudflare) is that everything breaks in weird ways according to your specific uses of a technology. The things listed above work for companies, until they don't. There's few things tha…

> ClickHouse come to mind for wholly different use cases than a time series database.

ClickHouse works fine as a TSDB if you don't mind getting a little dirty

Re: M3DB, a distributed timeseries database

#89

Post by [deleted] »

pas wrote: post #24
hkarthik wrote: post #20

Earlier quoted context omitted.

I can give you an ex-insiders view on this. Uber made an early strategic decision to invest in on-premise infrastructure due to fears that either Amazon or Google would enter the on-demand market as competitors and bring their cloud infrastructure to bear and potentially squeeze us for costs. Azure wasn’t much of an option during this time. This decision limited our adoption of cloud native solutions like SpannerDB a…

Netflix loves Cassandra, right? [0][1] So could someone describe why it wasn't a great fit for Uber? How come it was easier to invent the wheel in Go compared to cobbling together something with Cassandra/ES/Kafka (or other Java gadgets from the Hadoop ecosystem)? [0]: https://netflixtechblog.com/scaling-time-series-data-storage... [1]: https://www.datastax.com/resources/video/cassandra-netflix-a...

[deleted]

Re: M3DB, a distributed timeseries database

#90

Post by manigandham »

buro9 wrote: post #12
tnolet wrote: post #10

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.

It's a database for a metric platform. Think of OpenTSDB and Prometheus. Or for a better comparison think of Thanos https://thanos.io/ As to whether they could fulfil Uber's needs, the thing about scale (real massive scale - I work at Cloudflare) is that everything breaks in weird ways according to your specific uses of a technology. The things listed above work for companies, until they don't. There's few things tha…

Clickhouse (and other columnstore RDBMS) are all perfectly fine for time-series and usually better than the standard options because they have SQL querying.
Post reply on HN