Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

M3DB, a distributed timeseries database

m3db.io

Post reply on HN
Print view View original

131–138 of 138 posts

Re: M3DB, a distributed timeseries database

#131

Post by jjeaff »

carlisle_ wrote: post #98
jjeaff wrote: post #96

Earlier quoted context omitted.

There seems to be a pervasive misconception by the very employees at Uber, that they built their own chat platform. When in reality, and someone please correct me if I am wrong, uChat was a white labeled Mattermost. I have heard that the team that put it together actually tried to hide that fact from the company (for the glory, I guess). But that could be apocryphal.

That's entirely not true. The team was always forthcoming about the fact it was Mattermost, at least to other engineers. Mattermost didn't work out of the box, and certainly not the way and at the scale Uber needed it to. I'm not overly familiar with the technical details, but one thing in particular stands out as an example. There was a Town Hall channel that every user had to be a member of. This unfortunately did…

I dug up the comment I saw that mentioned this. They also claim to be an Uber employee.

https://news.ycombinator.com/item?id=19101617

The fact that uChat is commonly mentioned by Uber employees as being the "custom chat solution we built in-house". It leads me to believe that comment that the team tried to hide that it was built on open source.

I don't doubt for a second that scaling Mattermost for a huge organization like Uber was a big undertaking. But it seems disingenuous for people to always mention that Uber built uChat when it should be more like "Uber put a lot of work into Mattermost to scale it up."

Re: M3DB, a distributed timeseries database

#132

Post by pas »

remote_phone wrote: post #85
pas wrote: post #24

Earlier quoted context omitted.

Netflix loves Cassandra, right? [0][1] So could someone describe why it wasn't a great fit for Uber? How come it was easier to invent the wheel in Go compared to cobbling together something with Cassandra/ES/Kafka (or other Java gadgets from the Hadoop ecosystem)? [0]: https://netflixtechblog.com/scaling-time-series-data-storage... [1]: https://www.datastax.com/resources/video/cassandra-netflix-a...

It was an epic failure because you need a team to support and guide Cassandra use properly but no one wanted to do the grunt work. The VP of infrastructure MM openly called it “toil vs talent”, meaning those that did the grunt work would be held in high esteem and get yearly bonuses, but the promotions would go to those with “talent”, ie creating new things. When people are openly and stupidly incentivized like this,…

Oh, wow, that sounds worse than bad. Thanks for elaborating on the "root cause" too!

Re: M3DB, a distributed timeseries database

#133

Post by pas »

roskilli wrote: post #76
ckdarby wrote: post #58

Earlier quoted context omitted.

> which was too expensive for Uber to run (since the dataset is in petabytes). Ok, but I am fairly confident Netflix also is at that kind of scale. Netflix has a section on Atlas's documentation about how they get around this: https://github.com/Netflix/atlas/wiki/Overview#cost They also did this nice video that outlines their entire operation including how they do rollups: https://www.youtube.com/watch?v=4RG2DUK03_0…

I want to first say, I have a great amount of respect for Netflix's engineering and for Atlas itself, it's great that it exists and is more accessible than other scalable in-memory TSDBs open sourced by large companies. A few of my thoughts on this, and this has come up before. Firstly Netflix self-identifies it is expensive to run an in-memory TSDB for metrics - for instance Roy's talk on Atlas mentions this as such…

Thanks for the FOSDEM link. I know the videos are out, but just looking at the schedule to find the interesting talks took more time than I wanted to spend on it. (The conference became so huge.)

Maybe Bjorn's talk has the answers, but would you mind explaining how histograms are easy to aggregate? Don't you need either fixed buckets or raw data to produce a new histogram over a different dataset? (I know there are tricks to get great estimates, but naturally every re-aggregation would add larger and larger +/- intervals, no?)

Re: M3DB, a distributed timeseries database

#134

Post by johncolanduoni »

StreamBright wrote: post #114
richieartoul wrote: post #4

Earlier quoted context omitted.

Chronosphere, a startup founded by two of the early M3 engineers, just raised 11 million dollars to build a monitoring platform based around M3DB: https://techcrunch.com/2019/11/05/chronosphere-launches-with... Uber also uses M3DB extensively internally and the project is nowhere near being abandoned or on life support: https://github.com/m3db/m3/commits/master

>> While the founders, CEO Martin Mao and CTO Rob Skillington, were working at Uber, they recognized a gap in the monitoring industry, particularly around cloud-native technologies like containers and microservices What is the actual gap that is not addressed with one or all of the following? - Prometheus / Grafana [1] - Datadog [2] - Cloudwatch [3] 1. https://docs.docker.com/config/thirdparty/prometheus/ 2. https://…

Prometheus doesn’t have a good built in story for horizontal scaling of storage and queries, or long term storage. That’s why there’s M3DB, Cortex, etc. which let you send metrics to a set of Prometheus servers which then write to a cluster that handles broad queries and tiered storage. So these are less a competitor to Prometheus and Grafana and more an augmentation of it since it supports both emitting metrics in Prometheus form and querying them with PromQL.

Datadog gets very expensive beyond a certain point; much bigger than most startups, but much smaller than Uber.

Cloudwatch is usually used as a source of metrics rather than a destination. It doesn’t have as rich a data model as Prometheus for custom metrics, and has a lot of quite limiting restrictions (like only 10 tags per metric).

Re: M3DB, a distributed timeseries database

#135

Post by sieabahlpark »

bfung wrote: post #106
richieartoul wrote: post #4

Earlier quoted context omitted.

Chronosphere, a startup founded by two of the early M3 engineers, just raised 11 million dollars to build a monitoring platform based around M3DB: https://techcrunch.com/2019/11/05/chronosphere-launches-with... Uber also uses M3DB extensively internally and the project is nowhere near being abandoned or on life support: https://github.com/m3db/m3/commits/master

Yep, another way to pre-market/pre-signal to investors. A pretty common story these days: 1. I built an X to solve Unicorn U’s problem. 2. Open source it, give talks on it 3. Leave Unicorn U and start a company based on X 4. ... 5. Profit(???) Also, debatable whether or not Unicorn U actually needed a freshly built X instead of using existing tools/tech.

Seems genius imo

Re: M3DB, a distributed timeseries database

#136

Post by heliodor »

rossjudson wrote: post #59
heliodor wrote: post #22

When the Android app is broken in so many easy-to-fix ways that blatantly interfere with usability, how does a company allow its developers to spend time on making custom internal tools or even spend time open-sourcing them? The company has so much money and yet seems so utterly mismanaged.

Sounds like off-the-shelf tooling just didn't work. What's your solution for that?

In the grander scheme that I'm discussing, the solution you're asking about is to gather no more metrics than the off-the-shelf tools allow and spend the engineering effort on fixing the simple bugs that prevent customers from getting cars. Why do they need a perfect ton of metrics while they can't fix simple things such as:

- the car icon doesn't move as location updates come in (as evidenced by the route line getting shorter)

- the on-screen keyboard does not allow me to type anything after the first message (no other app on my phone has this problem)

- after I rate a driver, the app shows a map and none of the UI. I have to kill the app and restart it in order for it to be usable again. Picture this: I'm trying to get a ride, I open the app, I get nagged to rate the last driver. I agree just to be nice to the driver. (I should skip instead and get back to my task of getting a ride). After putting up with the nagware, the app fails 100% and I cannot complete the original task!

Alternatively, if they demand the collection of so many metrics, why don't they collect the metrics that would show them just how pathetically broken their Android app is?

Re: M3DB, a distributed timeseries database

#137

Post by StreamBright »

johncolanduoni wrote: post #134
StreamBright wrote: post #114

Earlier quoted context omitted.

>> While the founders, CEO Martin Mao and CTO Rob Skillington, were working at Uber, they recognized a gap in the monitoring industry, particularly around cloud-native technologies like containers and microservices What is the actual gap that is not addressed with one or all of the following? - Prometheus / Grafana [1] - Datadog [2] - Cloudwatch [3] 1. https://docs.docker.com/config/thirdparty/prometheus/ 2. https://…

Prometheus doesn’t have a good built in story for horizontal scaling of storage and queries, or long term storage. That’s why there’s M3DB, Cortex, etc. which let you send metrics to a set of Prometheus servers which then write to a cluster that handles broad queries and tiered storage. So these are less a competitor to Prometheus and Grafana and more an augmentation of it since it supports both emitting metrics in P…

Thanks for the detailed explanation. I agree with you on all points. I also think that even with the limitations you mentioned one company can find suitable monitoring for almost every scenario. I guess there is still a big enough market for M3DB where you need a huge amount of metrics, horizontal scalability and efficient long term storage.

Re: M3DB, a distributed timeseries database

#138

Post by 1996 »

valyala wrote: post #127
1996 wrote: post #88

Earlier quoted context omitted.

> ClickHouse come to mind for wholly different use cases than a time series database. ClickHouse works fine as a TSDB if you don't mind getting a little dirty

There is a TSDB solution if you don't want getting a little dirty - VictoriaMetrics [1]. It is built on the same principles as ClickHouse [2]. [1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast... [2] https://medium.com/@valyala/how-victoriametrics-makes-instan...

"-retentionPeriod - retention period in months for the data. Older data is automatically deleted. Default period is 1 month."

Not sure I want that in a TSDB!

"Reducing disk space usage by deleting unneded time series. This doesn't work as expected, since the deleted time series occupy disk space until the next merge operation, which can never occur."

Ouch. But ok, disk space is cheap.

The killer point: it seems to be purely json based- no SQL of any kind. I'm not sure about that. A lot of code would have to be changed to fit that model.

Post reply on HN