Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

M3DB, a distributed timeseries database

m3db.io

Post reply on HN
Print view View original

21–30 of 138 posts

Re: M3DB, a distributed timeseries database

#21

Post by abvdasker »

clircle wrote: post #9

Does "Time Series Database" mean anything technical, or is this just some Uber marketing? In statistics, time series has a technical meaning.

A time series database is specialized for use cases where the data and query patterns are solely temporal in nature and must show the latest data in real-time (performance metrics/monitoring and stock prices come to mind). Relational and NoSQL databases tend to degrade rapidly with these query patterns at scale (think of the complexity of SQL queries to bucket rows by timestamp).

https://en.m.wikipedia.org/wiki/Time_series_database

Re: M3DB, a distributed timeseries database

#22

Post by heliodor »

When the Android app is broken in so many easy-to-fix ways that blatantly interfere with usability, how does a company allow its developers to spend time on making custom internal tools or even spend time open-sourcing them? The company has so much money and yet seems so utterly mismanaged.

Re: M3DB, a distributed timeseries database

#23

Post by roskilli »

tnolet wrote: post #10

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.

As per sibling comment, they do most definitely work until they don’t. M3 actually started with ElasticSearch and Cassandra for index and storage respectively but then were replaced with M3DB. I mentioned the FOSDEM talk elsewhere in the thread but you might be interested in the evolution segment where it’s mentioned ”With M3DB 7x less servers from Cassandra, while increasing RF=2 to RF=3” and something that’s not on the slides but is in the talk is a reference to an order of magnitude reduction in operational overhead (incidents/oncall debugging). Both slides and video is linked from the FOSDEM talk’s page https://fosdem.org/2020/schedule/event/m3db/.

Re: M3DB, a distributed timeseries database

#24

Post by pas »

hkarthik wrote: post #20
tnolet wrote: post #10

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.

I can give you an ex-insiders view on this. Uber made an early strategic decision to invest in on-premise infrastructure due to fears that either Amazon or Google would enter the on-demand market as competitors and bring their cloud infrastructure to bear and potentially squeeze us for costs. Azure wasn’t much of an option during this time. This decision limited our adoption of cloud native solutions like SpannerDB a…

Netflix loves Cassandra, right? [0][1] So could someone describe why it wasn't a great fit for Uber? How come it was easier to invent the wheel in Go compared to cobbling together something with Cassandra/ES/Kafka (or other Java gadgets from the Hadoop ecosystem)?

[0]: https://netflixtechblog.com/scaling-time-series-data-storage... [1]: https://www.datastax.com/resources/video/cassandra-netflix-a...

Re: M3DB, a distributed timeseries database

#25

Post by [deleted] »

reichardt wrote: post #3
missosoup wrote: post #2

Uber has started many projects that ended up getting open sourced. And many of them are now either abandoned or on life support. H3 comes to mind as something we almost ended up using but luckily avoided. These open-sourcings seem a bit like PR pieces with no guarantees of any support or evolution after being published.

Why do you consider H3 to be on life support? It's basically a finished spec with actively developed implementations. https://github.com/uber/h3

[deleted]

Re: M3DB, a distributed timeseries database

#27

Post by roskilli »

pas wrote: post #24
hkarthik wrote: post #20

Earlier quoted context omitted.

I can give you an ex-insiders view on this. Uber made an early strategic decision to invest in on-premise infrastructure due to fears that either Amazon or Google would enter the on-demand market as competitors and bring their cloud infrastructure to bear and potentially squeeze us for costs. Azure wasn’t much of an option during this time. This decision limited our adoption of cloud native solutions like SpannerDB a…

Netflix loves Cassandra, right? [0][1] So could someone describe why it wasn't a great fit for Uber? How come it was easier to invent the wheel in Go compared to cobbling together something with Cassandra/ES/Kafka (or other Java gadgets from the Hadoop ecosystem)? [0]: https://netflixtechblog.com/scaling-time-series-data-storage... [1]: https://www.datastax.com/resources/video/cassandra-netflix-a...

Netflix actually built their own metrics time series store called Atlas for similar reasons to Uber building M3DB (FOSDEM talk mentions hardware reduction and oncall reduction), however open source Atlas only has an in-memory store component which was too expensive for Uber to run (since the dataset is in petabytes).

https://github.com/Netflix/atlas

Re: M3DB, a distributed timeseries database

#29

Post by api »

missosoup wrote: post #2

Uber has started many projects that ended up getting open sourced. And many of them are now either abandoned or on life support. H3 comes to mind as something we almost ended up using but luckily avoided. These open-sourcings seem a bit like PR pieces with no guarantees of any support or evolution after being published.

Many large companies push this stuff out for publicity and recruitment. Sometimes employees are encouraged to spend a little bit of their time on it or to brand extracurricular activities with the company name for publicity.

The test for open source is if it keeps getting maintained and supported for years. That only happens when the project is a core business effort, has some direct means of support (e.g. dual licensing or SaaS), or happens to be one of the few genuinely volunteer driven large scale open source projects.

Re: M3DB, a distributed timeseries database

#30

Post by whyreplicate »

tnolet wrote: post #10

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.

Cassandra and ElasticSearch would probably have been fine except that Uber dramatically under-provisioned the hardware used for them. The database redundancy was so low that any minor hardware issue could quickly turn into a major outage for all of Uber's monitoring services.
Post reply on HN