Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

M3DB, a distributed timeseries database

m3db.io

Post reply on HN
Print view View original

31–40 of 138 posts

Re: M3DB, a distributed timeseries database

#31

Post by cfors »

mamon wrote: post #19
missosoup wrote: post #6

Earlier quoted context omitted.

I agree with everything you say. But without any certainty around the roadmap, support, and longterm commitment by Uber to maintain these projects, they're nothing more than interesting repos amongst a sea of interesting repos. The way Uber brands them suggests that they're suitable for use in production environments, but so far that hasn't been the case with anything they open sourced outside a narrow envelope that…

Isn't that kind of the point of open-sourcing your internal tools? You don't want to be bothered with maintenance and support, so you're hoping that some anonymous volounteers will do that for you :)

Maybe I’m a cynic of all big corp companies but if you’ve ever worked with a big corp open source department that’s almost the entire point is to build PR for the engineering department. Same goes for tech blogs. These things will be PR pieces first, and valid production tools/frameworks second (mostly).

Re: M3DB, a distributed timeseries database

#32

Post by roskilli »

abvdasker wrote: post #21
clircle wrote: post #9

Does "Time Series Database" mean anything technical, or is this just some Uber marketing? In statistics, time series has a technical meaning.

A time series database is specialized for use cases where the data and query patterns are solely temporal in nature and must show the latest data in real-time (performance metrics/monitoring and stock prices come to mind). Relational and NoSQL databases tend to degrade rapidly with these query patterns at scale (think of the complexity of SQL queries to bucket rows by timestamp). https://en.m.wikipedia.org/wiki/Time_…

I touch on this a little in the podcast I did with Jeff[0], but it boils down to OpenTSDSB for us when we benchmarked could only do low tens of thousands of writes per second per node, whereas M3DB is hyper optimized and can do hundreds of thousands to millions of writes per second per node depending on compute/disk.

Also with a fast inverted index we were able to achieve much faster query times than OpenTSDB at scale.

[0]: https://softwareengineeringdaily.com/2019/08/21/time-series-...

Re: M3DB, a distributed timeseries database

#34

Post by roskilli »

whyreplicate wrote: post #30
tnolet wrote: post #10

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.

Cassandra and ElasticSearch would probably have been fine except that Uber dramatically under-provisioned the hardware used for them. The database redundancy was so low that any minor hardware issue could quickly turn into a major outage for all of Uber's monitoring services.

Well if you’re going to run at RF2 and push them when they can only do 60,000 writes per second vs multiple hundreds of thousands per second with specialized software on the same hardware.

It’s hard to justify using tens of millions of dollars more of hardware more to run Cassandra.

Re: M3DB, a distributed timeseries database

#35

Post by refset »

abvdasker wrote: post #21
clircle wrote: post #9

Does "Time Series Database" mean anything technical, or is this just some Uber marketing? In statistics, time series has a technical meaning.

A time series database is specialized for use cases where the data and query patterns are solely temporal in nature and must show the latest data in real-time (performance metrics/monitoring and stock prices come to mind). Relational and NoSQL databases tend to degrade rapidly with these query patterns at scale (think of the complexity of SQL queries to bucket rows by timestamp). https://en.m.wikipedia.org/wiki/Time_…

Note that temporal databases are also a thing, so it's probably wise to avoid using the word "temporal" when discussing time series databases. As far as I know kdb+ is the only technology that has a foot in both camps.

https://en.m.wikipedia.org/wiki/Temporal_database

Re: M3DB, a distributed timeseries database

#36

Post by cube2222 »

tnolet wrote: post #10

I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.

Having deployed m3 recently, I’ve not found an alternative which is cost effective and fast at the same time. Granted, it uses a lot of memory, but other than that I’ve been incredibly happy with it.

Re: M3DB, a distributed timeseries database

#37

Post by lopsidedBrain »

mamon wrote: post #19
missosoup wrote: post #6

Earlier quoted context omitted.

I agree with everything you say. But without any certainty around the roadmap, support, and longterm commitment by Uber to maintain these projects, they're nothing more than interesting repos amongst a sea of interesting repos. The way Uber brands them suggests that they're suitable for use in production environments, but so far that hasn't been the case with anything they open sourced outside a narrow envelope that…

Isn't that kind of the point of open-sourcing your internal tools? You don't want to be bothered with maintenance and support, so you're hoping that some anonymous volounteers will do that for you :)

Pretty much every successful open source project that people pay attention to is one that has had long-term support behind it. Linux, Mozilla, gcc, clang, git. Almost always, that support begins with the original author.

Protects that don't do that are therefore unlikely to remain interesting for long.

Re: M3DB, a distributed timeseries database

#39

Post by carlisle_ »

missosoup wrote: post #6
gtirloni wrote: post #5

Earlier quoted context omitted.

It's opensource. Why should Uber give any guarantees? They are not in the business of selling software. Unless Uber is actively blocking contributions, it's not Uber's fault if no community formed around something they opensourced. As for this being a PR piece, they could have achieved the same with just a detailed blog post and no code. It looks like a expensive PR piece if they have to opensource work that took pro…

I agree with everything you say. But without any certainty around the roadmap, support, and longterm commitment by Uber to maintain these projects, they're nothing more than interesting repos amongst a sea of interesting repos. The way Uber brands them suggests that they're suitable for use in production environments, but so far that hasn't been the case with anything they open sourced outside a narrow envelope that…

>It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case.

Former Uber engineer here. I can assure you that while our engineering team was massive, there was anything but too little work. If anything most engineers were massively overtaxed. Whether or not the work we were undertaking was meritious and valuable is an entire branch of philosophy I'm pretty sure.

Part of the struggle at big companies is that a lot of existing solutions just don't work. Let me use an example with chat. A few years ago Slack was evaluated as a replacement for HipChat, since Atlassian's outages had finally started affecting us during our own outages.

Everybody wanted to go to Slack, but the cost of Slack was tremendously prohibitive and the state of the service then (as I was told) was such that it could not support a company of Uber's size. Tremendous effort would have been undertaken by Slack to support Uber and they didn't want to expend that effort for a single customer. This was late 2015 early 2016.

There were tons of options, but ultimately an in-house chat software was created. At the time it seemed required to make our own highly reliable chat, considering how distributed engineering teams were. I think if you talk to anybody without the background of how chat evolved at Uber they would think the in-house chat project would have been a boondoggle.

Not all over-scoped engineering projects are actually so noble. There was certainly a ton of "reinventing wheels" going on. There was significantly more "these problems are really hard and I only have bad solutions."

Though if the result is ultimately, "nothing more than interesting repos amongst a sea of interesting repos" sign me up.

Re: M3DB, a distributed timeseries database

#40

Post by closeparen »

cfors wrote: post #31
mamon wrote: post #19

Earlier quoted context omitted.

Isn't that kind of the point of open-sourcing your internal tools? You don't want to be bothered with maintenance and support, so you're hoping that some anonymous volounteers will do that for you :)

Maybe I’m a cynic of all big corp companies but if you’ve ever worked with a big corp open source department that’s almost the entire point is to build PR for the engineering department. Same goes for tech blogs. These things will be PR pieces first, and valid production tools/frameworks second (mostly).

In my experience software gets written in the first place for the usual internal reasons. Corporate or individual prestige may be the driving factor in open sourcing, though, rather than a genuine interest in having it used externally.
Post reply on HN