91–100 of 138 posts
I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.
So how does this compare to e.g. Clickhouse?
Clickhouse is an analytic column-based RDBMS. It's not a timeseries database. Each class of product is used to solve different problems.
Any database can handle it, and columnstore RDBMS are designed to store and query trillion-row tables with full SQL functionality. The only advantage a "time-series" database gives you is some time-based query operators (like gap filling, last value, smoothing, etc). Those are now being added to SQL support for RDBMS so there's really nothing to be gained from a time-series database anymore.
Earlier quoted context omitted.
Cassandra and ElasticSearch would probably have been fine except that Uber dramatically under-provisioned the hardware used for them. The database redundancy was so low that any minor hardware issue could quickly turn into a major outage for all of Uber's monitoring services.
Well if you’re going to run at RF2 and push them when they can only do 60,000 writes per second vs multiple hundreds of thousands per second with specialized software on the same hardware. It’s hard to justify using tens of millions of dollars more of hardware more to run Cassandra.
Earlier quoted context omitted.
Well if you’re going to run at RF2 and push them when they can only do 60,000 writes per second vs multiple hundreds of thousands per second with specialized software on the same hardware. It’s hard to justify using tens of millions of dollars more of hardware more to run Cassandra.
What do you think of ScyllaDB then? Would it have been able to do the job
This makes it tough to use solely either ScyllaDB, ClickHouse or Cassandra for that matter for metrics workloads at scale since they need to find a needle in a haystack - a few thousand time series amongst a set of millions to billions, where users only specify a subset of the dimensions on the metrics in any order they want to. This is hard to do without an inverted index.
Earlier quoted context omitted.
I agree with everything you say. But without any certainty around the roadmap, support, and longterm commitment by Uber to maintain these projects, they're nothing more than interesting repos amongst a sea of interesting repos. The way Uber brands them suggests that they're suitable for use in production environments, but so far that hasn't been the case with anything they open sourced outside a narrow envelope that…
>It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case. Former Uber engineer here. I can assure you that while our engineering team was massive, there was anything but too little work. If anything most engineers were massively overtaxed.…
I have heard that the team that put it together actually tried to hide that fact from the company (for the glory, I guess). But that could be apocryphal.
Earlier quoted context omitted.
I accidentally left this point out. In retrospect it's easy to say Uber made the wrong decision to make uChat but it was one of few options at the time.
Seems like a huge point to leave out. Are you affiliated with Uber?
Earlier quoted context omitted.
>It seems like Uber had too big of an engineering department with too little work to do, so they started reinventing wheels. Which is cool if they're willing to support them in the long term, but so far that hasn't proven to be the case. Former Uber engineer here. I can assure you that while our engineering team was massive, there was anything but too little work. If anything most engineers were massively overtaxed.…
There seems to be a pervasive misconception by the very employees at Uber, that they built their own chat platform. When in reality, and someone please correct me if I am wrong, uChat was a white labeled Mattermost. I have heard that the team that put it together actually tried to hide that fact from the company (for the glory, I guess). But that could be apocryphal.
Mattermost didn't work out of the box, and certainly not the way and at the scale Uber needed it to. I'm not overly familiar with the technical details, but one thing in particular stands out as an example. There was a Town Hall channel that every user had to be a member of. This unfortunately did not scale, and not enough ACLs were available to limit all the ways users could use this universal room. Eventually they really fixed the problem, but it was a tremendous pain point for a long time. There were a lot of fundamentally "less than great" things about Mattermost that had to get updated to work for Uber.
There was the amusing time employees found out anybody could change the topic in the room, even if our chat permissions had been disabled. It was absolute chaos for at least an hour, I can't remember if it actually negatively impacted the deployment though it sounds vaguely familiar.
It's pretty telling of employees that badmouth the uChat team. That team ultimately was trying to do what they thought was best for the company, even if at the time it seemed like they bit off more than they could chew. There was no other engineering team so directly visible and exposed to the entire company internally like they were. People dismissive of their efforts are generally not used to the difficulty of making so many very vocal customers happy all at the same time, and could be more sympathetic.
I get Uber is huge. But honestly, there was nothing out there that could fulfill there use case? Cassandra, ElasticSearch, Influx, etc.? I might be completely wrong, but I just highly doubt that.
It's a database for a metric platform. Think of OpenTSDB and Prometheus. Or for a better comparison think of Thanos https://thanos.io/ As to whether they could fulfil Uber's needs, the thing about scale (real massive scale - I work at Cloudflare) is that everything breaks in weird ways according to your specific uses of a technology. The things listed above work for companies, until they don't. There's few things tha…
150m-200m events/minute and about 20-30 trillion (10^12) events stored. Doubling about every 12-18 months or so.
While it's true that things start to creak at scale, this has worked remarkably well for us so far. I doubt M3DB is somehow magical in this regard.
Earlier quoted context omitted.
Thanks for details! Really appreciate it. > It also uses a magnitude of storage space less than raw prometheus AFAIK, Prometheus compression is about 1.2-3 bytes per datapoint. A magnitude less is 0.12-0.3 bytes - are these numbers correct?
Here you have the specifics: https://m3db.github.io/m3/m3db/architecture/engine/ I admit I’ve exaggerated a bit as Prometheus doesn’t support downsampling, in m3db I only keep 2 weeks of data at full resolution, 2 months at lower, and 5 years at even lower.