Live data from Hacker News

How Netflix uses Druid for realtime insights

netflixtechblog.com

31–40 of 60 posts

Re: How Netflix uses Druid for realtime insights

#31

I wonder if materialize.io could handle such workloads at this stage.

Materialize, today, looks to be limited to running on a single node. https://materialize.io/docs/overview/architecture Netflix's workload would likely exhaust the resources of even a vertically-scaled single node.

Mind you, given the “Timely Dataflow” abstraction Materialize operates on top of, if you give it a query that only requires certain result rows from one of its mat views, then Materialize is only going to compute the intermediate rows (and, further back, retrieve the source rows) required to “render” the particular result-rows you ask for. (Sort of like how Excel, in memory-constrained conditions, only computes the intermediate cell-values required to render the cells currently in view, and thus, if an intermediate cell requires an `XmlHttpRequest` call to resolve, that call won’t fire if the cell’s value doesn’t need resolving yet.)

Because of that, you don’t really need to scale Materialize in a sharding sense. You can just have a bunch of “the same” Materialize node (i.e. every node just freestanding clone of a template node, with exactly the same sources and matviews) and then hit them with the parts of a map-reduce query launched by, say, Citus—where Citus was thinking it was talking to a bunch of Citus shard nodes each holding a table-shard named X, but was actually talking to Materialize nodes each holding a matview named X. As long as the query sent from the map-reduce job to each node is constrained in its WHERE clause to only the part of the data it expects to get from that node—rather than relying on the node to know what data it has—then the Materialize nodes would each just do the work required to supply that data (including only pulling in the parts of the configured sources required to compute that result.)

That’s just my intuition from how Materialize presents itself as PG-wire-protocol compatible, though; I haven’t tried this myself, and there might be some footguns in the path of anyone really trying to implement it.

And, of course, this is all irrelevant the moment you write a query that needs a pure reduce (e.g. the computation of a current finite-state-machine state over an event-stream source) rather than a map-reduce. Druid/Clickhouse/etc. can probably “scale” those, in at least the Hadoop “move the job around between each serial stage, so each stage has data-locality for the data of that stage” sense; while Materialize would give you no benefit at all in such a job over just querying a plain PG view defined on top of a Foreign Data Wrapper source.

Re: How Netflix uses Druid for realtime insights

#32
post #14

Earlier quoted context omitted.

Clickhouse is much more resource efficient in many cases but is less flexible and importantly extensible than Druid. Druid can easily be extended through available 3rd party extensions and you can write your own to implement custom serialisation formats, aggregations, connect to new streaming systems, read directly from whatever cold storage you have etc. In the Clickhouse model you have to work out a lot more of tha…

In our experience, on a significantly smaller scale, Clickhouse is vastly easier to operate compared to Druid, with all of its various components that all have various knobs and dials to configure and have to be orchestrated.

Druid committer here. Fwiw, Druid was designed to run on huge clusters and that really shows up in the multi-process architecture. The idea is that if you separate the components needed for ingestion, historical processing, query routing, and coordination, then there are two benefits: they don't interfere with each other (spikes in ingestion load won't interfere with ability to query historical data), and also you can scale each one individually for your workload. You could even auto-scale some of them. For example, the original Druid cluster was operated with load-based auto-scaling for the ingestion processes.

That being said we are currently working on reducing the number of processes to 4 (from the current 6) for a "standard" setup. The main reason is that at smaller scale there isn't as much of a purpose to having a larger number of processes.

We're also working on removing some of the knobs. Actually, depending on what version you originally looked at, many of them might already be gone.

Re: How Netflix uses Druid for realtime insights

#33

I wonder if materialize.io could handle such workloads at this stage.

materialize has to be able to keep all of its state in memory which only makes sense from a cost perspective for workloads which have a high value to space ratio. fine grained user behavior data typically just isn't that valuable.

Re: How Netflix uses Druid for realtime insights

#34
post #32

Earlier quoted context omitted.

In our experience, on a significantly smaller scale, Clickhouse is vastly easier to operate compared to Druid, with all of its various components that all have various knobs and dials to configure and have to be orchestrated.

Druid committer here. Fwiw, Druid was designed to run on huge clusters and that really shows up in the multi-process architecture. The idea is that if you separate the components needed for ingestion, historical processing, query routing, and coordination, then there are two benefits: they don't interfere with each other (spikes in ingestion load won't interfere with ability to query historical data), and also you ca…

It's been a few years since we evaluated Druid. It's great to hear that you're simplifying things, especially for smaller setups!

Re: How Netflix uses Druid for realtime insights

#36
post #14
post #8

Earlier quoted context omitted.

Can confirm clickhouse is generally faster across most typical workloads

Clickhouse is much more resource efficient in many cases but is less flexible and importantly extensible than Druid. Druid can easily be extended through available 3rd party extensions and you can write your own to implement custom serialisation formats, aggregations, connect to new streaming systems, read directly from whatever cold storage you have etc. In the Clickhouse model you have to work out a lot more of tha…

Druid committer here. (Also, I think we've met before in SF!)

One thing I wanted to add with regard to performance. Druid does indeed get a big boost from the fact that it uses inverted indexes for filtering. It also gets a boost from having a wide variety of approximate algorithms you can use if you want (for things like topN, count distinct, set difference/intersection, quantiles, etc). But straight scan performance has been improving quite a bit recently too.

The biggest change related to straight scan perf is fully vectorizing the query engine, which is partially done as of the latest release (0.17): https://druid.apache.org/docs/latest/querying/query-context..... In benchmarks, the implementation so far has been posting row scan rate improvements in the 2-3x range. I expect we'll be able to round it out and have it work for all queries over the next couple of releases. The multiples involved mean this is quite meaningful if you do a lot of straight scans.

There's plenty of other stuff going on too: our latest release added parallel merging of large result sets. Our next one (0.18) is going to add a new, more efficient hash aggregation engine. That next release is also going to add a JOIN operator -- not perf related, but probably the number one most requested feature.

Re: How Netflix uses Druid for realtime insights

#37

> During software updates, we enable the new version for a subset of users and ... compare how the new version is performing vs the previous version. Any regression in the metrics gives us a signal to abort the update and revert How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run?

> How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run?

That's assuming that the update contains a UI/UX change. A lot of updates that will roll out won't include that, they'll be fixing or optimising things.

Re: How Netflix uses Druid for realtime insights

#38

"To keep reading this story, create a free account." Seriously? I know it's the Medium hustle but someone at Netflix should know better.

I think one cool feature Medium can introduce is enterprise permissioning. If you have an enterprise plan and pay Medium oodles of money, you should maintain your own permissioning control and who sees your articles while still having access to the Medium social network. Best of both worlds.

> oodles of money

Medium isn't worth oodles, though. Looking at Squarespace and Wordpress pricing, for a company's tech blog, it's worth $25-$100 per month. Unless there's value from the Medium brand...

Re: How Netflix uses Druid for realtime insights

#39

"To keep reading this story, create a free account." Seriously? I know it's the Medium hustle but someone at Netflix should know better.

I guess it costs slightly less to hire someone who can manage a basic word editor, than someone who can manage a self-hosted website.

Re: How Netflix uses Druid for realtime insights

#40
post #11

So this is behind paywall. I am sure Netflix is not marking this story to be behind paywall and Medium can not put this story behind paywall without consent from Netflix.

You can avoid this issue on Medium and lots of other sites using the firefox add-on Bypass Paywalls: https://github.com/iamadamdev/bypass-paywalls-firefox
Post reply on HN