Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Netflix uses Druid for realtime insights

netflixtechblog.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: How Netflix uses Druid for realtime insights

#51

Post by manigandham »

tjoff wrote: post #50
manigandham wrote: post #49

Earlier quoted context omitted.

People are interested in the content. No need for HN to start filtering because the blog service. You can avoid the site if you want.

There most certainly is a pressing need for sites such as HN to start care about stuff like this. If people start to care about being exploited the content will move to a more ethical site. Or it might influence medium.

People don't care though, because they can make their own choices but they're clearly still sharing and reading on medium. You're asking for HN to arbitrarily enforce your wishes on top of what users are doing.

Re: How Netflix uses Druid for realtime insights

#52

Post by tjoff »

manigandham wrote: post #51
tjoff wrote: post #50

Earlier quoted context omitted.

There most certainly is a pressing need for sites such as HN to start care about stuff like this. If people start to care about being exploited the content will move to a more ethical site. Or it might influence medium.

People don't care though, because they can make their own choices but they're clearly still sharing and reading on medium. You're asking for HN to arbitrarily enforce your wishes on top of what users are doing.

They do, and it is hardly my wishes alone.

If 1% don't care and share the crap the other 99% still have to suffer for it.

Re: How Netflix uses Druid for realtime insights

#53

Post by jpgvm »

gianm wrote: post #36
jpgvm wrote: post #14

Earlier quoted context omitted.

Clickhouse is much more resource efficient in many cases but is less flexible and importantly extensible than Druid. Druid can easily be extended through available 3rd party extensions and you can write your own to implement custom serialisation formats, aggregations, connect to new streaming systems, read directly from whatever cold storage you have etc. In the Clickhouse model you have to work out a lot more of tha…

Druid committer here. (Also, I think we've met before in SF!) One thing I wanted to add with regard to performance. Druid does indeed get a big boost from the fact that it uses inverted indexes for filtering. It also gets a boost from having a wide variety of approximate algorithms you can use if you want (for things like topN, count distinct, set difference/intersection, quantiles, etc). But straight scan performanc…

Good point regarding aggregations, especially if done at ingest time w/rollup they really make a big difference. Clickhouse has aggregations too but they are done at merge time in the background.

Vectorized query engine and JOINs sounds awesome.

(We did meet in SF! Beer hall!)

Re: How Netflix uses Druid for realtime insights

#54

Post by barrkel »

gianm wrote: post #36
jpgvm wrote: post #14

Earlier quoted context omitted.

Clickhouse is much more resource efficient in many cases but is less flexible and importantly extensible than Druid. Druid can easily be extended through available 3rd party extensions and you can write your own to implement custom serialisation formats, aggregations, connect to new streaming systems, read directly from whatever cold storage you have etc. In the Clickhouse model you have to work out a lot more of tha…

Druid committer here. (Also, I think we've met before in SF!) One thing I wanted to add with regard to performance. Druid does indeed get a big boost from the fact that it uses inverted indexes for filtering. It also gets a boost from having a wide variety of approximate algorithms you can use if you want (for things like topN, count distinct, set difference/intersection, quantiles, etc). But straight scan performanc…

I was reading the details on inverted index usage in Druid, but what is described seems to be bitmap indexes, not inverted indexes.

Inverted indexes map distinct values in a column to a list of document ids containing the value. Bitmap indexes map distinct values to an array of booleans the same length as the number of documents, with true for presence and false for absence. Both index types can be highly compressed, of course.

Can you clarify what Druid is using?

Re: How Netflix uses Druid for realtime insights

#55

Post by frankmcsherry »

polskibus wrote: post #23

I wonder if materialize.io could handle such workloads at this stage.

Hi, I work at Materialize!

It's early days, but I can confirm that it's a bit hard to know out of the gates whether Materialize can handle such workloads. In particular, unless I missed it the query workload isn't really discussed in the post, and that is all that matters.

Druid makes a few compromises that Materialize isn't willing to make. For example AFAIK Druid doesn't support deletions or modifications in "realtime", which means they can track "min/max" style queries much more efficiently, but they lose out on some other features (connecting pipelines of these views where you might have to "retract" a prior min or max). This could certainly let it scale more, but also means that you learn a few months down the road that it doesn't do everything you hoped it would.

No joins in Druid is also reportedly a bit of a pain. Instead, you get to pre-denormalize your data. No need with Materialize (and hey, feel free to push down reductions through the joins, rather than denormalize and then reduce).

Short version, Materialize is definitely a "higher sophistication" data processing play. Whether that works out well remains to be seen! I'll slap up a blog post later today with a worked example.

Re: How Netflix uses Druid for realtime insights

#56

Post by quodlibetor »

jeffffff wrote: post #33
polskibus wrote: post #23

I wonder if materialize.io could handle such workloads at this stage.

materialize has to be able to keep all of its state in memory which only makes sense from a cost perspective for workloads which have a high value to space ratio. fine grained user behavior data typically just isn't that valuable.

> materialize has to be able to keep all of its state in memory

For now. We have a pretty good idea of what needs to be done to shed state to disk, and have designed to be able to implement it. We expect it to "just" be a matter of putting in the engineering effort.

Re: How Netflix uses Druid for realtime insights

#57

Post by quodlibetor »

derefr wrote: post #31
adamfeldman wrote: post #28

Earlier quoted context omitted.

Materialize, today, looks to be limited to running on a single node. https://materialize.io/docs/overview/architecture Netflix's workload would likely exhaust the resources of even a vertically-scaled single node.

Mind you, given the “Timely Dataflow” abstraction Materialize operates on top of, if you give it a query that only requires certain result rows from one of its mat views, then Materialize is only going to compute the intermediate rows (and, further back, retrieve the source rows) required to “render” the particular result-rows you ask for. (Sort of like how Excel, in memory-constrained conditions, only computes the i…

> if you give it a query that only requires certain result rows from one of its mat views, then Materialize is only going to compute the intermediate rows

This is absolutely correct!

> You can just have a bunch of “the same” Materialize node (i.e. every node just freestanding clone of a template node, with exactly the same sources and matviews) and then hit them with the parts of a map-reduce query

This should work, but we have been thinking about it/testing it differently internally. In general you should be able to create materialized views on different "shards" that have different `where` conditions, allowing you to control memory that way. This technique does require data that is actually partitionable in this way, same as it must be partitionable in mapreduce.

> this is all irrelevant the moment you write a query that needs a pure reduce

Of course, with materialize's sinks you can spin up a bunch of `materialized`s and connect them for a final reduce after data has gone through e.g. kafka or shared files. Being able to write joins and aggregates across heterogenous sources makes this kind of workload actually pretty pleasant.

Re: How Netflix uses Druid for realtime insights

#58

Post by rad_gruchalski »

tjoff wrote: post #52
manigandham wrote: post #51

Earlier quoted context omitted.

People don't care though, because they can make their own choices but they're clearly still sharing and reading on medium. You're asking for HN to arbitrarily enforce your wishes on top of what users are doing.

They do, and it is hardly my wishes alone. If 1% don't care and share the crap the other 99% still have to suffer for it.

That’s assuming the 99% care to suffer. I don‘t, I see a post from wall street journal, I don‘t click. I know it’s a paywall. People have a choice to click or not.

I do agree that medium is a bit of a mess where one is „less privileged“ when signed in.

But what we should be asking for is the companies like Netflix not using medium in a first place.

Re: How Netflix uses Druid for realtime insights

#59

Post by tjoff »

rad_gruchalski wrote: post #58
tjoff wrote: post #52

Earlier quoted context omitted.

They do, and it is hardly my wishes alone. If 1% don't care and share the crap the other 99% still have to suffer for it.

That’s assuming the 99% care to suffer. I don‘t, I see a post from wall street journal, I don‘t click. I know it’s a paywall. People have a choice to click or not. I do agree that medium is a bit of a mess where one is „less privileged“ when signed in. But what we should be asking for is the companies like Netflix not using medium in a first place.

Ban medium from HN and companies like netflix will switch in 5 minutes.

Re: How Netflix uses Druid for realtime insights

#60

Post by gianm »

barrkel wrote: post #54
gianm wrote: post #36

Earlier quoted context omitted.

Druid committer here. (Also, I think we've met before in SF!) One thing I wanted to add with regard to performance. Druid does indeed get a big boost from the fact that it uses inverted indexes for filtering. It also gets a boost from having a wide variety of approximate algorithms you can use if you want (for things like topN, count distinct, set difference/intersection, quantiles, etc). But straight scan performanc…

I was reading the details on inverted index usage in Druid, but what is described seems to be bitmap indexes, not inverted indexes. Inverted indexes map distinct values in a column to a list of document ids containing the value. Bitmap indexes map distinct values to an array of booleans the same length as the number of documents, with true for presence and false for absence. Both index types can be highly compressed,…

Logically, an array of booleans and a set of integers are equivalent. So in the Druid developer community we usually use the terms interchangeably. But to be precise, our indexes are all stored as bitmaps and compressed with bitmap compression libraries.
Post reply on HN