Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Netflix uses Druid for realtime insights

netflixtechblog.com

Post reply on HN
Print view View original

21–30 of 60 posts

Re: How Netflix uses Druid for realtime insights

#21

Post by danielbln »

jpgvm wrote: post #14
modarts wrote: post #8

Earlier quoted context omitted.

Can confirm clickhouse is generally faster across most typical workloads

Clickhouse is much more resource efficient in many cases but is less flexible and importantly extensible than Druid. Druid can easily be extended through available 3rd party extensions and you can write your own to implement custom serialisation formats, aggregations, connect to new streaming systems, read directly from whatever cold storage you have etc. In the Clickhouse model you have to work out a lot more of tha…

In our experience, on a significantly smaller scale, Clickhouse is vastly easier to operate compared to Druid, with all of its various components that all have various knobs and dials to configure and have to be orchestrated.

Re: How Netflix uses Druid for realtime insights

#22

Post by tjoff »

rad_gruchalski wrote: post #18
zerop wrote: post #17

Earlier quoted context omitted.

I am already logged in and it still is a featured story. It's counted against 3 features story I can read in a month. So it is behind paywall. C

Then log out. It’s not a paywall. Log out and you can read it. Medium is playing you.

Proper response: delete account and don't visit medium again.

Even better: sites such as hn should never allow links to sites employing dark patterns.

Re: How Netflix uses Druid for realtime insights

#24

Post by 2rsf »

devmunchies wrote: post #3

> During software updates, we enable the new version for a subset of users and ... compare how the new version is performing vs the previous version. Any regression in the metrics gives us a signal to abort the update and revert How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run?

regressions due to "users are used to" usually goes away after a while, you'll see a temporary dip in the results.

If you are convinced it's better for the long tun then what's the point in measuring ?

Re: How Netflix uses Druid for realtime insights

#25

Post by shaklee3 »

jpgvm wrote: post #14
modarts wrote: post #8

Earlier quoted context omitted.

Can confirm clickhouse is generally faster across most typical workloads

Clickhouse is much more resource efficient in many cases but is less flexible and importantly extensible than Druid. Druid can easily be extended through available 3rd party extensions and you can write your own to implement custom serialisation formats, aggregations, connect to new streaming systems, read directly from whatever cold storage you have etc. In the Clickhouse model you have to work out a lot more of tha…

I think what you're getting at can be accomplished with materialized views in clickhouse now. Most queries that might be fast with inverted indices can be solved that way.

Also, I don't think they use bloom filters for the index as far as I can tell from the documentation. There is certainly an option to use a bloom filter aggregator on a table for faster counts, but it's not the default. If you're referring to the fact that count () is not precise, there's a exact count function too. This is my speculation, though, and you may be fight.

Re: How Netflix uses Druid for realtime insights

#26

Post by jpgvm »

danielbln wrote: post #21
jpgvm wrote: post #14

Earlier quoted context omitted.

Clickhouse is much more resource efficient in many cases but is less flexible and importantly extensible than Druid. Druid can easily be extended through available 3rd party extensions and you can write your own to implement custom serialisation formats, aggregations, connect to new streaming systems, read directly from whatever cold storage you have etc. In the Clickhouse model you have to work out a lot more of tha…

In our experience, on a significantly smaller scale, Clickhouse is vastly easier to operate compared to Druid, with all of its various components that all have various knobs and dials to configure and have to be orchestrated.

Yes this is definitely also true.

Druid complexity is coming down a bit compared to where it started. These days you need brokers, middlemanagers and historicals - for queries, ingestion and storage respectively.

In the past to do batch ingestion it also required Hadoop but there is now a native parallel batch ingestion system that runs on the middlemanagers as worker tasks that can read from S3/GCS/existing Druid segments.

Druid is by far the more complex but you get a lot for it and with k8s it's not as hard to run/manage as it was in the past.

Re: How Netflix uses Druid for realtime insights

#27

Post by jpgvm »

shaklee3 wrote: post #25
jpgvm wrote: post #14

Earlier quoted context omitted.

Clickhouse is much more resource efficient in many cases but is less flexible and importantly extensible than Druid. Druid can easily be extended through available 3rd party extensions and you can write your own to implement custom serialisation formats, aggregations, connect to new streaming systems, read directly from whatever cold storage you have etc. In the Clickhouse model you have to work out a lot more of tha…

I think what you're getting at can be accomplished with materialized views in clickhouse now. Most queries that might be fast with inverted indices can be solved that way. Also, I don't think they use bloom filters for the index as far as I can tell from the documentation. There is certainly an option to use a bloom filter aggregator on a table for faster counts, but it's not the default. If you're referring to the f…

Yeah bloom filters aren't used by default my bad, by default you get no indices at all! I was thinking about the tokenbf/ngrambf indices. These help a ton in improving more sparse queries.

I will need to check out the materialised views. :)

Re: How Netflix uses Druid for realtime insights

#28

Post by adamfeldman »

polskibus wrote: post #23

I wonder if materialize.io could handle such workloads at this stage.

Materialize, today, looks to be limited to running on a single node. https://materialize.io/docs/overview/architecture

Netflix's workload would likely exhaust the resources of even a vertically-scaled single node.

Re: How Netflix uses Druid for realtime insights

#30

Post by yingw787 »

PedroBatista wrote: post #29

"To keep reading this story, create a free account." Seriously? I know it's the Medium hustle but someone at Netflix should know better.

I think one cool feature Medium can introduce is enterprise permissioning. If you have an enterprise plan and pay Medium oodles of money, you should maintain your own permissioning control and who sees your articles while still having access to the Medium social network. Best of both worlds.
Post reply on HN