Building real-time ETL pipelines with Apache Kafka

#1

Post by flippingbits » Fri, Feb 11, 2022, 1:44 PM UTC

Building real-time ETL pipelines with Apache Kafka
datacater.io

Re: Building real-time ETL pipelines with Apache Kafka

#2

Post by johanneskanybal » Fri, Feb 11, 2022, 2:29 PM UTC

basically an ad.

Re: Building real-time ETL pipelines with Apache Kafka

#3

Post by cam0 » Fri, Feb 11, 2022, 2:53 PM UTC

johanneskanybal wrote: post #2

basically an ad.

I wouldn't say that. Kafka is everywhere in the enterprise and it's adoption is still growing rapidly. This blog post is a decent overview for non-technical people at the very least.

Re: Building real-time ETL pipelines with Apache Kafka

#4

Post by r90t » Fri, Feb 11, 2022, 3:20 PM UTC

johanneskanybal wrote: post #2

basically an ad.

agree

Re: Building real-time ETL pipelines with Apache Kafka

#5

Post by iratewizard » Fri, Feb 11, 2022, 3:24 PM UTC

johanneskanybal wrote: post #2

basically an ad.

HN used to be strict about its "no self promotion" and "it must be interesting" rules. Dang probably got tired of constantly fighting about it. Now everyone with a blog treats HN like a dumping ground.

Edit: note that this is the CEO of the site. His account was inactive since 2012, and then in the past 7 months started dumping an ad every month here.

Re: Building real-time ETL pipelines with Apache Kafka

#6

Post by mfollert » Fri, Feb 11, 2022, 3:26 PM UTC

johanneskanybal wrote: post #2

basically an ad.

HN adblocker when?

Re: Building real-time ETL pipelines with Apache Kafka

#7

Post by j45 » Fri, Feb 11, 2022, 3:26 PM UTC

johanneskanybal wrote: post #2

basically an ad.

It might be ad like, but there’s a pretty handy diagram explaining Kafka to beginners in there.

I’ve been working to grok Kafka the past little while and that diagram, if accurate stands out from most of the blabber on extended YouTube videos going for ad revenue.

Re: Building real-time ETL pipelines with Apache Kafka

#8

Post by sumtechguy » Fri, Feb 11, 2022, 3:33 PM UTC

j45 wrote: post #7

johanneskanybal wrote: post #2

basically an ad.

It might be ad like, but there’s a pretty handy diagram explaining Kafka to beginners in there. I’ve been working to grok Kafka the past little while and that diagram, if accurate stands out from most of the blabber on extended YouTube videos going for ad revenue.

I always recommend the oreilly book 'kafka the definitive guide'. https://kafka.apache.org/books-and-papers

Re: Building real-time ETL pipelines with Apache Kafka

#9

Post by LittlePeter » Fri, Feb 11, 2022, 3:43 PM UTC

ETL on streams? To be pedantic, does that even make sense? Extract, transform and load... a batch of data. But a stream?

Re: Building real-time ETL pipelines with Apache Kafka

#10

Post by j45 » Fri, Feb 11, 2022, 4:48 PM UTC

sumtechguy wrote: post #8

j45 wrote: post #7

Earlier quoted context omitted.

It might be ad like, but there’s a pretty handy diagram explaining Kafka to beginners in there. I’ve been working to grok Kafka the past little while and that diagram, if accurate stands out from most of the blabber on extended YouTube videos going for ad revenue.

I always recommend the oreilly book 'kafka the definitive guide'. https://kafka.apache.org/books-and-papers

Thank you! Have so many oreillly books not sure why I didn’t think of it :)