Streaming Messages from Kafka into Redshift in Near Real-Time
engineeringblog.yelp.com
Streaming Messages from Kafka into Redshift in Near Real-Time
1–10 of 12 posts
Re: Streaming Messages from Kafka into Redshift in Near Real-Time
#2[deleted]
Re: Streaming Messages from Kafka into Redshift in Near Real-Time
#3With a rate of 300 small messages a second it was enough for me to write in 10k batches.
Had a small writer script that kept the buffer in memory and did batch insert.
Re: Streaming Messages from Kafka into Redshift in Near Real-Time
#4Cut out Kafka by writing directly to S3 and bulk loading from S3 directory (optimal for Redshift). The article never details what "near real-time" means, which is bothersome.
Re: Streaming Messages from Kafka into Redshift in Near Real-Time
#5Cut out Kafka by writing directly to S3 and bulk loading from S3 directory (optimal for Redshift). The article never details what "near real-time" means, which is bothersome.
According to the article, Yelp had 7 different data sources and similar number of targets.
If they wrote a loader for each combination, they'd end up with 49 combinations. Not to mention 7 loaders to write every time they add an app.
With Kafka - they just need to connect each thing to Kafka - 14 connectors instead of 49.
This is pretty much the scenario Kafka was invented for, and you get stream processing for free: https://engineering.linkedin.com/distributed-systems/log-wha...
Re: Streaming Messages from Kafka into Redshift in Near Real-Time
#6The proliferation of software using words with a strong, precise, pre-existing meaning is making some of these headlines difficult to read... My first impression was that there is a space telescope I was unaware of whose copious data was being converted into redshift measurements of galaxies. Sadly, it has nothing to do with space news. Not sure whether to laugh or sigh.
Re: Streaming Messages from Kafka into Redshift in Near Real-Time
#7The proliferation of software using words with a strong, precise, pre-existing meaning is making some of these headlines difficult to read... My first impression was that there is a space telescope I was unaware of whose copious data was being converted into redshift measurements of galaxies. Sadly, it has nothing to do with space news. Not sure whether to laugh or sigh.
Thank god I'm not the only one who was baffled at first by the headline.
Re: Streaming Messages from Kafka into Redshift in Near Real-Time
#8Cut out Kafka by writing directly to S3 and bulk loading from S3 directory (optimal for Redshift). The article never details what "near real-time" means, which is bothersome.
According to the article, Yelp had 7 different data sources and similar number of targets. If they wrote a loader for each combination, they'd end up with 49 combinations. Not to mention 7 loaders to write every time they add an app. With Kafka - they just need to connect each thing to Kafka - 14 connectors instead of 49. This is pretty much the scenario Kafka was invented for, and you get stream processing for free:…
You only need 1 loader for directory loading json. Yelp already has an ETL (and more transforms) for combinatorial normalization of format (from log files to events pretty much covers the spectrum).
Redshift will create columns (within some restrictions about nested arrays) which generally have to be avoided, however you get the data into redshift, from json. Kafka is a process/time wasteful step in almost every redshift loading scenario, given the current state of AWS services. Test for yourself over a few billion messages at various message sizes from 1k to 1M, if you get the chance.
Kafka is great for a message queue if you can't write to S3 directly or as a buffer to deal gracefully with S3 hiccups, for high frequency throughput to redshift.
Re: Streaming Messages from Kafka into Redshift in Near Real-Time
#9How would redshift compare to Yandexs' Clickhouse[1] for this kind of architecture?
Re: Streaming Messages from Kafka into Redshift in Near Real-Time
#10Google BigQuery has a Streaming API specifically for this reason. Up to 100,000 rows per second per table, available immediately for analysis. Interestingly, with BigQuery batch or stream ingest uses different resources than query, so your query performance doesn't degrade due to ingest.
(Work on Google Cloud)