Streaming Cassandra at WePay

#1

Post by matchagreentea » Sun, Jul 14, 2019, 12:21 AM UTC

Streaming Cassandra at WePay
wecode.wepay.com

Re: Streaming Cassandra at WePay

#2

Post by [deleted] » Sun, Jul 14, 2019, 12:50 AM UTC

[deleted]

Re: Streaming Cassandra at WePay

#3

Post by georgewfraser » Sun, Jul 14, 2019, 2:09 AM UTC

There's some excellent choices in this design, but using Kafka in the middle makes no sense. The Cassandra change-data segments are written in batches, and BigQuery loads in batches. So the author is splitting up change-data segments into individual rows, feeding those rows into Kafka, and then re-batching them on the BQ side for ingestion.

The use of a streaming system adds a tremendous amount of complexity for no benefit, because the original data source is batch. A simpler solution would be to process the data in 5-minute batches and use a BigQuery MERGE statement to apply that "patch" to the existing table.

You really shouldn't use Kafka unless you have a true streaming data source and a hard latency requirement of < 5 minutes. It's much easier to work with data in small batches, using local disk and blob stores like S3 as your queue.

Re: Streaming Cassandra at WePay

#4

Post by [deleted] » Sun, Jul 14, 2019, 2:36 AM UTC

[deleted]

Re: Streaming Cassandra at WePay

#5

Post by dillonmckay » Sun, Jul 14, 2019, 2:41 AM UTC

georgewfraser wrote: post #3

There's some excellent choices in this design, but using Kafka in the middle makes no sense. The Cassandra change-data segments are written in batches, and BigQuery loads in batches. So the author is splitting up change-data segments into individual rows, feeding those rows into Kafka, and then re-batching them on the BQ side for ingestion. The use of a streaming system adds a tremendous amount of complexity for no b…

I recently had a junior dev propose using Kafka as a business ‘event’ system, to trigger or be triggered by webhooks, in order to integrate our API with a third party CRM.

Re: Streaming Cassandra at WePay

#6

Post by tony1234 » Sun, Jul 14, 2019, 2:45 AM UTC

good job

Re: Streaming Cassandra at WePay

#7

Post by matchagreentea » Sun, Jul 14, 2019, 2:47 AM UTC

georgewfraser wrote: post #3

There's some excellent choices in this design, but using Kafka in the middle makes no sense. The Cassandra change-data segments are written in batches, and BigQuery loads in batches. So the author is splitting up change-data segments into individual rows, feeding those rows into Kafka, and then re-batching them on the BQ side for ingestion. The use of a streaming system adds a tremendous amount of complexity for no b…

OP here.

Thanks for the feedback! I totally agree with you that using a streaming approach seems like an overkill. Actually making the pipeline truly real-time is part of our future work for Cassandra CDC (This is mentioned in Part 2 of this blogpost, which is expected to be published next week). In Cassandra 4.0, the CDC feature is improved to allow real-time parsing of commit logs. We are running 3.X in production, however we plan to add 4.0 support in the future.

Re: Streaming Cassandra at WePay

#8

Post by nightshadetrie » Sun, Jul 14, 2019, 3:43 AM UTC

georgewfraser wrote: post #3

There's some excellent choices in this design, but using Kafka in the middle makes no sense. The Cassandra change-data segments are written in batches, and BigQuery loads in batches. So the author is splitting up change-data segments into individual rows, feeding those rows into Kafka, and then re-batching them on the BQ side for ingestion. The use of a streaming system adds a tremendous amount of complexity for no b…

Alot of engineers at WePay come from LinkedIn, so there's a special love for Kafka.

Re: Streaming Cassandra at WePay

#9

Post by voidfunc » Sun, Jul 14, 2019, 3:50 AM UTC

dillonmckay wrote: post #5

georgewfraser wrote: post #3

There's some excellent choices in this design, but using Kafka in the middle makes no sense. The Cassandra change-data segments are written in batches, and BigQuery loads in batches. So the author is splitting up change-data segments into individual rows, feeding those rows into Kafka, and then re-batching them on the BQ side for ingestion. The use of a streaming system adds a tremendous amount of complexity for no b…

I recently had a junior dev propose using Kafka as a business ‘event’ system, to trigger or be triggered by webhooks, in order to integrate our API with a third party CRM.

What's your point? Your comment comes off as dismissive and incomplete. Why was the juniors suggestion bad? It may have totally been a garbage suggestion, but if it was explain the reasoning.

Re: Streaming Cassandra at WePay

#10

Post by fasthandle » Sun, Jul 14, 2019, 4:24 AM UTC

This was a remarkably detailed and honestly confusing post from Tencent, until I noticed the Chase logo at the top, and it was not by Tencent. (The WeChat team in Tencent there is very small, close, integrated).

Pretty major name clash here, WeChat pay aka WePay, perhaps WeChat Pay, and well, another WePay.

For those not familiar, WeChat, somewhat a WhatsApp+/Facebood Messenger of China, probably earlier mover however riding from QQ, transact billions of yuan in payments every day and together with AliPay is promoting a cashless society (which I think is wrong, I like cash).