Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streaming Cassandra at WePay

wecode.wepay.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: Streaming Cassandra at WePay

#11

Post by georgewfraser »

matchagreentea wrote: post #7
georgewfraser wrote: post #3

There's some excellent choices in this design, but using Kafka in the middle makes no sense. The Cassandra change-data segments are written in batches, and BigQuery loads in batches. So the author is splitting up change-data segments into individual rows, feeding those rows into Kafka, and then re-batching them on the BQ side for ingestion. The use of a streaming system adds a tremendous amount of complexity for no b…

OP here. Thanks for the feedback! I totally agree with you that using a streaming approach seems like an overkill. Actually making the pipeline truly real-time is part of our future work for Cassandra CDC (This is mentioned in Part 2 of this blogpost, which is expected to be published next week). In Cassandra 4.0, the CDC feature is improved to allow real-time parsing of commit logs. We are running 3.X in production,…

What does the 4.0 api look like? I would home that they would expose a high-level interface for change data, similar to SQL Server or Snowflake streams. Most databases leave this as an afterthought, which is so unfortunate.

Re: Streaming Cassandra at WePay

#12

Post by haecceity »

fasthandle wrote: post #10

This was a remarkably detailed and honestly confusing post from Tencent, until I noticed the Chase logo at the top, and it was not by Tencent. (The WeChat team in Tencent there is very small, close, integrated). Pretty major name clash here, WeChat pay aka WePay, perhaps WeChat Pay, and well, another WePay. For those not familiar, WeChat, somewhat a WhatsApp+/Facebood Messenger of China, probably earlier mover howeve…

Why do you think cash is preferable to electronic payment? I personally hate carrying change.

Re: Streaming Cassandra at WePay

#13

Post by _cairn »

voidfunc wrote: post #9
dillonmckay wrote: post #5

Earlier quoted context omitted.

I recently had a junior dev propose using Kafka as a business ‘event’ system, to trigger or be triggered by webhooks, in order to integrate our API with a third party CRM.

What's your point? Your comment comes off as dismissive and incomplete. Why was the juniors suggestion bad? It may have totally been a garbage suggestion, but if it was explain the reasoning.

Not the author of the relevant post but assuming he's tailing off of the first response I can only surmise they meant that in their case Kafka was unnecessary/a bad choice due to the restraints posed (<5 minute delay, streaming input, etc). Agree that without context it's an incomplete comment that does come off as dismissive, ("junior dev") seems to be meant as belittling/insulting.

Re: Streaming Cassandra at WePay

#14

Post by j-tyler »

fasthandle wrote: post #10

This was a remarkably detailed and honestly confusing post from Tencent, until I noticed the Chase logo at the top, and it was not by Tencent. (The WeChat team in Tencent there is very small, close, integrated). Pretty major name clash here, WeChat pay aka WePay, perhaps WeChat Pay, and well, another WePay. For those not familiar, WeChat, somewhat a WhatsApp+/Facebood Messenger of China, probably earlier mover howeve…

WePay predates WeChat by three years.

https://en.wikipedia.org/wiki/WePay

https://en.wikipedia.org/wiki/WeChat

Re: Streaming Cassandra at WePay

#15

Post by matchagreentea »

georgewfraser wrote: post #11
matchagreentea wrote: post #7

Earlier quoted context omitted.

OP here. Thanks for the feedback! I totally agree with you that using a streaming approach seems like an overkill. Actually making the pipeline truly real-time is part of our future work for Cassandra CDC (This is mentioned in Part 2 of this blogpost, which is expected to be published next week). In Cassandra 4.0, the CDC feature is improved to allow real-time parsing of commit logs. We are running 3.X in production,…

What does the 4.0 api look like? I would home that they would expose a high-level interface for change data, similar to SQL Server or Snowflake streams. Most databases leave this as an afterthought, which is so unfortunate.

http://cassandra.apache.org/doc/latest/operating/cdc.html

The high-level idea is hard-linking the commit log segment file in cdc directory as it is being written + have a separate index file to keep track of offset.

Re: Streaming Cassandra at WePay

#16

Post by pfarnsworth »

matchagreentea wrote: post #15
georgewfraser wrote: post #11

Earlier quoted context omitted.

What does the 4.0 api look like? I would home that they would expose a high-level interface for change data, similar to SQL Server or Snowflake streams. Most databases leave this as an afterthought, which is so unfortunate.

http://cassandra.apache.org/doc/latest/operating/cdc.html The high-level idea is hard-linking the commit log segment file in cdc directory as it is being written + have a separate index file to keep track of offset.

Relying on an eventual consistency database sounds like a huge challenge in terms of getting absolute consistency between Cassandra and Kafka. There seems to be many edge cases that could cause the data in Kafka to get out of sync, it doesn't sound very reliable. Especially if you want to rely on Kafka for event sourcing later on, which is sounds like you're planning on in the future. For a payments company that sounds very risky.

What happens if you write a row to Cassandra but that row is invalidated because of failed writes to the other nodes in the cluster (ex. the write was QUORUM and it succeeded on the node that you're reading from but failed on all other nodes). In Cassandra, when an error occurs, there are no rollbacks, and the row gets read-repaired. It sounds like that row will get outputted to Kafka. How do you "disappear" data written into Kafka if it gets read-repaired?

Re: Streaming Cassandra at WePay

#17

Post by benjaminwootton »

Curious if you also looked at Aurora over Cassandra being as you are in the AWS ecosystem?

We are an AWS aligned consultancy who are seeing lots of take up of Aurora as it’s MySQL API compatible but with better performance and fully managed.

There’s an interesting pattern here using Lambda for CDC between Aurora and Redshift, though not sure how the cost would scale at WePay scale https://aws.amazon.com/blogs/big-data/combine-transactional-...

Re: Streaming Cassandra at WePay

#18

Post by chucky_z »

benjaminwootton wrote: post #17

Curious if you also looked at Aurora over Cassandra being as you are in the AWS ecosystem? We are an AWS aligned consultancy who are seeing lots of take up of Aurora as it’s MySQL API compatible but with better performance and fully managed. There’s an interesting pattern here using Lambda for CDC between Aurora and Redshift, though not sure how the cost would scale at WePay scale https://aws.amazon.com/blogs/big-dat…

Aurora and Redshift do not scale well enough for companies at their scale (only 64TB for Aurora last I checked, and Redshift falls over near the 100TB level). They'd be looking at DynamoDB.

Re: Streaming Cassandra at WePay

#19

Post by matchagreentea »

pfarnsworth wrote: post #16
matchagreentea wrote: post #15

Earlier quoted context omitted.

http://cassandra.apache.org/doc/latest/operating/cdc.html The high-level idea is hard-linking the commit log segment file in cdc directory as it is being written + have a separate index file to keep track of offset.

Relying on an eventual consistency database sounds like a huge challenge in terms of getting absolute consistency between Cassandra and Kafka. There seems to be many edge cases that could cause the data in Kafka to get out of sync, it doesn't sound very reliable. Especially if you want to rely on Kafka for event sourcing later on, which is sounds like you're planning on in the future. For a payments company that soun…

great question actually. There is a nuance in cassandra regarding to read-repair. If you issue a QUORUM read and the read itself failed (say, because 2/3 nodes are down), but succeeded on 1 node; and later on you read again when all three nodes are up, the failed write will actually be resurrected during read repair due to last write wins conflict resolution.

This is why it is recommended to always retry when a write fails. This way data might arrive at BQ slightly before cassandra itself for a failed write, but eventually they will be consistent.

That said, at WePay cassandra is being used by services which do not have to be strongly consistent at the moment. Not to say it’s not possible, as there are companies that use Cassandra’s light weight transaction feature for CAS operations.

As for consistency between kafka and cassandra, agree that its hard to make it 100% consistent (due to a number of limitations cassandra poses). This is why we tend to be pretty careful about which services should use cassandra, and specify clearly what is the expectation and SLA for cassandra cdc.

Re: Streaming Cassandra at WePay

#20

Post by easytiger »

haecceity wrote: post #12
fasthandle wrote: post #10

This was a remarkably detailed and honestly confusing post from Tencent, until I noticed the Chase logo at the top, and it was not by Tencent. (The WeChat team in Tencent there is very small, close, integrated). Pretty major name clash here, WeChat pay aka WePay, perhaps WeChat Pay, and well, another WePay. For those not familiar, WeChat, somewhat a WhatsApp+/Facebood Messenger of China, probably earlier mover howeve…

Why do you think cash is preferable to electronic payment? I personally hate carrying change.

One big benefit is decentralisation.
Post reply on HN