Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streaming Cassandra at WePay

wecode.wepay.com

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Streaming Cassandra at WePay

#31

Post by agacera »

KaiserPro wrote: post #30

Interesting article! One question, how are your backups? We moved away from datastax for a number of reasons, one of the biggest was the lifecycle manager. For us, it silently failed to make backups, created backups that were unrecoverable, and other generally nasty stuff. The worst part was with every update, something broke that should have been caught be QA. It felt like nobody was regularly testing their backups,…

Can you please elaborate a little bit more of how you replaced it by postgres? Because it is strange that a single box of postgres in way less powerful instance type would perform the same as your cassandra cluster. This kind of seems that the first solution was way over engineered or was built for different requirements.

Re: Streaming Cassandra at WePay

#33

Post by marcus_holmes »

I'm really curious why MySQL was dropped instead of being optimised?

SQL databases can (and do) scale to high volumes, and would have avoided a lot of re-engineering. What was the final blocker for WePay that meant that optimising SQL could go no further?

Re: Streaming Cassandra at WePay

#34

Post by Thaxll »

marcus_holmes wrote: post #33

I'm really curious why MySQL was dropped instead of being optimised? SQL databases can (and do) scale to high volumes, and would have avoided a lot of re-engineering. What was the final blocker for WePay that meant that optimising SQL could go no further?

They don't really scale since MySQL / PG don't have built in features for horizontal scaling / sharding ect ... If you want to make it scale you have to do a lot of things manually / use third party tools / solutions.

Re: Streaming Cassandra at WePay

#35

Post by ledneb »

Looking at the "double-writing" option, this talk by Udi Dahan ("Reliable Messaging Without Distributed Transactions") could be of interest to you https://vimeo.com/showcase/3784159/video/111998645

It's still a "double write" but you may not need distributed transactions if you're happy with at-least-once writes to Kafka and deduplication on the consuming side.

Re: Streaming Cassandra at WePay

#36

Post by KaiserPro »

agacera wrote: post #31
KaiserPro wrote: post #30

Interesting article! One question, how are your backups? We moved away from datastax for a number of reasons, one of the biggest was the lifecycle manager. For us, it silently failed to make backups, created backups that were unrecoverable, and other generally nasty stuff. The worst part was with every update, something broke that should have been caught be QA. It felt like nobody was regularly testing their backups,…

Can you please elaborate a little bit more of how you replaced it by postgres? Because it is strange that a single box of postgres in way less powerful instance type would perform the same as your cassandra cluster. This kind of seems that the first solution was way over engineered or was built for different requirements.

Oh it was a total mistake, Fortunately it wasn't mine. But I did have to support it and migrate away from it.

Cassandra session are quite heavy, We have a large farm that spins up, does stuff and closes down. So thats the first problem. (yes we used kafka to pipeline the data in, and that worked well, but...)

It _used_ to be a very heavy write/read ratio. But as time went on, we needed to read more and more thing concurrently.

Because its "distributed" and basically a glorified token ring system, throughput drops dramatically as load increases.

We are not inserting that much data, just lots and lots of records. We then do a geospatial query later on to pull that data back. postGIS is far better at handling this, compared the datastax graph layer + solr(ie, the full datastax "stack" ).

But honestly, we could have coped with that, if the backups worked. That and shipping code with a 4 year old CVE that could have been easily remedied if they'd bothered to do an automated scan.

Every point release would involved 1-5 days of hard work from me. considering the support cost was > my wage, that stung quite a lot.

Re: Streaming Cassandra at WePay

#37

Post by fasthandle »

haecceity wrote: post #12
fasthandle wrote: post #10

This was a remarkably detailed and honestly confusing post from Tencent, until I noticed the Chase logo at the top, and it was not by Tencent. (The WeChat team in Tencent there is very small, close, integrated). Pretty major name clash here, WeChat pay aka WePay, perhaps WeChat Pay, and well, another WePay. For those not familiar, WeChat, somewhat a WhatsApp+/Facebood Messenger of China, probably earlier mover howeve…

Why do you think cash is preferable to electronic payment? I personally hate carrying change.

Cash is faster.

It is anonymous. It is private. It feels good.

It does not depend on an internet connection which is really important for low-probability high-impact events.

It does not depend on 3rd parties.

Re: Streaming Cassandra at WePay

#38

Post by marcinzm »

marcus_holmes wrote: post #33

I'm really curious why MySQL was dropped instead of being optimised? SQL databases can (and do) scale to high volumes, and would have avoided a lot of re-engineering. What was the final blocker for WePay that meant that optimising SQL could go no further?

SQL on it's own scales to a point, that point being however large you can make a single machine. Which can be quiet small depending on the data sizes in question. Beyond that you need to use something like Vitess (which the article mentions) but that comes with it's own overheads and caveats. You no longer have SQL really, you have Vitess. A couple years ago a company I was at tried to use Vitess but found too many edge cases and too much operational overhead.

Re: Streaming Cassandra at WePay

#39

Post by twic »

vijaybritto wrote: post #23

Sorry, Off topic here: Why are there three scrollbars for the window??!

Redundancy. Only one of the scrollbars is actually active at the moment, but if it fails, a leadership election will take place, and one of the others will take over as the active.

Re: Streaming Cassandra at WePay

#40

Post by chucky_z »

gregorygoc wrote: post #22
chucky_z wrote: post #18

Earlier quoted context omitted.

Aurora and Redshift do not scale well enough for companies at their scale (only 64TB for Aurora last I checked, and Redshift falls over near the 100TB level). They'd be looking at DynamoDB.

I don’t think any managed AWS service scales well for any bigger enterprise which have to handle a decent amount of traffic. Once you reach certain threshold you soon hit some throttling and limits BS and AWS solution is just to throw more money at them.

DynamoDB scales a lot. It falls into the "throw money at it," bucket though, and you need to have a pattern that shards well with it. If you have a reason they will basically always increase your limits, too.
Post reply on HN