Live data from Hacker News

Moving persistent data out of Redis

githubengineering.com

61–70 of 77 posts

Re: Moving persistent data out of Redis

#61
post #10

Earlier quoted context omitted.

FWIW we went through a very similar process to that documented here by Github (~3 months ago). It was entirely due to operational reasons and nothing to with shortcomings in Redis itself. MySQL was the master record for 99% of our data while Redis was the master record for the other 1% (as it happens it was also a kind of activity stream). Having the single 'master' reference for our data reduced complexity to a degr…

What method are you using to replicate from MySQL binlog to various other systems?

FWIW, I've used github.com/siddontang/go-mysql to successfully replicate from MySQL to DynamoDB. Currently not using GTIDs and looking into that next.

Re: Moving persistent data out of Redis

#63
post #62

Has Github stopped allowing free searching for code? All I get is "Must include at least one user, organization, or repository". I think that its a greater problem than the speed of its streams.

I believe this restriction was put in place to limit searching for accidentally committed AWS keys.

Re: Moving persistent data out of Redis

#64
post #3

Wow I'm learning today that Github used Redis for persistent data, now that they moved away :-) Anyway very happy that Redis helped to run such an important site. From the blog post it looks like that for certain things to move away from Redis was hard even if they are very skilled with MySQL, this is a good thing from the POV of Redis since it means that Redis allows to model certain things easily. However they want…

From the projects I've been involved in, the reason is simply that we don't want to have 2 persistent storage systems. There's a need for a fast cache system, and there's a need for a reliable – as in certainty above speed – database. The former is usually Redis, and the latter most often needs to be a full-blown SQL database to handle the required complexity of larger applications.

It's just easier to have one single source of truth. Please don't change Redis into a large SQL database. :)

Re: Moving persistent data out of Redis

#65
post #64
post #3

Wow I'm learning today that Github used Redis for persistent data, now that they moved away :-) Anyway very happy that Redis helped to run such an important site. From the blog post it looks like that for certain things to move away from Redis was hard even if they are very skilled with MySQL, this is a good thing from the POV of Redis since it means that Redis allows to model certain things easily. However they want…

From the projects I've been involved in, the reason is simply that we don't want to have 2 persistent storage systems. There's a need for a fast cache system, and there's a need for a reliable – as in certainty above speed – database. The former is usually Redis, and the latter most often needs to be a full-blown SQL database to handle the required complexity of larger applications. It's just easier to have one singl…

Thanks! No plans to change it into an SQL database :-) Actually the idea is to focus more in the caching/streaming area.

Re: Moving persistent data out of Redis

#66
post #21

We've recently had to move away from redis for persistent data storage at work too - opting instead to write a service layer ontop of cassandra for storing data. Redis was tremendous in our journey up there - but one of the shortcomings is that it isn't as easy to scale-up as cassandra is if you haven't designed your system to scale-up on redis from when it was built (which we didn't) - instead of re-architecting for…

yes

Re: Moving persistent data out of Redis

#67
post #54

Wonder why they didn't use Cassandra for this use case.

Last year I was setting up a trial of Cassandra for something, going through the usual swearing of a new tool not quite working as expected (eg by default picking a random port for inter-node communication)... and the next desk over, a non-tech colleague called Cassandra kept hearing me mutter angrily about 'cassandra' and wondered what she'd done. Whoops :)

Do you sit close to Ezekiel as well?

Though having someone sit near me not knowing what I am/we are working on would surprise me. But it does happen especially if there are hotdesks nearby for people from other offices to work on temporarily. I do swear loudly often so probably not a good choice to have those too near me...

Re: Moving persistent data out of Redis

#68

Earlier quoted context omitted.

just wondering why Mongo is considered a part of "JS". Is it because of the MEAN stack?

Part of it is JSON as the storage format. Another part is its Node driver. The whole API fit it well. It understood async programming. It felt JS-like. The input and output were JSON instances. Finally, yea, from what I know of him "web scale" did play part in the decision. Oh those heady days.

For data that's mostly to do with the the API provided by the particular mongodb driver, than mongodb itself. Mongo stores and transmits BSON, not JSON. Most mongo drivers expose an API that serialises your data to BSON for writes and wraps the BSON data with a JSON-like interface for reads.

Re: Moving persistent data out of Redis

#69
post #58

Earlier quoted context omitted.

>instead of re-architecting for a redis-cluster setup, we decided to move the component to a clustered microservice written in go, that sits as a memory-cache & write buffer infront of cassandra for hot, highly mutated data. Somehow setting up a Redis cluster and doing whatever you have to do to distribute/shard your keys effectively (which afaik is not much) does sound a little more efficient than rewriting a cluste…

My post above describes the main reason for moving from redis - the fact that data for inactive users doesn't need to be memory perpetually. :P

Cool. I look forward to the post that reveals the unique properties of Cassandra that ended up making it the most practical data store for your use case.

I understand that Cassandra et al exist to solve real problems that someone out there has experienced, and I seek to throw no shade on the great engineers who make these fine products. I am, however, somewhat dubious that these niche products are applicable in the vast majority of cases where they're deployed. I strongly believe, and I think the data would bare out on this, that when it gets down to brass tacks, most people are integrating such specialized tools into generic products to either a) make life at the office more exciting; b) beef up resume points for their next job application cycle; or c) both.

Someone in our company wrote a blog post pretending to justify the move to a niche datastore. He's very proud of it and makes several spurious, nonsensical justifications in it. The truth is that MySQL would've been many times more practical along all axes, except the one this guy cares most about, which involve his personal career ambitions.

This move was partially under the radar so objections couldn't be raised and full backups were not properly arranged. It cost the company a lot of money not only in time and infrastructure, but also in the recovery process that had to be undertaken by real data experts (or nearest we had at the time, at least) when the cluster was destroyed by one of his careless scripts. :)

Second nightmare, currently ongoing: shifting everything to docker/k8s, which, for just one example among a very long laundry list of complaints, only got support for directly addressing app servers behind a load balancer last month, as a beta feature (in k8s nomenclature, that's "Version 1.5 has a beta StatefulSets feature to make Pods in a ReplicaSet uniquely addressable from inside the cluster! Don't forget to make a Headless Service and Persistent Volume." Exhausted yet? Just wait.).

Why are we switching to something that lacks such basic functionality (we're like 3 versions behind, so we can't use it)? If I told you, I'd have to kill you, but it sure makes our resumes pretty.

I'm all for learning, experimentation, and doing things for fun. We are on Hacker News after all. I guess I've just developed a taste for a stable production ethos that, to co-opt a scriptural term, is not "blown about by every wind of [tech fad]". I crave a company that makes its decisions based on a significant and real cost-benefit analysis that shows substantial unique benefits and sufficient maturity to a tech before jumping on the bandwagon. I guess I just want some sanity.

As it stands, people just pretend that these justifications exist by making up some mumbo-jumbo about "dude JavaScript on the backend is like really event-driven, brah!"

Re: Moving persistent data out of Redis

#70
post #63
post #62

Has Github stopped allowing free searching for code? All I get is "Must include at least one user, organization, or repository". I think that its a greater problem than the speed of its streams.

I believe this restriction was put in place to limit searching for accidentally committed AWS keys.

AWS keys are already being detected and the account owners get notified when they are found. Private key material was a bigger issue at one point in time (i.e dotfiles/.ssh/id_rsa).
Post reply on HN