Sidekiq Pro is great, we're paying for it! 10k a year I think.
But for people who are interested in alternatives, I'd also suggest Good Job (runs on Postgresql).
Mike Perham, Creator of Sidekiq: From Employment to Independence
41–50 of 115 posts
Re: Mike Perham, Creator of Sidekiq: From Employment to Independence
#42Sidekiq is one of my favorite things about using Ruby, I include it in my short list every time a "Why use Ruby?" question gets asked. I miss it desperately when I work in other languages. I plan on giving Faktory a shot ASAP.
Mike has always struck me as a great guy, really the kind of person who you're happy to know is building one of your favorite tools. In a world of starving open source contributors and mega corps throwing around weight, his success with Sidekiq stands out. It makes me very happy to see.
Re: Mike Perham, Creator of Sidekiq: From Employment to Independence
#43Earlier quoted context omitted.
Idempotence doesn't solve this problem. The jobs are all idempotent. The problem is that jobs will never be retried if a crash occurs. This doesn't happen at a high rate, but it happens more than zero times per week for us. We pay for Sidekiq Pro and have superfetch enabled so we are protected. If we didn't do so we'd need to create some additional infra to detect jobs that were never properly run and re-run them.
Crashes are under your control though. They’re not caused by sidekiq. And you could always add your own crash recovery logic, as you say. To me that makes it a reasonable candidate for a pro feature. It’s hard to get this right though. No matter where the line gets drawn, free users will complain that they don’t get everything for free.
How are crashes under your control? Again they aren't talking about uncaught exceptions, but crashes. So maybe the server gets unplugged, the network disconnects, etc.
Re: Mike Perham, Creator of Sidekiq: From Employment to Independence
#44Earlier quoted context omitted.
how often do your workers crash? i rely heavily on sidekiq and don't think I see this very often, if ever.
We process around 50M sidekiq jobs a day across a few hundred workers on a heavily autoscaled infrastructure. Over the past week there were 2 jobs that would have been lost if not for superfetch. It's not a ton, but it's not zero. And when it comes to data durability the difference between zero and not zero is usually all that matters. Edit for additional color: One of the most common crashes we'll see is OutOfMemory…
Let me get this straight, you're complaining about eight 9s of reliability?
50,000,000 * 7 = 350,000,000
2 / 350,000,000 = 0.000000005714286
1 - (2 / 350,000,000) = 0.999999994285714 = 99.999999%
> It's not a ton, but it's not zero. And when it comes to data durability the difference between zero and not zero is usually all that matters.
If your system isn't resilient to 2 in 350,000,000 jobs failing I think there is something wrong with your system.
Re: Mike Perham, Creator of Sidekiq: From Employment to Independence
#45Earlier quoted context omitted.
Crashes are under your control though. They’re not caused by sidekiq. And you could always add your own crash recovery logic, as you say. To me that makes it a reasonable candidate for a pro feature. It’s hard to get this right though. No matter where the line gets drawn, free users will complain that they don’t get everything for free.
How are crashes under your control? Again they aren't talking about uncaught exceptions, but crashes. So maybe the server gets unplugged, the network disconnects, etc.
To me 'crash' means any unexpected termination, whether it's caused by an uncaught exception, OOM, or hardware/network issues.
I guess you can say that hardware issues on your host aren't under your control, but it's under your control to find a host that doesn't have these issues. And not even a full-on ACID database is going to be 100% reliable if you yank the power cord at the wrong moment.
Re: Mike Perham, Creator of Sidekiq: From Employment to Independence
#46Earlier quoted context omitted.
Well Sidekiq is free to use. It's only the pro version that he charges and the free version code is open source. I don't see the problem in having that kind of business model, it still allows the community to thrive and offers entreprises a way to have premium support. Plus it allows him to invest more time in maintaining the free version.
I have no problem paying for the Pro version, but one if its marketing pitches is "enhanced reliability", which is a wild marketing spin on "the free version will lose jobs in fairly common scenarios". In sidekiq without super_fetch (a paid feature), any jobs in progress when a worker crashes are lost forever. If a worker merely encounters an exception the job will be put back on the queue and retried but a crash mea…
I wish this was prominently documented. Most people new to Sidekiq have no idea that the job will be lost forever if you simply hard kill the worker. I have seen a couple of instances where the team had Sidekiq Pro, but they had not enabled reliable fetch because they were unaware of this problem
Re: Mike Perham, Creator of Sidekiq: From Employment to Independence
#47This is one of the best businesses I have heard of. Even 6-7 years back, Sidekiq was grossing $80k/month [1]. I imagine it must be much more now. No servers to maintain, no employees, minimal support work and Mike has the complete freedom to work on updates whenever he prefers. Almost entire revenue minus payment processing costs must be profit. Imagine making $1M / yr working only 10-20hrs per week (Don't wish to pr…
Hangfire and Oban are two other background job systems which have a commercial aspect. AFAIK Hangfire has been around almost as long as Sidekiq and is still actively supported.
I'm closer to $10m than $1m in annual revenue now.
My take on Sidekiq's secret sauce: a job system is a distributed system. Most of Sidekiq's commercial features are available as OSS gems but the complexity sneaks up on you as you integrate 3-6 of those features together. Building your own almost always leads to a worse system than the mature, well-debugged system which I have curated.
Re: Mike Perham, Creator of Sidekiq: From Employment to Independence
#48I guess Matsumoto should have charged for Ruby. I'm sure Mike Perham and Derek Kraan would have been glad to pay for that and that the Ruby community would be in a strong and healthy shape.
People need to eat man, doing something for free is a luxury not many possess. We can see how messed up the financial situation is for a lot of Open Source devs and their lives would be better if they were charging money instead of subsisting off grants and donations. I don't even necessarily disagree with your point that without being free those things wouldn't have taken off but we need to find a way to strike a ba…
I personally try to spend money or use ad supported anything that is open-source just to help someone else eat. It really hit home for me about five years ago when the author of a WoW addon[0] couldn't develop anymore because of his financial and life situation.
So many communities across the web rely on people putting in their spare hours for free just to enjoy things. Whether it's spreadsheets in Eve, Addons and Weak Auras in WoW, forum analysis posts, or whatever goes on in the depths of pvpoke, so much free labor underpins massive parts of the world today.
I would love something that I could donate x money to per month and then based on usage, have it dole out to all the content providers with perhaps a minimum per month. It just seems daunting to do that as a) not a crypto scheme and b.) across all the various creator landscapes.
[0]https://www.polygon.com/2018/9/25/17901552/world-of-warcraft...
Re: Mike Perham, Creator of Sidekiq: From Employment to Independence
#49Earlier quoted context omitted.
From the JS community, this is sadly the level of snark we expect. It doesn't have to be like this. Your community tried to be better once. Have you forgotten? https://jasonfleetwoodboldt.com/courses/stepping-up-rails/ma...
There seems to be an ever-present high-level of defensiveness from the Ruby community over JS/React. Rails has had a profound impact on web development, promoting fast starts and shipping quickly. Companies like Shopify and Twitter benefited greatly from this. But the framework is 18 years old. The web that it was created for has changed significantly.
It’s changing back. SPAs are on the way out. Check back in 5 or 10 years - Rails will still be here.
Re: Mike Perham, Creator of Sidekiq: From Employment to Independence
#50This is one of the best businesses I have heard of. Even 6-7 years back, Sidekiq was grossing $80k/month [1]. I imagine it must be much more now. No servers to maintain, no employees, minimal support work and Mike has the complete freedom to work on updates whenever he prefers. Almost entire revenue minus payment processing costs must be profit. Imagine making $1M / yr working only 10-20hrs per week (Don't wish to pr…
> I can't really outsource much of my support work because it is so technical and specialized.
Yeah, I had a Sidekiq question on StackOverflow and Mike Perham answered himself couple of days later.