Mike Perham, Creator of Sidekiq: From Employment to Independence • Hacker News Board

Re: Mike Perham, Creator of Sidekiq: From Employment to Independence

#91

Post by phamilton » Fri, Apr 14, 2023, 11:34 PM UTC

mperham wrote: post #87

phamilton wrote: post #8

Earlier quoted context omitted.

I have no problem paying for the Pro version, but one if its marketing pitches is "enhanced reliability", which is a wild marketing spin on "the free version will lose jobs in fairly common scenarios". In sidekiq without super_fetch (a paid feature), any jobs in progress when a worker crashes are lost forever. If a worker merely encounters an exception the job will be put back on the queue and retried but a crash mea…

The free version acts exactly like Resque, the previous market leader in Ruby background jobs. If it was good enough reliability for GitHub and Shopify to use for years, it was good enough for Sidekiq OSS too. Here's Resque literally using `lpop` which is destructive and will lose jobs. https://github.com/resque/resque/blob/7623b8dfbdd0a07eb04b19...

> If it was good enough reliability for GitHub and Shopify to use for years, it was good enough for Sidekiq OSS too.

Great point, and thanks for chiming in. I wonder if containerization has made this more painful (due to cgroups and OOMs). The comments here are basically some people saying it's never been a problem for them and some people saying they encounter it a lot (in containerized environments) and have had to add mitigations.

Either way, my observation is a lot of people not paying for Sidekiq Pro should. I hope you can agree with that.

Re: Mike Perham, Creator of Sidekiq: From Employment to Independence

#92

Post by KyeRussell » Fri, Apr 14, 2023, 11:59 PM UTC

danielrhodes wrote: post #40

rektide wrote: post #24

Earlier quoted context omitted.

From the JS community, this is sadly the level of snark we expect. It doesn't have to be like this. Your community tried to be better once. Have you forgotten? https://jasonfleetwoodboldt.com/courses/stepping-up-rails/ma...

There seems to be an ever-present high-level of defensiveness from the Ruby community over JS/React. Rails has had a profound impact on web development, promoting fast starts and shipping quickly. Companies like Shopify and Twitter benefited greatly from this. But the framework is 18 years old. The web that it was created for has changed significantly.

Probably DHH’s continued prominence in the community.

Re: Mike Perham, Creator of Sidekiq: From Employment to Independence

#93

Post by ZephyrBlu » Sat, Apr 15, 2023, 12:13 AM UTC

phamilton wrote: post #52

ZephyrBlu wrote: post #44

Earlier quoted context omitted.

Let me get this straight, you're complaining about eight 9s of reliability? 50,000,000 * 7 = 350,000,000 2 / 350,000,000 = 0.000000005714286 1 - (2 / 350,000,000) = 0.999999994285714 = 99.999999% > It's not a ton, but it's not zero. And when it comes to data durability the difference between zero and not zero is usually all that matters. If your system isn't resilient to 2 in 350,000,000 jobs failing I think there is…

This isn't about 2 in 350,000,000 jobs failing. It's about 2 jobs disappearing entirely. It's not reliability we're talking about, it's about durability. For reference, S3 has eleven 9s of durability. Every major queuing system solves this problem. RabbitMQ uses unacknowledged messages which are pinned to a tcp connection, so when that connection drops before acknowledging them they get picked up by another worker. S…

If you want to focus on durability then I think your complaint makes even less sense. Somehow I doubt S3 is primarily backed by Redis.

I think it's fair to assume that something backed by Redis is not durable by default because that's not what Redis is known for, whereas the other options you listed are known for their resiliency and durability. I wouldn't view Sidekiq as a similar product to RabbitMQ and SQS.

Also, Sidekiq Pro uses more advanced Redis features to enable super_fetch lending to the assumption that by default Redis is not durable: https://www.bigbinary.com/blog/increase-reliability-of-backg....

Re: Mike Perham, Creator of Sidekiq: From Employment to Independence

#94

Post by philsnow » Sat, Apr 15, 2023, 12:42 AM UTC

andris9 wrote: post #56

Mike Perham's Sidekiq was what I modeled my own business, EmailEngine, after. I took my decade's worth of expertise in email protocol implementations (I also run the open-source Nodemailer library) and built an email gateway app my customers can download and run on their machines or servers. Just like Sidekiq Pro, the customers need an active subscription to run it, but the app runs on their premises, and they mainta…

You have a bullet point about compliance / hosting concerns. I'm curious, do you have any idea how many of your customers care about that?

Do you have any concern that a customer is going to look under the covers and re-implement your product? I would guess the overwhelming majority of customers are going to be happy to just consume the API and go about their own business, but some products attract copycats, and handing out code / build artifacts makes that easier.

The tradeoff you've made makes a ton of sense if you're not concerned about copycats.

Re: Mike Perham, Creator of Sidekiq: From Employment to Independence

#95

Post by andris9 » Sat, Apr 15, 2023, 4:34 AM UTC

philsnow wrote: post #94

andris9 wrote: post #56

Mike Perham's Sidekiq was what I modeled my own business, EmailEngine, after. I took my decade's worth of expertise in email protocol implementations (I also run the open-source Nodemailer library) and built an email gateway app my customers can download and run on their machines or servers. Just like Sidekiq Pro, the customers need an active subscription to run it, but the app runs on their premises, and they mainta…

You have a bullet point about compliance / hosting concerns. I'm curious, do you have any idea how many of your customers care about that? Do you have any concern that a customer is going to look under the covers and re-implement your product? I would guess the overwhelming majority of customers are going to be happy to just consume the API and go about their own business, but some products attract copycats, and hand…

Compliance issues are a real concern. For example, I have some customers who switched from Nylas to EmailEngine because Nylas closed their Canada DC, even though EmailEngine is an inferior product. I'm not really worried about copycats, as it's a niche and complex project. There are literally hundreds of email RFCs, some as long as a book, and you would need to have at least an overview of these. You can already use IMAP client libraries in any language, but what EmailEngine does is cover all the weird edge cases, like handling sequence numbers as identifiers. Those who have the skill likely have better things to do, and those who don't probably find easier things to copy.

Re: Mike Perham, Creator of Sidekiq: From Employment to Independence

#96

Post by josegonzalez » Sat, Apr 15, 2023, 6:26 AM UTC

danielrhodes wrote: post #40

rektide wrote: post #24

Earlier quoted context omitted.

From the JS community, this is sadly the level of snark we expect. It doesn't have to be like this. Your community tried to be better once. Have you forgotten? https://jasonfleetwoodboldt.com/courses/stepping-up-rails/ma...

There seems to be an ever-present high-level of defensiveness from the Ruby community over JS/React. Rails has had a profound impact on web development, promoting fast starts and shipping quickly. Companies like Shopify and Twitter benefited greatly from this. But the framework is 18 years old. The web that it was created for has changed significantly.

The framework was first conceived 18 years ago, but continues to evolve. The framework has also changed significantly to deal with functionality that modern web applications need, in much the same way the Javascript community has evolved to better suit the needs of the modern web application.

Re: Mike Perham, Creator of Sidekiq: From Employment to Independence

#97

Post by andris9 » Sat, Apr 15, 2023, 6:43 AM UTC

mperham wrote: post #74

andris9 wrote: post #56

Mike Perham's Sidekiq was what I modeled my own business, EmailEngine, after. I took my decade's worth of expertise in email protocol implementations (I also run the open-source Nodemailer library) and built an email gateway app my customers can download and run on their machines or servers. Just like Sidekiq Pro, the customers need an active subscription to run it, but the app runs on their premises, and they mainta…

Congratulations, that’s great. I’m always happy to hear of more successful indie developers.

Thank you for your inspiration!

Re: Mike Perham, Creator of Sidekiq: From Employment to Independence

#98

Post by satvikpendem » Sat, Apr 15, 2023, 8:41 AM UTC

andris9 wrote: post #65

alberth wrote: post #62

Earlier quoted context omitted.

Unsolicited feedback, always link to your business in your posts. I googled “emailengine” and it’s wasn’t super obvious which was your business since so much paid ads exist for that search.

Oh, yeah, I forgot my pitch. The link is https://emailengine.app - EmailEngine acts as a mail client, basically the same way Thunderbird runs on desktop, or the iPhone Mail on the phone, but instead of a GUI it has REST API and instead of desktop notifications it sends JSON webhooks. And instead of a single email account, it can manage thousands of accounts.

How does it compare to eg Nylas? I was looking into that for an email app I wanted to make, but they were enterprise only it seems. I also found your business but I didn't want to self host just to try it out.

Re: Mike Perham, Creator of Sidekiq: From Employment to Independence

#99

Post by satvikpendem » Sat, Apr 15, 2023, 8:46 AM UTC

philsnow wrote: post #94

andris9 wrote: post #56

Mike Perham's Sidekiq was what I modeled my own business, EmailEngine, after. I took my decade's worth of expertise in email protocol implementations (I also run the open-source Nodemailer library) and built an email gateway app my customers can download and run on their machines or servers. Just like Sidekiq Pro, the customers need an active subscription to run it, but the app runs on their premises, and they mainta…

You have a bullet point about compliance / hosting concerns. I'm curious, do you have any idea how many of your customers care about that? Do you have any concern that a customer is going to look under the covers and re-implement your product? I would guess the overwhelming majority of customers are going to be happy to just consume the API and go about their own business, but some products attract copycats, and hand…

If you're worried about copying the code, I could see having the self hosted artifact be compiled or otherwise obfuscated would probably deter most people who wanted to reverse engineer it, since it would take significantly more time and effort to open up a compiled app than something just made in JS for example.

If you're worried about copying the API design, well, the implementation is the hard part, not necessarily the API design, which the Google vs Oracle lawsuit also showed, interestingly enough.

Re: Mike Perham, Creator of Sidekiq: From Employment to Independence

#100

Post by satvikpendem » Sat, Apr 15, 2023, 8:51 AM UTC

andris9 wrote: post #95

philsnow wrote: post #94

Earlier quoted context omitted.

You have a bullet point about compliance / hosting concerns. I'm curious, do you have any idea how many of your customers care about that? Do you have any concern that a customer is going to look under the covers and re-implement your product? I would guess the overwhelming majority of customers are going to be happy to just consume the API and go about their own business, but some products attract copycats, and hand…

Compliance issues are a real concern. For example, I have some customers who switched from Nylas to EmailEngine because Nylas closed their Canada DC, even though EmailEngine is an inferior product. I'm not really worried about copycats, as it's a niche and complex project. There are literally hundreds of email RFCs, some as long as a book, and you would need to have at least an overview of these. You can already use…

Who are your main customer segments? I ask because I'm implementing Microsoft, Google and Apple email support for my app and Microsoft and Google have an API now, while Apple is still a pain. Beyond that, for custom email providers, I'm not sure who else is major enough to implement for. I guess I'm asking, couldn't your customer do something like I'm doing, to cover 80% of the use cases by using the Microsoft and Google APIs and spending some time using an IMAP client for Apple iCloud, and discarding the other smaller percentage of other email providers?

Granted, it still does kind of suck to implement even these three, but I'm sure it would likely suck less if I were part of a bigger enterprise who could afford Nylas for example, which I looked into but seemed too enterprisey for my needs.