Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Deep Dive into Solid Queue for Ruby on Rails

blog.appsignal.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: A Deep Dive into Solid Queue for Ruby on Rails

#21

Post by rubyfan »

charcircuit wrote: post #16
rubyfan wrote: post #8

Earlier quoted context omitted.

Don’t be silly. Bad choices are made in all sorts of languages and teams - this has nothing to do with language. High pressure situations can lead teams to make choices they don’t always foresee as bad until after they are paying the consequences.

Sure bad choices are made everywhere, but I was essentially claiming that when a community has a hammer, they will see nails.

Queues are not a ruby specific thing, nor are they particularly pervasive within Rails apps. Having a good framework to handle them doesn’t make it the only tool in the tool belt. On the contrary, the fact that Rails has good tools to fit many different types of system architecture needs is a counterpoint of your assertion.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#22

Post by WillPostForFood »

aaronblohowiak wrote: post #19
trevorhinesley wrote: post #9

Earlier quoted context omitted.

I’ve never felt like “throw everything into a queue” was a mindset within the Ruby community, nor have we done that at my companies. And multi-region is a business decision.

Resque was a staple for a long long time. In the jvm world, throw everything into Kafka is also a staple of a lot of "enterprise" shops. Or SQS for AWS places I've worked at. I think it is not a ruby language thing, but a certain kind of architecture thing.

True that it is not uncommon to use Sidekiq or Resque , but Rails 8 is going to be the first version to ship with a queuing system (SolidQueue), later this year. So queueing has been an add-on for 20 years. I don't think it is quite a staple.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#23

Post by zdragnar »

throwaway493943 wrote: post #14
nkraft11 wrote: post #2

I briefly worked at a YC company that was a ruby shop. Their answer to every performance problem was to stick it on a queue. There were, I don’t know, dozens of them. Then they decided they needed to be multi-region, because reasons. But the queues weren’t set up to be multi-region, so they built an entirely new service that’s job was to decide which queue in which region jobs needed to go on. So now you had jobs cri…

I'm going to be brave (but still use a throwaway) and ask the dumb question - what is wrong with putting things in queues to help with performance problems? If some endpoint is too slow to return a response to the frontend within a reasonable time, enqueueing it via a worker makes sense to me. That doesn't cover all performance issues but it handles a lot of them. You should also do things like optimize SQL queries,…

There's two primary areas that I've seen teams get bitten by this personally:

1) Designers don't understand that things are going to happen async, and the UI ends up wanting to make assumptions that everything is happening in real time. Even if it works with the current design, it's one small change away from being impossible to implement.

This is a general difficulty with working in eventually consistent systems, but if you're putting something in a queue because you're too lazy to optimize (rather than the natural complexity of the workload demanding it) you're going to be hurting yourself unnecessarily.

2) Errors get swallowed really easily. Instead of being properly reported to the team and surfaced to a user in a timely manner, the default setting of some configurations to just keep retrying the job later means if you're not monitoring closely you'll end up with tens of thousands of jobs retrying over and over at various intervals.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#24

Post by hobs »

throwaway493943 wrote: post #14
nkraft11 wrote: post #2

I briefly worked at a YC company that was a ruby shop. Their answer to every performance problem was to stick it on a queue. There were, I don’t know, dozens of them. Then they decided they needed to be multi-region, because reasons. But the queues weren’t set up to be multi-region, so they built an entirely new service that’s job was to decide which queue in which region jobs needed to go on. So now you had jobs cri…

I'm going to be brave (but still use a throwaway) and ask the dumb question - what is wrong with putting things in queues to help with performance problems? If some endpoint is too slow to return a response to the frontend within a reasonable time, enqueueing it via a worker makes sense to me. That doesn't cover all performance issues but it handles a lot of them. You should also do things like optimize SQL queries,…

Pretty simple - people often think queues are magic, and an unbounded queue is just your previous problem but now way worse; what happens when the queue "gets full"?

But if you are just smoothing out some work its pretty normal, just make sure you are modeling things instead of putting it in the magic queue.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#25

Post by crowcroft »

teyc wrote: post #17
throwaway493943 wrote: post #14

Earlier quoted context omitted.

I'm going to be brave (but still use a throwaway) and ask the dumb question - what is wrong with putting things in queues to help with performance problems? If some endpoint is too slow to return a response to the frontend within a reasonable time, enqueueing it via a worker makes sense to me. That doesn't cover all performance issues but it handles a lot of them. You should also do things like optimize SQL queries,…

Queues have several problems - if the caller is http it may timeout and retry, leading to more jobs being queued - the caller may no longer care because it took so long and the work is wasted - if the caller is called from a queue it can cause cascades - you can fill a disk up and crash the system

To me the question is, so what's a better alternative? At least queues can be designed to handle timeouts, errors, and flakey APIs.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#26

Post by nkraft11 »

zdragnar wrote: post #23
throwaway493943 wrote: post #14

Earlier quoted context omitted.

I'm going to be brave (but still use a throwaway) and ask the dumb question - what is wrong with putting things in queues to help with performance problems? If some endpoint is too slow to return a response to the frontend within a reasonable time, enqueueing it via a worker makes sense to me. That doesn't cover all performance issues but it handles a lot of them. You should also do things like optimize SQL queries,…

There's two primary areas that I've seen teams get bitten by this personally: 1) Designers don't understand that things are going to happen async, and the UI ends up wanting to make assumptions that everything is happening in real time. Even if it works with the current design, it's one small change away from being impossible to implement. This is a general difficulty with working in eventually consistent systems, bu…

Error handling was a huge issue, along with other weird distributed system bugs. Backed up queues, job shedding, thundering herds, you name it. When you have jobs on queues kicking off new jobs on different queues, tracing issues is just miserable. Sure, it's not a problem of ruby per se, but engineers would basically just throw their hands up and say "ruby can't handle this" and sidekiq became the One True Way™.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#27

Post by sbarre »

AstroBen wrote: post #20
teaearlgraycold wrote: post #4

Earlier quoted context omitted.

Ruby and the age of “I don’t care what type this variable is, it quacks like a duck!” is over and dead. Improvements to type systems have shown there is a better way to do software development.

Static typing isn't free. Dynamic typing is perfectly viable to build any sized software - there's living proof, far from dead So we're only left with personal opinion

Or the standard "it depends" answer that everyone eventually realizes is the only correct answer. ;-)

Re: A Deep Dive into Solid Queue for Ruby on Rails

#28

Post by nkraft11 »

rubyfan wrote: post #6
nkraft11 wrote: post #2

I briefly worked at a YC company that was a ruby shop. Their answer to every performance problem was to stick it on a queue. There were, I don’t know, dozens of them. Then they decided they needed to be multi-region, because reasons. But the queues weren’t set up to be multi-region, so they built an entirely new service that’s job was to decide which queue in which region jobs needed to go on. So now you had jobs cri…

Yeah this is not the fault of ruby. Sounds more like bad choices that could be made with any language or framework.

You're totally correct. It's not a problem of ruby per se, but engineers would basically just throw their hands up and say "ruby can't handle this" and sidekiq became the One True Way™. What ensued was the most byzantine software architecture I've ever seen.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#29

Post by pqdbr »

Great writeup. I'd love to know more about how the Supervisor works, and how it. "fork[s] a separate process for each supervised worker/dispatcher/scheduler".

In a Rails app served with Puma, I've always had a hard time understanding what would be the canonical way for having a loop doing some periodic work.

I know Puma has plugin support but I don't see much documentation there.

Forking a process / threads is something that we're used having Rails / Puma take care for us.

Pressed for time and without having time to deep dive, we ended up settling with sidekiq-cron, and it's been serving us so nicely.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#30

Post by RangerScience »

throwaway493943 wrote: post #14
nkraft11 wrote: post #2

I briefly worked at a YC company that was a ruby shop. Their answer to every performance problem was to stick it on a queue. There were, I don’t know, dozens of them. Then they decided they needed to be multi-region, because reasons. But the queues weren’t set up to be multi-region, so they built an entirely new service that’s job was to decide which queue in which region jobs needed to go on. So now you had jobs cri…

I'm going to be brave (but still use a throwaway) and ask the dumb question - what is wrong with putting things in queues to help with performance problems? If some endpoint is too slow to return a response to the frontend within a reasonable time, enqueueing it via a worker makes sense to me. That doesn't cover all performance issues but it handles a lot of them. You should also do things like optimize SQL queries,…

I think your question is well-asked, and I lament any work environments that have led you to think asking a question like this would be A Problem(tm).

IMO - there’s a lot of things that queues are an excellent answer to. Potentially including performance.

But - queues (generally and among other things) solve the problem of “this will take some time AND the user doesn’t need an immediate response.”

If that’s not your problem, then using queues might not be the solution. If it’s something that’s taking too long and the user DOES need a response, then (as you say) optimizing is what you should try, not queues. Or some product redesign so the user either doesn’t need an immediate response. Or finding a way to split up the part producing an immediate response and the part that takes awhile.

For example: validating uploaded bulk data is in the right “shape”, and then enqueuing the full validation and insertion.

Also really really avoid jobs that enqueue jobs. Sometimes they’re necessary (spacing out some operation on chunks of a group; or a job that ONLY spawns other jobs) but mostly they’re a route to spaghetti.

Post reply on HN