Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Deep Dive into Solid Queue for Ruby on Rails

blog.appsignal.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: A Deep Dive into Solid Queue for Ruby on Rails

#41

Post by Lio »

zdragnar wrote: post #23
throwaway493943 wrote: post #14

Earlier quoted context omitted.

I'm going to be brave (but still use a throwaway) and ask the dumb question - what is wrong with putting things in queues to help with performance problems? If some endpoint is too slow to return a response to the frontend within a reasonable time, enqueueing it via a worker makes sense to me. That doesn't cover all performance issues but it handles a lot of them. You should also do things like optimize SQL queries,…

There's two primary areas that I've seen teams get bitten by this personally: 1) Designers don't understand that things are going to happen async, and the UI ends up wanting to make assumptions that everything is happening in real time. Even if it works with the current design, it's one small change away from being impossible to implement. This is a general difficulty with working in eventually consistent systems, bu…

These are good points, in answer to them:

1. Yes this is true but Rails now comes with nice support for async UI built to push updates to the browser via Hotwire and Turbo.

You’d need something like that anyway anytime you’re calling an external service you don’t control.

2. Again this is also a good point but even running every request synchronously you still need good error logging because you don’t want to share details of an error with your frontend.

With background jobs you definitely need to be on top of monitoring and retry logic. I also think you need to be very careful about idempotency amd retry logic.

I see that as the engineering trade offs for that pattern. There’s very little in the way of silver bullets in engineering; different solutions just come with different trade offs.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#42

Post by Lio »

riffraff wrote: post #38
ch4s3 wrote: post #15

Earlier quoted context omitted.

Maybe, the cycle has happened before and maybe come back around again. Dynamic typing is really nice when most of your data looks like bags of strings. Compilers and tools just don’t add a lot when you’re passing around glorified blobs of stringy json-like stuff. Type gymnastics can eat a lot of time where you could otherwise be shipping something useful.

I would argue when you're passing around stringy JSON-like thingies is when typing is most useful :) You're not going to misuse an API that takes a Person or Cart, but mixing up two hashes cause you used two different strings as keys can happen easily. (I do think dynamic typing is mostly fine, but I do wish ruby had optional static typing with some nice syntax instead of RBS)

It’s on the way thankfully.

I’m really excited about Sorbet getting behind the new RBS-inline comment syntax and the prospect of both runtime and static analysis as optional tools when needed.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#43

Post by jaredsohn »

ch4s3 wrote: post #15
teaearlgraycold wrote: post #4

Earlier quoted context omitted.

Ruby and the age of “I don’t care what type this variable is, it quacks like a duck!” is over and dead. Improvements to type systems have shown there is a better way to do software development.

Maybe, the cycle has happened before and maybe come back around again. Dynamic typing is really nice when most of your data looks like bags of strings. Compilers and tools just don’t add a lot when you’re passing around glorified blobs of stringy json-like stuff. Type gymnastics can eat a lot of time where you could otherwise be shipping something useful.

I haven't really felt a need for static typing in my job's Rails app (small team, and I've been working in this codebase a really long time) but I think LLMs can be a huge help for automating the type gymnastics.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#44

Post by [deleted] »

RangerScience wrote: post #30
throwaway493943 wrote: post #14

Earlier quoted context omitted.

I'm going to be brave (but still use a throwaway) and ask the dumb question - what is wrong with putting things in queues to help with performance problems? If some endpoint is too slow to return a response to the frontend within a reasonable time, enqueueing it via a worker makes sense to me. That doesn't cover all performance issues but it handles a lot of them. You should also do things like optimize SQL queries,…

I think your question is well-asked, and I lament any work environments that have led you to think asking a question like this would be A Problem(tm). IMO - there’s a lot of things that queues are an excellent answer to. Potentially including performance. But - queues (generally and among other things) solve the problem of “this will take some time AND the user doesn’t need an immediate response.” If that’s not your…

[deleted]

Re: A Deep Dive into Solid Queue for Ruby on Rails

#46

Post by atmosx »

charcircuit wrote: post #7
rubyfan wrote: post #6

Earlier quoted context omitted.

Yeah this is not the fault of ruby. Sounds more like bad choices that could be made with any language or framework.

Culture around a language influences what choices are made.

Funny part: I initially thought you were referring to the word “Byzantine” itself, which tends to carry a negative connotation in English, mostly due to historical bias. But you’re actually talking about Ruby!

If we were to take Byzantine in a more accurate, historical sense, something truly “Byzantine” should be evolving, enduring, top-tier, and built to last for 1k years.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#47

Post by yxhuvud »

nkraft11 wrote: post #2

I briefly worked at a YC company that was a ruby shop. Their answer to every performance problem was to stick it on a queue. There were, I don’t know, dozens of them. Then they decided they needed to be multi-region, because reasons. But the queues weren’t set up to be multi-region, so they built an entirely new service that’s job was to decide which queue in which region jobs needed to go on. So now you had jobs cri…

Bad architecture can happen in any language. I don't see how the language choice could ever protect you against the described structural problem you built.

Also you will see that the answer to most actual performance problems tend to be queues even in other languages. At least in mature places - mostly because it is possible to inspect what a queue is doing. Though it will of course be a problem if it is part of a big spaghetti architecture.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#48

Post by nkraft11 »

rubyfan wrote: post #21
charcircuit wrote: post #16

Earlier quoted context omitted.

Sure bad choices are made everywhere, but I was essentially claiming that when a community has a hammer, they will see nails.

Queues are not a ruby specific thing, nor are they particularly pervasive within Rails apps. Having a good framework to handle them doesn’t make it the only tool in the tool belt. On the contrary, the fact that Rails has good tools to fit many different types of system architecture needs is a counterpoint of your assertion.

Sort of. In this case the lack of multithreading led engineers to using sidekiq as a stand in.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#49

Post by ch4s3 »

riffraff wrote: post #38
ch4s3 wrote: post #15

Earlier quoted context omitted.

Maybe, the cycle has happened before and maybe come back around again. Dynamic typing is really nice when most of your data looks like bags of strings. Compilers and tools just don’t add a lot when you’re passing around glorified blobs of stringy json-like stuff. Type gymnastics can eat a lot of time where you could otherwise be shipping something useful.

I would argue when you're passing around stringy JSON-like thingies is when typing is most useful :) You're not going to misuse an API that takes a Person or Cart, but mixing up two hashes cause you used two different strings as keys can happen easily. (I do think dynamic typing is mostly fine, but I do wish ruby had optional static typing with some nice syntax instead of RBS)

> You're not going to misuse an API that takes a Person or Cart, but mixing up two hashes cause you used two different strings as keys can happen easily.

This is more or less trivial to catch and fix, I'm just not sure a type system is worth it's weight for that kind of case.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#50

Post by j0rd72 »

hschne wrote: post #33

Author here. What a pleasant surprise to see this on HN! Happy to answer any questions.

Thanks for the write-up, I feel like I understand Solid Queue quite well, now.

I suppose my primary question is: What does this do better than Sidekiq+Redis; or, why should I convert my Sidekiq jobs to use Solid Queue? I'm curious also if there are comparisons of performance anywhere.

All-in-all, though, it looks technically quite promising!

Post reply on HN