Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Deep Dive into Solid Queue for Ruby on Rails

blog.appsignal.com

Post reply on HN
Print view View original

11–20 of 50 posts

Re: A Deep Dive into Solid Queue for Ruby on Rails

#11

Post by trevorhinesley »

morkalork wrote: post #10
trevorhinesley wrote: post #9

Earlier quoted context omitted.

I’ve never felt like “throw everything into a queue” was a mindset within the Ruby community, nor have we done that at my companies. And multi-region is a business decision.

Doesn't Ruby, like Python, have a GIL? I always found that one is enough to encourage some "premature scalable architecture"

It does have a GIL. You’re not wrong, but by that same logic, there’s pitfalls when using multi-threading as well, even in languages where it’s native (e.g., Elixir).

Regardless, in my experience, when you run into scenarios that need queueing, multi-threading, etc., you need to know what you’re doing.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#14

Post by throwaway493943 »

nkraft11 wrote: post #2

I briefly worked at a YC company that was a ruby shop. Their answer to every performance problem was to stick it on a queue. There were, I don’t know, dozens of them. Then they decided they needed to be multi-region, because reasons. But the queues weren’t set up to be multi-region, so they built an entirely new service that’s job was to decide which queue in which region jobs needed to go on. So now you had jobs cri…

I'm going to be brave (but still use a throwaway) and ask the dumb question - what is wrong with putting things in queues to help with performance problems?

If some endpoint is too slow to return a response to the frontend within a reasonable time, enqueueing it via a worker makes sense to me.

That doesn't cover all performance issues but it handles a lot of them. You should also do things like optimize SQL queries, cache in redis or the db, perhaps run multiple threads within an endpoint, etc. but I don't see anything wrong with specifically having dozens of workers/queues. We have that in my work's Rails app.

Happy to hear how I can do things better if I'm missing something.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#15

Post by ch4s3 »

teaearlgraycold wrote: post #4
nkraft11 wrote: post #2

I briefly worked at a YC company that was a ruby shop. Their answer to every performance problem was to stick it on a queue. There were, I don’t know, dozens of them. Then they decided they needed to be multi-region, because reasons. But the queues weren’t set up to be multi-region, so they built an entirely new service that’s job was to decide which queue in which region jobs needed to go on. So now you had jobs cri…

Ruby and the age of “I don’t care what type this variable is, it quacks like a duck!” is over and dead. Improvements to type systems have shown there is a better way to do software development.

Maybe, the cycle has happened before and maybe come back around again. Dynamic typing is really nice when most of your data looks like bags of strings. Compilers and tools just don’t add a lot when you’re passing around glorified blobs of stringy json-like stuff. Type gymnastics can eat a lot of time where you could otherwise be shipping something useful.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#16

Post by charcircuit »

rubyfan wrote: post #8
charcircuit wrote: post #7

Earlier quoted context omitted.

Culture around a language influences what choices are made.

Don’t be silly. Bad choices are made in all sorts of languages and teams - this has nothing to do with language. High pressure situations can lead teams to make choices they don’t always foresee as bad until after they are paying the consequences.

Sure bad choices are made everywhere, but I was essentially claiming that when a community has a hammer, they will see nails.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#17

Post by teyc »

throwaway493943 wrote: post #14
nkraft11 wrote: post #2

I briefly worked at a YC company that was a ruby shop. Their answer to every performance problem was to stick it on a queue. There were, I don’t know, dozens of them. Then they decided they needed to be multi-region, because reasons. But the queues weren’t set up to be multi-region, so they built an entirely new service that’s job was to decide which queue in which region jobs needed to go on. So now you had jobs cri…

I'm going to be brave (but still use a throwaway) and ask the dumb question - what is wrong with putting things in queues to help with performance problems? If some endpoint is too slow to return a response to the frontend within a reasonable time, enqueueing it via a worker makes sense to me. That doesn't cover all performance issues but it handles a lot of them. You should also do things like optimize SQL queries,…

Queues have several problems - if the caller is http it may timeout and retry, leading to more jobs being queued - the caller may no longer care because it took so long and the work is wasted - if the caller is called from a queue it can cause cascades - you can fill a disk up and crash the system

Re: A Deep Dive into Solid Queue for Ruby on Rails

#18

Post by rubyfan »

throwaway493943 wrote: post #14
nkraft11 wrote: post #2

I briefly worked at a YC company that was a ruby shop. Their answer to every performance problem was to stick it on a queue. There were, I don’t know, dozens of them. Then they decided they needed to be multi-region, because reasons. But the queues weren’t set up to be multi-region, so they built an entirely new service that’s job was to decide which queue in which region jobs needed to go on. So now you had jobs cri…

I'm going to be brave (but still use a throwaway) and ask the dumb question - what is wrong with putting things in queues to help with performance problems? If some endpoint is too slow to return a response to the frontend within a reasonable time, enqueueing it via a worker makes sense to me. That doesn't cover all performance issues but it handles a lot of them. You should also do things like optimize SQL queries,…

You’re not missing anything and are correct in that there are plenty of reasons to use queues and defer work that can be handled asynchronously outside of a request/response. This is not specific to ruby or any language for that matter.

The parent indicated the cross region dynamic required extra routing logic and introduced debugging problems.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#19

Post by aaronblohowiak »

trevorhinesley wrote: post #9
charcircuit wrote: post #7

Earlier quoted context omitted.

Culture around a language influences what choices are made.

I’ve never felt like “throw everything into a queue” was a mindset within the Ruby community, nor have we done that at my companies. And multi-region is a business decision.

Resque was a staple for a long long time. In the jvm world, throw everything into Kafka is also a staple of a lot of "enterprise" shops. Or SQS for AWS places I've worked at. I think it is not a ruby language thing, but a certain kind of architecture thing.

Re: A Deep Dive into Solid Queue for Ruby on Rails

#20

Post by AstroBen »

teaearlgraycold wrote: post #4
nkraft11 wrote: post #2

I briefly worked at a YC company that was a ruby shop. Their answer to every performance problem was to stick it on a queue. There were, I don’t know, dozens of them. Then they decided they needed to be multi-region, because reasons. But the queues weren’t set up to be multi-region, so they built an entirely new service that’s job was to decide which queue in which region jobs needed to go on. So now you had jobs cri…

Ruby and the age of “I don’t care what type this variable is, it quacks like a duck!” is over and dead. Improvements to type systems have shown there is a better way to do software development.

Static typing isn't free. Dynamic typing is perfectly viable to build any sized software - there's living proof, far from dead

So we're only left with personal opinion

Post reply on HN