I believe it’s the last one. I use queues more than ever.
Almost every AWS architecture diagram will have a queue.
SQS is extremely stable, mature and cheap. Integrates with so many services and comes out of the box with good defaults.
Ask HN: Why do message queue-based architectures seem less popular now?
121–130 of 376 posts
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#122I like a lot of the answers, but something else I'd add: lots of "popular" architectures from the late 00s and early 2010s have fallen by the wayside because people realized "You're not Google. Your company will never be Google." That is, there was a big desire around that time period to "build it how the big successful companies built it." But since then, a lot of us have realized that complexity isn't necessary for…
To be fair, I worked on multiple projects removing queues at Google, so it's more than just that.
and mandates that virtually all new projects not directly use borg/kubernetes.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#123I really hope that people are slowly starting to understand that using kafka and turning it in a single point of failure (yes, it fails) of your architecture is not a good idea.
This pattern has it's uses, but if you are using it everywhere, every time you have some sort of notifications because "it's easy" or whatever, you are likely doing it wrong and you will understand this at some point and it will not be pleasant.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#124Earlier quoted context omitted.
Kubernetes brings more than just being Google. In a way it’s also an ecosystem.
That‘s nice, but the question is if you (i.e. your company) needs this ecosystem.
Need no. But it’s nice especially since I have the know how.
Creating a new deployment is just super fast. Developers can deploy their own apps etc.
And then there is Helm.
If we only decide by “need” then most of the time we also wouldn’t need object oriented programming.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#125it's disk speeds.
suddenly everyone could scale much much more, but by then they were moving to the cloud and execs don't understand two buzzwords at the same time.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#126I believe it’s the last one. I use queues more than ever. Almost every AWS architecture diagram will have a queue. SQS is extremely stable, mature and cheap. Integrates with so many services and comes out of the box with good defaults.
This is my experience too. I frequently require a single queue for some very specific task but it does not form part of a greater application.
My current work (data engineering) is such that I don't have a relational database instance that it would make sense to use instead. SQS is very straightforward and just works.
None of this is to say that I would necessarily advocate for a user-facing app to be _based_ on a queue-based architecture.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#127Earlier quoted context omitted.
To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to h…
Helpful comment! If I could pick your brain... I'm looking at a green field implementation of a task system, for human tasks - people need to do a thing, and then mark that they've done it, and that "unlocks" subsequent human tasks, and near as I can tell the overall task flow is a DAG. I'm currently considering how (if?) to allow for complex logic about things like which tasks are present in the overall DAG - things…
Not GP or specifically Airflow user; but my approach is to have a fixed job graph, and unnecessary jobs immediately succeed. And indeed, jobs are external executables, with all the skip/no skip logic executed therein.
If nothing else, it makes it easy to understand what actually happened and when - just look at job logs.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#128Kafka is everywhere. But I guess there is a point where hyped new tech becomes boring and mainstream, and people stop writing about it.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#129Speaking from my own experience message queues haven’t disappeared as much as have been abstracted away. For example enqueue to SQS + poll became invoke server less process. There is a message queue in there somewhere just that it’s not as exposed. Or take AWS SNS which IMO is one level of abstraction higher than SQS. It became so feature rich that it can practically replace SQS. What might have disappeared is those…
good point. tcp was justified at some point in it's birth a queueing component. today nobody dates to think of it as such.
Re: Ask HN: Why do message queue-based architectures seem less popular now?
#130The only place I find the complexity of message queues worth the trouble is in the embedded world. The limited resources make messaging a sensible way to communicate across devices with a fairly agnostic perspective on what runs on the devices apart from the message queue.
Most of us in the desktop computing world don't actually need the distribution, reliability features, implementation-agnostic benefits of a queue. We can integrate our code very directly if we choose to. It seems to me that many of us didn't for a while because it was an exciting paradigm, but it rarely made sense in the places I encountered it.
There are certainly cases where they're extremely useful and I wouldn't want anything else, but again, this is typically in settings where I'm very constrained and need to talk to a lot of devices rather than when writing software for the web or desktop computers.
As for your last point, the Internet of Things is driven by message queues (like MQTT), so depending on the type of work you're doing, message queues are all over the place but certainly not exciting to write about. It's day-to-day stuff that isn't rapidly evolving or requiring new exciting insights. It just works.