311–320 of 376 posts
I have a couple of datapoints. One project I know of started with message queues for request response pattern. It performed poorly because Windows Service Bus writes messages to a database. That increased latency for a UI heavy application. Second project used message queues but the front end was a HTTP API. When overloaded the API timed out at 30 seconds but the job was still in the queue and wasn’t cancelled. It le…
MSMQ?
We are now based firmly in the Azure landscape and Event Grids provide us with an effective team service boundary for other teams to consume our events all with the appropriate RBAC. Internal team Azure Service Bus is the underlying internal team driver for building decoupled and resilient services where we have to guarantee eventual consistency between our internal system landscape and the external SaaS services we actively leverage. At this scale it works very effectively, especially when pods can be dropped at any point within our k8s clusters.
I have a couple of datapoints. One project I know of started with message queues for request response pattern. It performed poorly because Windows Service Bus writes messages to a database. That increased latency for a UI heavy application. Second project used message queues but the front end was a HTTP API. When overloaded the API timed out at 30 seconds but the job was still in the queue and wasn’t cancelled. It le…
> Windows service Bus?? MSMQ?
Earlier quoted context omitted.
We also have much much bigger single machines available for reasonable money. So a lot of reasonable workloads can fit in one machine now that used to require a small cluster
This! You NEEDED to scale horizontally because machines were just doing too much. I remember when our Apache boxes couldn’t even cope doing SSL so we had a hardware box doing it on ingress!
Because it’s massive over engineering of dubious value, unless you are Uber etc
I find it useful in small doses even at small scale. When you have only one of a thing processing stuff it's useful to isolate that behind queues so you can bounce it and lose nothing: "pause" the incoming queue (new messages stack up,) let the pipeline drain, do whatever you must, and then un-pause. Nothing is lost, despite the backend being simple minded and not fault tolerant. The key here is "small dose". Very li…
2) Serverless, e.g. AWS lambdas can be joined with step functions instead and scale without a queue.
3) People have been burned. Multiple configurations, multiple queues, multiple libraries and languages, multiple backing stores, multiple serialisation standards and bugs - it's just overly complex engineering a distributed system for small to medium business. YAGNI.
4) Simpler architectures, e.g. microservices for better or worse tend to be fat monoliths made more efficient, mostly in my experience because those in charge don't actually don't understand the pattern, but the side effect is fewer queues compared to real microservices arch .
O/T I cringe whenever I hear our tech architects discuss our arch as microservices and datamesh. The former because it's not (as above, it's multiple small services) the latter also a problem because datamesh is an antiquated pattern that's better filled with segregated schemas on a distributed database, and scoped access per system to the data each needs, instead of adapter patterns, multiple dbs with slightly different schemas and fascades/specialised endpoints all the fucking way down.
During heavy load the queue bloats up to a few million messages, then drains off over time. Or it spawns a few hundred lambdas to chow all the messages down...depending on what we want.
On AWS for example, you use SQS and a sprinkling of SNS, or perhaps Kinesis for a few things and you're good. There isn't a lot to talk about there, so the queues no longer become the center of the design.
Message-queue based architectures are great for data processing, but not great for interactive web sites, and if most people are building interactive web sites, then the choices seem a little obvious. I still design event systems for data processing (especially with immutable business data where you have new facts but still need to know that you were "wrong" or had a different picture at some earlier time). But for most apps... you just don't need it.
Earlier quoted context omitted.
I’ve only ever seen a single dev team managing their own K8s cluster. If by deploy you mean “they merge a branch which triggers a lot of automation that causes their code to deploy,” you don’t need K8s for that. Don’t get me wrong, I like K8s and run it at home, and I’d take it any day over ECS or the like at work, but it’s not like you can’t achieve a very similar outcome without it.
Of course. And I can achieve a Webserver in C. Doesn’t mean it’s the best way given the circumstances. There are many ways in tech to achieve the same result. I don’t understand why people constantly need to point that out. I also don’t understand why k8s ruffles so many feathers. Reminds me a bit of Linux vs windows vs Mac debates.
I’m a fan of graduated difficulty. Having complex, powerful systems should require that you understand them, else when they break – and they will, because they’re computers – you’ll be utterly lost.
Earlier quoted context omitted.
Helpful comment! If I could pick your brain... I'm looking at a green field implementation of a task system, for human tasks - people need to do a thing, and then mark that they've done it, and that "unlocks" subsequent human tasks, and near as I can tell the overall task flow is a DAG. I'm currently considering how (if?) to allow for complex logic about things like which tasks are present in the overall DAG - things…
Not GP or specifically Airflow user; but my approach is to have a fixed job graph, and unnecessary jobs immediately succeed. And indeed, jobs are external executables, with all the skip/no skip logic executed therein. If nothing else, it makes it easy to understand what actually happened and when - just look at job logs.
Closed - Passed
Closed - Failed
Closed - Waived
When you hit that Waived state, it should include a note explaining why it was waived. This could be “parent transaction dropped below threshold amount, so we don’t need this control” or “Executive X signed off on it”.
I’m not sure about the auto-skip thing you propose, just from a UX perspective. I don’t want my task list cluttered up with unnecessary things. Still, I am struggling with precisely where to store the business logic about which tasks are needed when. I’m leaning towards implementing that in a reporting layer. Validation would happen in the background and raise warnings, rather than hard stopping people.
The theory there is that the people doing the work generally know what’s needed better than the system does. Thus the system just provides gentle reminders about the typical case, which users can make the choice to suppress.