A simple core.async job system in Clojure
blog.janetacarr.com
A simple core.async job system in Clojure
1–10 of 18 posts
Re: A simple core.async job system in Clojure
#2Thank you for this post Janet.
I think my technical perspectives have moved in a similar direction: keep things extremely simple with minimum moving parts.
I maintained (automated upgrades) a RabbitMQ cluster and while it is powerful software it is operationally expensive. For a side project you probably just batch process in a cron.
If I were to take the approach in this blog post I would want everyone on the team to be extremely familiar with the model of task running: stuck jobs, timeouts, duplicate jobs, client disconnects and retries, stuck "poison" jobs seem like issues you might face.
Re: A simple core.async job system in Clojure
#3I appreciate this emphasis on simplicity and I recommend it to others. I've seen trends in the other direction which I think are dangerous. I notice that as some companies adopted a microservices approach there was a tendency to allow in more technologies than necessary.
"Premature polyglot programming" is a disease that afflicts certain startups. While any sufficiently big company will be polyglot, a small startup needs to stay focused on a limited tech stack, for as long as possible. After all, each new technology requires someone on staff who has the skill for that technology, and when your startup is small, your talent pool will also be small, and so it becomes common that you only have 1 person on the team who knows how to run some particular technology (Kafka, RabbitMQ, Redshift, DynamoDB, MongoDB, Tornado, etc).
So you end up with a lot of single point of failures, where the "single point of failure" is the single engineer who knows how to run the technology that you've made critical to the company. Be wary. Avoid this if possible.
I notice, especially, as the tech industry developed better tools for managing complex devops situations (Docker, Kubernetes, Terraform) there was a tendency from some engineers to think "Nowadays it is easy to run 10 different technologies, therefore we should run 10 different technologies." Be wary of this.
Janet Carr's emphasis on simplicity is something we should all imitate.
Re: A simple core.async job system in Clojure
#4Thank you for this post Janet. I think my technical perspectives have moved in a similar direction: keep things extremely simple with minimum moving parts. I maintained (automated upgrades) a RabbitMQ cluster and while it is powerful software it is operationally expensive. For a side project you probably just batch process in a cron. If I were to take the approach in this blog post I would want everyone on the team t…
I'm glad you liked it!
I'm using a PaaS, so I didn't want to pay the extra money for a cron job. Maybe not a wise a choice, but here we are.
Totally agree about the issues I might face. I'm glad that the Clojure REPL is a thing, so I can test out all of a job's functionality before sending it off to async land.
Re: A simple core.async job system in Clojure
#5I appreciate this emphasis on simplicity and I recommend it to others. I've seen trends in the other direction which I think are dangerous. I notice that as some companies adopted a microservices approach there was a tendency to allow in more technologies than necessary. "Premature polyglot programming" is a disease that afflicts certain startups. While any sufficiently big company will be polyglot, a small startup n…
Wow, thanks for the kind words.
I've been bitten by the ops hellscape of microservices and various tech many times throughout my career, and it's definitely shaped how I think about designing and building software now.
Re: A simple core.async job system in Clojure
#6I appreciate this emphasis on simplicity and I recommend it to others. I've seen trends in the other direction which I think are dangerous. I notice that as some companies adopted a microservices approach there was a tendency to allow in more technologies than necessary. "Premature polyglot programming" is a disease that afflicts certain startups. While any sufficiently big company will be polyglot, a small startup n…
I agree, and a lot of this applies to large companies as well.
Of course larger orgs will have higher tolerance for a wider set of technologies, but this tolerance is not infinite, and choosing tech because that one team was a fan creates staffing problems over time, especially as the original devs move on. Eventually you get “estate sprawl” that’s difficult to manage.
Choosing to minimise dependencies is definitely a (good) choice. The key is the right tech for the task at hand. Equally org-wide “use Oracle for everything” mandates aren’t helpful.
Balancing these at a large org definitely needs good engineering leadership!
Re: A simple core.async job system in Clojure
#7I appreciate this emphasis on simplicity and I recommend it to others. I've seen trends in the other direction which I think are dangerous. I notice that as some companies adopted a microservices approach there was a tendency to allow in more technologies than necessary. "Premature polyglot programming" is a disease that afflicts certain startups. While any sufficiently big company will be polyglot, a small startup n…
You can use this same line of reasoning to avoid using Clojure entirely.
Re: A simple core.async job system in Clojure
#8I appreciate this emphasis on simplicity and I recommend it to others. I've seen trends in the other direction which I think are dangerous. I notice that as some companies adopted a microservices approach there was a tendency to allow in more technologies than necessary. "Premature polyglot programming" is a disease that afflicts certain startups. While any sufficiently big company will be polyglot, a small startup n…
You can use this same line of reasoning to avoid using Clojure entirely.
Yeah but that's not nearly as fun!
Re: A simple core.async job system in Clojure
#9Re: A simple core.async job system in Clojure
#10core.async was actually the "killer app" that sold Clojure to me. I've always liked Go's CSP-style concurrency, but I wanted a proper functional language to go with it. It turns out that Channels/Queues are just an insanely good abstraction for getting threads to talk to each other. The closest general-purpose library I've found that helps bring that to the rest of the world has been ZeroMQ, which is great but not as easy and nice to use as core.async.
That said, I really don't think RabbitMQ is so bad. The default docker configuration for it is fine for most cases you're likely to come across for all but the biggest applications, and I do think that having the ability to have jobs restart/requeued when a worker crashes out of the box is worth the tradeoffs for slightly increased complexity. I usually just use a single docker compose and glue everything together in one big ol' YAML.
Still, there's obvious value in avoiding dependencies, so it of course depends on the size and scope of your project. If you think you're going to end up distributing this over 100 nodes, something like RabbitMQ or ActiveMQ might be worth playing with, but if you're just doing relatively small (at you appear to be in this project), it's probably the correct choice to mimic whatever behavior you need with core.async.