Ask HN: The Worst Engineer I Know Is Demanding We Start Using Celery (Python)

I work with an engineer who consistently advocates for terrible architecture and design patterns with a ton of confidence. Every time one of his ideas is implemented it turns into a flaming bag of complexity dog shit, and everyone else is left holding the bag to clean up.

His new tirade is demanding we start using celery to solve every problem we have. I instinctively want to dismiss him out of turn given his previous track record of thinking himself into a pretzel, however, I don't want to fall into a trap myself of negative thinking.

The way I see it, we would already be sending async job messages to a durable queue like SQS. It seems extremely trivial to me to run a service that polls the queue for new messages and dispatches the jobs on its own from there (as I have a done with great success a number of times in other environments). What is the benefit of pulling in all the complexity of celery for these simple asynchronous tasks?

Sorry, I cannot add much on Celery.

If "every time one of his ideas is implemented it turns into a flaming bag of complexity dog shit, and everyone else is left holding the bag to clean up," then it might be better to discuss how you make and stick to decisions in the team, rather than following your fellow engineer down his fad-of-the-day.

Sounds like your company/team have some serious communication issues. These things do happen in all companies regardless of size - but from what I understand, this is a repeat event.

Mentioning these sorts of concerns with other people in the company (especially higher-ups) is crucial. They can offer you the guidance on how to handle the situation far more effectively than some strangers on a forum.

I have had many instances of these sort of situations. Half the time you just suck it up and do it (if the person is more senior or it's a topic you're not so confident about) and the other half you can usually find some other strategy like meeting in the middle.

Maybe Celery is the best option? Have a team-wide meeting to discuss his side of the story. Sure, he might be one to just advocate constantly, but give him his pedestal to speak. If his arguments are shaky, he will step down from the pedestal on his own when he realises that this decision is inefficient and unnecessary. If he's not the founder/CEO then talk to the founder/CEO or anyone above this guy so that they can help you put him in his place.

Sorry I couldn't be of technical help but I hope this feedback will help tackle your team's root problem - communication.

You are hitting the nail on the head with the communication issues. Truth be told, I am more senior and the rest of the leadership team understands that this individual needs a lot of management and shouldn't be allowed to implement stuff on their own.

"If his arguments are shaky, he will step down from the pedestal on his own when he realises that this decision is inefficient and unnecessary." - The sad part about this person is that when presented with this type of feedback, they always double and triple down despite the facts on the board, it is really odd and risky behavior to be honest.

At the end of the day, I am the person who has to put him in his place, so I want to make sure I am not missing a major point of consideration with celery.

> It seems extremely trivial to me to run a service that polls the queue for new messages and dispatches the jobs on its own from there

Are you handling retries, timeouts, disconnected clients, logging failures, dealing with perma-failing jobs, priorities, execution metrics, etc.?

You may or may not be interested in those features, but that's why projects like celery exist. It's not "extremely trivial" to replicate them all. If you have only extremely simple tasks that you don't critically depend on, that's fine to use simple solutions instead.

If you want a constructive discussion about things like that, the questions are: what problem are we trying to solve, what features we need today/soon, how long will the implementation take, what is the risk in not changing anything.

> Every time one of his ideas is implemented it turns into a flaming bag of complexity dog shit

How sure are you it's due to the idea itself rather than due to the implementation? If it really is due to the ideas and it repeated, that's a management problem more than that person problem. Maybe start chatting to them?