Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku has been running a second copy of my scheduler instance

openfolder.sh

Post reply on HN
Print view View original

51–60 of 132 posts

Re: Heroku has been running a second copy of my scheduler instance

#51

Post by andybak »

andrewstuart wrote: post #40

>> So is it Celery then? This awesome library has its fair share of complaints about duplicated task execution — most are due to misconfiguration. I scanned dozens of Stack Overflow threads and GitHub issues in Celery’s repository. Very few were relevant to my case, and none of the fixes were. I was so annoyed by the pain of configuring Celery I stopped my project and wrote my own job queuing system. Job and message…

Hell yeah. And people it them for low-load situations where they could just use a db table or even just spawining a new process. 30k lines of code and several binary dependencies to run a handful of tasks because "it's the best way to do it".

Re: Heroku has been running a second copy of my scheduler instance

#52

Post by rcarmo »

maushu wrote: post #4

This is why I use the git commit short hash as part of the identification of the service (eg: container name). Allows for quick check of which is the running code version and identify problems like these.

Wouldn’t help in this case if you redeployed the same version. I usually log the PID and some form of host ID.

Re: Heroku has been running a second copy of my scheduler instance

#55

Post by gsa »

shibel wrote: post #46
lorenzk wrote: post #23

Not sure if you have done this already but: If you rotate your database, redis and email provider passwords/tokens, the zombie process might still be running but will not have any effects any more.

OP here: that’s a good idea that did come up to mind (specifically rotating the Redis queue credentials). I was just sure Heroku’s gonna take care of this shit by now. BTW, I’m not 100% sure it’ll work: if Heroku updates that zombie dyno’s environment variables, it may still have access to the new credentials :/ (My Redis instance is also a Heroku add-on)

Rotate a token, but also mangle it and change to the correct one in code. eg: change database url from `postgres://` to `invalid://` and replace it in code.

Re: Heroku has been running a second copy of my scheduler instance

#56

Post by echelon »

lloydatkinson wrote: post #49
echelon wrote: post #41

Earlier quoted context omitted.

Netlify did this to me too! I have my startup with them and pay them a hefty bill. I had my personal websites on a second account (that should have been a free plan - one deploy every three years, next to zero traffic), and they deleted them and continue to bill me. I've had multiple customer service interactions asking them to stop billing me and they're nothing short of rude and unhelpful. "We've told you the dozen…

I have been mulling over switching from Netlify to Vercel for a while for my personal site too. Netlify has weird defaults for everything. It caches almost nothing by default. I had to create a Netlify headers file to tell them that actually yes I would like my CSS, JS, and fonts cached. Absolutely absurd I had to do that. Even for font files. Because as we all know, the font files change so often that we should neve…

> It caches almost nothing by default.

Now I have to go check and see if it's not caching our fonts. Geez, I just assumed...

> fear of changing DNS settings

I recently had to migrate DNS to Cloudflare to stop a 40,000 QPS DDoS attack. DNS migration was slow, but painless. As long as you plan accordingly and set up the new destination in advance, it should be fine. Hosting DNS at a DNS provider will give you extra flexibility in the future.

Re: Heroku has been running a second copy of my scheduler instance

#57

Post by tannhaeuser »

lloydatkinson wrote: post #49
echelon wrote: post #41

Earlier quoted context omitted.

Netlify did this to me too! I have my startup with them and pay them a hefty bill. I had my personal websites on a second account (that should have been a free plan - one deploy every three years, next to zero traffic), and they deleted them and continue to bill me. I've had multiple customer service interactions asking them to stop billing me and they're nothing short of rude and unhelpful. "We've told you the dozen…

I have been mulling over switching from Netlify to Vercel for a while for my personal site too. Netlify has weird defaults for everything. It caches almost nothing by default. I had to create a Netlify headers file to tell them that actually yes I would like my CSS, JS, and fonts cached. Absolutely absurd I had to do that. Even for font files. Because as we all know, the font files change so often that we should neve…

> fear of changing DNS settings

Is that really a thing, with HNers of all people? You just use your registrar's/nameserver operator's web tool to point your DNS name to a new IP. Transferring your DNS to a new registrar might be a bit more involved but is guaranteed to be also possible by domain market regulation. Or maybe it is a problem of cache invalidation and/or lack of control over exact timing of DNS switchover? Or, idk, possibly Google-owned TLDs like .dev require Google Domains as registrar? Or does Google Domains (or GoDaddy or other big registrar) make transferring your domain difficult in a dark pattern way? Doubt it though, if even changing pointed to IPs appears difficult, which however shouldn't be something a registrar would have an interest in making difficult.

Re: Heroku has been running a second copy of my scheduler instance

#58

Post by echelon »

tannhaeuser wrote: post #57
lloydatkinson wrote: post #49

Earlier quoted context omitted.

I have been mulling over switching from Netlify to Vercel for a while for my personal site too. Netlify has weird defaults for everything. It caches almost nothing by default. I had to create a Netlify headers file to tell them that actually yes I would like my CSS, JS, and fonts cached. Absolutely absurd I had to do that. Even for font files. Because as we all know, the font files change so often that we should neve…

> fear of changing DNS settings Is that really a thing, with HNers of all people? You just use your registrar's/nameserver operator's web tool to point your DNS name to a new IP. Transferring your DNS to a new registrar might be a bit more involved but is guaranteed to be also possible by domain market regulation. Or maybe it is a problem of cache invalidation and/or lack of control over exact timing of DNS switchove…

> Is that really a thing, with HNers of all people?

You want it to be hitless. Unfortunately DNS can take days to fully propagate and you may not see mistakes until it's too late to fix them. This can cause horrifying outages.

HNers should be respectful of DNS changes and plan accordingly.

Re: Heroku has been running a second copy of my scheduler instance

#59

Post by dentarg »

Probably related to https://status.heroku.com/incidents/2531 ("On April 6, 2023, between 21:20 UTC and 22:30 UTC, our Common Runtime customers were unable to provision dynos in the US and EU regions. This issue impacted starting/scaling dynos in existing apps, one-off dynos, builds, and release phases.")

Re: Heroku has been running a second copy of my scheduler instance

#60

Post by rmnclmnt »

Great read, thanks!

I haven't heavily used Celery anymore for the past few years, but I remember trying to setup a HA Celery Beat schedule process with Kubernetes in 2019 and hitting this exact same issue! There was an alternative scheduler[1] storing its state in Redis instead of a local file, after a few fixes it was working pretty well: multiple beat processes running concurrently and not firing twice a scheduled task.

Don't know if that still works though.

[1] https://github.com/mixkorshun/celery-beatx

Post reply on HN