Heroku has been running a second copy of my scheduler instance • Hacker News Board

Re: Heroku has been running a second copy of my scheduler instance

#81

Post by scott_w » Tue, Apr 18, 2023, 12:04 PM UTC

zakki wrote: post #66

echelon wrote: post #58

Earlier quoted context omitted.

> Is that really a thing, with HNers of all people? You want it to be hitless. Unfortunately DNS can take days to fully propagate and you may not see mistakes until it's too late to fix them. This can cause horrifying outages. HNers should be respectful of DNS changes and plan accordingly.

You can change your DNS TTL to let say 5 minutes before you move to new IP. You can change it 3 days before the movement(?).

A story from 10 years ago providing services to the education sector:

I did exactly what you suggested, even leaving extra time (a full weekend!) for the DNS changes to propagate. What happened?

Turns out local authorities don't all respect DNS TTL settings and we had a major outage. I had to on-the-fly learn how to configure iptables to act as a proxy for the new server.

The "proxy" was still receiving requests 2 weeks later.

Re: Heroku has been running a second copy of my scheduler instance

#82

Post by abatilo » Tue, Apr 18, 2023, 12:23 PM UTC

In 2017, I worked at a company that built a websocket based app and ran it on Heroku. We had this exact same problem and all 5 of the engineers at the time spent all day until we realized that we had socket connections still connected to some "ghost dyno".

I can't remember exactly what we did but I think we just kept deploying over and over again until the old one finally went away.

Re: Heroku has been running a second copy of my scheduler instance

#83

Post by LegitShady » Tue, Apr 18, 2023, 12:30 PM UTC

joering2 wrote: post #39

ranting-moth wrote: post #38

Earlier quoted context omitted.

File a chargeback each time they charge your card. The merchant has to pay the transaction fees both ways and a fee for the chargeback.

But don't do it if its Google. I had been charged for g-suite for domain that expired and someone else picked it up (and parked), long story short after 3 months of back-and-forth with "support" I gave up and started with chargebacks. It was 7 of them, each month one for $6, until one sunny day I woke up and my own gmail account, my youtube TV and Google Play on 2 Android devices all got blocked with messages popping…

That's a good reason to never use Google ever. They did you a favor.

Re: Heroku has been running a second copy of my scheduler instance

#84

Post by SOLAR_FIELDS » Tue, Apr 18, 2023, 12:49 PM UTC

ranting-moth wrote: post #45

joering2 wrote: post #39

Earlier quoted context omitted.

But don't do it if its Google. I had been charged for g-suite for domain that expired and someone else picked it up (and parked), long story short after 3 months of back-and-forth with "support" I gave up and started with chargebacks. It was 7 of them, each month one for $6, until one sunny day I woke up and my own gmail account, my youtube TV and Google Play on 2 Android devices all got blocked with messages popping…

Yes, chargeback is pretty much a nuclear option. But in his case they'd already closed his account and are not responding, so time for the nukes.

Just a heads up, Salesforce owns Heroku, so if OP did this he had better be willing to lose access to anything else Salesforce related. Like Slack or Quip

Re: Heroku has been running a second copy of my scheduler instance

#85

Post by jmuguy » Tue, Apr 18, 2023, 1:23 PM UTC

8n4vidtmkvmk wrote: post #19

teaearlgraycold wrote: post #13

I vote for moving to Render! My god I wish I could short Heroku stock.

Looks really expensive. And weird pricing. I click "pricing" and it gives me per user costs. Wat? I thought I was hosting stuff, not managing teams. Click on +Computing costs and it's like $0! $0! $0! Check the fineprint and its like 4GB RAM and 2 CPUs is $85/mo, or you can get the same at DigitalOcean for $24. Then add Postgres for another $95. Or if you're cheap like me, you self-host that in your $24 droplet and p…

Its been a little while since I looked at Render and their pricing definitely used to be a lot simpler. Charging per user is just stupid. And then weird compute costs with plans like "Pro Plus" and "Pro Ultra". "Web Services" and "Private Services". WTF is this shit.

Re: Heroku has been running a second copy of my scheduler instance

#86

Post by danjac » Tue, Apr 18, 2023, 1:28 PM UTC

andrewstuart wrote: post #40

>> So is it Celery then? This awesome library has its fair share of complaints about duplicated task execution — most are due to misconfiguration. I scanned dozens of Stack Overflow threads and GitHub issues in Celery’s repository. Very few were relevant to my case, and none of the fixes were. I was so annoyed by the pain of configuring Celery I stopped my project and wrote my own job queuing system. Job and message…

If you just need to send a set of emails once a week, the lowly cron job is fine. You don't need celerybeat.

RQ or Huey are otherwise fine for anything more complicated. I've rarely run into a situation where Celery is better, but teams just use it because it's the "default".

Re: Heroku has been running a second copy of my scheduler instance

#87

Post by aeyes » Tue, Apr 18, 2023, 1:37 PM UTC

Even though it won't help you now, instead of celery-once I highly recommend using the celery-redbeat scheduler which uses a distributed lock to ensure that only one scheduler instance is active. It's plug&play and compatible with all the traditional task scheduling methods which beat supports.

Re: Heroku has been running a second copy of my scheduler instance

#88

Post by dclowd9901 » Tue, Apr 18, 2023, 1:53 PM UTC

shibel wrote: post #47

dclowd9901 wrote: post #7

Author and me have so much in common. From journalism to software engineering and our formative career shift years were spent on home made music apps. Now I’m wondering just how many of us are out there.

Author here. Small world! Would love to see your app if you don’t mind sharing. I’m @SHxKM on Twitter if you’d like to connect.

Unfortunately it’s abandonware now. It was called Showhopping and it would use the Scrobbler API and a list of your favorite bands and find out if any bands you might like (including the ones you listed) were playing in a given area any time soon. You could give it a location and a time range and it would give a list of shows playing on a map with a percentage back based on the likelihood that you’d enjoy the show. It was very lightweight with no sign up and I had it in mind people could use it like Yelp essentially, just ad-hoc.

https://lifehacker.com/showhopping-finds-upcoming-shows-arou...

Anyway, nice to meet you! I enjoyed your write up.

Re: Heroku has been running a second copy of my scheduler instance

#89

Post by EntrePrescott » Tue, Apr 18, 2023, 1:59 PM UTC

Kinda refreshing - within the ever growing genre of stories about Heroku's downfall into junk status - to find such an article with a higher level of detail and analysis of the origins of the problem.

There's one thing that somewhat surprised me though: this line:

> "In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development"

Was it? Really? I mean, I'm not saying the author did anything wrong in choosing Heroku at the time (after all, who could have known back then what a dumpster fire Heroku would turn out to be), and the commercial success Heroku achieved before going down the gutter shows that it must have had some good reasons going for it, at least on paper before the users are confronted with problems that make them judge the situation differently…

… but I don't really get where that "pretty much the only option" vibe would come from. Maybe It's just me having completely different selection criteria, but from my perspective, there were lots of other options to "get started with web-development" and Heroku was just one among many. I don't see at all by what measure Heroku would have seemed like "pretty much the only option"?

Re: Heroku has been running a second copy of my scheduler instance

#90

Post by jazzyjackson » Tue, Apr 18, 2023, 2:08 PM UTC

EntrePrescott wrote: post #89

Kinda refreshing - within the ever growing genre of stories about Heroku's downfall into junk status - to find such an article with a higher level of detail and analysis of the origins of the problem. There's one thing that somewhat surprised me though: this line: > "In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development" Was it? Really? I mean, I'm not saying the auth…

Assuming they wanted a webserver + database and wanted to learn web stuff, not server admin / unix stuff to configure a LAMP stack or whathaveyou + certbot etc. Deploying on your own domain of legitimately daunting without a lot of prerequisite knowledge. Heroku automated a lot of that and some people learned it as the only way to deploy without learning the manual process first, and that's fine IMO.