Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

community.influxdata.com

Post reply on HN
Print view View original

181–190 of 278 posts

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#181

Post by XCabbage »

StopHammoTime wrote: post #58

It seems this probably happened due to some regulation or other. The sunset date for the service should have been a month prior so that influx could have kept the data legally until the 30th in case of this situation happening. They wanted to have the euros flowing in right until the last minute. 1. Flash messages on all user facing consoles. 2. No new resource able to be created for 6 mo this. 3. Emails. 4. Service…

There's no regulatory consideration involved as far as I can tell. On Slack at https://influxcommunity.slack.com/archives/CH8TV3LJG/p168894... they explain the shutdown thus:

> "The region did not get enough usage or growth to make it economically viable to operate, so it became necessary for InfluxData to discontinue service in those regions."

So it's worse than you believe. Yes, the handling is a scandal for all the reasons you say. But they weren't even pushed into this by some regulatory issue; it's pure cost-cutting.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#182

Post by willvarfar »

RedShift1 wrote: post #156
willvarfar wrote: post #153

Earlier quoted context omitted.

(not GP) I've been bitten by the old Influx and had to migrate to something we could trust... (Influx basically tacitly admitted that the original architecture was pretty poo and they've since swapped out the engine (twice?), but it smells a bit like mongodb trying to reinvent itself and distance itself from all the early web-scale claims, so I'm kinda skeptical). So I've rolled our own with MySQL+tokudb, but that's…

I use Timescale and I can recommend it. The reason why I'm still using influx too (1.7) is because it's unmatched in its data storage efficiency and query performance. You can get close with Timescale, but its main power is having the query power of PostgreSQL, if you have room for the extra hardware resources it requires.

(My memory is that we kept on influx including 1.7, but it was a while ago now so memory might be fuzzy)

I guess influx perf and efficiency is really depending a lot on your data shape then :)

Our experience was that performance dropped off a cliff if you had too much data, too much tagset cardinality, or else your query was too broad. And when it failed, it lost data.

In fact, it lost data generally. When we were replacing it we dual-ran an ACID DB version (which, with tokudb, was fast enough to keep up (although we didn't index every tag column)). So we did a diff and discovered just small random holes in the influx data that we'd never noticed before.

We had other considerations when we went mysql, as in, we were already using it. If shopping for a standalone solution to start a new project on, I'm thinking Timescale is the go-to these days?

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#183

Post by kawsper »

sofixa wrote: post #131
galleywest200 wrote: post #78

Earlier quoted context omitted.

We self host influxdb, never had this problem.

Which problem? Of the massive breaking changes between 0.8 and later, and then between 1.x and 2.x? Not to mention InfluxQL to Flux? Also, they did remove clustering in the open source version which was a very poor move from a PR perspective. And in my view, they have never recovered from it - years ago it was Prometheus vs InfluxDB for (non-SaaS) observability metrics, nowadays the only question is which backend for…

And now they've released 3.0.0 in their cloud, which they claim is backwards compatible, but lets see.

I sometimes wonder if vendors realize that they put their customers into a buying mode when they do this, when our options are:

- Upgrade to the new version of product X.

- Change to vendor or tooling completely as we're already changing everything.

We might pick another system if we feel like it is more stable.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#184

Post by steveBK123 »

SkyPuncher wrote: post #47
pauldix wrote: post #29

Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influx…

Wow, that's pretty pathetic and your attitude "we can't help our customers" is even more damning. Email is not reliable enough to simply rely on a few email blasts for this. I would expect: * Those 3 "email blast" notifications. I'm guessing one of two things happened here: * You sent them as an "email blast" from a marketing-type email service. These hit email filters because they came from a known spam IP. * You se…

It's crazy the last email blast was ~6 weeks ago.

And the shutdown isn't even a month / quarter end, so seems even less like a billing cycle thing.

There's so many more mature ways to do a graceful paid service shutdown. Disable reads first so people get errors & contact support. Then disables writes as well. Somewhere a few weeks later you can consider deleting data.

As others have said, I've worked at megacorps with internal systems that had more mature migrations than this. Honestly I have run internal apps at sub-1000 employee firms where we did various forms of scream tests and soft deletes before MONTHS later, even daring to delete data on disk.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#185

Post by petercooper »

tredre3 wrote: post #4

> According to the support, the notification emails to the users were sent on Feb 23, Apr 6 and May 15th. However, we did not receive those at all. If true, this is concerning. One message getting lost in spam understandable. But three over 6 months would imply they're being blacklisted and/or their mail sender is simply broken. Do serious companies not have canaries or other checks in place to ensure their notificat…

But three over 6 months would imply they're being blacklisted and/or their mail sender is simply broken.

Plus some enterprises change their mail policy out of nowhere. I have a lot of subscribers from enterprise companies, but sometimes every mail delivered to some company comes back with something like "Your address is not on our whitelist. We only accept approved emails." and every subscriber is hard bounced off the list and that's that. (They aren't paying me, though, so less of a big deal.)

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#186

Post by gtirloni »

sofixa wrote: post #131
galleywest200 wrote: post #78

Earlier quoted context omitted.

We self host influxdb, never had this problem.

Which problem? Of the massive breaking changes between 0.8 and later, and then between 1.x and 2.x? Not to mention InfluxQL to Flux? Also, they did remove clustering in the open source version which was a very poor move from a PR perspective. And in my view, they have never recovered from it - years ago it was Prometheus vs InfluxDB for (non-SaaS) observability metrics, nowadays the only question is which backend for…

> they did remove clustering in the open source version which was a very poor move from a PR perspective. And in my view, they have never recovered from it

I still remember this. We were ready to standardize on InfluxDB when we got a taste of their business practices.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#188

Post by gtirloni »

So many easy ways that this could have been avoided. sigh

- Phone calls

- Scream tests

- Monitor services still in use. Contact these customers individually

- ...

Not a single individual said "Gee, people are still using that DC, should we really destroy it?"

Either this shows Influx is really naive and inexperienced or... they are in deep trouble cash-wise and were working in panic mode to cut costs.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#189

Post by dangoodmanUT »

It should not be understated how bad this is. Your #1 expectation as a cloud database provider is to keep data safe and recoverable.

I hope for at least their sake they took a backup of everyone's DB that could be restored in another region, but based on the fact that they didn't do a scream test, I doubt they thought about this either.

This must have been forced upon by upper management, because there is no way someone along the chain to actually delete data did not suggest a scream test. No way someone didn't say "this is a terrible idea, email is not reliable".

Adding Influx right next to GCP of providers I'm never using. Self-hosting is the way, and use ClickHouse.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#190

Post by dangoodmanUT »

pauldix wrote: post #29

Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influx…

> I realize that it's not ideal that we've shut down this system

Not ideal???

You backed up everyone's DB and moved that to another region so they can just restore and change DB endpoints, right?

I don't believe that someone along the chain didn't suggest a scream test or similar. If they did, they must have been ignored.

Post reply on HN