Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

community.influxdata.com

Post reply on HN
Print view View original

231–240 of 278 posts

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#231

Post by jacquesm »

dangoodmanUT wrote: post #189

It should not be understated how bad this is. Your #1 expectation as a cloud database provider is to keep data safe and recoverable. I hope for at least their sake they took a backup of everyone's DB that could be restored in another region, but based on the fact that they didn't do a scream test, I doubt they thought about this either. This must have been forced upon by upper management, because there is no way some…

This is pretty much corporate suicide. I really don't understand what they are trying to achieve with this and their attitude in this thread is baffling.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#232

Post by jacquesm »

heyoni wrote: post #202
d33 wrote: post #194

Earlier quoted context omitted.

In case anyone else is wondering: > The Scream Test is simple – remove it and wait for the screams. If someone screams, put it back. The Scream Test can be applied to any product, service or capability – particularly when there is poor ownership or understanding of it’s importance. https://www.v-wiki.net/scream-test-meaning/

Maybe this is the scream test…just done badly

No, it's gone. For it to be a scream test they'd have to be able to retrieve the data somehow and they can not.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#233

Post by TeMPOraL »

jacquesm wrote: post #111
WirelessGigabit wrote: post #102

Earlier quoted context omitted.

You know, I don't really care that mortgages get sold, but the fact that I need to go out of my way every time to make sure I update my information in their system is astounding.

Technically that shouldn't even be your problem.

Neither should be a random third party defrauding a bank, but banks are masters at outsourcing work and responsibility onto their customers, and that's why it's called "identity theft" and suddenly I am at fault somehow.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#234

Post by TeMPOraL »

fbdab103 wrote: post #72
SkyPuncher wrote: post #66

Earlier quoted context omitted.

You worded that better than I could have. I ignore most of my vendor's emails because they're simply trying to spam me at this point.

I was also thinking that salesmen will cold-call/email me at least three times before they give up. People with whom I have no business relationship try harder than a corporation taking someone's money.

Welcome to modern economy. This applies to B2C, and as we can see it, B2B too. Getting a new customer is all the shitty vendors care about. Keeping existing customers is apparently just cost.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#235

Post by yencabulator »

KomoD wrote: post #212
arp242 wrote: post #128

Earlier quoted context omitted.

> we made our best efforts to notify affected users You call three emails (the last of which was almost 2 months ago) "best efforts"? I had to read your message three times because this is so reality-defying preposterous I just couldn't believe I didn't miss anything. How about warnings on the dashboard? How about an intentional error (or limited service interruption) so that people would log in to their dashboard?

> How about warnings on the dashboard? They did have a warning on the dashboard, problem is a lot of people don't check the dashboard because they don't need to, as they just view everything through grafana, etc. They also had a notice on the status page

Getting those people's attention would be what the intentional errors are for.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#236

Post by sofixa »

flagrant_taco wrote: post #203
flagrant_taco wrote: post #191

Earlier quoted context omitted.

If it's backwards compatible why would they bump the major version? Either there should have been a breaking change in there or they don't understand symantic versioning

Not sure why the downvotes here, a database service misusing semantic version is itself a red flag in my opinion. If major releases don't indicate a breaking change I'm not as confident in what might be part of a minor or patch release.

To be fair, they have a bunch of associated tooling like their query language and UI, so it might have been those that have merited a bump to 3.x

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#237

Post by TeMPOraL »

MarkSweep wrote: post #82
jlund-molfese wrote: post #38

Earlier quoted context omitted.

I even (sort of) do this when I'm deprecating something which my team is the only user of, because sometimes it's hard to tell if something's really unused! First shut off the VPC access while leaving all the other infrastructure and data intact, wait a week or two and see if everything breaks, then get rid of everything else

There is another variant of this: if you can show that the code you are deleting never worked, there is no need to do a scream test. That is, if anyone cared about the code you are deleting, they would have already been screaming.

I would be careful with that. Maybe they did scream, but you haven't heard it, and they worked around the issue. Or maybe they did their workaround without saying anything. Or maybe you're wrong about your code not working. It actually may be working in some way that you don't know of, but is useful to someone.

To use an ecosystem analogy, once you expose your software to the world beyond your own dev environment, even internally, you'll eventually find that something colonized it - much like everything on this planet that isn't being actively and regularly scrubbed.

In my own career, I've seen cases of this. For example, once we were tweaking a little embedded database that supported a half-finished feature meant for internal use, and only then we (as in everyone in the dev team) learned that somehow, the QA & deployment support people got wind of it, and were scripting against exposed parts of that DB for a good year. And, it turns out, it wasn't the only part of the software that we thought of as incidental phenotype (or didn't think of at all), and the other team considered stable behavior.

See also the so-called Hyrum's Law: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody."

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#238

Post by mst »

tetha wrote: post #214
mst wrote: post #152

Earlier quoted context omitted.

When ovh sunsetted a class of VPSen and I'd completely failed to notice they were going to do that, I asked nicely in the support ticket I'd sent in and they turned it back on for a few days while I shifted the data to a replacement (which was still an ovh VPS, it had been Just Working long enough that I didn't feel like I'd been mistreated, more lulled into complacency by the lack of problems). I think requiring a t…

I mean this is why you do these projects on two different timelines: The internal timeline and the external timeline. Externally you communication: Different announcements each month, final notices at T+5M, System will be deleted at T+6M, data will be lost at that point and so on. Internally (at least at work) such a timeline is more that at T+6M, we cut access to the systems. Afterwards, systems not accessed for 2-4…

Yep, and in certain in house situations it's best to keep a backup around for ~13 months in case there's an obscure business process that only gets done once a year. (I'm aware that some people reading that sentence are going to go wtf at the idea that that's anybody's problem except whoever didn't tell you said business process even existed, but if it's a sufficiently critical finance or HR thing it tends to rapidly become everybody's problem so I like to have options)

Agree absolutely wrt complacency, I believe I asked for less than a week because I actively preferred a situation where I had to get on it immediately.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#239

Post by tracker1 »

shin_lao wrote: post #218
StopHammoTime wrote: post #58

It seems this probably happened due to some regulation or other. The sunset date for the service should have been a month prior so that influx could have kept the data legally until the 30th in case of this situation happening. They wanted to have the euros flowing in right until the last minute. 1. Flash messages on all user facing consoles. 2. No new resource able to be created for 6 mo this. 3. Emails. 4. Service…

It's a cost-cutting measure that reeks of a company trying to cut costs as fast as possible.

They still should have at the very least done a backup of each customer DB in those regions and created an option to download and/or restore to a new region and kept those for at least 30-90 days.

A scream test would have been a better option in addition to the above.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#240

Post by visionik »

pauldix wrote: post #29

Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influx…

Three emails are not "best efforts". 4.5 months notice is not "best effort".

My opinion on best effort: I founded, ran, and sold a SaaS company used by some of the most well known companies in the world. Our "best effort" was a minimum of 12 months notice, with a six month grace period afterwards. Emails weekly. Phone calls at least once a month. Reach out to customer leadership if no response. Then scream test as others suggested.

Post reply on HN