Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

community.influxdata.com

Post reply on HN
Print view View original

71–80 of 278 posts

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#71

Post by pmh »

js2 wrote: post #34
pauldix wrote: post #29

Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influx…

Paul: I'm surprised you didn't do a scream test. Not everyone is going to see those emails and even those that do may not understand what they are reading. Internally at my company we always do scream tests as part of our EOL process because we know we can't reach everyone, even our own employees. https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-... Fun story: my mortgage got sold last year. Not the first…

> scream test

I didn't realize there was an actual name for this, so I'll add another for the thread that I haven't seen brought up: email is evil in operations (EIEIO, like the nursery rhyme)

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#72

Post by fbdab103 »

SkyPuncher wrote: post #66
imglorp wrote: post #62

Earlier quoted context omitted.

Another problem is that service providers frequently poison the email channel with important sounding engagement dreck and we are now conditioned to ignore it. * Important: migrate to this new feature immediately or you risk missing out!! Vs * Important: migrate your data immediately or you risk losing it!!

You worded that better than I could have. I ignore most of my vendor's emails because they're simply trying to spam me at this point.

I was also thinking that salesmen will cold-call/email me at least three times before they give up. People with whom I have no business relationship try harder than a corporation taking someone's money.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#73

Post by amluto »

pauldix wrote: post #33
chillfox wrote: post #30

Earlier quoted context omitted.

Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications. I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.

We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.

> After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss.

Using billing contacts for this is a mess. The billing contact could be an accounts payable department that will check the invoice against the contract or PO and pay it. It could be an outsourced office that has no idea what a database is. It could be someone who only catches up on email once every few weeks.

What is isn’t in a technical contact who knows what the shutdown of a database means.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#74

Post by thelittlenag »

I've seen better communications around company-internal services that have been deprecated and for which a replacement exists that we need to migrate to. Heck, I've seen this a couple of times.

We had even tried out Influx a few different times. It was always ok, but never quite good enough. Now with this, I think, this seals the deal on me ever considering Influx either as a product or as a service.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#75

Post by devrand »

SkyPuncher wrote: post #47
pauldix wrote: post #29

Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influx…

Wow, that's pretty pathetic and your attitude "we can't help our customers" is even more damning. Email is not reliable enough to simply rely on a few email blasts for this. I would expect: * Those 3 "email blast" notifications. I'm guessing one of two things happened here: * You sent them as an "email blast" from a marketing-type email service. These hit email filters because they came from a known spam IP. * You se…

> Ideally, an automated transfer to another region with automated forwarding. It's okay to have poor performance, but it's not okay to go "poof" entirely.

If the data is moving between countries then this is not an option. Your clients may have legal or contractual obligations with respect to data location.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#76

Post by scosman »

remram wrote: post #27
scosman wrote: post #24

Earlier quoted context omitted.

If you are shutting it down, you can pay for the grace period. Period.

Just shut it down for real (after proper early warnings), so you save on compute and no one is confused about the state, and offer data retrieval for the grace period. Brownouts are great for API changes, but not very useful before a full shutdown.

You assume warnings reach users. Some people miss emails. Fewer miss a service going offline. Keeping data after shutdown is a good backup.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#77

Post by ksajadi »

Unfortunately we’ve been bitten by influx operation issues a few times before. We adopted influxDB long time ago and always had to deal with breaking changes for each upgrade and every time we had an issue their answer would be: upgrade the latest version and see if it continues.

Then recently they made a change to Telegraf that broke all our data collection because they changed the environment variable replacer and their Jsonnet parser broke.

Now this. Shutting down a region without brownouts and only emails is not operationally acceptable.

We’ve moved on from influxDB for a while and only rely on telegraf now.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#78

Post by galleywest200 »

ksajadi wrote: post #77

Unfortunately we’ve been bitten by influx operation issues a few times before. We adopted influxDB long time ago and always had to deal with breaking changes for each upgrade and every time we had an issue their answer would be: upgrade the latest version and see if it continues. Then recently they made a change to Telegraf that broke all our data collection because they changed the environment variable replacer and…

We self host influxdb, never had this problem.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#79

Post by kelnos »

pauldix wrote: post #33
chillfox wrote: post #30

Earlier quoted context omitted.

Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications. I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.

We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.

Ignore for a moment what you think your customers should have done, and look at the actual outcome. Some customers did not know about the shutdown and deletion, and have now lost data. You telling them "well you should have read your email" is not going to satisfy them, even if you think it should.

All you've done is told your customers that their data isn't safe with your service. This was an easily-avoidable "own goal" situation.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#80

Post by vbezhenar »

pierat wrote: post #50
pauldix wrote: post #33

Earlier quoted context omitted.

We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.

As a CTO, you should make yourself aware of what a scream test is https://www.v-wiki.net/scream-test-meaning/ Basically, you just turn stuff off, and make people scream, while they can turn it back on. You could have did this a month ago, as a critical warning of impending termination of systems. You didn't do that. Instead, you sent a few emails, which itself isn't even a guaranteed delivery. Again, as a CTO, you sh…

You can't turn system off if customer paid for it. That's the breach of contract.

If you keep customer data after payment period, you're losing money.

So scream test will cost you money.

They decided that this cost is not appropriate payment to compensate for possible reputation loss.

Whether that's a good business decision or not, time will tell.

Post reply on HN