Live data from Hacker News

InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

community.influxdata.com

41–50 of 278 posts

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#41
post #33

Earlier quoted context omitted.

Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications. I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.

We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.

As a buyer I have come to expect good vendors to design systems so mistakes (my team or yours) don't cost me sleep or you business.[3] i.e.

- they do soft-deletes before hard

- have robust access control systems and partitioning - so we don't have to give access to everyone in the org to object model with full r/w

- don't instantly nuke the account if a payment goes astray or delayed - try to reach out before to a point of contact before pulling the plug, payment systems can be messy for all sorts of reason, ask before assuming the worst.

- customer managers who can connect couple of times a year which usually benefits the vendor as upsells happens on good % of those connects.

- also small things like training, certification

- Deprecation of service is handled slowly(1 Yr would be expected) and in multiple phases with multiple modes of communication.

Not all companies can move fast to plan and execute a major change in location like this in 4 months, bare minimum you would have to consider

  - End customers (your customer's customers) may need to be notified and may need to sign off

  - Compliance and GDPR DPA changes - both end customers and internal ones

  - DR, BCP concerns have to be planned for , not all GCP regions are equivalent.

  - Documentation and certifications like SoC, ISO, PCI, HIPAA etc usually mean ton of paperwork to modify

  - SRE/Devops may have to move other services along with telemetry on InfluxDB, may need network whitelisting from their  customers, things typically break when moving, need to plan dry runs, rollbacks and so on.
A better way to handle service closure would be to shut down but not delete on the planned date[1] , and offer data export separately for few weeks/month after[2].

You can definitely do better than shutting down service and deleting data at the same time .

[1] I would do this for internal customers let alone external paying ones

[2] You could have even charged for this to offset any costs, most customers wouldn't have a problem paying if they really needed it.

[3] Not trying to imply InfluxDB is doing these things, or isn't a good vendor, these are some criteria I have come to measure new vendors by.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#42
post #34
post #29

Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influx…

Paul: I'm surprised you didn't do a scream test. Not everyone is going to see those emails and even those that do may not understand what they are reading. Internally at my company we always do scream tests as part of our EOL process because we know we can't reach everyone, even our own employees. https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-... Fun story: my mortgage got sold last year. Not the first…

[deleted]

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#43

Earlier quoted context omitted.

Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications. I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.

Why do customers believe that they don’t need to read their emails?

I missed an email that a train I was scheduled for had changed schedules on a trip earlier in the year. Why? Because Eurostar sends me maybe weekly email marketing messages that get filtered to one of my Google tabs because I maybe take them every couple years. It's probably unreasonable to expect that I'll see a reasonably last minute update though I'm not sure what a good alternative is.

I get probably 100+ emails a day that hit my inbox in some form and occasionally fairly important ones get mixed in with the mostly dross though Gmail does a pretty good job overall.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#44
post #33

Earlier quoted context omitted.

Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications. I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.

We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.

[deleted]

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#46
This could have been easily mitigated with a giant red ugly banner "YOUR DATA WILL BE LOST IN X DAYS. MIGRATE NOW".

Three emails clearly wasn't enough right? Now their name is in the dumps, customers are pissed and my only exposure to influxdb is a negative one.

I hope other saas guys learn this very expensive lesson.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#47
post #29

Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influx…

Wow, that's pretty pathetic and your attitude "we can't help our customers" is even more damning. Email is not reliable enough to simply rely on a few email blasts for this.

I would expect:

* Those 3 "email blast" notifications. I'm guessing one of two things happened here:

  * You sent them as an "email blast" from a marketing-type email service. These hit email filters because they came from a known spam IP.

  * You sent them as a transactional email, but blasted them too quickly and got pegged for spam. Never hit the inbox.
* Increasingly common "you haven't migrated emails" if you still detect traffic on these instances. This is pretty critical since some companies might not realize they have affected They should, but things get complex.

* Ideally, an automated transfer to another region with automated forwarding. It's okay to have poor performance, but it's not okay to go "poof" entirely.

* A soft-delete at the deadline, with 90 to 180 days to finalize migration. If this is costing you dearly, then drive prices up, but don't hard delete data.

Frankly, the last one is the real issue. It's literally unbelievable that a database provider didn't soft-delete. Further, I would expect that you'd be able to migrate these to another region to get customers back up an running.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#48
post #33

Earlier quoted context omitted.

Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications. I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.

We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.

I know that this is a stressful time and it’s all hindsight, but there are two different contact methods that don’t rely on email available to you in migrations like this:

Going read-only, waiting 2 weeks, and then deleting. The contact method is peoples’ alerting systems as writes stop working.

Putting a message on your service dashboard indicating the upcoming action. The contact method is exactly what it sounds like, and it’s the only other place you can stick text and know for sure all your customers can access it.

It will probably help customer relations if you don’t hide behind the defense of only having email - there are a few strategies for this that you can use in the future. Best of luck on the road ahead - I know this must be a particularly stressful time.

Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion

#50
post #33

Earlier quoted context omitted.

Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications. I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.

We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.

As a CTO, you should make yourself aware of what a scream test is

https://www.v-wiki.net/scream-test-meaning/

Basically, you just turn stuff off, and make people scream, while they can turn it back on. You could have did this a month ago, as a critical warning of impending termination of systems.

You didn't do that.

Instead, you sent a few emails, which itself isn't even a guaranteed delivery. Again, as a CTO, you should know that email is not a guaranteed deliverable.

Post reply on HN