251–260 of 278 posts
This could have been easily mitigated with a giant red ugly banner "YOUR DATA WILL BE LOST IN X DAYS. MIGRATE NOW". Three emails clearly wasn't enough right? Now their name is in the dumps, customers are pissed and my only exposure to influxdb is a negative one. I hope other saas guys learn this very expensive lesson.
According to other comments, apparently there was a banner, but it was tiny and green instead. Pure genious.
Earlier quoted context omitted.
It's unusual to read that InfluxDB is fast and efficient. Did you try VictoriaMetrics? It usually needs 10x less RAM than InfluxDB for the same workload, especially when the number of active time series is high. It also uses less CPU and disk space on the same production workload. [1] [1] https://valyala.medium.com/insert-benchmarks-with-inch-influ...
I've heard of VictoriaMetrics before but haven't had time to play with it. InfluxDB is also now ingrained in a production system so replacing it is not straightforward. The query language is also different meaning everything that uses it will need to be updated too, and coming from a mainly SQL background, PromSQL/MetricsQL looks oddly weird.
[1] https://valyala.medium.com/promql-tutorial-for-beginners-9ab...
TLDR: “We don’t plan to delete your data again in the forseeable future”
Full quote: “You can sign up for a new account here InfluxDB Cloud using a different region. We want to assure you that there are no more scheduled shutdowns planned. Therefore, once you have created the new account and begin writing to it, we do not foresee any data loss going forward”
No more information was provided in the mail.
Earlier quoted context omitted.
No one is saying that existing mortgages would be re-priced. Those are contracts and you can't unilaterally change them. What I and others are saying is that if you changed the obligations of buyers such that mortgages became more expensive to service, that those servicing cost increases would ultimately be borne by the mortgage borrowers rather than eaten by mortgage lenders out of the goodness of their hearts. > th…
> What I and others are saying is that if you changed the obligations of buyers such that mortgages became more expensive to service, that those servicing cost increases would ultimately be borne by the mortgage borrowers rather than eaten by mortgage lenders out of the goodness of their hearts. I got what you and others are saying but I've been trying - and failing, apparently - to point out that the costs for a mor…
If it made zero difference, they would Fedex the documents, but they dont.
It should not be understated how bad this is. Your #1 expectation as a cloud database provider is to keep data safe and recoverable. I hope for at least their sake they took a backup of everyone's DB that could be restored in another region, but based on the fact that they didn't do a scream test, I doubt they thought about this either. This must have been forced upon by upper management, because there is no way some…
This is pretty much corporate suicide. I really don't understand what they are trying to achieve with this and their attitude in this thread is baffling.
The least nonsense thing I can think of is that they weren't paying their bills. They weren't paying rent, the landlord locked them out and repo'd their servers, or something similar. (perhaps they were inspired by Elon Musk's recent antics?)
If that were the case, they would not disclose that that's what happened. If they disclosed that, all of their other customers would immediately begin migrating their data; not tomorrow, not next week, now.
If there were any excuse they would give it. "We were hacked!" "It was a disgruntled ex-employee!" "The datacenter burned down!" "It's those dirty EU data laws!" etc.
Shutting down the data center and deleting all the data (without migrating) at the same exact time and that was Plan A--nah I don't believe that.
Earlier quoted context omitted.
As a buyer I have come to expect good vendors to design systems so mistakes (my team or yours) don't cost me sleep or you business.[3] i.e. - they do soft-deletes before hard - have robust access control systems and partitioning - so we don't have to give access to everyone in the org to object model with full r/w - don't instantly nuke the account if a payment goes astray or delayed - try to reach out before to a po…
> don't instantly nuke the account if a payment goes astray or delayed Hetzner deleted my server just one week after my payment due date. My credit card failed the payment for some reason. I didn’t notice this because I was ill with Covid. They sent me one email (or at least, I received only one email) as a warning. I only realized the server was gone when my services stopped working. I’m not sure if such a short war…
Perhaps it's because I pay for several bare metal servers, or because I have a business account with them. Perhaps it's because I pay their invoices by bank transfer manually instead of by credit card. Who knows! You have made me wary of changing to a credit card now, because those do fail from time to time!
What worries me more is Hetzner's reputation for suddenly dropping customers with no warning and no way to retrieve data from the servers. That's always on the back of my mind.
It should not be understated how bad this is. Your #1 expectation as a cloud database provider is to keep data safe and recoverable. I hope for at least their sake they took a backup of everyone's DB that could be restored in another region, but based on the fact that they didn't do a scream test, I doubt they thought about this either. This must have been forced upon by upper management, because there is no way some…
I agree. This should be an indication to all current users that they should no longer trust InfluxData with their business. The CTO seems to have been checked out for a long time (just look at how little developer engagement there is on here) and the CEO seems to have no idea how to run a DBaaS. The fact that nobody else from the company has stepped in to try and defuse this should terrify anyone who has data on Infl…
It's the same "we 'tried'" message they have here. Even worse, this wasn't a regulatory shut-down, this was a lack of demand decision. They had 100% control over the timing and means of the shut-down. They didn't even keep backups! They just deleted everything.
Some highlights from the blog. It reads like a "cover my ass" to the board, rather than fixing problems for customers.
* > Over the years, two of the regions did not get enough demand to justify the continuation of those regional services.
* In other words, they had no external pressure. They just shut this down entirely on their own accord.
* > via our community Slack channel, Support, and forums, we soon realized that our communication did not register with everyone
* In other words, "we didn't look at any metrics or usage data. How could we have possibly known people were still relying on this?"
* That's literally unbelievable. They didn't even keep backups! They deleted those too! Even it the region is going down, I'd expect backups to be maintained for their SLA.
Earlier quoted context omitted.
Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications. I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.
We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.
Please make sure you can implement a TCP-type communication with your customers for this kind of critical movement.
Earlier quoted context omitted.
Just shut it down for real (after proper early warnings), so you save on compute and no one is confused about the state, and offer data retrieval for the grace period. Brownouts are great for API changes, but not very useful before a full shutdown.
You assume warnings reach users. Some people miss emails. Fewer miss a service going offline. Keeping data after shutdown is a good backup.
Earlier quoted context omitted.
This is pretty much corporate suicide. I really don't understand what they are trying to achieve with this and their attitude in this thread is baffling.
I completely agree with you regarding corporate suicide. The rest of my post is complete speculation. The least nonsense thing I can think of is that they weren't paying their bills. They weren't paying rent, the landlord locked them out and repo'd their servers, or something similar. (perhaps they were inspired by Elon Musk's recent antics?) If that were the case, they would not disclose that that's what happened. I…