Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influx…
InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
221–230 of 278 posts
Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
#222Earlier quoted context omitted.
Something similar happened to me when I lived at a condo in Boston and the management company changed and I somehow missed the memo. I kept sending checks to the old management company for a few months before the problem was discovered. Unfortunately, someone at the old management company apparently had a bit of a gambling habit and petty-thefted that money away, and I never saw it again even though she lost in a law…
Was the employee the defendant, not the company?
Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
#223Earlier quoted context omitted.
Of course not, but financial products are priced and offered with a financial outcome (usually a margin) in mind. If you make mortgage processing more expensive, you'll find the offers for origination are worse than if mortgage processing were less expensive.
Sorry, it's a ridiculous argument. Mortgages get bought and sold all the time and clearly the buyers are on the hook for the communications costs and these do not pre-emptively get priced into future products by the sellers. I'm sure that there are situations where your argument has merit but this isn't one of those.
> these [communications costs] do not pre-emptively get priced into future products by the sellers
They 100% do get priced in. Whenever you buy a product, you're paying all the costs of that product. When someone originates a mortgage, they're aware of the secondary market for mortgages. If that secondary market is eroded by a significant increase in communications costs, that reduces the willingness of a secondary buyer to bid for your book of mortgages. That erosion reduces your projected profit on originating, so you take a little longer to lower your offered rate to 5.250%, or you charge a bit higher origination fee, or whatever to ensure you maintain a viable business. So long as these fees hit the entire market, the other originators are all making the same calculations.
It seems odd that you [seem to] think that money for these costs would just result in reduced profits for the financial services companies rather than in increased borrowing costs.
Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
#224Earlier quoted context omitted.
Wow, that's pretty pathetic and your attitude "we can't help our customers" is even more damning. Email is not reliable enough to simply rely on a few email blasts for this. I would expect: * Those 3 "email blast" notifications. I'm guessing one of two things happened here: * You sent them as an "email blast" from a marketing-type email service. These hit email filters because they came from a known spam IP. * You se…
> Ideally, an automated transfer to another region with automated forwarding. It's okay to have poor performance, but it's not okay to go "poof" entirely. If the data is moving between countries then this is not an option. Your clients may have legal or contractual obligations with respect to data location.
Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
#225Earlier quoted context omitted.
What I don't understand is why they have not auto migrated all customer data to the closest region. Why ask the customers to do this migration instead of transparently handling it for them? yes the cost may be different but this was their decision, they should absorb these costs until the customers decide.
Auto migrating data to the closest region cannot be done without legal implications if that region is another country.
Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
#226Deprecation of services is a common occurrence at AWS and many other tech companies. But it's never taken lightly. A mandatory step in this process is analyzing usage logs. We need to ensure customers have transitioned to the alternative. If they haven't, we reach out. We understand why. The idea of simply "nuking" customer data without a viable alternative is unthinkable.
The InfluxDB incident brings to light the ongoing debate around soft vs. hard deletion. It's unacceptable for a hard delete to be the first step in any deprecation process. A clear escalation process is necessary: notify the customer, wait for explicit acknowledgement, disable their APIs for a short period, extend this period if necessary, soft delete for a certain period, notify again, and only then consider a hard delete.
The so-called ["scream test"](https://www.v-wiki.net/scream-test-meaning/) is not a viable strategy for a cloud service provider. Proactive communication and customer engagement are key.
This incident is a wake-up call. It underscores the importance of data durability and effective, respectful customer communication in cloud services and platform teams. Communication is more than three cover-your-ass emails; it's caring about your customers.
Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
#227Earlier quoted context omitted.
I use Timescale and I can recommend it. The reason why I'm still using influx too (1.7) is because it's unmatched in its data storage efficiency and query performance. You can get close with Timescale, but its main power is having the query power of PostgreSQL, if you have room for the extra hardware resources it requires.
(My memory is that we kept on influx including 1.7, but it was a while ago now so memory might be fuzzy) I guess influx perf and efficiency is really depending a lot on your data shape then :) Our experience was that performance dropped off a cliff if you had too much data, too much tagset cardinality, or else your query was too broad. And when it failed, it lost data. In fact, it lost data generally. When we were re…
I agree with GP about storage efficiency, which is superb. Query performance is good as long as a single query doesn’t deal with more than ~dozens of series. And $deity help you if you want to do hourly roll-ups of all series for a short time range, as RAM usage is wildly unpredictable. Storage is optimized for long reads of a single series, not for short reads of many series (but in fairness, you have to choose one or the other, that’s just the physics of the thing).
If I were starting from scratch, I’d probably pick Timescale. Or maybe DuckDB… I wonder if it would work for our use case.
Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
#228It should not be understated how bad this is. Your #1 expectation as a cloud database provider is to keep data safe and recoverable. I hope for at least their sake they took a backup of everyone's DB that could be restored in another region, but based on the fact that they didn't do a scream test, I doubt they thought about this either. This must have been forced upon by upper management, because there is no way some…
The CTO seems to have been checked out for a long time (just look at how little developer engagement there is on here) and the CEO seems to have no idea how to run a DBaaS. The fact that nobody else from the company has stepped in to try and defuse this should terrify anyone who has data on InfluxData's cloud.
This is the beginning of the end. It seems like all of the good people have left the company, and being willing to destroy credibility to cut costs is a clear sign that the company is running on fumes.
So, now is the time - find your alternative, whether it's Timescale, QuestDB, VictoriaMetrics, ClickHouse, or just self-hosting.
Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
#229Earlier quoted context omitted.
Because if a company has 20000 engineers which one is the one that gets the vendors email? Answer is usually none and the email to that address goes to /dev/null. Or suppose an employee did have the email on file and left the company. Or suppose people assume vendor email are spam because they’re almost always spam.
What? Each and everyone of our suppliers have a dedicate address their notification are sent to. Those automatically goes into the service desk as a ticket and is read by the service desk team, which can either correct billing information if required, or escalate to the correct team if action is require or there's any doubt about the content of the email. If you have 20.000 engineers (or even 200) you have a function…
Your model is a smart one. It’s smart enough it tells me you’re either a small company or a newer company, or both, or a company with a rarely together vendor management team.
Re: InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion
#230Earlier quoted context omitted.
Sorry, it's a ridiculous argument. Mortgages get bought and sold all the time and clearly the buyers are on the hook for the communications costs and these do not pre-emptively get priced into future products by the sellers. I'm sure that there are situations where your argument has merit but this isn't one of those.
No one is saying that existing mortgages would be re-priced. Those are contracts and you can't unilaterally change them. What I and others are saying is that if you changed the obligations of buyers such that mortgages became more expensive to service, that those servicing cost increases would ultimately be borne by the mortgage borrowers rather than eaten by mortgage lenders out of the goodness of their hearts. > th…
I got what you and others are saying but I've been trying - and failing, apparently - to point out that the costs for a mortgage in case of a sale are born by the buyers who are not even the same kind of institutions as the parties that sell them, and so they are in no position to charge the subjects, nor are the sellers going to price the mortgages any higher in the future because the costs aren't born by them.
There is no such thing as 'projected profits' that go into this because mortgages can be sold (and are sold) more than once, the number of times is not known when they are issued first. And it is going to be only a small fraction of the audience that is going to be hard to reach for whatever reason. The presumption that there is some kind of free market mechanism that will ultimately pass those costs back to the original mortgage underwriters is not in any way evidenced by present day mortgage prices. On the sum total of mortgages out there and the - exorbitant - profits they create for the lenders we're talking about such small amounts that it will make zero difference.
A 'significant increase in communications costs' would translate into that being something that is some noticeable percentage of the total yield over the remaining time and it just simply isn't. Typical mortgage rates and amounts utterly dwarf the costs of a one time notification, especially if you don't have to notify everybody like that, and you can try cheaper channels first until you have a hit. Besides, the original mortgage 'service charge' is already a large multiple of the various costs and tends to be mostly pure profit for the initial lender.
It seems odd that you [seem to] think that mortgages are priced such that the mere cost of communications is going to show up in the prices, they are amongst the most profitable financial products.