21–30 of 65 posts
Aside from mitigating local dev accidentally pointing to the prod db, if you have the db accessible externally means it’s susceptible to network attacks and password attacks
From the company's homepage: "deliver marketing emails at scale" Maybe this company doesn't need to exist, and shouldn't.
I remember as a child thinking adults had everything under control. That they know what they're doing. I guess I assumed a day would come when I too would know. That day never came. It's easy to think when you look at shiny websites and the first paragraph of this comment that other adults do know. But I'm often reminded of the truth: nobody knows. Everyone is always operating at least slightly outside their comfort zone or, in the case of the article, wildly.
From the company's homepage: "deliver marketing emails at scale" Maybe this company doesn't need to exist, and shouldn't.
Many companies are already solving the same problem, and then resend comes into the picture with the support of YC. They are doing everything except sending the mails.
Earlier quoted context omitted.
That single sentence contains multitudes: * Production should be immutable * No one doing dev in a dev environment should have such trivial access to prod * Are there still good reasons for a migration to drop all tables? I guess it's for the dev environment to etch-a-sketch to a known state? Yikes.
> No one doing dev in a dev environment should have such trivial access to prod It’s the new and hip ‘cloud’! Probably using planetscale or something like that, which (last I checked, maybe it changed but wasn’t on), doesn’t even have ip protections outside the mysql user settings (while bad, would’ve protected them). > Are there still good reasons for a migration to drop all tables? We haven’t found any.
Earlier quoted context omitted.
Many companies are already solving the same problem, and then resend comes into the picture with the support of YC. They are doing everything except sending the mails.
They do, or do not operate MX and deliver SMTP traffic to other mail servers?
But yeah, both the incident and the report are really tough to read. It would be great if they can do a follow-up with further actions they're taking.
There's a neo-bank called Revolut that allegedly at one point had just two teams: "go fast" and "don't screw it up". I feel like an infrastructure play needs some dedicated hires in camp 2.
Fortunately we had backups available.
Often this isn't a problem with the individual developer itself, but points to a problem with the organization. Frankly most developers shouldn't have access to a production database, let alone mutable access.
One major concern is loss of data, but another is privacy.
It's so frustrating to see this happening when there are tools that solve this like Snaplet (I'm a founder), and Replibyte that allow you to generate or obfuscate data for usage in dev-environments, and Neon that allows you to branch your database.
If they’re small then you can see it happening where someone was logged into prod using some environment variables to sort out some issue - probably didn’t even update production, just a few queries - and then went back to work. Hours later they run some script that does DROP DATABASE from that same shell they used to troubleshoot, which takes a little longer than usual… Anyway I can totally see it happening to me in…