71–80 of 222 posts
The most important lesson was about reducing the alerts and sleeping better. It's important to be able to leave your laptop at the office and not worry about things going wrong. This post gave me flashbacks of when I made websites for others and hosted them myself. Never again!
Alex! Great to see you on HN.
Earlier quoted context omitted.
I read a Substack article almost every day and still consistently get a little jolt of rage when the overlay fades in. I guess they do it because it works, and if the average person felt the same way, they wouldn't be doing it?
> if the average person felt the same way, they wouldn't be doing it? You might think this, but I tend to doubt it's true... people making these decisions end up being in their own little bubble, and not really have a good idea what people's actual response is. That said, I find Substack's pop-up tolerable: I am after all getting someone's work for free, and I'd rather have a simple pop-up that I know I can get rid o…
If I could have my time again I'd focus on enablement of the broader engineering group through an incident management process, readiness exercises, some aggregate incident analyses for the org to learn from and leaning into observability. Infrastructure is kind of a solved problem for common use cases today, just requires the expertise. With our ducks in a row I'd next have look to a GRC function for the compliance b…
This is the problem however for many (older) companies. They either don't care, or quite literally don't know the infrastructure solutions out there which can save literally thousands of hours per year of headache. Sure, for many companies with legacy systems they have a "dont fix what isn't broken" mindset, but from what I've seen, I always ask, if shipping and modifying new versions of a system takes hours or even days to complete, is the system really not 'broken'? I guess I never realized it, but having automated and clean infrastructure with tests and uptime metrics is a must-have for me on anything I build going forward. Take 2-3 weeks to save months of headache.
> We were doubling the size of our customer base every 9 months, and pretty soon this meant we needed more servers.
I've found that a moderately sized 4 core 'droplet' from Digital Ocean handles nearly anything I can throw at it (at least in terms of the web application world) - I see you product is doing employee management / scheduling - is there heavy compute / resources associated? Or is it more due to privacy requirements (i.e. you need a siloed instance / server per client?). Not trying to sound smug at all, just genuinely curious where the load comes from.
Get a 403 Forbidden when I visit https://www.tanda.co/
> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…
I don’t think it’s unreasonable to expect that multiple channels of communication will be tried before cutting off service, especially not for services provided to a business.
For larger businesses with thousands of dollars of revenue per month? Sure. For small $50 charges? No way.
Earlier quoted context omitted.
Not exactly. Some countries have laws in place that prevent a provider from removing service due to non-payment to a business. Instead they must follow a process, including physical paper notices, before removing service. DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.
DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty. DO should abide by the law of the country they operate in. If that means they can shut off services for non-payment then that's what they're entitled to do. By the sounds of it they gave plenty of warning and 30 days service they weren't getting paid for. Email is the expected form of business commu…
I would not expect Digital Ocean to alter the MOTD configuration of my Linux box because I forget to check my emails. That would be an invasion of privacy in my opinion.
Would be interested to know what your hosting bill was like? We recently moved everything in-house and started hosting our own servers. Having physical access to the machine (and the network HW) pretty much eliminated all of our dev ops (1). We expected the HW to fail often and such, but in 2 years the only maintenance we had to do was add more disks to the RAID array and resize the partition a few times. Even though…
It's around 150k USD/month at the moment. Would love to learn more about your setup. Email me if you're interested in chatting. Email in my profile.
> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…