Ask HN: What's your "it's not stupid if it works" story?

I’ve worked at a company where our remote access was over a super slow modem line but the machine did have access to the internet.

At Google almost 20 years ago, a bunch of our machines (possibly with slightly bespoke CPUs?) were behaving oddly. These machines were mostly in use for serving Google's web index, so almost the entire RAM was devoted to index data; the indexserver processes were designed to be robust against hardware failure, and if they noticed any kind of corruption they'd dump and reload their data. We noticed that they were dumping/reloading massively more often than they'd expect.

Eventually the cause was narrowed down to that, randomly when the machine was stressed, the second half (actually, the final 2052 bytes) of some physical page in memory would get zeroed out. This wasn't great for the indexservers but they survived due to the defensive way that they accessed their data. But when we tried to use these new machines for Gmail, it was disastrous - random zeroing of general process code/data or even kernel data meant things were crashing hard.

We noticed from the kernel panic dumps (Google had a feature that sent kernel panics over the network to a central collector, which got a lot of use around this time) that a small number of pages were showing up in crash dump registers far more often than would statistically be expected. This suggested that the zeroing wasn't completely random. So we added a list of "bad pages" that would be forcefully removed from the kernel's allocator at boot time, so those pages would never be allocated for the kernel or any process. Any time we saw more than a few instances of some page address in a kernel panic dump, we added it to the list for the next kernel build. Like magic, this dropped the rate of crashes down into the noise level.

The root cause of the problem was never really determined (probably some kind of chipset bug) and those machines are long obsolete now. But it was somehow discovered that if you reset the machine via poking some register in the northbridge rather than via the normal reset mechanism, the problem went away entirely. So for years the Google bootup scripts included a check for this kind of CPU/chipset, followed by a check of how the last reset had been performed (via a marker file) and if it wasn't the special hard reset, adding the marker file and poking the northbridge to reset again. These machines took far far longer than any other machines in the fleet to reboot due to these extra checks and the double reboot, but it worked.

> since you can check again for the actual matches.

Can you explain this?

When I was younger I learned that Sed was turing complete. So I did was any young woman would do. I build an entire search engine in Sed. But it wasn't like a useless little search tool that provided bad search capability for a website, no, nearly every page (minus a few like the about page) of a site was nothing more than a presentation on top of the search query results. Several thousand hardcoded "known" pages and infinite possible pages depending on user searches. Because it was the foundation of the site, search worked unlike most websites searches of the era (~2005). This site happily ran for about a decade with surges of traffic now and then before a server migration and too little time prevented its continued existence.

Adult me is both horrified and impressed at this creation.

I have a website, blog, a cdn, and a domain level email inbox running off AWS SES.

You email specific email addresses which get processed as web pages, blog posts, or attachments extracted to the cdn. All other emails sent to your domain sit inside the inbox. You can also send out emails from any address.

The blog and webpages are all SEO optimised so you can share the link on say Twitter and it will unfurl the link and read the meta tags.

You can also forward specific emails to a special address to be shared or bookmarked in your browser.

The entire thing runs off Lambdas, S3, Cognito, and AWS SES, nary a database. I use pug template files to format content extracted from emails.

To make this work, I had to do a deep dive into how Gmail’s email composer translates actions into HTML tags, then align the templates to these behaviours.

For a while, I had a handful of paying customers which paid my AWS bills. Right now, I’m down to one customer and the rest of my uses are for personal projects.

I learnt a lot in the process - from templating to SEO to S3 usage to Lambdas and got a very usable domain level email inbox and blog out of it. The CDN and static pages are a little less useful but building them too was quite fun.

Btw, highly recommend nodemailer as a module for email parsing.