way back in the late 90s stone age of interactive ad agencies, we were doing our first really big gig for hp. it was a demo shipping out to retail stores showcasing one of their products -- a run of 30,000 stamped cd roms.
i was the one developing the macromedia director app running on the cd.
we were on-time.
we were ready to send them out the door.
it was awesome.
and then we tested the rom outside of our network...
in some far-off corner of code, i had baked in a hard reference to one of our file servers on our network for some streaming assets. the cd failed as soon as you put it in the drive due to that reference to the missing file.
by the time we discovered this, we'd already glass-mastered and stamped 30,000 discs to the tune of $40k or so. or, about $6k per employee. in a company that booked about $50k the previous year. where i worked for free for 9 months.
so, my line of code cost our little company the equivalent of almost all of our previous year's revenue -- not profit, but revenue.
we, of course, had to make the run again -- only this time at the emergency rush prices. and this time, we were running late.
we managed to book some time in the middle of the night at the stamping plant. it was 4am. i had a courier standing over my shoulder watching me run the final build again, this time without the dreaded line of code -- which broke other things i had to fix when i removed it -- before he could take it.
i finished testing. ejected the disc. handed it to the courier, who started running as he was placing it into its case. he drove like hell to make it to the airport where we counter-to-countered it on a 2-hour, 6am flight to vegas for stamping.
oh, and it almost got even worse from there. almost.
we didn't know if they would be able to stuff the cds into the packaging because this was an emergency run and they didn't have the people available.
so...
we were actually on our way to rent a uhaul which we calculated we could drive to vegas just in time for the stamping run to finish. from there, we would load the discs on their spindles, and 4 of us were going to sit in the back of the van, stuffing 30,000 discs while we drove the uhaul to palo alto. from vegas. yes, stuffing discs in the back of a traveling uhaul.
we even had the patio furniture from one of the employees yards already picked out to sit in while we were in the back of the truck.
luckily, the plant managed to squeeze in our packaging (at rush pricing, of course) and all we needed to do was have one of our guys take them as luggage on a later flight that day to the bay area instead.
as to a couple, big lessons learned?
1) i can honestly tell you, i've never, ever had a hard-coded, local network link in anything i've shipped since and never will again. always test off-network. especially these days with mobile apps and their on-off-network states.
2) a strong, non-finger-pointing team is where you need to be. i felt appropriately awful, but we handled it as a team and proceeded to grow that little company to about $40 million a year before a merger.
p.s. oh, and next time, remind me to tell you about the time i ran a database query on production that nuked the entire website for the publicly-traded software company which relied on -- wait for it -- the website to do all its commerce.