>He created a 100.000 line ETL tool that I replaced with 1000 lines one, that also run 10-100 times faster. I did it while explained everything to all of the team members so everybody was onboard.
Data import processes often seem to get shuffled to people usually least prepared to understand how to handle them.
2 anecdotes:
1. worked someplace years ago who would take in customer data, transform it in to our structure, then put in to mysql (2004 mysql). larger projects... this ended up taking 24 hours in some cases. I spent some time (a few hours), and had the whole thing working in under an hour. Working with a couple other folks there they got it down to around 20-25 minutes. The big secret was .... dropping indexes, importing chunks in to memory (vs disk), then copying the memory table to disk after a chunk. This was first met with "that can't work... we've already optimized everything... etc". Until it was demonstrated.
2. Colleague worked at a company which was primarily email marketing. They'd take in email lists, "import" them (which created some related metadata records, etc). The guy who wrote this importer had 'rewritten it to be faster' a year earlier, and larger imports (we're talking like... 10k emails) were taking days. Onboarding a client of 30k emails would take 2 weeks. It was literally insane code. My colleague rewrote the whole thing to work the 30k emails in under an hour, documented the code, had tests demonstrating weird edge cases... "Can't trust it - original dev needs to review it first". He (my colleague) was part of a group of people who was let go a couple months later (and was also approaching 40).
The phrase "big data" was also taking hold as a buzzword around the time of incident 2 (2 years ago maybe?). People think anything that can't fit in to excel is somehow 'big data'.