Agree 99% except this statement:
> Embrace duplication - don't unnecessary add loops or abstractions
I’ll usually make a function or perhaps tiny class as soon as I start reusing bits of code.
Apart from that, agree as stated. At my previous job (Python shop), a lot of the data engineers came from a Java background, and had a tendency to think top-down. Many things were over engineered ‘just because we might need it’:
- Factory classes used only once or twice in entire code base
- Lets make an AbstractReaderInterface because we might want to abstract the file type or location later (while 100% of files are Parquet on S3)
I’ve really enjoyed using dataclasses and Pydantics BaseModels prolifically, and adding type hints (coupled with type checks in CI).
Model the data, write a well structured imperative workflow, set up CI, write unit tests, enforce typing. Add OOP if needed, then close ticket.