Lessons Learned from Two Years as a Data Scientist

I felt like this article was a bit light on data scientist specific advice, and while I am not one, I do herd them for a living, so thought I'd put some random thoughts together:

1) Quite often you are not training a machine to be the best at something. You're training a machine to help a human to be the best at something. Be sure to optimise for this when necessary.

2) Push predictions, don't ask others to pull them. Focus on decoupling your data science team and their customers early on. The worst thing that can happen is duplicating logic, first to craft features during training, and later to submit those features to an API from clients. Even if you hide the feature engineering behind the API, this can either slow down predictions, or still require bulky requests from the client in the case of sequence data. Instead, stream data into your feature store, and stream predictions out onto your event bus. Then your data science team can truly be a black box.

3) Unit test invariants in your model's outputs. While you can't write tests for exact outputs, you can say "such and such a case should output a higher value than some other case, all things being equal". When your model disagrees, do at least consider that the model may be correct though.

4) Do ablation tests in reverse, and unit test each addition to your model's architecture to prove it helps.

5) Often you will train a model on historical data, and content yourself that all future predictions will be outside this training set. However, don't forget that sometimes updates to historical data will trigger a prediction to be recalculated, and this might be overfit. Sometimes you can serve cached results, but small feature changes make this harder.

6) Your data scientists are probably the people who are most intimate with your data. They will be the first to stumble on bugs and biases, so give them very good channels to report QA issues. If you are a lone data scientist in a larger organisation, seek out and forge these channels early.

7) Don't treat labelling tools as grubby little hacked together apps. Resource them properly, make sure you watch and listen to the humans building and using them.

8) Have objective ways of comparing models that are thematically similar but may differ in their exact goal variables. If you can't directly compare log loss or whatever like-for-like, find some more external criteria.

9) Much of your job is building trust in your models with stakeholders. Don't be afraid to build simple stuff that captures established intuitions before going deep - show people the machine gets the basics first.

10) If you're struggling to replicate a result from a paper, either with or without the original code, welcome to academia.

Probably not earth shattering stuff, I grant you.

That modern Python example scream regression to me. Why not simply cars = []?

This obsession with killing perfectly good languages with strongly typed hints is completely undermining the point.

> in fact Google doesn't allow any production-level projects to be written in python due to safety concerns.

Then why did you write it as a fact?

Check out ipdb instead of pdb. It’s pdb but with the ipython repl instead of python’s

I think you're in the minority thinking that Python is "killed" by adding static (not strong, Python is already strongly typed) type hints. They are also optional, so you're free to not use them.

To go back to the code example, if you want to express the same thing in Python as in Java, you have to add a type hint to be able to statically check the code. A good thing about Python is that you can choose when and if you want to use a static type hint, while in Java you're forced to use them.