Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons Learned from Two Years as a Data Scientist

dawndrain.github.io

Post reply on HN
Print view View original

61–70 of 89 posts

Re: Lessons Learned from Two Years as a Data Scientist

#61

Post by jeeeb »

thom wrote: post #21

I felt like this article was a bit light on data scientist specific advice, and while I am not one, I do herd them for a living, so thought I'd put some random thoughts together: 1) Quite often you are not training a machine to be the best at something. You're training a machine to help a human to be the best at something. Be sure to optimise for this when necessary. 2) Push predictions, don't ask others to pull them…

Good list. One thing I'd add, which you kind of hint at:

Good practices from software engineering are just as applicable to Data Science. In particular:

Notebooks are great for performing an EDA, and testing out new concepts. They're not great for running production code. Put your non-once off code in regular source code files and source control it.

Break your code into separately testable and composable functions. Write unit tests to verify behavior where you can. Speaking from experience you all most certainly will find bugs.

Implement a peer review process for the methodology used and the code. Approaches should be explainable and justifiable. Bugs and poor assumptions can lead to incorrect results.

Focus on making your model training process end-to-end reproducible. Document the training data used. Document the configuration used. Link back to the commit hash of the exact code used. Make sure your environment is reproducible.

Re: Lessons Learned from Two Years as a Data Scientist

#63

Post by listenallyall »

This reads like a pretty "wet-behind-the-ears" professional who doesn't know what he doesn't know.

> There's no Java awfulness like ... instead it's just `cars = []`

I mean, there's very good reasons for static typing. And if he was using Kotlin, he could specify whether the variable `cars` was itself immutable and whether the list was immutable (`val/var cars : List/MutableList`

> notebooks

yea, Jupyter kernels exist for almost every language. This is not a Python advantage.

> debugging

Good IDEs have the ability to set breakpoints, inspect variables, test methods, etc.

> type hints

"oh, forget what I said earlier about how Java had ugly boilerplate, now I have an `import` and a type def after all" - except nothing here is actually enforced

> parallelism

parallelism is relative... a lot of compiled JVM code will run much faster than Python to start with, and even with `multiprocessing`, Python won't catch up (and JVM languages have their own concurrency solutions, of course)

> I've put a large chunk of my money in leveraged index funds and etfs.

Written by a person who's never seen the slightest hint of a bear market, or rising interest rates. That's ok, you wouldn't be the first smart person to be seduced by leverage: https://www.investopedia.com/terms/m/myron-scholes.asp

> Stimulants like caffeine, adderall, and modafinil are magic... People do stay on adderall and modafinil indefinitely

Look, I'm no doctor (and I'm aware I'm out of the loop on things like this), but mental & concentration stimulants are the kinds of things associated with old people, not recent graduates.

Re: Lessons Learned from Two Years as a Data Scientist

#64

Post by hazbot »

listenallyall wrote: post #63

This reads like a pretty "wet-behind-the-ears" professional who doesn't know what he doesn't know. > There's no Java awfulness like ... instead it's just `cars = []` I mean, there's very good reasons for static typing. And if he was using Kotlin, he could specify whether the variable `cars` was itself immutable and whether the list was immutable (`val/var cars : List /MutableList ` > notebooks yea, Jupyter kernels ex…

True, but that didn't stop me from finding parts interesting, and just generally enjoying the earnestness.

Re: Lessons Learned from Two Years as a Data Scientist

#65

Post by krrrh »

ZephyrBlu wrote: post #17
krrrh wrote: post #6

Earlier quoted context omitted.

The leverage means they go up faster, but they also go down faster. In regular investing you feel good when you get a modest return, and bad when you experience a modest loss. With leveraged funds you feel like a genius when the market goes up and like a complete moron when you get wiped out.

I understand how leverage works, just not in the context of an index fund or ETF. From your description it sounds like the leverage is baked into it, so it's kind of like "safe" leverage in that you can't lose more than you have.

This link explains it pretty well. There are other advantages from regular forms of margin too.

https://www.investopedia.com/terms/l/leveraged-etf.asp

Re: Lessons Learned from Two Years as a Data Scientist

#67

Post by dawndrain »

listenallyall wrote: post #63

This reads like a pretty "wet-behind-the-ears" professional who doesn't know what he doesn't know. > There's no Java awfulness like ... instead it's just `cars = []` I mean, there's very good reasons for static typing. And if he was using Kotlin, he could specify whether the variable `cars` was itself immutable and whether the list was immutable (`val/var cars : List /MutableList ` > notebooks yea, Jupyter kernels ex…

Certainly lots of things to learn still.

Going point by point:

It's not "he" it's "she".

I don't think it's controversial that you can be much more concise with python. My experience first learning Java was that everything was 2-3x as verbose as in python. The difference is smaller if you're using type hints in python, but it's still more concise.

I talked about repl's/notebooks for other languages. They're still an especially great tool for python/data science since they make it very easy to visualize data and share analyses.

I played around with breakpoints in pycharm and I don't think it would work for me. You need to run your code from pycharm in debug mode for the breakpoint to trigger, whereas I always run things from the command line or a notebook.

I believe you that there are times when python is slower. At least it's not noticeably slower when simply calling C behind the scenes or when you're i/o blocked anyway.

Re investing, I mean, everyone has seen phenomenal returns since they were born: this century is unprecedented. Also there was the pandemic crash very recently, so everyone has experienced an extremely harsh (albeit brief) bear market too. LTCM was like 100x-leveraged, which I would not advocate for, since you'll almost certainly get wiped out if you hold that position for more than a few hours...

Eh, lots of kids have add, and like 10% of college students used adderall in 2016 according to the first hit on google. In any case they've been magic for me the few times I've tried them e.g. working 12+ good hours in a day.

Re: Lessons Learned from Two Years as a Data Scientist

#68

Post by tfehring »

bt3 wrote: post #38
throwaway98797 wrote: post #31

Good stuff, excpet: > This is dangerous advice if you go all in on this.

SPXL also has an expense ratio over 1%, which will eat away at earnings unless in the best of bull rushes (now).

The expense ratios of leveraged ETFs are nothing compared to the volatility drag. There are far cheaper and more effective ways than leveraged ETFs for buy-and-hold investors to obtain leverage, notably LEAPs and index futures. (Disclaimer: Not investing advice, do your own research, etc.)

Re: Lessons Learned from Two Years as a Data Scientist

#69

Post by tomnipotent »

listenallyall wrote: post #63

This reads like a pretty "wet-behind-the-ears" professional who doesn't know what he doesn't know. > There's no Java awfulness like ... instead it's just `cars = []` I mean, there's very good reasons for static typing. And if he was using Kotlin, he could specify whether the variable `cars` was itself immutable and whether the list was immutable (`val/var cars : List /MutableList ` > notebooks yea, Jupyter kernels ex…

This is an oddly snarky response to someone just sharing their experience, but you seem to be reading it matter-of-factly. She's upfront about having just graduated, and having 2-years of experience with a math background and not a CS background.

She clearly covered a lot of good ground in that time, and even took the time to write a 6000+ word article.

Re: Lessons Learned from Two Years as a Data Scientist

#70

Post by dhanna »

dawndrain wrote: post #67
listenallyall wrote: post #63

This reads like a pretty "wet-behind-the-ears" professional who doesn't know what he doesn't know. > There's no Java awfulness like ... instead it's just `cars = []` I mean, there's very good reasons for static typing. And if he was using Kotlin, he could specify whether the variable `cars` was itself immutable and whether the list was immutable (`val/var cars : List /MutableList ` > notebooks yea, Jupyter kernels ex…

Certainly lots of things to learn still. Going point by point: It's not "he" it's "she". I don't think it's controversial that you can be much more concise with python. My experience first learning Java was that everything was 2-3x as verbose as in python. The difference is smaller if you're using type hints in python, but it's still more concise. I talked about repl's/notebooks for other languages. They're still an…

It’s magic because you have no tolerance. That goes away after a few weeks of continuous use.
Post reply on HN