Earlier quoted context omitted.
I see this same attitude about TDD adoption - teams in my company say things like “testing is for lackeys / that work is beneath us”, I.e. they see that as the responsibility of QA testers who are less important in their view. This is short sighted, arrogant and encourages similar problems with superiority complexes. TDD is still controversial in some circles, but engineers who have a deep understanding of both tests…
TDD is fine when you have a specification to work to. A lot of software development in the real world is quite "experimental". Requirements are poor so devs need to provide what is essentially a prototype and receive feedback until it is good enough.
We don't need data scientists, we need data engineers
361–367 of 367 posts
Re: We don't need data scientists, we need data engineers
#362Re: We don't need data scientists, we need data engineers
#363Earlier quoted context omitted.
Ideas are so cheap and easy. Implementation is a long hard road. And where you learn your idea was vague enough that it had almost no value. And only through painstaking iteration can you turn it into something with value.
> Ideas are so cheap and easy. I doubt this.
Re: We don't need data scientists, we need data engineers
#364Earlier quoted context omitted.
What's the market price for an idea? If it's greater than zero, let me know. I have notebooks full of them. Business ideas. Project ideas. Political ideas. Social ideas. I generally can't give 'em away, much less sell them. Why? Everybody has their own ideas, and they like 'em better. And the ones in my notebooks don't have what really matters: validation.
I’m interested, do you have a blog?
Re: We don't need data scientists, we need data engineers
#365Earlier quoted context omitted.
Nothing precludes you from doing integer arithmetic in a dynamic language. I’m not a quant and this isn’t my area of expertise, but, for example, I’m pretty sure various differential equation solving methods depend on variables taking on continuous values, so floating point basically must be used. Understanding the impact of that is definitely very important. Analogously, I frequently run into numerical precision iss…
>Nothing precludes you from doing integer arithmetic in a dynamic language. You would be surprised. The second you use pandas with a custom data type (let alone any other library you'd want to use) it can randomly auto convert it to a float. Furthermore identifying when it randomly converts the type on you is a pain. >so floating point basically must be used. Quants tend to use fixed precision types. It is like a flo…
That's a pandas (and maybe numpy) issue, not a dynamic language issue. (If you want to generalize from the specific libraries more accurately than “dynamic language”, it's “using a low-level library whose type system doesn't match the host language type system” issue.
> Quants tend to use fixed precision types. It is like a float in every way, except base 10 instead of base 2 so there is no floating point error.
No, a type that is like binary floating point in every way except base 10 instead of base 2 would be decimal floating point, not fixed point. Decimal fixed point is different from binary floating point in more ways than base.
Re: We don't need data scientists, we need data engineers
#366Earlier quoted context omitted.
I used to work at a legacy automaker and you’d be shocked at how much ML has changed certain areas of the business. It used to take an entire department to sort warranty claims and it’s now mostly automated. Aluminum part defects are now spotted automatically on the plant floor. Don’t even get me started with telematics data. Most software isn’t consumer facing but just because you don’t see it doesn’t mean it’s not…
I wouldn't think of a system that would automate the processing of warranty claims as ML. That's mostly applying the policy/rules to each claim. However, finding defects in aluminum parts that involves using computer vision, would absolutely be a ML solution.
Re: We don't need data scientists, we need data engineers
#367I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…