Goodbye, data science • Hacker News Board

Re: Goodbye, data science

#361

Post by angstrom » Wed, Nov 30, 2022, 2:18 AM UTC

EdwardDiego wrote: post #346

JoeAltmaier wrote: post #305

Earlier quoted context omitted.

Well, not so easily. The chances per-capita are in the sub-percent range at 30.

Unless you're exposed to a carcinogenic chemical...

Nah, not even. I have a mutation called CDH1 that happens to be pathogenic and predisposes me to a greater than 40% chance of stomach cancer. It's a dominant gene which means it has a 50% chance I've passed it onto my daughter as well.

That cancer is what's known as a Hereditary Diffuse Gastric Cancer gene (HDGC). It just so happens that the E-cadherin control that suppresses those cancer cells is not processed properly. The diffuse part is what makes it particularly tricky. It's on the surface of the stomach epithelial cells and progresses from there. The only solution is a total gastrectomy (prophylactic if you do it early). No carcinogen necessary. It's found in populations all over the world and pathogenic lines don't even have to be related. The mutation can occur independently in the germline and is passed on. As long as you reproduce before it kills you nature really doesn't care.

Fun side fact. It also predisposes carriers to 70% chance of breast cancer. As a result many of those diagnosed are women who then find out they need to also have their stomachs removed.

Re: Goodbye, data science

#362

Post by quickthrower2 » Wed, Nov 30, 2022, 2:55 AM UTC

icedchai wrote: post #333

quickthrower2 wrote: post #203

Earlier quoted context omitted.

Software Engineering is one of the few knowledge working areas where you can actually test the result in various ways as a layman. You can flush the toilet before paying the plumber to a large extent and hire another counter-team called QA. QA themselves are tested by future production bugs. In other disciplines it is way more fuzzy. If you are in the conclusion business and there isn’t a clear path to test your conc…

Unfortunately, I haven't worked at a company with dedicated QA in the past 5+ years, maybe longer. QA is often seen as a side job for engineers and product teams.

Oh! Dedicated QA makes a big difference, especially when they take leadership and are willing to get involved in lets call it qa-ops: improving automated testing and such like.

Re: Goodbye, data science

#363

Post by esparrohack » Wed, Nov 30, 2022, 3:00 AM UTC

adamsmith143 wrote: post #281

esparrohack wrote: post #77

Earlier quoted context omitted.

Yeah but in the end it’s just code. And even better, just R. The business value comes from the stats guy.

When the R/stats guy quits and you have to figure out which of his 7 notebooks to run in which order and which local files need to be in which local directories to run correctly and which versions of each package are now broken and which code you need to rewrite to fix it you start to realize the value he produced was clicking a lot of buttons in the right order and that overall this doesn't scale at all.

Yeah, but I meant that because the business value is in the stats, and there is such low quality of stats in the field to begin with, it’s borked no matter what.

There’s no point in fixing it. You can just pretend like you did. But if the stat work is quality, then it’s worth the effort to optimize.

Re: Goodbye, data science

#364

Post by layman51 » Wed, Nov 30, 2022, 3:01 AM UTC

drgiggles wrote: post #58

Unfortunately it seemed pretty clear from the start that this is what data science would turn into. Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result. For research and analysis data scientists must have a strong understanding of underlying statistical theo…

> Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result.

That's a good way of putting it. I remember in my first calculus-based probability+statistics class in college, I felt incredibly challenged by the theory. I wondered why there are so many probability distributions out there, why the standard stats formulas look like they do, what "kernel density estimation" even is, etc.

On the other hand, my data science course did include some theory, but a big part of it was also learning how to type the right commands in R to perform the "featured analysis of the week" on a sample data set. Something about these lab exercises felt off because it felt more like training rather than education. The professor expressed something along the lines that if we wanted to go far with this in the future, he would expect us to design the algorithms behind the function calls. I think the analogy he used was "baking a cake from scratch rather than buying a ready made one at the store."

Re: Goodbye, data science

#365

Post by eigen » Wed, Nov 30, 2022, 3:58 AM UTC

lrc wrote: post #216

scottLobster wrote: post #156

Earlier quoted context omitted.

Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response…

My own thought (I know there is a great deal of room for disagreement) is that the J6 crowd saw no consequences for the attempts to obstruct the Brett Kavanaugh confirmation and thought the rules had changed. One of them was shot in the neck, many others are still incarcerated two years later despite a clear constitutional right to a speedy trial. I'd rather the Capitol Police had just cracked heads at this point. Yo…

> You might think one protest was more justified than another, but the differential in response works to dissolve confidence in the fair application of the law.

in 2018, the capitol was open to the public, no one broke in. 78 were arrested in the Capitol on Oct 5, 2018 and charged with Crowding, Obstructing, or Incommoding [1].

in 2022, the Capitol was closed to the public and people broke in. 12 people were arrested on Jan 6, 2021 and charged with Unlawful Entry or Assaulting a Police Officer [2].

Assaulting a Police Office is a felony; Crowding, Obstructing, or Incommoding is a misdemeanor. Seems there was a differential in severity of breaking the law as well.

[1] https://www.uscp.gov/media-center/press-releases/us-capitol-...

[2] https://www.uscp.gov/media-center/press-releases/us-capitol-...

Re: Goodbye, data science

#366

Post by SkyPuncher » Wed, Nov 30, 2022, 4:16 AM UTC

spaniard89277 wrote: post #22

Correct me if I'm wrong because I'm on the receiving end of such models, but I feel that many times a couple of linear regressions, surveys and qualitative work with customers could land much better results. I say so because I've had time to read some of the reports that DS teams produce to drive decisions in my BIGCORP and it makes very little sense most of the times. And we suffer from it because we have direct con…

Actual, human based decisions will almost always win out.

Data is only helpful when it is directly and clearly tied to the problem.

* Good: "Our customers are complaining of random drop-outs. We've noticed X% of requests to Y service take longer than Z time. We believe that's the problem".

* Bad: "Companies who are most successful on our platform upload X things in their first Z days. We must find a way for everyone to upload X things in Z days".

Re: Goodbye, data science

#367

Post by 29athrowaway » Wed, Nov 30, 2022, 4:16 AM UTC

The field of software engineering learned to punish cowboy coding and prima donnas. Now it's the turn of data science.

Re: Goodbye, data science

#368

Post by insightcheck » Wed, Nov 30, 2022, 4:19 AM UTC

PragmaticPulp wrote: post #196

scottLobster wrote: post #156

Earlier quoted context omitted.

Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response…

Consequences often catch up slowly. It took years for Elizabeth Holmes to be sentenced because it takes time to collect evidence, build an airtight case, and give people their due process. As I get older, I'm actually noticing more and more consequences catching up with people, albeit slowly. The people I knew who drank heavily through their 20s and 30s are in much worse shape than basically anyone who made an effort…

To add another piece of evidence, while previous poster noted that the initial response by police officers on January 6 seemed less violent than they could have been (though even then, one person was shot and killed), the US Department of Justice is continuing to publish press releases about charges of people involved in the January 6 Capitol attack (at https://www.justice.gov/news , with full records with dates at https://www.justice.gov/usao-dc/capitol-breach-cases). The charges for many of the people involved caught up eventually, though it took time.

Separately, to put a positive spin on this, it often takes time for positive habits to pay off. When picking up a positive habit (e.g. exercise and especially learning a new technical skill such as a language), oftentimes much of the reward doesn't come until far later. This is important to keep in mind, especially if one has self-doubts or even a lack of encouragement for trying to adopt a new positive habit in one's life.

Re: Goodbye, data science

#369

Post by ims » Wed, Nov 30, 2022, 4:32 AM UTC

adamsmith143 wrote: post #73

xyzelement wrote: post #11

// it was often personally unfulfilling (e.g. tuning a parameter to make the business extra money). He lost me here. Something I've always loved about being an engineer (and now in product) is that something small we do/tweak can have big impact. If you tuned a parameter and that actually had tangible impact on the business, that's like the best case scenario and should be celebrated (vs doing some cool rocket scienc…

If I work my ass off and my model recommends a few extra units, I don't see another dime, so where's the motivation?

This guy is a straight-shooter with upper-management written all over him.

Re: Goodbye, data science

#370

Post by insightcheck » Wed, Nov 30, 2022, 4:32 AM UTC

ZephyrBlu wrote: post #352

PragmaticPulp wrote: post #196

Earlier quoted context omitted.

Consequences often catch up slowly. It took years for Elizabeth Holmes to be sentenced because it takes time to collect evidence, build an airtight case, and give people their due process. As I get older, I'm actually noticing more and more consequences catching up with people, albeit slowly. The people I knew who drank heavily through their 20s and 30s are in much worse shape than basically anyone who made an effort…

The problem is that time value is extremely relevant. If it takes 10-20 years for consequences to catch up, the person is likely to have already built up an unassailable lead that the consequence barely dents. > He literally had to leave the state and go somewhere new to escape his past network and get new jobs after 10 years of this. That's not even that bad of a consequence. It sounds like his strategy was worth it…

> The problem is that time value is extremely relevant. If it takes 10-20 years for consequences to catch up, the person is likely to have already built up an unassailable lead that the consequence barely dents.

I largely agree with you: the big names attached to the resume, the pay, and the effort spent on interviewing skills likely offset the negatives of the reputation (though I also intuitively don't like it because the strategy is rather self-centred).

However, the consequence is rather significant if he has roots. It's harder to pack up and move if one has a romantic partner who is settled into a job at a particular place, and you could also possibly be leaving family and friends. Sometimes one has to move, but typically one has the option to come back, which wouldn't be practical for the person in question. It's still plausibly worth it for the person if he didn't have roots and collected a lot of compensation, but especially when one is older (the commenter mentioned 10 years of workin experience), moves can be tougher.