Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Review of "Statistics" by Freedman, Pisani, and Purves (2017)

cadlag.org

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Review of "Statistics" by Freedman, Pisani, and Purves (2017)

#21

Post by kjellsbells »

Awesome as FP&P is, I still think there is a gap in the market for a stats book for non technical students.

I have a mental image of a Tufte-like book that aims to profoundly sharpen the students' BS-detector. That is, teach the student by deliberately showing broken things, and then guide the reader: can they spot how things are broken? What might they try to fix first? How might these fixes themselves have flaws? How might people try to hide issues? And so on.

Its my assertion that non technical people have, or can be trained to have, excellent BS detection skills even if they dont speak the mathematical languages.

The worst outcome, one we have today, is that those students are dazzled and confused by the mathematical discourse, but believe they have to obey, so they end up believing in a formulaic Statistics God that is fed p values and other detritus and spits out Insight in return: when in fact, it does nothing of the sort.

Re: Review of "Statistics" by Freedman, Pisani, and Purves (2017)

#22

Post by mnky9800n »

ivansavz wrote: post #13

I've been working on a introductory STATS book for the past couple of years and I totally understand where the OP is coming from. There are so many books out there that focus on technique (the HOW), but don't explain the reasoning (the WHY). I guess it wouldn't be a problem if the techniques being taught in STATS101 were actually usable in the real world. A bit like driving a car: you don't need to know how internal…

I wrote this paper a few years ago about this for education researchers: https://journals.aps.org/prper/abstract/10.1103/PhysRevPhysE...

This paper probably seems obvious to a lot of people but i found when i gave talks about things and read and reviewed papers people typically didn't know basic things like why you might leave some data out as a test set, why some models work better than others, when you use logistic regression versus linear regression, etc.

Re: Review of "Statistics" by Freedman, Pisani, and Purves (2017)

#23

Post by mnky9800n »

joshdavham wrote: post #18

> Much of the power of statistics is in common sense, amplified by appropriate mathematical tools, and refined through careful analysis. I hate to be contrarian, but even though I have a degree in statistics, I feel like much of statistics/probability actually violates common sense. In fact, it's probably the most unintuitive field that I'm familiar with. Many of the readers will probably be familiar with the Monty H…

can you give examples of how statistics violates common sense?

Re: Review of "Statistics" by Freedman, Pisani, and Purves (2017)

#24

Post by Jorge1o1 »

mnky9800n wrote: post #23
joshdavham wrote: post #18

> Much of the power of statistics is in common sense, amplified by appropriate mathematical tools, and refined through careful analysis. I hate to be contrarian, but even though I have a degree in statistics, I feel like much of statistics/probability actually violates common sense. In fact, it's probably the most unintuitive field that I'm familiar with. Many of the readers will probably be familiar with the Monty H…

can you give examples of how statistics violates common sense?

A classic example would be that if you have a test (say for cancer) with a false positive rate of “””only””” 5%, and your disease has an incidence of say 1 in 1000.

Let’s say that you get a positive diagnosis for the disease, and you ask someone the question:

What is the probability you actually have the disease?

Most people will say 95% or 99%, but your actual probability of having the disease in this example is <2%

Re: Review of "Statistics" by Freedman, Pisani, and Purves (2017)

#25

Post by mdanger007 »

joshdavham wrote: post #18

> Much of the power of statistics is in common sense, amplified by appropriate mathematical tools, and refined through careful analysis. I hate to be contrarian, but even though I have a degree in statistics, I feel like much of statistics/probability actually violates common sense. In fact, it's probably the most unintuitive field that I'm familiar with. Many of the readers will probably be familiar with the Monty H…

Common sense is overrated. Shakespeare, Neumann, Da Vinci… great thinkers didn’t think common common sense caught up with them

Re: Review of "Statistics" by Freedman, Pisani, and Purves (2017)

#26

Post by wiktor-k »

by256 wrote: post #4

I read/worked through Freedman's Statistics a couple of years ago and I walked away from it a different person. I always recommend it when someone asks for a good book to learn statistics from. However, it did leave me craving some of the maths that the authors intentionally left out to make the material more accessible. Freedman's more advanced book, Statistical Models, has you derive many of the results from the fi…

> I read/worked through Freedman's Statistics a couple of years ago and I walked away from it a different person.

Wholeheartedly agree. It took me a year to slowly go through all chapters and this experience really influenced my way of thinking.

I wish other topics had books of that level of quality.

Re: Review of "Statistics" by Freedman, Pisani, and Purves (2017)

#27

Post by solveit »

lupire wrote: post #15
djoldman wrote: post #6

Most statistics classes are not taught to people who will be professional statisticians. I agree whole-heartedly with this: > The book by Freedman, Pisani, and Purves is the one I would have liked to teach from, and it was the book I drew upon the most in prepping my own lectures, as an antidote to the overwrought and confused style of my assigned text. The authors maintain the underlying attitude that statistics is…

Like calculus for the engineer, statistics is primarily for the social scientist. It is an applied mathenatics, or a form of physics (math in the realm world). Math fans tend to discount and dismiss applied statistics as being not math, in a way that they don't do for physics, for some reason I don't fully grasp. I think it's because statistics gets a bad reputation from the legions of terrible social scientists in t…

> Math fans tend to discount and dismiss applied statistics as being not math, in a way that they don't do for physics, for some reason I don't fully grasp.

It's because we're secretly afraid that the physicists are smarter than us.

Less facetiously, physicists keep discovering things that lead to new mathematics we would never have dreamed of ourselves, so we have a healthy respect for how insightful they can be.

Re: Review of "Statistics" by Freedman, Pisani, and Purves (2017)

#28

Post by sedev »

Jorge1o1 wrote: post #24
mnky9800n wrote: post #23

Earlier quoted context omitted.

can you give examples of how statistics violates common sense?

A classic example would be that if you have a test (say for cancer) with a false positive rate of “””only””” 5%, and your disease has an incidence of say 1 in 1000. Let’s say that you get a positive diagnosis for the disease, and you ask someone the question: What is the probability you actually have the disease? Most people will say 95% or 99%, but your actual probability of having the disease in this example is <2%

Unfortunately that well-worn example usually only proves that "false positive" as a technical term fails to match people's intuitions. The underlying problem about the base rate is important to teach, but it's easy for well-meaning people to try and teach the base rate lesson but fail by instead teaching a bullshit gotcha about the definition of "false positive."

Re: Review of "Statistics" by Freedman, Pisani, and Purves (2017)

#29

Post by ivansavz »

mnky9800n wrote: post #22
ivansavz wrote: post #13

I've been working on a introductory STATS book for the past couple of years and I totally understand where the OP is coming from. There are so many books out there that focus on technique (the HOW), but don't explain the reasoning (the WHY). I guess it wouldn't be a problem if the techniques being taught in STATS101 were actually usable in the real world. A bit like driving a car: you don't need to know how internal…

I wrote this paper a few years ago about this for education researchers: https://journals.aps.org/prper/abstract/10.1103/PhysRevPhysE... This paper probably seems obvious to a lot of people but i found when i gave talks about things and read and reviewed papers people typically didn't know basic things like why you might leave some data out as a test set, why some models work better than others, when you use logistic…

Nice. I see you cover hierarchical (multilevel) linear models, which is already a big step up from the techniques normally covered in STATS101.

The general advice about measuring/comparing models also seems useful.

Re: Review of "Statistics" by Freedman, Pisani, and Purves (2017)

#30

Post by mnky9800n »

ivansavz wrote: post #29
mnky9800n wrote: post #22

Earlier quoted context omitted.

I wrote this paper a few years ago about this for education researchers: https://journals.aps.org/prper/abstract/10.1103/PhysRevPhysE... This paper probably seems obvious to a lot of people but i found when i gave talks about things and read and reviewed papers people typically didn't know basic things like why you might leave some data out as a test set, why some models work better than others, when you use logistic…

Nice. I see you cover hierarchical (multilevel) linear models, which is already a big step up from the techniques normally covered in STATS101. The general advice about measuring/comparing models also seems useful.

Yeah I tried to make the point that even in the case of multi level models you still should consider their ability to predict because otherwise how can you trust the model understands the underlying correlation structure of your data? That’s because many people had been advocating for these models dogmatically while presenting very poor fit statistics (r^2<0.2) while making big claims. Since I finished my phd I calmed down a bit haha. Now I just run workshops and conferences instead. And I try to present statistics and machine learning as building a lab apparatus. Once the model is built then you can ask it research questions. But simply building the model is not research.
Post reply on HN