Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exam Credit for Knowing What You Do Not Know

daviddfriedman.blogspot.com

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Exam Credit for Knowing What You Do Not Know

#31

Post by datburg »

mikecsh wrote: post #15

As an electronic engineering student, our exams were negatively marked. If you got an answer wrong you incurred (-0.25 * $availablePoints). If you didn't answer the question you scored 0. We despised this at the time.. Now a medical student, we don't have negative marking. I feel it's actually very important in this discipline to know your limitations and not pass-off as though you know more than you do. I have howev…

Do not worry. We know the ocean of knowledge they throw/threw us in with full expectation of us drinking in all that new material while being surrounded with sharks and there is a heavy storm that keeps a person from taking a breath.

Still Medical school kicks ass and life after it is colorless. Also, why are you complaining? With 4 Subjects tested every week year 2, 40 genes implicated in ovarian cancer and their frequencies are over the top. The only time I was disappointed was when a concept slipped because of the lack of waling minutes to master. Concepts are more important now as I see it in an age when any symptom or fact that slips can be double checked. Best years of my life!

Re: Exam Credit for Knowing What You Do Not Know

#32

Post by hexane360 »

Cyph0n wrote: post #27
whack wrote: post #18

There was a Decision Analysis class at Stanford that took this idea to an extreme. For each multiple choice option, you had to indicate the percentage confidence of that option being correct. The percentages for all 4 options had to add up to 100%. The score you got for each question = 1 + ln(confidence-assigned-for-correct-answer). So if you were completely sure and gave the correct answer 100%, you would get a scor…

Why would anyone ever put 0% then? I would just redefine "impossible" to be an arbitrary percentage that guarantees I don't nuke the question. Quick question: is confidence taken in percent form (ln(90)) or decimal form (ln(0.9))? Edit: I am aware of the implication of having percent values, guys/gals. But if it's in decimal form, a 5% confidence level will end up losing you the current question AND three others.

If it was in percent form, a correct answer with 100% confidence would be 1 + ln(100) = 5.605 pts. Only ln(1) = 0.

Re: Exam Credit for Knowing What You Do Not Know

#33

Post by derefr »

Cyph0n wrote: post #27
whack wrote: post #18

There was a Decision Analysis class at Stanford that took this idea to an extreme. For each multiple choice option, you had to indicate the percentage confidence of that option being correct. The percentages for all 4 options had to add up to 100%. The score you got for each question = 1 + ln(confidence-assigned-for-correct-answer). So if you were completely sure and gave the correct answer 100%, you would get a scor…

Why would anyone ever put 0% then? I would just redefine "impossible" to be an arbitrary percentage that guarantees I don't nuke the question. Quick question: is confidence taken in percent form (ln(90)) or decimal form (ln(0.9))? Edit: I am aware of the implication of having percent values, guys/gals. But if it's in decimal form, a 5% confidence level will end up losing you the current question AND three others.

I assume that was part of the lesson: in Bayesian decision theory, an agent that assigns 0.0 or 1.0 confidence to something can no longer update on new evidence about that thing. Saying something is 0% likely means being so sure about that thing, that nothing—no matter how amazing—could or would ever change your mind.

Basically, no human is actually ever 0.0 or 1.0 sure of anything. Assign five or ten nines of confidence, but not 0.0 or 1.0. (It's a lot clearer when you remove the log scale and notice that a 1.0 confidence-value translates to you saying you're "infinitely confident", which seems ridiculous.)

Re: Exam Credit for Knowing What You Do Not Know

#34

Post by cornstalks »

Cyph0n wrote: post #27
whack wrote: post #18

There was a Decision Analysis class at Stanford that took this idea to an extreme. For each multiple choice option, you had to indicate the percentage confidence of that option being correct. The percentages for all 4 options had to add up to 100%. The score you got for each question = 1 + ln(confidence-assigned-for-correct-answer). So if you were completely sure and gave the correct answer 100%, you would get a scor…

Why would anyone ever put 0% then? I would just redefine "impossible" to be an arbitrary percentage that guarantees I don't nuke the question. Quick question: is confidence taken in percent form (ln(90)) or decimal form (ln(0.9))? Edit: I am aware of the implication of having percent values, guys/gals. But if it's in decimal form, a 5% confidence level will end up losing you the current question AND three others.

I'm not whack, but it must be in decimal form, or else putting 100% on a correct answer would yield ~5.6 instead of 1 (which whack says would be the score).

Re: Exam Credit for Knowing What You Do Not Know

#35

Post by derefr »

silverphoenix wrote: post #7
epalm wrote: post #2

I was a Computer Science student and teaching assistant at the University of Toronto for a number of years (about a decade ago), and the 20% rule was in full effect: "On a test, if you write 'I don't know', you get 20% of the marks for that question" As a student, this challenged you to know and understand what you know, and what you don't, and promoted the idea that knowing what you don't know is, in itself, valuabl…

I recall there was another professor at UofT during my time there that used a scheme of "Everyone starts with 30% on each question, every relevant and correct statement you write earns you marks, every irrelevant or incorrect statement loses you marks."

It'd be interesting to use that approach without having each question need to be limited by asymptotically approaching a 100% per-question score. You could do so well on a question (proving things the teacher didn't expect the students to have learned at that level) that it would help you out with your scores on other questions.

Probably I'd just let the questions be scored with integers, start everyone off at 3, +1 for good points, -1 for bad points, and then take the Euclidean distance of all their question-scores to find out the total "length" of the reach the student has into good-answer space.

Re: Exam Credit for Knowing What You Do Not Know

#36

Post by _yosefk »

mathattack wrote: post #30

Side note - if the article sounds uber-rational econ, it may be in part because the author is the son of Milton Friedman. Here is his first post on the topic. http://daviddfriedman.blogspot.com/2012/11/the-use-of-old-ex... There is a great quote at the end. And, for a last comment ... . I like to say that being a professor is better than working for a living, except when grading exams. One reason is that grading exam…

While it's hard to argue that it runs in the family to some extent in this case, David Friedman is an econ thinker in his own right and certainly goes way further into what most would call extreme libertarian/"uber-rational econ" territory than Milton Friedman ever did. In fact, I'd say that "The Machinery of Freedom" ought to entitle David Friedman to the simpler introduction "...because the author is David Friedman."

Re: Exam Credit for Knowing What You Do Not Know

#37

Post by mikecsh »

datburg wrote: post #31
mikecsh wrote: post #15

As an electronic engineering student, our exams were negatively marked. If you got an answer wrong you incurred (-0.25 * $availablePoints). If you didn't answer the question you scored 0. We despised this at the time.. Now a medical student, we don't have negative marking. I feel it's actually very important in this discipline to know your limitations and not pass-off as though you know more than you do. I have howev…

Do not worry. We know the ocean of knowledge they throw/threw us in with full expectation of us drinking in all that new material while being surrounded with sharks and there is a heavy storm that keeps a person from taking a breath. Still Medical school kicks ass and life after it is colorless. Also, why are you complaining? With 4 Subjects tested every week year 2, 40 genes implicated in ovarian cancer and their fr…

I wasn't complaining, but thanks for this!

Re: Exam Credit for Knowing What You Do Not Know

#38

Post by hectormalot »

whack wrote: post #18

There was a Decision Analysis class at Stanford that took this idea to an extreme. For each multiple choice option, you had to indicate the percentage confidence of that option being correct. The percentages for all 4 options had to add up to 100%. The score you got for each question = 1 + ln(confidence-assigned-for-correct-answer). So if you were completely sure and gave the correct answer 100%, you would get a scor…

I had a professor that used a slightly different approach. A correct answer would yield you 1pt, but any of the 3 incorrect answers would subtract 0.5pts. Expected average result for guessing is negative, so he incentivized people to leave questions open if they were unsure.

Re: Exam Credit for Knowing What You Do Not Know

#39

Post by CaseyJParker »

hectormalot wrote: post #38
whack wrote: post #18

There was a Decision Analysis class at Stanford that took this idea to an extreme. For each multiple choice option, you had to indicate the percentage confidence of that option being correct. The percentages for all 4 options had to add up to 100%. The score you got for each question = 1 + ln(confidence-assigned-for-correct-answer). So if you were completely sure and gave the correct answer 100%, you would get a scor…

I had a professor that used a slightly different approach. A correct answer would yield you 1pt, but any of the 3 incorrect answers would subtract 0.5pts. Expected average result for guessing is negative, so he incentivized people to leave questions open if they were unsure.

The expected average for a straight guess with absolutely zero exposure to the material would still be slightly positive. If nothing else, there's intuition and ingenuity, which place a person just a touch above a truly random guess-generating-machine.

Re: Exam Credit for Knowing What You Do Not Know

#40

Post by catern »

whack wrote: post #18

There was a Decision Analysis class at Stanford that took this idea to an extreme. For each multiple choice option, you had to indicate the percentage confidence of that option being correct. The percentages for all 4 options had to add up to 100%. The score you got for each question = 1 + ln(confidence-assigned-for-correct-answer). So if you were completely sure and gave the correct answer 100%, you would get a scor…

Yes, there's a class on the same subject at CMU with the same policy. Here's the document they handed out to explain the policy: http://cmu.io/~sbaugh/midterm_grading_function.pdf
Post reply on HN