Live data from Hacker News

From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

fertrevino.com

1–10 of 102 posts

From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#1
I recently worked on running a thorough healthcare eval on GPT-5. The results show a (slight) regression in GPT-5 performance compared to GPT-4 era models.

I found this to be an interesting finding. Here are the detailed results: https://www.fertrevino.com/docs/gpt5_medhelm.pdf

From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
fertrevino.com

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#5
post #3

Interesting topic, but I'm not opening a PDF from some random website. Post a summary of the paper or the key findings here first.

It's hacker news. You can handle a PDF.

I approve of this level of paranoia, but I would just like to know why PDFs are dangerous (reasonable) but HTML is not (inconsistent).

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#8
post #5

Earlier quoted context omitted.

It's hacker news. You can handle a PDF.

I approve of this level of paranoia, but I would just like to know why PDFs are dangerous (reasonable) but HTML is not (inconsistent).

PDFs can run almost anything and have an attack surface the size of Greece's coast.

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#9
post #5

Earlier quoted context omitted.

I approve of this level of paranoia, but I would just like to know why PDFs are dangerous (reasonable) but HTML is not (inconsistent).

PDFs can run almost anything and have an attack surface the size of Greece's coast.

That's not very different than web browsers, but usually security concerned people just disable scripting functionality and such in their viewer (browser, pdf reader, rtf viewer, etc) instead of focusing on the file extension it comes in.

I think pdf.js even defaults to not running scripts in PDFs by default (would need to double check), if you want to view it in the browser's sandbox. Of course there's still always text rendering based security attacks and such but, again, there's nothing unique to that vs a webpage in a browser.

Post reply on HN