Live data from Hacker News

PDF to Text, a challenging problem

marginalia.nu

201–206 of 206 posts

Re: PDF to Text, a challenging problem

#201

Earlier quoted context omitted.

I got what you're saying the first time, and you still seem to be entirely missing the point. Immutability means that an object cannot be modified after it's created, and any changes result in a new object rather than altering the original. You're saying "well, look, I can modify this pdf and I can even undo my changes...", what I'm saying is that whenever you modify a PDF, you're essentially creating a new file rath…

The people who act as if PDFs are legally immutable are not performing MD5 comparisons. Also, that isnt even an intention of the file format as far as I can see, its mostly a byproduct of cruft and backwards compatibility. No one would call .doc immutable because its very difficult to move an image and then restore that image to the original location. In this context, people will save something out as pdf and store i…

I feel like I'm talking to a toddler, sigh. Let me try again.

Immutability doesn't mean that an "object cannot be modified", it means that in order to modify an object, you must create a new (clone) object. That's all what I meant to say. Sure, you can get pedantic or otherwise and say "yes, pdfs are immutable; or no, pdfs aren't immutable in some contexts", etc., and depending on the point of view, both of these claims could be correct — I'm not arguing about the specifics.

I'm just saying that your explanation of why you think pdfs are not immutable hinges on an incorrect idea of what immutability actually is.

There's a rigorous definition for "immutability" in computer science, e.g., strings in many programming languages are immutable, but that doesn't mean you can't manipulate them, it just means that operations that appear to modify strings actually create new string objects.

The greatest illustration for immutability is imbued in programming languages with immutability-by-default, e.g., Clojure. Once someone groks the basics, it becomes really clear what that thing is about.

Re: PDF to Text, a challenging problem

#202
post #197

Earlier quoted context omitted.

I'm not sure I understand what you're saying? PDF isn't a machine-readable format for most kinds of data and keeping inherent court costs down is always a concern because it keeps the courts fair to the poor.

I’m saying that most jurisdictions likely already do require data to be machine-readable, but when you run into PDFs, it isn’t a document dump (which courts don’t look kindly upon), but is instead a product of mixed parts convention and motivated laziness.

You're saying two mutually exclusive things. Either it's required to be machine readable or it's PDF: it can't be both.

Re: PDF to Text, a challenging problem

#203

Earlier quoted context omitted.

The people who act as if PDFs are legally immutable are not performing MD5 comparisons. Also, that isnt even an intention of the file format as far as I can see, its mostly a byproduct of cruft and backwards compatibility. No one would call .doc immutable because its very difficult to move an image and then restore that image to the original location. In this context, people will save something out as pdf and store i…

I feel like I'm talking to a toddler, sigh. Let me try again. Immutability doesn't mean that an "object cannot be modified", it means that in order to modify an object, you must create a new (clone) object. That's all what I meant to say. Sure, you can get pedantic or otherwise and say "yes, pdfs are immutable; or no, pdfs aren't immutable in some contexts", etc., and depending on the point of view, both of these cla…

> I feel like I'm talking to a toddler, sigh.

Me too, but I'm done. Have fun!

Re: PDF to Text, a challenging problem

#204

Earlier quoted context omitted.

Have you ever used Acrobat? Not "Acrobat Reader", but regular Acrobat, the most popular PDF editor. It's from Adobe, and it definitely does not "resist" edits.

I got what you're saying the first time, and you still seem to be entirely missing the point. Immutability means that an object cannot be modified after it's created, and any changes result in a new object rather than altering the original. You're saying "well, look, I can modify this pdf and I can even undo my changes...", what I'm saying is that whenever you modify a PDF, you're essentially creating a new file rath…

> I got what you're saying the first time,

That wasn't me. Multiple people were taking the time to help you understand.

Re: PDF to Text, a challenging problem

#205
post #163

Earlier quoted context omitted.

That looks like a pretty good starting point, thanks. I've been dabbling in vision models but need a much higher degree of accuracy than they seem able to provide, opting instead for more traditional techniques and handling errors manually.

For non-table documents a fine tuned yolov8 + tesseract with _good_ image pre-processing has basically a zero percent error rate on monolingual texts. I say basically because the training data has worse labels than what the multi-model system gives out in the cases that I double checked manually. But no one reads the manual on tesseract and everyone ends up feeding it garbage, with predictable results. Tables are an…

The first thing I did when I saw this thread was ctrl-f for doclaynet :)

I've been at this problem since 2013, and a few years ago turned my findings into more of a consultancy than a product. See HTTPS://pdfcrun.ch

However, due to various events, I burned out recently and took a permie job, so would love to stick my head in the sand and play video games in my spare time, but secretly hoping you'd see this and to hear about your work.

Re: PDF to Text, a challenging problem

#206
post #205

Earlier quoted context omitted.

For non-table documents a fine tuned yolov8 + tesseract with _good_ image pre-processing has basically a zero percent error rate on monolingual texts. I say basically because the training data has worse labels than what the multi-model system gives out in the cases that I double checked manually. But no one reads the manual on tesseract and everyone ends up feeding it garbage, with predictable results. Tables are an…

The first thing I did when I saw this thread was ctrl-f for doclaynet :) I've been at this problem since 2013, and a few years ago turned my findings into more of a consultancy than a product. See HTTPS://pdfcrun.ch However, due to various events, I burned out recently and took a permie job, so would love to stick my head in the sand and play video games in my spare time, but secretly hoping you'd see this and to hea…

There's not much to say.

Doclaynet is the easy part and with triple the usual resolution the previous gen of yolo models have solved document segmentation for every document I've looked at.

The hard part is the table segmentation. I don't have the budget to do a proper exploration of hyper parameters for the gridformer models before starting a $50,000 training run.

This is a back burner project along with speaker diarization. I have no idea why those haven't been solved since they are very low hanging fruit that would release tens of millions in productivity when deployed at scale, but regardless I can't justify buying a Nvidia DGX H200 and spending two months exploring architectures for each.

Post reply on HN