`uv run marker_single “path/to/file.pdf”` auto dumps markdown, images, and a json map of the PDF to a folder in the venv and gives you the path.
It’s like magic.
PDF to Text, a challenging problem
191–200 of 206 posts
Re: PDF to Text, a challenging problem
#192Earlier quoted context omitted.
I appreciate the clarification. Should have been more precise with my terminology. That being said, I think I'm talking about the forest of PDFs. When I said PDFs have a "markup-like structure," I was talking from my experience manually writing PDFs from scratch using Adobe's spec. PDFs definitely have a structured, hierarchical format with nested elements that looks a lot like markup languages conceptually. The obje…
Markup is only indirectly related to hierarchical structure. “Markup” means that there is text that is being “marked up” with additional attributes (styling, structure information, metadata, …). This is how HTML and XML work, and also languages like TeX, Troff, and Markdown. For example, in the text “this is some text”, you can mark up the word “some” as being emphasized, as in “this is some text”. The general princi…
I was quoting the article there about JSON/YAML, not making that claim myself.
Did you take a look at the article I linked? It shows visual examples of hand-coded PDFs that demonstrate the structural similarities I am talking about.
Thanks for the clarification on terminology. I could have been clearer and more precise. I referred to "DOM-like structures" as an analogy for the hierarchical nature of PDF objects, not to claim HTML is DOM.
My core point wasn't about the technical definition of markup languages, but about the structural similarity between PDF's object model and hierarchical formats.
When coding a PDF document by hand, you work with nested structures using delimiters like ">" that create hierarchical relationships between objects - which has practical parallels to working with nested elements in other formats.
The forest vs. trees metaphor was to acknowledge that while PDFs aren't primarily markup formats (the trees), they do share structural characteristics with hierarchical formats (the forest) based on my hands-on experience with manual PDF creation.
Hope that helps clarify things a bit.
Re: PDF to Text, a challenging problem
#193Earlier quoted context omitted.
Should have added "looks good on screen and on paper", "stores text compactly" and "multiple pages supported" :) And yes, that's a pretty easy set of hurdles. I wish we'd standardized on DjVu instead. Re "PDF's are immutable." - that's not a psychological fallacy, that's a primary advantage of PDFs. If I wanted mutable format, I'd take an odt (or rtf or a doc). "Output only" format allows one to use the very latest v…
PDFs are not really immutable. I use Okular all the time to write my "notes" (it's just text that you can place anywhere) on top of a PDF form and then print out a new completely filled out PDF. The only thing I do by hand is sign the physical paper.
Your understanding of immutability feels skewed here. Every time you annotate the PDF, it creates a new version. Even when you overwrite the same file, the structure of the original document changes, therefore creating a new document, ultimately making it "the ship of Theseus.pdf"
Sure, someone may try using the same argument, applying it to .doc and .txt documents, yet there is a general consensus saying that pdfs were designed to "resist the change". You can probably self-illustrate the point by making changes to a .txt document and then removing your changes - the md5 of the file would remain the same.
Re: PDF to Text, a challenging problem
#194Earlier quoted context omitted.
PDFs inherently are a markup / xml format, the standard is available to learn from. It's possible to create the same PDF in many, many, many ways. Some might lean towards exporting a layout containing text and graphics from a graphics suite. Others might lean towards exporting text and graphics from a word processor, which is words first. The lens of how the creating app deals with information is often something that…
uh. There is very little XML and the spec is a thousand pages long.
Clarified above - referring to the visual side of coding PDFs by hand.
https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...
Re: PDF to Text, a challenging problem
#195Earlier quoted context omitted.
Microsoft is one of the bigger contributors to this. Like -- why does excel have a feature to export to PDF, but not a feature to do the opposite? That export functionality really feels like it was given to a summer intern who finished it in two weeks and never had to deal with it ever again.
It does have a feature to do the opposite. You can, in theory, extract tabular data from PDFs with Excel (note: only on the Windows version; this function isn’t available in macOS Excel). In practice I’ve found it to be extremely unreliable, and I suspect this may be because the optional metadata that semantically defines a table as a table is missing from the errant PDF. It’ll still look like a table when rendered,…
Yeah. The "extremely unreliable" part of that is the stinker. Some of the exports I get through FOIA are thousands and thousands of pages, so the unreliability really compounds really quickly. It's frustrating, because there are many things Microsoft could do with PDFs to make that a non-problem. But it's consistently been a naive implementation that doesn't consider newlines.
Re: PDF to Text, a challenging problem
#196Earlier quoted context omitted.
When we get an alternative that can: (1) be stored in a single file (2) Allow tables, images and anything else that can be shown on a piece paper (3) Won't have animation, fold-out text, or anything that cannot be be shown on a piece of paper (4) won't require Javascript or access to external sites that means never.. We've got lucky we at least got PDF before "web designers" made (3) impossible, and marketers made (4…
> (3) Won't have animation, fold-out text, or anything that cannot be be shown on a piece of paper > (4) won't require Javascript or access to external sites So about that... https://opensource.adobe.com/dc-acrobat-sdk-docs/library/jsa...
that's the power of legacy. Adobe may think they can add junk to PDF like Javascript support, or lmz's "3D PDF" link below, but since PDFs viewers have a diverse ecosystem, those features won't have a great adoption.
And this is actually pretty great, maybe even the best part of PDFs! Companies _know_ that publishing PDF that require 3d-graphics or Javascript means many people won't be able to see them, so they publish good, static PDFs, maintaining virtuous cycle.
Re: PDF to Text, a challenging problem
#197Earlier quoted context omitted.
> The solution is to make it illegal to do this: wiretap data, for example, should be provided in a standardized machine-readable format. There's no ethical reason for simple technical friction to be affecting the outcomes of criminal proceedings. I can’t speak to wiretaps specifically, but when it comes to the legal field, this is usually already how it operates. GDPR, for example, makes specific provisions that use…
I'm not sure I understand what you're saying? PDF isn't a machine-readable format for most kinds of data and keeping inherent court costs down is always a concern because it keeps the courts fair to the poor.
I’m saying that most jurisdictions likely already do require data to be machine-readable, but when you run into PDFs, it isn’t a document dump (which courts don’t look kindly upon), but is instead a product of mixed parts convention and motivated laziness.
Re: PDF to Text, a challenging problem
#198Earlier quoted context omitted.
PDFs are not really immutable. I use Okular all the time to write my "notes" (it's just text that you can place anywhere) on top of a PDF form and then print out a new completely filled out PDF. The only thing I do by hand is sign the physical paper.
Your understanding of immutability feels skewed here. Every time you annotate the PDF, it creates a new version. Even when you overwrite the same file, the structure of the original document changes, therefore creating a new document, ultimately making it "the ship of Theseus.pdf" Sure, someone may try using the same argument, applying it to .doc and .txt documents, yet there is a general consensus saying that pdfs w…
Have you ever used Acrobat? Not "Acrobat Reader", but regular Acrobat, the most popular PDF editor. It's from Adobe, and it definitely does not "resist" edits.
Re: PDF to Text, a challenging problem
#199Earlier quoted context omitted.
Your understanding of immutability feels skewed here. Every time you annotate the PDF, it creates a new version. Even when you overwrite the same file, the structure of the original document changes, therefore creating a new document, ultimately making it "the ship of Theseus.pdf" Sure, someone may try using the same argument, applying it to .doc and .txt documents, yet there is a general consensus saying that pdfs w…
Have you ever used Acrobat? Not "Acrobat Reader", but regular Acrobat, the most popular PDF editor. It's from Adobe, and it definitely does not "resist" edits.
I got what you're saying the first time, and you still seem to be entirely missing the point. Immutability means that an object cannot be modified after it's created, and any changes result in a new object rather than altering the original.
You're saying "well, look, I can modify this pdf and I can even undo my changes...", what I'm saying is that whenever you modify a PDF, you're essentially creating a new file rather than truly "undoing" changes in the original. PDFs have complex internal structures with metadata, object references, and possibly compression that make bit-perfect restoration challenging.
Unlike plain text files where changes can be precisely tracked and reversed at the character level, PDFs don't easily support this kind of granular reversibility. Even "undoing" in PDF editors often means generating yet another variant rather than returning to the exact binary state of the original.
Take a look at how Git stores PDFs - when the delta approach doesn't work efficiently since even small logical changes can result in significantly different binary files with completely different checksums, it stores EVERY version of the same document in a separate blob object.
When you annotate a pdf and then later change your mind, undo all the annotations and save it — only to your eyes it may look the same as the original — in digital reality, it will be a different file.
Re: PDF to Text, a challenging problem
#200Earlier quoted context omitted.
Have you ever used Acrobat? Not "Acrobat Reader", but regular Acrobat, the most popular PDF editor. It's from Adobe, and it definitely does not "resist" edits.
I got what you're saying the first time, and you still seem to be entirely missing the point. Immutability means that an object cannot be modified after it's created, and any changes result in a new object rather than altering the original. You're saying "well, look, I can modify this pdf and I can even undo my changes...", what I'm saying is that whenever you modify a PDF, you're essentially creating a new file rath…
The people who act as if PDFs are legally immutable are not performing MD5 comparisons.
Also, that isnt even an intention of the file format as far as I can see, its mostly a byproduct of cruft and backwards compatibility.
No one would call .doc immutable because its very difficult to move an image and then restore that image to the original location.
In this context, people will save something out as pdf and store it because they dont think it cannot be modified.
But as has been rightly pointed out, thats not the case.