Live data from Hacker News

Twenty Years of Pandoc

pandoc.org

41–50 of 55 posts

Re: Twenty Years of Pandoc

#41

> by writing N parsers (“readers”) and M renderers (“writers”), one could support N × M conversions. Beautiful writeup for a wonderful project. In an age of vibe-coding hype it's also so nice to see how things can be extended and snowball in usefulness when things are built correctly, by hand, from basic principles. > Perhaps, then, in the future, people will no longer have a need for tools like pandoc. I think we wi…

I think the art is about figuring out and defining what the underlying technical principles you want to leverage in order to provide a relevant software solution are. LLMs used naively might do that, but they probably won't. The work is still in figuring out what a useful domain representation of the problem is - and then you can unleash the llms for the boilerplate.

Re: Twenty Years of Pandoc

#43

Love pandoc, I use it for all sorts of stuff…here is my minimal static site generator: find . -name '*.md' -type f -exec sh -c ' for file do out="docs/${file#./}" out="${out%.md}.html" mkdir -p "$(dirname "$out")" pandoc --quiet --template template.html "$file" -o "$out" done ' sh {} +

I do this too, but I use cmake to switch the backend between pandoc and lowdown [0]. Lowdown is not a drop-in replacement for pandoc flavored markdown, but it's between 10-100X faster.

It's very handy to switch when tinkering with template changes because I can convert ~ 1000 markdown files instantaneously on my mid 2010s desktop.

[0]: https://kristaps.bsd.lv/lowdown/

Re: Twenty Years of Pandoc

#45
I've always taken particular note of how wisely scope-limited Pandoc is. Markup that aligns with natural language convention[1], is tightly converted, but the further from natural language, the less fidelity Pandoc can promise. Until, at the DITA or S1000D stage of "this ain't natlang, brah", Pandoc says "forget it" and just won't even pretend that such markup is even convertible.

Because, spoiler, it's not.

Constructs like tables and bibliographies challenge natural language markup - resulting in an explosion of different formalisms - but component content system artifacts for transclusion and conditionals shatter any pretense that these file types are "documents" at all. Both of those artifacts must draw formal structure from outside of language, i.e., from their own product / domain. They're parts of a system that make documents, but are not themselves documents or natural language. They are meaningless - or, worse, full of wrong meaning - outside of their runtime environment inside an explicit knowledge domain. Something that newer component content formats like Typst recognize explicitly.

The proof of all this is, as they say, in the pudding. What do people write documents in today? Well, they stick to natural language formats, sometimes they let the document system handle tables in some bespoke way, but conditionals are viewed with justified suspicion. DITA and S1000D projects, and the cursed migrations that lead to them, are sparse and driven almost exclusively by regulatory requirements, or, more often, program offices misreading regulatory requirements[0].

And here we all are in the LLM age, where natural language is being vindicated in ways both awe-inspiring and devastating. While component content systems force an LLM to expand its context window to the entire repository to make sense of any single sentence.

The crap of all this is, this is stuff that computer / information science has known since at least the 1980s. There are papers written about it. But high-complexity component content systems are sellable to non-technical writer groups because they don't see the tripwires in the fundamentals, or they think[2] that their product domain is so structured that the tripwires can be rigged as structure.

[0] No, converting to a pile of S1000D 040As doesn't magically fix your MTAs or your ILS or anything else.

[1] I do realize that proximity to natural language is correlate, not cause. Markdown converts well as a low-power notation whose instances denote values; it reads like natural language because that's what low-power does. The operative variable is whether the artifact denotes a document or a function from configuration to documents. AsciiDoc with `ifdef::[]` and `include::[]` converts every bit as badly as DITA, although without the fundamental nonsense of XSD and Horn's Information Mapping.

[2] Almost always wrongly

Re: Twenty Years of Pandoc

#46

"The choice of Haskell has also led to a high quality and low volume of contributors" I feel this influence of choosing a tech stack and its impact on self selected and auto-reenforced culture is most often underestimated. From my own experience, at a time I was (involuntarily) working in Java, and when .Net was released, from a pure technical point of view it was like a breath of fresh air. Java was suffering from o…

"few applicants, most pretty good" was true for us as well when hiring Haskell developers at Freckle (https://tech.freckle.com/) - in some ways it was scary: we're away from the herd, the numbers aren't on our side, what if we ever need to hire large volumes of people? Instead of getting 200 JavaScript devs in the pipeline, you would get maybe 10-20 Haskellers. What if the person you're looking for is not in that pool? Intuitively, you assume that more is always better.

In other ways it made filtering through candidates quite simple: half of the people who applied were really good, the other either language astronauts or folks fascinated with the tooling who didn't genuinely want to move fast to produce commercially-viable software, they wanted to tinker. You just needed to figure out which bucket the person was in.

Re: Twenty Years of Pandoc

#48
I remember noting "djot" when John M. created it but hadn't really looked into it before now. After reading through the design goals and differences from Markdown, I can see we'd have been better off if somehow djot had been able to come first. Gruber has said he thinks Markdown being underspecified was a feature, but as a user trying to mix and match different tools it drove me nuts.

Re: Twenty Years of Pandoc

#49

"The choice of Haskell has also led to a high quality and low volume of contributors" I feel this influence of choosing a tech stack and its impact on self selected and auto-reenforced culture is most often underestimated. From my own experience, at a time I was (involuntarily) working in Java, and when .Net was released, from a pure technical point of view it was like a breath of fresh air. Java was suffering from o…

"few applicants, most pretty good" was true for us as well when hiring Haskell developers at Freckle ( https://tech.freckle.com/ ) - in some ways it was scary: we're away from the herd, the numbers aren't on our side, what if we ever need to hire large volumes of people? Instead of getting 200 JavaScript devs in the pipeline, you would get maybe 10-20 Haskellers. What if the person you're looking for is not in that p…

Most systems are not large enough to need 200 developers. Even when they are, you hire those 200 over time. If you have 200 developers (or 200 anything) you need to have a good training and mentoring program.

Language is not the problem. If someone is a senior developer (not just has the title because of years of service), you can teach them Haskell on the job for little cost. Sure it will take them a few years to be an expert in the language, but most problems they need to solve don't need language experts, just someone good enough. And Haskell is a different language, most often you are hiring for a language that is only slightly different from ones they already know.

Re: Twenty Years of Pandoc

#50
post #19

Pandoc is awesome. My fav usage is configuring git to use it to normalize binary docs to markdown (like a .docx) so they can be diffed. Works amazing for redlining contacts.

I do this too. If you use e.g. GitHub you can build this into your PR checks so that any PRs containing docx file changes get a CI bot comment with the diff from the docx.
Post reply on HN