Rewriting LaTeX in Pure Rust • Hacker News Board

Re: Rewriting LaTeX in Pure Rust

#101

Post by todd8 » Mon, Dec 07, 2020, 3:08 PM UTC

The title is a bit misleading.

LaTeX is a macro package written using the TeX macro system, so I doubt that LaTeX itself is being rewritten in Rust. TeX, created by Donald Knuth, is now over 40 years old. It has a number of remarkable characteristics:

* TeX is essentially bug free. It can be a challenging program to use, but in my experience I was always consoled by the knowledge that if something wasn't working it was my fault (in contrast to my use of other document processing systems and word processors). See [1].

* The TeX program is extraordinarily well documented. Knuth has an entire book which is the literate program comprises TeX. See [2].

* TeX was originally written in a version of Pascal, the source code was written using a literate style and processed using tools the Knuth wrote (Tangle and Weave). This was later machine translated into C.

* TeX hasn't changed. It reached version 3 in 1989 and Knuth essentially froze it at this point. One of its great values is that documents written in TeX will always produce the same output, whether written last century, this century, or next century. We will always be able to see what the original author wanted.

* TeX is unusually capable. Knuth is obviously a perfectionist. Unhappy with the state of digital typography when he wrote TeX he designed an entire toolchain for creating fonts call MetaFont. I used it to design my company's first unique logo back in the 90's. (I only needed five characters.) See [3]. Knuth did this before Postscipt was invented.

* TeX is public domain. Before open source was a thing, Knuth put TeX in the public domain.

* TeX is the Lingua Franca used by Mathematicians, Physicists, Computer Scientists and others for books, academic papers, and even presentations; nothing else comes close.

* TeX has a number of widely used forks, notably XeTeX and LuaTeX. XeTeX added Unicode support and LuaTeX is an attempt to make TeX and it's companion LaTeX more easily programmable. See [4] and [5].

* TeX has a powerful macro system. This is how packages are added to TeX. The most widely used such package is LaTeX itself. LaTeX is not written in C or Pascal, it is written as a complex set of macros.

* Tex's macro system is great for its intended purpose, but not so great for how it is currently used. Knuth originally used the macros to define layouts of books and papers. Things like line spacing font choice for page numbers and such can be easily encapsulated as TeX macros. TeX's macro system is powerful enough that even complex packages like LaTeX could be written in it. Today, there are packages like TikZ/Pgf that are very sophisticated and complex programs written entirely in TeX macros. TikZ/Pgf is a powerful graphics, drawing, and even animation system. It's manual is 1271 pages long. See [6]

Where does this leave the originally linked project called Tectonic? It's not obvious from the github repository, but the user web site is a bit more informative, see [7]. This "rewrite" seems to rely on XeTeX and provides some command line operations that streamline the invocation of XeTeX. I don't understand its value proposition.

For me, what's really needed is a better language than TeX's macro system for writing extensions. A few years ago I was writing TeX code (macros) to format spine labels for the books in my personal library. I used the number of pages in the book to adjust the size and layout of the call number and 2D barcode printed on the label. All of this was quite challenging in TeX's macro system, so I wrote a simple Python program that generated the LaTeX output that resulted in a page of labels that were easy to print. Maybe I could have done it in LuaTeX using Lua, but at the time LuaTeX was so poorly documented (how ironic) that I couldn't figure out how to do it.

The adoption of some programming language that could guarantee backward compatibility indefinitely and a software package written in it that was compatible with LaTeX2e or LaTeX3 and replaced them. This language should not be TeX's macro system. The right language, perhaps something like ML or Scheme, would make programmed typesetting so much easier.

[1] https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.43801907..., behind a paywall but copies are easily available.

[2] https://www-cs-faculty.stanford.edu/~knuth/abcde.html

[3] https://en.wikipedia.org/wiki/Metafont

[4] https://en.wikipedia.org/wiki/XeTeX

[5] https://en.wikipedia.org/wiki/LuaTeX

[6] http://mirrors.ibiblio.org/CTAN/graphics/pgf/base/doc/pgfman...

[7] https://tectonic-typesetting.github.io/en-US/

Re: Rewriting LaTeX in Pure Rust

#102

Post by colejohnson66 » Mon, Dec 07, 2020, 3:12 PM UTC

marktangotango wrote: post #83

This “x implemented in y” is a curious thing we are seeing with growing frequency nowadays (imo). On the one hand, good for these folks, to be rewriting something old in a new way (be it language, library, or framework). On the other hand, could this be an indicator that we’ve reached a certain level where it is easier to rehash the ideas of the past than create new work or invent new ideas or applications? Maybe som…

What’s next is obviously JS or even WASM to typeset on the client side! /s

Re: Rewriting LaTeX in Pure Rust

#103

Post by scns » Mon, Dec 07, 2020, 3:12 PM UTC

diarrhea wrote: post #53

SiempreViernes wrote: post #41

Earlier quoted context omitted.

You don't play very many new games do you?

Being on HN and talking about TeXLive, I was thinking in terms of Linux, for which those 90GB blockbuster games don't exist (?). So I did a mental jump there, because to me TeXLive is the "native" Linux TeX distribution, and Miktex the "native" Windows distribution.

XCom 2 runs natively on Linux (Thanks Feral!). Needs > 70GB of disk space with all extensions installed.

Re: Rewriting LaTeX in Pure Rust

#104

Post by drran » Mon, Dec 07, 2020, 3:28 PM UTC

colejohnson66 wrote: post #100

throwaway890369 wrote: post #93

Earlier quoted context omitted.

For those unfamiliar, Knuth's versioning scheme approaches pi and therefore never reaches 4.0 [0] > In his seminal text layout system, TeX, and his equally brilliant typeface design system, METAFONT, Donald uses a versioning number system that asymptotically approaches perfection. The version numbers of TeX approach π (the current version is 3.14159265) and the version numbers of METAFONT approach e. [0] http://senti…

You’re leaving out the best part! When he passes (hopefully not anytime soon[a]) the version number of TeX will become pi. Any and all bugs will become “features” at that point. Same with METAFONT; it’ll become e. [a]: I’d love to get a full and complete set of The Art of Computer Programming (all volumes) written first (on his own time, of course), but I doubt that’ll actually happen. According to Wikipedia, there’s…

Someone may change his name to D. E. Knuth and continue to improve code. License allows that.

Re: Rewriting LaTeX in Pure Rust

#105

Post by OJFord » Mon, Dec 07, 2020, 3:37 PM UTC

kgwgk wrote: post #69

scoopertrooper wrote: post #54

Earlier quoted context omitted.

O'rly? https://github.com/tectonic-typesetting/tectonic/blob/master...

It seems you're linking to wrappers to the underlying https://github.com/tectonic-typesetting/tectonic/tree/master...

As I understand/recall - the intention was always to be a rust rewrite; the starting point was 'let's take all the C and wrap it in `unsafe`'.

The issue linked up-thread introduces this fork (~ a year ago) titling it '[convert to rust] everything' (emphasis mine), and going on to talk about 'pure rust' (again).

I think the main repo's issue at least would've been a clearer submission than the fork; I initially had the same reaction as above.

Re: Rewriting LaTeX in Pure Rust

#106

Post by returningfory2 » Mon, Dec 07, 2020, 3:44 PM UTC

sega_sai wrote: post #62

I'm not sure rewriting Latex in XYZ is a good idea. The whole thing is a mess and a pain to use (and I use it multiple times a week). Sure the documents often look good, but when they don't or there is an error, fixing it just becomes a nightmare. I wish somebody would design a modern typesetting system that would accept say latex equations but without 50 years of cruft and levels of macroses upon macroses.

> I wish somebody would design a modern typesetting system...

In theory yes, but I think the problem is that it will never be adopted. Too much of academia is built on the deep assumption that LaTeX is the only possible format that people write documents in.

I think one possible way out is to start a new from-scratch TeX engine that is built in such a way that a new typesetting language (or "front end" if you like) can be added to it. This may lead to some reasonable slow migration strategies.

Re: Rewriting LaTeX in Pure Rust

#107

Post by ThinkBeat » Mon, Dec 07, 2020, 3:49 PM UTC

grey-area wrote: post #50

jfk13 wrote: post #48

Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and g…

The advantage of this approach (as opposed to a rewrite), is that you reproduce all the bugs and misfeatures that people in the wild depend on. You can then add a test suite and start refactoring and gradually move to a codebase you're happy with, while breaking few or no users on the way. Another approach is to split it into blocks and replace parts of the system, but whether that is feasible depends on the software…

Is is not clear if all the Bugs will manifest in the same manner after the translation.

Re: Rewriting LaTeX in Pure Rust

#108

Post by MaxBarraclough » Mon, Dec 07, 2020, 3:49 PM UTC

Xylakant wrote: post #81

MaxBarraclough wrote: post #67

Earlier quoted context omitted.

> littered with unsafe blocks Good points. It's unfortunate that headlines rarely distinguish safe Rust from unsafe, when so much of the advantage of Rust depends on it. You may even get a hostile response for asking about it ( https://news.ycombinator.com/item?id=24141493 )

I think you're getting your hostile answers for implying that unsafe rust is equivalent to C. It's most definitely not. Using unsafe grants you certain powers, but it does not disable all of rusts features - for example the borrow checker is not turned off by unsafe. See https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html#unsa...

Safety doesn't just come from language features, it also comes from the language disallowing dangerous actions. Rust's unsafe mode opens the door to undefined behaviour, of the sort that plagues so many C/C++ codebases (buffer overflows etc). A program written in safe Rust offers far better assurances than a program making heavy use of unsafe Rust: safe Rust is unable to result in undefined behaviour. (Bugs in the compiler and standard library may still cause mischief, but that's another matter, the intent of the safe Rust subset is to be guaranteed free from UB.)

You may be right that a program written in 100% unsafe Rust might still be less prone to undefined behaviour than a program written in C, but that's not my point. Excessive use of unsafe features undermines the considerable safety advantages that Rust offers over C, and it's regrettable when this is disregarded.

Re: Rewriting LaTeX in Pure Rust

#109

Post by returningfory2 » Mon, Dec 07, 2020, 3:54 PM UTC

jfk13 wrote: post #48

Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and g…

I absolutely agree, but would go even further.

When you say Xetex "not a solid piece of engineering on which to build the future", I think the same thing also applies to the original Tex engine written by Knuth. By modern software engineering standards, the original Tex implementation is a nightmare. It's enormously difficult to extend or add new features, and this has resulted in (1) comparatively few extensions being made and (2) when such extensions have been made (e.g. Xetex), they are very difficult technically.

Re: Rewriting LaTeX in Pure Rust

#110

Post by koenigdavidmj » Mon, Dec 07, 2020, 4:02 PM UTC

jfk13 wrote: post #48

Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and g…

If I recall correctly, they did a machine-assisted conversion of the Go compiler from C to Go using a similar technique. They wrote a compiler for C to Go, only bothering to handle the particulars of that one codebase, and changing the C if required. This emitted very un-idiomatic Go code, which they could then clean up as required.

What they got immediately was a few classes of bug gone. No more memory leaks, and NPEs and out-of-bounds errors are now defined to fail in a nicer way. Then they could spend time making their new code more idiomatic.