Rewriting LaTeX in Pure Rust • Hacker News Board

Re: Rewriting LaTeX in Pure Rust

#131

Post by svat » Tue, Dec 08, 2020, 2:07 AM UTC

returningfory2 wrote: post #109

jfk13 wrote: post #48

Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and g…

I absolutely agree, but would go even further. When you say Xetex "not a solid piece of engineering on which to build the future", I think the same thing also applies to the original Tex engine written by Knuth. By modern software engineering standards, the original Tex implementation is a nightmare . It's enormously difficult to extend or add new features, and this has resulted in (1) comparatively few extensions be…

I agree, but one also can't blame Knuth: he wrote the program the way he knew best (he's a machine-code programmer at heart), under the constraints at the time (portability at various academic sites circa 1980 practically dictated Pascal, then Pascal's limitations required a preprocessor like WEB, etc). In fact, the earlier (TeX78) implementation in the SAIL language was written less monolithically, as a bunch of separate modules.

He also did his best to make the implementation and source code understandable, publishing the program in print as an extensively documented/commented book (another reason for WEB), gave a workshop of 12 lectures about the implementation of the program, even had a semester-long course at Stanford with that book (program source code) as textbook (with exercises and exam problems). He also wrote TeX with hooks and some of its core functionality written as extensions using those hooks, hoping it would show others how to extend it. He has multiple times expressed surprise that more people didn't write their own versions of TeX. “Rewriting a typesetting system is fairly easy.” He seems to have overestimated the ability of others to read his code.

If anything, I think a lesson from the TeX situation is that one's work can be too good: if he had simply published the algorithms at a high level (only the Knuth-Plass line-breaking algorithm was published as an independent paper) then maybe others would have implemented/combined them in interesting ways, but by publishing the entire source code and offering rewards for bugs etc, TeX got a (deserved) reputation as a very high quality stable and bug-free codebase and everyone wanted to use literally TeX itself. What's worse is that for a few years after it was created, TeX was possibly more widely available and more portable (what with its TRIP test and all that) than any single programming language (one had a much higher chance of TeX macros working consistently everywhere TeX was used, than code written in say C or Pascal): so it must have seemed natural to write large things like LaTeX entirely in TeX macros. As “TeX macros” wasn't designed or intended as a full-fledged programming language, we can see the effects today.

Re: Rewriting LaTeX in Pure Rust

#132

Post by alexpovel » Tue, Dec 08, 2020, 8:18 AM UTC

2ion wrote: post #123

leephillips wrote: post #87

Earlier quoted context omitted.

I’ve been a luatex advocate in the past¹, but I use xetex instead, unless I need the Lua integration. The memory handling is the reason. I find that for documents with a lot of fonts, luatex eats all the memory available and then crashes, taking a huge amount of time to do so, whereas xetex just breezes through the same document. [1] https://lwn.net/Articles/731581/

What's troublesome for me is that I have been using * xetex when I needed a font that was not easily achievable in pdftex over the past decade * pdftex for everything else because microtype(TM) just works(TM) (even though kerning can be done using fontspec and font features in xetex). I've tried luatex multiple times over the past decade, it was mostly just too slow. Now luatex is fast. But I have no idea if I now "s…

> basically every scientific paper I read is set using pdftex

That is only because their templates are years behind the curve and they are slow to update. It is not an argument for the advantages of pdftex, aside from its stability, gained over many decades.

LuaTeX has been nothing but stable for me, so from a technical standpoint, there is no reason not to switch.

As far as scientific papers go, the publishers and editors probably value stability and backward-compatibility (I would).

Re: Rewriting LaTeX in Pure Rust

#133

Post by selfhoster11 » Tue, Dec 08, 2020, 10:12 AM UTC

steveklabnik wrote: post #95

selfhoster11 wrote: post #92

Earlier quoted context omitted.

The more I read about legacy (and actively maintained) project refactoring, the more firmly I find myself agreeing that the gradual replacement is the right way to go about things in nearly every case. Whether we like it or not, the old system is a source of truth about how things are done, so the only way to preserve this knowledge fully is to copy the whole thing as-is and then "restate" parts of that knowledge in…

If you’re interested in this topic, you should read “How to work effectively with legacy code” by Michael Feathers.

It's been on my reading list for a while, I'll take this as a reminder to read it :)

Re: Rewriting LaTeX in Pure Rust

#134

Post by kazinator » Tue, Dec 08, 2020, 3:59 PM UTC

grey-area wrote: post #50

jfk13 wrote: post #48

Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and g…

The advantage of this approach (as opposed to a rewrite), is that you reproduce all the bugs and misfeatures that people in the wild depend on. You can then add a test suite and start refactoring and gradually move to a codebase you're happy with, while breaking few or no users on the way. Another approach is to split it into blocks and replace parts of the system, but whether that is feasible depends on the software…

How do you not break people who depend on misfeatures, while eliminating misfeatures?

Hey, I think Windows drive letter names are misfeature.

Let's have a Windows-in-Rust and they will soon be a thing of the past.

Re: Rewriting LaTeX in Pure Rust

#135

Post by kazinator » Tue, Dec 08, 2020, 4:08 PM UTC

jfk13 wrote: post #48

Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and g…

> Simply wrapping the 1980s-era code

I just checked the Wikipedia page on XeTeX (not knowing what the heck that is at all). It's in fact actually something very recent in terms of the TeX timeline; it was released in 2004. It has Unicode support, which seems to be the big thing.

That is more recent than, I think, the last time I used TeX; which worked absolutely fine.

Someone rewriting TeX in Rust should work with Knuth's original Pascal sources, in my opinion, not some knock-off (and look at XeTeX behaviors and documentation in order to do the Unicode stuff in a compatible way).

Re: Rewriting LaTeX in Pure Rust

#136

Post by grey-area » Tue, Dec 08, 2020, 4:34 PM UTC

steveklabnik wrote: post #95

selfhoster11 wrote: post #92

Earlier quoted context omitted.

The more I read about legacy (and actively maintained) project refactoring, the more firmly I find myself agreeing that the gradual replacement is the right way to go about things in nearly every case. Whether we like it or not, the old system is a source of truth about how things are done, so the only way to preserve this knowledge fully is to copy the whole thing as-is and then "restate" parts of that knowledge in…

If you’re interested in this topic, you should read “How to work effectively with legacy code” by Michael Feathers.

That looks really interesting, thanks for recommending it here.

Re: Rewriting LaTeX in Pure Rust

#137

Post by deepsun » Tue, Dec 08, 2020, 5:37 PM UTC

kazinator wrote: post #135

jfk13 wrote: post #48

Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and g…

> Simply wrapping the 1980s-era code I just checked the Wikipedia page on XeTeX (not knowing what the heck that is at all). It's in fact actually something very recent in terms of the TeX timeline; it was released in 2004. It has Unicode support, which seems to be the big thing. That is more recent than, I think, the last time I used TeX; which worked absolutely fine. Someone rewriting TeX in Rust should work with Kn…

> the last time I used TeX; which worked absolutely fine.

I've heard that original TeX is the only software in the world that doesn't have any bugs (no one found so far).

Re: Rewriting LaTeX in Pure Rust

#138

Post by int_19h » Tue, Dec 08, 2020, 6:47 PM UTC

jabl wrote: post #51

jbergens wrote: post #49

Earlier quoted context omitted.

Rust seems to be a lot safer than C or C++ but I don't really understand why a documentation system must be written in a low level language. Why not use C#, Java, Go or something similar? Most users can't really need extreme performance.

I'm quite sure anybody who has written substantial documents with *TeX disagrees that "extreme performance" isn't needed.

The average document size hasn't increased in the last 30 years, though, did it? But hardware is several orders of magnitude faster.

Re: Rewriting LaTeX in Pure Rust

#139

Post by jfk13 » Tue, Dec 08, 2020, 7:19 PM UTC

deepsun wrote: post #137

kazinator wrote: post #135

Earlier quoted context omitted.

> Simply wrapping the 1980s-era code I just checked the Wikipedia page on XeTeX (not knowing what the heck that is at all). It's in fact actually something very recent in terms of the TeX timeline; it was released in 2004. It has Unicode support, which seems to be the big thing. That is more recent than, I think, the last time I used TeX; which worked absolutely fine. Someone rewriting TeX in Rust should work with Kn…

> the last time I used TeX; which worked absolutely fine. I've heard that original TeX is the only software in the world that doesn't have any bugs (no one found so far).

That's certainly not correct; while it is much closer to bug-free than most software, there have been numerous fixes since the original release (as well as a few enhancements).

See https://ctan.org/tex-archive/systems/knuth/dist/errata, particularly the file "tex82.bug".

Knuth will be reviewing bug reports and potentially issuing additional fixes again next year (see http://www.tug.org/texmfbug/).

Re: Rewriting LaTeX in Pure Rust

#140

Post by kazinator » Tue, Dec 08, 2020, 7:51 PM UTC

jfk13 wrote: post #139

deepsun wrote: post #137

Earlier quoted context omitted.

> the last time I used TeX; which worked absolutely fine. I've heard that original TeX is the only software in the world that doesn't have any bugs (no one found so far).

That's certainly not correct; while it is much closer to bug-free than most software, there have been numerous fixes since the original release (as well as a few enhancements). See https://ctan.org/tex-archive/systems/knuth/dist/errata , particularly the file "tex82.bug". Knuth will be reviewing bug reports and potentially issuing additional fixes again next year (see http://www.tug.org/texmfbug/ ).

I found one years ago that I didn't report. When I issued Ctrl-D on the interactive TeX prompt to bail out, it failed to issue a newline, leaving the operating system prompt juxtaposed to the right of the TeX prompt.

According to ISO C, "[w]hether the last line [of a text stream] requires a terminating new-line character is implementation-defined", so terminating the program without the last character written to stdout (a text stream) being a newline is not maximally portable.

That's a peculiar and possibly unique situation in the standard: whether or not a requirement exists is implementation-defined. Logically, that is as good as it being required, since any implementation can make it required. Those not making it required are just supplying a documented extension in place of undefined behavior.