Live data from Hacker News

Rewriting LaTeX in Pure Rust

github.com

121–130 of 148 posts

Re: Rewriting LaTeX in Pure Rust

#121

Earlier quoted context omitted.

I think you're getting your hostile answers for implying that unsafe rust is equivalent to C. It's most definitely not. Using unsafe grants you certain powers, but it does not disable all of rusts features - for example the borrow checker is not turned off by unsafe. See https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html#unsa...

Safety doesn't just come from language features, it also comes from the language disallowing dangerous actions. Rust's unsafe mode opens the door to undefined behaviour, of the sort that plagues so many C/C++ codebases (buffer overflows etc). A program written in safe Rust offers far better assurances than a program making heavy use of unsafe Rust: safe Rust is unable to result in undefined behaviour. (Bugs in the co…

> Safety doesn't just come from language features, it also comes from the language disallowing dangerous actions. Rust's unsafe mode opens the door to undefined behaviour, of the sort that plagues so many C/C++ codebases (buffer overflows etc). A program written in safe Rust offers far better assurances than a program making heavy use of unsafe Rust: safe Rust is unable to result in undefined behaviour.

Safety is not an absolute, it's a spectrum. No one denies that safe rust is better than unsafe rust on the safety scale.

> You may be right that a program written in 100% unsafe Rust might still be less prone to undefined behavior than a program written in C, but that's not my point. Excessive use of unsafe features undermines the considerable safety advantages that Rust offers over C, and it's regrettable when this is disregarded.

It's not disregarded. The point you are disregarding that when porting a C application to rust, unsafe rust is a step up from C, not step down from safe rust. Unless you choose to rewrite from ground up (which is infeasible in many places), you'll need unsafe rust, either for binding or by using tooling that converts the C sources to rust. But once you have unsafe rust, you already get all the help that the borrow checker brings and you can gradually shrink the unsafe code. It's a matter of practicality, you seem to be advocating for absolutes and I think that's earning you the down votes you're receiving.

Re: Rewriting LaTeX in Pure Rust

#122
post #52

Earlier quoted context omitted.

The advantage of this approach (as opposed to a rewrite), is that you reproduce all the bugs and misfeatures that people in the wild depend on. You can then add a test suite and start refactoring and gradually move to a codebase you're happy with, while breaking few or no users on the way. Another approach is to split it into blocks and replace parts of the system, but whether that is feasible depends on the software…

But the advantage of avoiding C specific bugs is shallow, because the Rust code contains lots of unsafe blocks. Bugfixes made to the original source are not automatically ported to the Rust code base. From my point of view the chosen porting "strategy" doesn't make much sense. It is more of a toy project to see what's possible. What would really make sense would be starting with an extensive test suite and trying to…

Everyone has a different way of handling this but if I had to, I would also take the approach described in this project: a variant of the strangle pattern of refactoring.

However, I don't know of a single one of these c to rust translation projects that has succeeded. I don't know of any large c to rust rewrite project that has succeeded so that doesn't help judge between rewrite patterns. It may just be that rewrites don't yield results at all.

Re: Rewriting LaTeX in Pure Rust

#123

Earlier quoted context omitted.

LuaLaTeX allocates memory as-needed, see section 3.4.1 in the manual [0] (and comments/answers in this thread [1]). Base TeX has an arbitrary, by modern standards low memory limit, leading to a whole class of errors plaguing unsuspecting users [2], and spawning entire extensions to deal with these limitations [3]. This is simply an artefact of times past and has no technical relevance nowadays. LuaTeX allows dynamic…

I’ve been a luatex advocate in the past¹, but I use xetex instead, unless I need the Lua integration. The memory handling is the reason. I find that for documents with a lot of fonts, luatex eats all the memory available and then crashes, taking a huge amount of time to do so, whereas xetex just breezes through the same document. [1] https://lwn.net/Articles/731581/

What's troublesome for me is that I have been using

* xetex when I needed a font that was not easily achievable in pdftex over the past decade * pdftex for everything else because microtype(TM) just works(TM) (even though kerning can be done using fontspec and font features in xetex).

I've tried luatex multiple times over the past decade, it was mostly just too slow. Now luatex is fast. But I have no idea if I now "should" use luatex over pdftex for best out-of-the-box results or not.

Unfortunately, switching to luatex is not a zero-effort (moving to polyglossia, using fontspec, maybe removing some magic in many-lines private templates, and so on).

For all I know, because I'm always curious and peek at PDF file properties as a hobby (if only to check which cool font that is), basically every scientific paper I read is set using pdftex. luatex usage in the wild is, as far as I perceive it, nil, outside of enthusiast luatex user spheres. I don't think this will change unless texlive drops pdftex (as it still ships ptex and even uptex, it probably won't for a very long time).

Re: Rewriting LaTeX in Pure Rust

#124
post #123

Earlier quoted context omitted.

I’ve been a luatex advocate in the past¹, but I use xetex instead, unless I need the Lua integration. The memory handling is the reason. I find that for documents with a lot of fonts, luatex eats all the memory available and then crashes, taking a huge amount of time to do so, whereas xetex just breezes through the same document. [1] https://lwn.net/Articles/731581/

What's troublesome for me is that I have been using * xetex when I needed a font that was not easily achievable in pdftex over the past decade * pdftex for everything else because microtype(TM) just works(TM) (even though kerning can be done using fontspec and font features in xetex). I've tried luatex multiple times over the past decade, it was mostly just too slow. Now luatex is fast. But I have no idea if I now "s…

Officially, luatex is the future. ConTeXt is based on it. I’ve heard that the kinds of problems I’m having are caused by its font-loading routines, and not the core parts of luatex, but without further research that doesn’t really help me.

Re: Rewriting LaTeX in Pure Rust

#126
post #48

Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and g…

> Simply wrapping the 1980s-era code in a "skin" of Rust syntax brings little value.

That's right for TeX, but XeTeX was first released in 2004.

Re: Rewriting LaTeX in Pure Rust

#127
post #126
post #48

Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and g…

> Simply wrapping the 1980s-era code in a "skin" of Rust syntax brings little value. That's right for TeX, but XeTeX was first released in 2004.

Yes, but the bulk of its code is the original TeX code, from 1984. It did not attempt to reimplement or modernise the core code. Until a few years ago, it was even still built from tex.web plus a set of change-files plus some C/C++ libraries. Nowadays, the main change-files have been merged into the WEB source, for easier management, but it's still the old TeX code at heart.

Actually, there was another intermediate stage: XeTeX is in effect a descendant of TeXgX, an extended version of TeX that integrated with the now-discontinued QuickDraw GX graphics and font technology on classic Mac OS. But anyhow, it's still a direct descendant of Knuth's code. (No criticism intended: TeX was -- and still is -- a fantastic piece of work, but its code is from a different era and was shaped by constraints that are irrelevant today.)

Re: Rewriting LaTeX in Pure Rust

#128
While the name “Rewriting LaTeX in Pure Rust” is very exciting, there are two problems:

• “LaTeX” is a set of macros (ab)using the macro/text-expansion feature of an underlying TeX engine (LuaTeX, XeTeX, pdTeX, original Knuth TeX, or this one), to provide things like cross-references, section numbering, etc. When you as a typical user use “LaTeX”, you're actually using these macros, along with a huge variety of macro “packages” written by many authors, that together add up to orders of magnitude more lines of code than the TeX engine itself (about 25000 lines originally: https://tex.stackexchange.com/a/505664). Most of the incompatibilities and error messages a typical user encounters when using LaTeX are caused not by the TeX engine but by these packages or LaTeX, which this project doesn't touch.

• This project comes about as:

•— tex.web, Knuth's original source code, written in his literate-programming system WEB (a preprocessor on top of Pascal),

•— extended to xetex.web (XeTeX), to provide Unicode support etc,

•— automatically tangled to Pascal code (macros expanded and constants replaced),

•— automatically translated to C code,

•— now automatically translated to Rust code, by wrapping the C code in `unsafe` Rust blocks.

So it has gone through multiple rounds of machine translation. Actually, I'm impressed that this project has made many improvements in undoing and restoring some of the WEB macros, since the last time this was mentioned here. (Compare a random comparison from 1 year ago: https://gist.github.com/627399d0150e66d211a264bc05b33beb with the same comparison today: https://gist.github.com/71e47f2276f9c6c030efe0e3357ef3bf) — two of the four differences I had pointed out in my comment then (https://news.ycombinator.com/item?id=21177367) are gone, and the only ones to mention are:

• The comments from the original are gone.

• Something like “cur_if:=subtype(p);” becomes “ cur_if = MEM[p].b16.s0 as i16;”

So there's still a long way to go. But even with all that, what would be more useful IMO are (in order of difficulty):

• Translate directly from WEB (xetex.web) to Rust. The original uses only a subset of Pascal so IMO it should be possible to write an automatic translator that preserves more of the original. Something along these lines has been done for translating WEB to CWEB by Martin Ruckert (see https://w3-o.cs.hm.edu/users/ruckert/public_html/web2w/index... and http://tug.org/TUGboat/Contents/listauthor.html#Ruckert,Mart... and his demo from TUG 2020). Alternatively, one could give up on XeTeX and start with the LuaTeX source code (in C).

• Understand how the TeX program is written/works, and write a more idiomatic translation in Rust: something along these lines is in progress as a one-person hobby project by Emily Eisenberg (one of the original developers of KaTeX): see https://github.com/xymostech/XymosTeX This one is most exciting to me, as I also have (not very useful so far) grand plans to understand TeX and to make it more understandable for everybody (https://shreevatsa.net/tex/program).

• As most of the code a user deals with is in the macro/LaTeX layer, modify TeX macro expansion to make it more “debuggable” (https://tex.stackexchange.com/a/384881, https://cstheory.stackexchange.com/a/40282): show full stack traces beyond what one gets with \errorcontextlines=\maxdimen (so file names and line numbers, by tracking the provenance of token lists), show arguments scanned (“passed in”) and expected/found types thereof, cleanly separate the macro expansion layer from the typesetting (line-breaking / page-breaking) layer, maybe even (JIT?) “compile” some of the lower-level or frequently-used macros — the eventual goal would be for the user to completely understand what is going on when they compile their LaTeX document, so that (1) they are less surprised by errors and know what to do, (2) they may be induced (if they're programmers) to do less in the (unsuitable) macro layer of the TeX engine and do more at the appropriate layer: either do it in a preprocessor that runs on the .tex file, or do it at the typesetting layer via something like LuaTeX's hooks (pre_linebreak_filter etc).

Anyway, this project could still end up getting there, so let's see! Interacting with the TeX community (the TeX StackExchange, mailing lists, write an article for TUGboat, etc) might also be useful.

Re: Rewriting LaTeX in Pure Rust

#129

Earlier quoted context omitted.

Safety doesn't just come from language features, it also comes from the language disallowing dangerous actions. Rust's unsafe mode opens the door to undefined behaviour, of the sort that plagues so many C/C++ codebases (buffer overflows etc). A program written in safe Rust offers far better assurances than a program making heavy use of unsafe Rust: safe Rust is unable to result in undefined behaviour. (Bugs in the co…

> Safety doesn't just come from language features, it also comes from the language disallowing dangerous actions. Rust's unsafe mode opens the door to undefined behaviour, of the sort that plagues so many C/C++ codebases (buffer overflows etc). A program written in safe Rust offers far better assurances than a program making heavy use of unsafe Rust: safe Rust is unable to result in undefined behaviour. Safety is not…

> Safety is not an absolute, it's a spectrum.

Use of the safe subset of Rust means the compiler and standard-library offer you a guarantee of the absence of undefined behaviour. That's an absolute guarantee of safety, under Rust's understanding of the word.

It doesn't give you a guaranteed absence of memory-leaks. It certainly doesn't give you a guarantee of whole-program correctness, as Rust isn't a formal verification framework. Both these properties are beyond the scope of 'safety' as Rust uses it.

> It's not disregarded.

It is. Projects are described as written in Rust, treating safe Rust and unsafe Rust equally.

> The point you are disregarding that when porting a C application to rust, unsafe rust is a step up from C, not step down from safe rust.

I explicitly acknowledged this.

> Unless you choose to rewrite from ground up (which is infeasible in many places), you'll need unsafe rust, either for binding or by using tooling that converts the C sources to rust.

Sure, no disagreement there.

> But once you have unsafe rust, you already get all the help that the borrow checker brings and you can gradually shrink the unsafe code.

Sure, and that's a good use of Rust's unsafe features.

> you seem to be advocating for absolutes

Not really. If I had meant to argue that Rust shouldn't have unsafe features, I'd have done so.

In an ideal world all code would be written in a way that completely closes the door on undefined behaviour, but we agree there are good reasons Rust includes its unsafe features, and there are good practical reasons to use them. I'm advocating for a greater emphasis on the use of the safe subset of Rust.

Written in Rust tells me something about the software. Written in 100% safe Rust tells me much more about the software. That's essentially my point.

This distinction doesn't arise with languages like C and JavaScript. All C is unsafe, and all JavaScript is safe. For languages like Rust and D, there's value in being upfront about the use of their safe subsets.

Re: Rewriting LaTeX in Pure Rust

#130

I'm not sure rewriting Latex in XYZ is a good idea. The whole thing is a mess and a pain to use (and I use it multiple times a week). Sure the documents often look good, but when they don't or there is an error, fixing it just becomes a nightmare. I wish somebody would design a modern typesetting system that would accept say latex equations but without 50 years of cruft and levels of macroses upon macroses.

> I wish somebody would design a modern typesetting system... In theory yes, but I think the problem is that it will never be adopted. Too much of academia is built on the deep assumption that LaTeX is the only possible format that people write documents in. I think one possible way out is to start a new from-scratch TeX engine that is built in such a way that a new typesetting language (or "front end" if you like) c…

In theory yes, but I think the problem is that it will never be adopted. Too much of academia is built on the deep assumption that LaTeX is the only possible format that people write documents in.

You're not wrong, but then again, this is not the biggest thing wrong with a lot of academic publishing, and it wouldn't be the biggest tower that a lot of academics are trying to topple. Indeed, breaking away from TeX and breaking away from the traditional journals and the toxic model around publishing in them might be happy partners.

Post reply on HN