Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xee: A Modern XPath and XSLT Engine in Rust

blog.startifact.com

Post reply on HN
Print view View original

71–80 of 245 posts

Re: Xee: A Modern XPath and XSLT Engine in Rust

#71

Post by jrpelkonen »

jerf wrote: post #29
airstrike wrote: post #8

What problems are {elegantly, neatly, best} solved by using XPath and XSLT today that would make them reasonable choices over alternatives?

XPath is a very nice language for querying over XML. Most places pitch it as a "declarative" syntax, but as I am quite skeptical of "declarative" as a concept, you can also look at the vast majority of the XPath standard as a way to imperatively drive a multicursor over an XML document, diving in out and out nodes and extracting bits of text and such, without having to write the equivalent code in your language to do…

It’s been a while since I’ve had to deal with XML, but I remember finding it fairly convenient to restructure XML documents with XSLT. Modifying the data in those documents, much less so. I think there’s a sweet spot.

Re: Xee: A Modern XPath and XSLT Engine in Rust

#72

Post by koito17 »

velcrovan wrote: post #67
vessenes wrote: post #13

This, thirty years later, is the best pitch for XML I’ve read. Essentially, it’s a slow moving, standards-based approach to data interoperability. I hated it the minute I learned about it, because it missed something I knew I cared about, but didn’t have a word for in the 90s - developer ergonomics. XML sucks shit for someone who wants to think tersely and code by hand. Seriously, I hate it with a fiery passion. Happ…

You should try using a LISP like Racket for XML. Because XML can be expressed directly as S-expressions, XML and LISP go together like peanut butter and jelly. Hello world (greeting ((attr "val") (href "#")) "Hello " (thing "world"))

In my experience, at least with Clojure, it's much more convenient to serialize XML into a map-like structure. With your example, the data structure would look like so.

  {:tag     :greeting
   :attrs   {:href "#" :attr "val"}
   :content ["Hello" {:tag :thing :content ["world"]}]}
Some people use namespaced keywords (e.g. :xml/tag) to help disambiguate keys in the map. This kind of data structure tends to be more convenient than dealing with plain sexps or so-called "Hiccup syntax". i.e.

  [:greeting {:href "#" :attr "val"} "Hello" [:thing "world"]]
The above syntax is convenient to write, but it's tedious to manipulate. For instance, one needs to dispatch on types to determine whether an element at some index is an attribute map or a child. By using the former data structure, one simply looks up the :attrs or :content key. Additionally, the map structure is easier to depth-first search; it's a one-liner with the tree-seq function.

I've written a rudimentary EPUB parser in Clojure and found it easier to work with zippers than any other data structure to e.g. look for elements with a ancestor.

Zippers are available in most programming languages, thankfully, so this advantage is not really unique to Clojure (or another Lisp). However, I will agree that something like sexps (or Hiccup) is more convenient than e.g. JSX, since you are dealing with the native syntax of the language rather than introducing a compilation step and non-standard syntax.

Re: Xee: A Modern XPath and XSLT Engine in Rust

#73

Post by [deleted] »

01HNNWZ0MV43FF wrote: post #36
infogulch wrote: post #26

There are many humongous XML sources. E.g. the Wikipedia archive is 42GB of uncompressed text. Holding a fully parsed representation of it in memory would take even more, perhaps even >100GB which immediately puts this size of document out of reach. The obvious solution is streaming, but streaming appears to not be supported, though is listed under Challenging Future Ideas: https://github.com/Paligo/xee/blob/main/ide…

Is that all in one big document?

[deleted]

Re: Xee: A Modern XPath and XSLT Engine in Rust

#74

Post by Finnucane »

therealmarv wrote: post #5

Great to see that somebody else creates a true open source XSLT 3 and XPATH 3 implementation! I worked on projects which refused to use anything more modern than XSLT & XPATH 1.0 because of lack of support in the non Java/Net World (1.0 = tech from 1999). Kudos to Saxon though, it was and is great but I wished there were more implementations of XSLT 2.0 & XPATH 2.0 and beyond in the open source World... both are so m…

I've worked on archive projects with complex TEI xml files (which is why when people say xml is bad and it should be all json or whatever, I just LOL), and fortunately, my employer will pay for me to have an editor (Oxygen) that includes the enterprise version of Saxon and other goodies. An open-source xml processing engine that wasn't decades out of date would be a big deal in the digital humanities world.

Re: Xee: A Modern XPath and XSLT Engine in Rust

#75

Post by notfed »

Does it preserve whitespace? Something that I always found asinine about XSLT is that it wipes out whitespace when transforming. Imagine you have thousands of corporate XML files in source control, and you want to tranform them all, performing some simple mutation. XSLT claims to be fit for this job, but in practice your diff is going to be full of unintentional whitespace mangling.

Re: Xee: A Modern XPath and XSLT Engine in Rust

#78

Post by ebruchez »

notfed wrote: post #75

Does it preserve whitespace? Something that I always found asinine about XSLT is that it wipes out whitespace when transforming. Imagine you have thousands of corporate XML files in source control, and you want to tranform them all, performing some simple mutation. XSLT claims to be fit for this job, but in practice your diff is going to be full of unintentional whitespace mangling.

XSLT will perform the transformations that you instruct it to do. It does not wipe out whitespace just on its own. Do you mean that you'd like facilities to nicely reindent the output?

Re: Xee: A Modern XPath and XSLT Engine in Rust

#79

Post by bambax »

faassen wrote: post #50
infogulch wrote: post #26

There are many humongous XML sources. E.g. the Wikipedia archive is 42GB of uncompressed text. Holding a fully parsed representation of it in memory would take even more, perhaps even >100GB which immediately puts this size of document out of reach. The obvious solution is streaming, but streaming appears to not be supported, though is listed under Challenging Future Ideas: https://github.com/Paligo/xee/blob/main/ide…

Anything could be supported with sufficient effort, but streaming hasn't been my priority so far and I haven't explored it in detail. I want to get XSLT 3.0 working properly first. There's a potential alternative to streaming, though - succinct storage of XML in memory: https://blog.startifact.com/posts/succinct/ I've built a succinct XML library named Xoz (not integrated into Xee yet): https://github.com/Paligo/xoz…

> I want to get XSLT 3.0 working properly first

May I ask why? I used to do a lot of XSLT in 2007-2012 and stuck with XSLT 2.0. I don't know what's in 3.0 as I've never actually tried it but I never felt there was some feature missing from 2.0 that prevented me to do something.

As for streaming, an intermediary step would be the ability to cut up a big XML file in smaller ones. A big XML document is almost always the concatenation of smaller files (that's certainly the case for Wikipedia for example). If one can output smaller files, transform each of them, and then reconstruct the initial big file without ever loading it in full in memory, that should cover a huge proportion of "streaming" needs.

Re: Xee: A Modern XPath and XSLT Engine in Rust

#80

Post by philipkglass »

infogulch wrote: post #53
wongarsu wrote: post #31

Earlier quoted context omitted.

100GB doesn't sound that out of reach. It's expensive in a laptop, but in a desktop that's about $300 of RAM and our supported by many consumer mainboards. Hetzner will rent me a dedicated server with that amount of ram for $61/month. If the payloads in question are in that range, the time spent to support streaming doesn't feel justified compared to just using a machine with more memory. Maybe reducing the size of t…

I just pulled the 100GB number out of nowhere, I have no idea how much overhead parsed xml consumes, it could be less or it could be more than 2.5x (it probably depends on the specific document in question). In any case I don't have $1500 to blow on a new computer with 100GB of ram in the unsubstantiated hope that it happens to fit, just so I can play with the Wikipedia data dump. And I don't think that's a reasonabl…

In the case of Wikipedia dumps there is an easy work-around. The XML dump starts with a small header and "" section. Then it's just millions of "page" documents for the Wiki pages.

You can read the document as a streaming text source and split it into chunks based on matching pairs of "" and "" with a simple state machine. Then you can stream those single-page documents to an XML parser without worrying about document size. This of course doesn't apply in the general case where you are processing arbitrary huge XML documents.

I have processed Wikipedia many times with less than 8 GB of RAM.

Post reply on HN