Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsoid in PHP, or There and Back Again

phabricator.wikimedia.org

Post reply on HN
Print view View original

51–60 of 75 posts

Re: Parsoid in PHP, or There and Back Again

#51

Post by _v7gu »

TicklishTiger wrote: post #26

Anybody else feeling that strict typing and long var names are not worth all the visual overload? Example: https://github.com/wikimedia/parsoid/blob/master/src/Parsoid... This is how I would write the function definition: function html2wikitext($config, $html, $options = [], $data = null) This how Wikimedia did it: public function html2wikitext( PageConfig $pageConfig, string $html, array $options = [], ?SelserData $…

Honestly, I think the tooling should allow you to write just

    function html2wikitext($config, $html, $options = [], $data)
and then infer the types, and subsequently enforce them. The typechecker should only complain when it can't infer the types (a-la strict). If you use tooling to reduce verbosity, you'll have best of both worlds simultaneously.

Re: Parsoid in PHP, or There and Back Again

#52

Post by hnzix »

b3kart wrote: post #44
hnzix wrote: post #41

Earlier quoted context omitted.

Strict typing has an overhead, but the payoff is apparent when working with a large dev team of variable skill levels. People can make a mess in any language but it's easier to decode with strict typing. For lone wolf coding or rapid prototyping the equation is different.

I've never worked in a large team that has used a dynamically typed language. To me it sounds like a nightmare, especially given that I know how little time is typically left for documentation. I see that it could work if you enforce type annotations. But then you might as well use a strongly typed language.

I've worked on a 15 person team using Perl that was actually pleasant because everyone was A grade, actively did code reviews and the two seniors were A+ and would kick your ass for bad code. But most teams aren't like that so would benefit from strong typing.

Re: Parsoid in PHP, or There and Back Again

#53

Post by Piskvorrr »

TicklishTiger wrote: post #26

Anybody else feeling that strict typing and long var names are not worth all the visual overload? Example: https://github.com/wikimedia/parsoid/blob/master/src/Parsoid... This is how I would write the function definition: function html2wikitext($config, $html, $options = [], $data = null) This how Wikimedia did it: public function html2wikitext( PageConfig $pageConfig, string $html, array $options = [], ?SelserData $…

You can then e.g. run a static analyzer over it, which can say things like 'in SomeOtherFile.php:130, you're passing $config="nope"; this is a string, not what the function expects to handle' or 'html2wikitext is supposed to return string, you're returning a DateTimeImmutable at line 160'. Same with the access modifiers: 'private function whatever() is unused'.

Further, documentation: "of course everybody knows what you're supposed and forbidden to pass into $data" - NOT. Even if it's just you writing the code: the you+1year will have trouble reading it (been there). Not even when it's supposed to be documented. If you have an explicit data structure, this becomes far more evident, even before any documentation (note: not replacing it).

I'm not interested in playing computer in my head any more, juggling internal state that's completely superfluous to me: am I a higher primate? Yes. Are higher primates tool users? Also yes. Should I let machines do the menial tasks for me, leaving me to do the creative ones? A hundred times yes.

(NB: this is not a silver bullet - e.g. won't help against logic errors - but it's a useful guard against going completely off the rails)

Re: Parsoid in PHP, or There and Back Again

#54

Post by plopz »

TicklishTiger wrote: post #26

Anybody else feeling that strict typing and long var names are not worth all the visual overload? Example: https://github.com/wikimedia/parsoid/blob/master/src/Parsoid... This is how I would write the function definition: function html2wikitext($config, $html, $options = [], $data = null) This how Wikimedia did it: public function html2wikitext( PageConfig $pageConfig, string $html, array $options = [], ?SelserData $…

I agree with you, I prefer your structure. I actually wish PHP hadn't added typing, if I wanted that I wouldn't have chosen PHP in the first place.

Re: Parsoid in PHP, or There and Back Again

#55

Post by acqq »

hnzix wrote: post #52
b3kart wrote: post #44

Earlier quoted context omitted.

I've never worked in a large team that has used a dynamically typed language. To me it sounds like a nightmare, especially given that I know how little time is typically left for documentation. I see that it could work if you enforce type annotations. But then you might as well use a strongly typed language.

I've worked on a 15 person team using Perl that was actually pleasant because everyone was A grade, actively did code reviews and the two seniors were A+ and would kick your ass for bad code. But most teams aren't like that so would benefit from strong typing.

And Perl has many aspects of strength compared with Python or Javascript. See for example:

https://news.ycombinator.com/item?id=22282080

and

https://news.ycombinator.com/item?id=22282306

For what comes out of the box with Perl regarding undefined variables. Perl has "my" "local" and "ours" for the start but even more: the array and hash variables have by design different syntax, which helps immensely: the sigils are of real help as a kind of written "type." It's like in old Basics string variables looked different from the numeric ones, and Larry Wall acknowledged he was inspired by that. I can go on and on. Perl looks hard to the uninitiated but it can produce much more stable code, in my experience, than the "typical" scripting languages like Python and Javascript. Stable in the sense that you know it will work after it compiles, not only once you test with every possible input.

But yes, the people using it should know learn enough before they start contributing, and in a lot of places it's preferred to have people who barely know the basics of what they use (because they are "cheap" and "easily replaceable").

Re: Parsoid in PHP, or There and Back Again

#56

Post by egeozcan »

Cthulhu_ wrote: post #47
TicklishTiger wrote: post #26

Anybody else feeling that strict typing and long var names are not worth all the visual overload? Example: https://github.com/wikimedia/parsoid/blob/master/src/Parsoid... This is how I would write the function definition: function html2wikitext($config, $html, $options = [], $data = null) This how Wikimedia did it: public function html2wikitext( PageConfig $pageConfig, string $html, array $options = [], ?SelserData $…

Your first function definition leans HEAVILY on implicit knowledge - what does $config mean? What does $html mean? What are valid properties of $config and $data? This is fine in smaller codebases, 'your' code, and code that you can read to a point where you can extrapolate these variables from the implementation, but this simply does not scale beyond a certain code size - or more importantly, a certain amount of con…

Why would you add a docblock on top of your already typed functions? Maybe just description would be cool but phpdoc with all the duplicated parameter definitions? I think it's unnecessary but would love to hear other perspectives.

Re: Parsoid in PHP, or There and Back Again

#57

Post by Eremotherium »

egeozcan wrote: post #56
Cthulhu_ wrote: post #47

Earlier quoted context omitted.

Your first function definition leans HEAVILY on implicit knowledge - what does $config mean? What does $html mean? What are valid properties of $config and $data? This is fine in smaller codebases, 'your' code, and code that you can read to a point where you can extrapolate these variables from the implementation, but this simply does not scale beyond a certain code size - or more importantly, a certain amount of con…

Why would you add a docblock on top of your already typed functions? Maybe just description would be cool but phpdoc with all the duplicated parameter definitions? I think it's unnecessary but would love to hear other perspectives.

You can add e.g. descriptions to your params if needed. Also phpdoc understands types like string[] for arrays and union types like (int|string) for untyped params.

Re: Parsoid in PHP, or There and Back Again

#58

Post by cscottnet »

I'm on the team. Part 2 of this post series should have lots of interesting technical details for y'all; be patient, I'm still writing it.

But to whet your appetite: we used https://github.com/cscott/js2php to generate a "crappy first draft" of the PHP code for our JS source. Not going for correctness, instead trying to match code style and syntax changes so that we could more easily review git diffs from the crappy first draft to the "working" version, and concentrate attention on the important bits, not the boring syntax-change-y parts.

The original legacy Mediawiki parser used a big pile of regexps and had all sorts of corner cases caused by the particular order in which the regexps were applied, etc.

Parsoid uses a PEG tokenizer, written with pegjs (we wrote a PHP backend to pegjs for this project). There are still a bunch of regexps scattered throughout the code, because they are still very useful for text processing and a valuable feature of both JavaScript and PHP as programming languages, but they are not the primary parsing mechanism. Translating the regexps was actually one of the more difficult parts, because there are some subtle differences between JS and PHP regexps.

We made a deliberate choice to switch from JS-style loose typing to strict typing in the PHP port. Whatever you may consider the long term merits are for maintainability, programming-in-the-large, etc, they were extremely useful for the porting project itself, since they caught a bunch of non-obvious problems where the types of things were slightly different in PHP and JS. JS used anonymous objects all over the place; we used PHP associative arrays for many of these places, but found it very worthwhile to take the time to create proper typed classes during the translation where possible; it really helped clarify the interfaces and, again, catch a lot of subtle impedance mismatches during the port.

We tried to narrow scope by not converting every loose interface or anonymous object to a type -- we actually converted as many things as possible to proper JS classes in the "pregame" before the port, but the important thing was to get the port done and complete as quickly as possible. We'll be continuing to tighten the type system -- as much for code documentation as anything else -- as we address code debt moving forward.

AMA, although I don't check hacker news frequently so I can't promise to reply.

Re: Parsoid in PHP, or There and Back Again

#59

Post by cscottnet »

cscottnet wrote: post #58

I'm on the team. Part 2 of this post series should have lots of interesting technical details for y'all; be patient, I'm still writing it. But to whet your appetite: we used https://github.com/cscott/js2php to generate a "crappy first draft" of the PHP code for our JS source. Not going for correctness, instead trying to match code style and syntax changes so that we could more easily review git diffs from the crappy…

If you want to dig through the history some: https://github.com/wikimedia/parsoid/blame/6eb00df3e090b20cc... Is a pretty good example of the porting technique. You'll see quite a decent number of lines are still unchanged from the "automatic conversion from JS". https://github.com/wikimedia/parsoid/commit/6eb00df3e090b20c... shows what the initial port process was like. Still quite a bit of work, but you'll see it's almost all "real" work that needs a human to think about things, not just mechanical syntax translation. The syntax translation part was done automatically.

Then https://github.com/wikimedia/parsoid/commits/master/src/Ext/... is a not-too-atypical view of the process after the "intial working port" was done (post Aug 2019). Some nasty bugs fixed (https://github.com/wikimedia/parsoid/commit/34fcb4241aa0f3a0... a GC bug in PHP!), some more subtle bugs (PHP's crazy behavior of '$' at the end of a regexp, unless you use the 'D' flag), etc.

If you look through the history earlier in 2019, you'll even see JS commits like https://github.com/wikimedia/parsoid/commit/2853a90ceda7cdfa... which are to the JS code (in production at the time) preparing the way for the PHP port. In that particular case, our tooling was doing offset conversion between JS UTF-16 and PHP UTF-8 as part of the output-testing-and-comparison QA framework we'd built for the port, and it was getting hugely confused by Gallery since Gallery was using "bogus" offsets into the source text. Since fixing the offsets was rather involved (the patchset for this commit in gerrit went through 56 revisions : https://gerrit.wikimedia.org/r/505319 ) the change was first done on the JS side, thoroughly tested, and deployed to production to ensure it had no inadvertent effects, before that now-better JS code was ported to PHP. It would have been a disaster to try to make this change in the PHP version directly during the port.

Re: Parsoid in PHP, or There and Back Again

#60

Post by lolphp111 »

Why? The editor needs a frontend in javascript anyways, so why mot handle this all in real time on the client?

Now they rewrote in PHP, thats probably one of the worst languages out there, and why not rewrite in something compiled if speed was the main reason for a rewrite?

For me PHP sits in the middle as a poor language, and still slow compared to any compiled languages. Also i would want to see some wasm vs php benchmarks they did before starting with php.

Lots of poor decisions from the wiki team.

Post reply on HN