Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Parser that Cracked the MediaWiki Code

dirkriehle.com

Post reply on HN
Print view View original

11–20 of 33 posts

Re: The Parser that Cracked the MediaWiki Code

#11

Post by VMG »

sigil wrote: post #9
Semiapies wrote: post #2

I hadn't realized that there were any parsing issues around MediaWiki's markup. 5000 lines of PHP? Eek.

It's worse. The MediaWiki PHP code doesn't implement a proper scanner and parser, it's a bunch of regexes around which the code has grown more or less organically. Silent compensation for mismatched starting and ending tokens abounds, and causes problems for all consumers of the markup, in the same way that lenient HTML parsers have. The difference is that Wikipedia, as the sole channel for editing markup, could have…

Interestingly markdown has the same problem.

Another example of the imperfect but working implementation winning.

Re: The Parser that Cracked the MediaWiki Code

#13

Post by sigil »

It's great to see people tackling this problem, but I wouldn't declare victory for sweble just yet ("The Parser That Cracked..."). There are other promising MediaWiki parser efforts out there.

For one, sweble is a Java parser, and I'm not sure this makes it a good drop-in replacement for the current MediaWiki PHP code. The DBPedia Project also has what looks like a decent AST-based Java parser [1]. I would be interested in a comparison between sweble and DBPedia's WikiParser.

I stumbled across a very nice MediaWiki scanner and parser in C a while ago [2]. It uses ragel [3] for the scanner; the parser is not a completely generic AST builder, but is rather specific to the problem of converting MediaWiki markup to some other wiki markup. It does do quite a bit of the parser work already though.

Presumably a PHP extension around a C or C++ scanner/parser could someday replace the current MediaWiki parsing code.

[1] http://wiki.dbpedia.org/DeveloperDocumentation/WikiParser?v=...

[2] http://git.wincent.com/wikitext.git

[3] http://www.complang.org/ragel/

Re: The Parser that Cracked the MediaWiki Code

#14

Post by mdaniel »

From reading the article, and especially the interesting comments thereon, it seems this problem is half a bogus "language" specification and half that the unwashed masses are inputting any damn thing they like and Wikipedia accepts it.

I suppose this is one of the knobs that must be tuned to balance between reproducible I/O and turning away meaningful contributions from the community.

Re: The Parser that Cracked the MediaWiki Code

#15

Post by sunir »

This is a breakthrough and a welcome one. From a end user point of view, it has a couple major implications.

First, I believe this reveals the complexity of the parser, which implies a complex syntax, which implies a complex user interface as felt by end users. A more complex the user interface may make it harder it is to attract new editors, although it's unclear (to me) if that is a fact.

Second, having an AST representation is awesome. It makes it possible to even think about building a path towards WYSIWYG or some other form of rich text editing. It was not really possible to build a WYSIWYG editor around the wiki syntax.

If you have an AST, you can also store the page as the AST since you can regenerate the wiki syntax from the AST for people who need text-based editors.

Re: The Parser that Cracked the MediaWiki Code

#16

Post by neilk »

This isn't the first alternative parser for MediaWiki content -- there are 28 rows in this table. (I just added Sweble's and my own project...)

http://www.mediawiki.org/wiki/Alternative_parsers#Known_impl...

Most of these are special purpose hacks. Kiwi and Sweble are the most serious projects I'm aware of, that have tried to generate a full parse.

However, few of these projects are useful for upgrading Wikipedia itself. Even the general parsers like Sweble are effectively special-purpose, since we have a lot of PHP that hooks into the parser and warps its behaviour in "interesting" ways. The average parser geek usually wants to write to a cleaner spec in, well, any language other than PHP. ;)

Currently the Wikimedia Foundation is just starting a MediaWiki.next project. Parsing is just one of the things we are going to change in major ways -- fixing this will make it much easier to do WYSIWYG editing or to publish content in ways that aren't just HTML pages.

(Obviously we will be looking at Sweble carefully.)

If this sounds like a fun project to you, please get in touch! Or check out the "Future" portal on MediaWiki.org.

http://www.mediawiki.org/wiki/Future

Re: The Parser that Cracked the MediaWiki Code

#18

Post by seanp2k »

VMG wrote: post #11
sigil wrote: post #9

Earlier quoted context omitted.

It's worse. The MediaWiki PHP code doesn't implement a proper scanner and parser, it's a bunch of regexes around which the code has grown more or less organically. Silent compensation for mismatched starting and ending tokens abounds, and causes problems for all consumers of the markup, in the same way that lenient HTML parsers have. The difference is that Wikipedia, as the sole channel for editing markup, could have…

Interestingly markdown has the same problem. Another example of the imperfect but working implementation winning.

Markdown...ugh. Let's just stick to DokuWiki or Mediawiki syntax for everything, please. If you need something more advanced than that, you should be using LaTeX. Actually, it'd be cool to build a working MediaWiki + Markdown => LaTeX converter....in something like Python.

Re: The Parser that Cracked the MediaWiki Code

#19

Post by knowtheory »

neilk wrote: post #16

This isn't the first alternative parser for MediaWiki content -- there are 28 rows in this table. (I just added Sweble's and my own project...) http://www.mediawiki.org/wiki/Alternative_parsers#Known_impl... Most of these are special purpose hacks. Kiwi and Sweble are the most serious projects I'm aware of, that have tried to generate a full parse. However, few of these projects are useful for upgrading Wikipedia its…

Hey Neilk!

Did you ever turn up anything regarding this? http://news.ycombinator.com/item?id=2216249

btw, neat js parser, i'll have to check it out. :)

Re: The Parser that Cracked the MediaWiki Code

#20

Post by sigil »

seanp2k wrote: post #18
VMG wrote: post #11

Earlier quoted context omitted.

Interestingly markdown has the same problem. Another example of the imperfect but working implementation winning.

Markdown...ugh. Let's just stick to DokuWiki or Mediawiki syntax for everything, please. If you need something more advanced than that, you should be using LaTeX. Actually, it'd be cool to build a working MediaWiki + Markdown => LaTeX converter....in something like Python.

> it'd be cool to build a working MediaWiki + Markdown => LaTeX converter....in something like Python.

For parsing MediaWiki in Python, check out mwlib [1], which was part of cooperation between Wikimedia Foundation and PediaPress. It's neither very complete nor very fast, but you might be able to hack up some LaTeX conversion with it.

[1] http://code.pediapress.com/wiki/wiki/mwlib

Post reply on HN