Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Parser that Cracked the MediaWiki Code

dirkriehle.com

Post reply on HN
Print view View original

21–30 of 33 posts

Re: The Parser that Cracked the MediaWiki Code

#21

Post by neilk »

knowtheory wrote: post #19
neilk wrote: post #16

This isn't the first alternative parser for MediaWiki content -- there are 28 rows in this table. (I just added Sweble's and my own project...) http://www.mediawiki.org/wiki/Alternative_parsers#Known_impl... Most of these are special purpose hacks. Kiwi and Sweble are the most serious projects I'm aware of, that have tried to generate a full parse. However, few of these projects are useful for upgrading Wikipedia its…

Hey Neilk! Did you ever turn up anything regarding this? http://news.ycombinator.com/item?id=2216249 btw, neat js parser, i'll have to check it out. :)

FYI the JS parser is broken for some cases, but it works great for most things you want from message strings.

As for your original question, I don't think there is a forum that tries to unite the left-brained and right-brained wikipedians. There is a bit of a divide. I'll send an email right now to someone who might know better.

We don't have contests per se to try to steer the community, other than I guess GSoC, or reaching out to developers that we think are already doing good things.

Re: The Parser that Cracked the MediaWiki Code

#22

Post by ZoFreX »

sigil wrote: post #13

It's great to see people tackling this problem, but I wouldn't declare victory for sweble just yet ("The Parser That Cracked..."). There are other promising MediaWiki parser efforts out there. For one, sweble is a Java parser, and I'm not sure this makes it a good drop-in replacement for the current MediaWiki PHP code. The DBPedia Project also has what looks like a decent AST-based Java parser [1]. I would be interes…

Given the complexity of Wikipedia's deployment compared to a typical MediaWiki installation, it really wouldn't be much effort to hook into a parser in say, Java rather than PHP, and would be well worth doing if it had significant benefits.

Of course, a PHP parser would still have to be maintained in parallel as not everyone would be able to do the Java option.

Re: The Parser that Cracked the MediaWiki Code

#23

Post by VMG »

seanp2k wrote: post #18
VMG wrote: post #11

Earlier quoted context omitted.

Interestingly markdown has the same problem. Another example of the imperfect but working implementation winning.

Markdown...ugh. Let's just stick to DokuWiki or Mediawiki syntax for everything, please. If you need something more advanced than that, you should be using LaTeX. Actually, it'd be cool to build a working MediaWiki + Markdown => LaTeX converter....in something like Python.

there is markdown2pdf written Haskell, which seems to have XeTeX as an intermediary step.

Personally I'd be happy to see any markup language becoming the default, regardless which one it is. Having a proper grammar would be a bonus.

Re: The Parser that Cracked the MediaWiki Code

#24

Post by tokenadult »

sunir wrote: post #15

This is a breakthrough and a welcome one. From a end user point of view, it has a couple major implications. First, I believe this reveals the complexity of the parser, which implies a complex syntax, which implies a complex user interface as felt by end users. A more complex the user interface may make it harder it is to attract new editors, although it's unclear (to me) if that is a fact. Second, having an AST repr…

A more complex the user interface may make it harder it is to attract new editors

There may be friction against gaining new editors from the user interface of the MediaWiki software, but I think the greatest barrier to participation by new editors is the hostile, drama-filled environment on many controversial topics on Wikipedia. My evidence for that is the decline in "unsustainable fashion"

http://strategy.wikimedia.org/wiki/Story_of_Wikimedia_Editor...

in the number of Wikipedian administrators, who presumably for the most part are people who know how to use Wikimedia software. Too many of best contributors (people who look up facts in reliable sources and edit articles for better readability) on Wikipedia feel attacked and that their time is wasted. I know a lot of dedicated hobbyists who quietly work on their hobby-related subjects putting together great articles, but on any subject that is controversial, and for which looking up reliable sources takes some effort, Wikipedia is becoming a war zone and is not improving in quality.

http://strategy.wikimedia.org/wiki/Strategic_Plan/Movement_P...

http://strategy.wikimedia.org/wiki/Strategic_Plan/Movement_P...

Re: The Parser that Cracked the MediaWiki Code

#25

Post by sigil »

ZoFreX wrote: post #22
sigil wrote: post #13

It's great to see people tackling this problem, but I wouldn't declare victory for sweble just yet ("The Parser That Cracked..."). There are other promising MediaWiki parser efforts out there. For one, sweble is a Java parser, and I'm not sure this makes it a good drop-in replacement for the current MediaWiki PHP code. The DBPedia Project also has what looks like a decent AST-based Java parser [1]. I would be interes…

Given the complexity of Wikipedia's deployment compared to a typical MediaWiki installation, it really wouldn't be much effort to hook into a parser in say, Java rather than PHP, and would be well worth doing if it had significant benefits. Of course, a PHP parser would still have to be maintained in parallel as not everyone would be able to do the Java option.

> Given the complexity of Wikipedia's deployment compared to a typical MediaWiki installation, it really wouldn't be much effort to hook into a parser in say, Java rather than PHP...

No doubt the incremental complexity for Wikipedia would be small in relative terms. I assume that argument would support a variety of proposals.

A solid scanner and parser in C/C++ would benefit a broader audience though. All the major scripting languages can be extended in C/C++. In fact, the ragel-based parser I mentioned earlier [1] was built to be used from within Ruby code.

[1] http://git.wincent.com/wikitext.git

Re: The Parser that Cracked the MediaWiki Code

#27

Post by car »

bjonathan wrote: post #4

Site down, here is a mirror: https://www.readability.com/articles/r9i55x6e cache version: http://webcache.googleusercontent.com/search?q=cache:8xjwEj-...

http://www.sweble.org is the actual Wikitext parser project homepage. Please go there until dirkriehle.com is back up.

Re: The Parser that Cracked the MediaWiki Code

#28

Post by ropers »

bjonathan wrote: post #4

Site down, here is a mirror: https://www.readability.com/articles/r9i55x6e cache version: http://webcache.googleusercontent.com/search?q=cache:8xjwEj-...

Or try Coral Cache:

http://dirkriehle.com.nyud.net/2011/05/01/the-parser-that-cr...

"It worked for me." ;-)

Re: The Parser that Cracked the MediaWiki Code

#29

Post by driehle »

pornel wrote: post #5
brianjolney wrote: post #3

link died. any mirrors?

Googlecache: http://webcache.googleusercontent.com/search?client=opera&#3...

It's back up at http://dirkriehle.com - the project site is actually http://sweble.org where under Crystalball Demo you can play with the parser without having to install anything.

Re: The Parser that Cracked the MediaWiki Code

#30

Post by lloeki »

seanp2k wrote: post #18
VMG wrote: post #11

Earlier quoted context omitted.

Interestingly markdown has the same problem. Another example of the imperfect but working implementation winning.

Markdown...ugh. Let's just stick to DokuWiki or Mediawiki syntax for everything, please. If you need something more advanced than that, you should be using LaTeX. Actually, it'd be cool to build a working MediaWiki + Markdown => LaTeX converter....in something like Python.

There are actually a bunch of Markdown parsers that create a DOM, or at least a proper tree (e.g markdown2 in PyPI). Markdown is outrageously simple compared to MediaWiki.

I actually wrote a subset-of-MediaWiki parser in C#, which stayed at the "subset" stage because of the ridiculous complexity and corner-cases that crop up even very early.

Post reply on HN