Live data from Hacker News

In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

online.wsj.com

61–70 of 78 posts

Re: In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

#61
post #31

Earlier quoted context omitted.

> The only people who would need this information are botanists I think that looking up plant species are exactly the kind of thing people would want to use an encyclopedia for.

These are (almost) one-sentence articles. They're not appropriate for an encyclopaedia. If anything, they should be in a 'list' article format. If you want these kind of one-sentence descriptors, they would be better served in a specialist publication.

Why isn't it appropriate for an encyclopaedia?

For example, the 1910 Encyclopaedia Britannica's complete entry for "denim" is "(an abbreviation of the serge de Nimes), the name originally given to a kind of serge. It is now applies to a stout twilled cloth made in various colours, usually of cotton, and used for overalls, &c."

The entry for "Gimli" is "In Scandinavian mythology, the great hall of heaven whither the righteous will go to spend eternity."

It's not hard to find more examples. But I don't think people considered the EB less of an encyclopaedia for its use of one-sentence descriptors.

The heart of the matter is that there's precious little difference between a dictionary and an encyclopedia. Indeed, the EB's full name is "The Encyclopædia Britannica: a Dictionary of Arts, Sciences, Literature and General Information"

To double check that it's not limited to the EB, I looked in Harmsworth's Universal encyclopedia. The entry for "fulcrum" is "(Lat. fulcrum, a prop) Fixed point in the mechanical system of a lever about which the lever can rotate. See Lever." The entry for "gumboil" is "Small abscess on the gum arising in most cases from decay at the root of a tooth."

See http://menvall.wordpress.com/2010/09/14/on-wikipedias-attemp... for an analysis of the distinction between the two, and the conclusion that "everything that is included in a dictionary also can be included in an encyclopedia, whereas all that is included in an encyclopedia either can or can’t be included in a dictionary. This relation is, however, completely misunderstood by some editors of Wikipedia."

(Had you written that it wasn't appropriate for Wikipedia, than that's a different issue. I speak now only of the broad category of "encyclopedia".)

Triple-checking, the entry for "gumboil" in Wikipedia (at http://en.wikipedia.org/wiki/Gumboil) redirects to "Intraoral dental sinus". The complete entry is two sentences long:

> Intraoral dental sinus (also termed a parulis and commonly, a gumboil) is an oral lesion characterized by a soft erythematous papule (red spot) that develops on the alveolar process in association with a non-vital tooth and accompanying dental abscess.[1] A parulis is made up of inflamed granulation tissue.

By your definition, this "(almost) one-sentence" article should be removed from WP, no?

Re: In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

#62

I think this is a good thing, but ideally no bot would be required: if no article in your language, show all the facts from wikidata, eg http://tools.wmflabs.org/reasonator/?q=Q1339

There's a proposal to create 'virtual' articles from Wikidata: http://meta.wikimedia.org/wiki/Wikidata/Notes/Article_genera...

The benefit of this is that articles would be dynamically synthesised from the latest data when the user requests them, and not actually created and added to the wiki. This would prevent the creation of a potentially infinite number of articles on subjects not significant enough to merit a write-up by a human author, so could be a way to combat the 'bot inflation' of article numbers.

I don't think anyone's started work on it yet, but if someone fancies it...

Re: In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

#63
post #34
post #29

Earlier quoted context omitted.

At least part of the problem is that he's generating what one might call 'info trash': he's taking highly structured information from databases, and turning it into natural-language prose, a data source of less value since it's less structured. These prose versions are now going to steadily fall out of sync with the original databases, be much more prominent in Wikipedia and Google, diverge from each other, be harder…

Sometimes I appreciate what you call "info trash." For example, I assume there is a bot that turns census data into articles for every incorporated community in the US, like this: http://en.wikipedia.org/wiki/Agency,_Missouri . I still think the article is useful as is, with just the map, data sheet, and demographics, and of course many incorporations have additional human-composed information added. I could imagine…

Looking at the history of that page,[0] it appears a couple of different bots have worked on it, with human intervention. (I suspect ``Ram-Man'' is an earlier version of ``Rambot'', but I could be wrong.)

I've read pages like that before, and it never once occurred to me that they were anything other than the result of sheer human bloody-mindedness. They're not `exciting', but they're very clearly written in an easily parseable way that doesn't scream ``machine-generated'' to me. If this is indicative, the quality of output of these bots is excellent, and a good use of automation --- let the bots fill out the dry factual stuff, and the humans write the less tangible, non-statistical stuff.

[0]: http://en.wikipedia.org/w/index.php?title=Agency,_Missouri&a...

Re: In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

#64

Why is he doing this in Swedish and two versions of Filipino? He apparently speaks English, so I assume he could adapt his code to do English entries as well.

The English wikipedia is less desperate for article counts because it's already got a huge number. Wherease the other language wikipedias are always looking for more content in their specific language. So, in essence, you could say that he'll get more issues by doing this on the English wikipedia.

Re: In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

#65
post #61
post #31

Earlier quoted context omitted.

These are (almost) one-sentence articles. They're not appropriate for an encyclopaedia. If anything, they should be in a 'list' article format. If you want these kind of one-sentence descriptors, they would be better served in a specialist publication.

Why isn't it appropriate for an encyclopaedia? For example, the 1910 Encyclopaedia Britannica's complete entry for "denim" is "(an abbreviation of the serge de Nimes ), the name originally given to a kind of serge. It is now applies to a stout twilled cloth made in various colours, usually of cotton, and used for overalls, &c." The entry for "Gimli" is "In Scandinavian mythology, the great hall of heaven whither the…

Firstly, just because you can find brief entries in other encyclopaedias doesn't mean that it's good form.

Secondly, I didn't demand complete excision with the kind of frothy fervor you're implying. I said a better format for these "X is type Y, discovered by Z, listed in Q" is collating them all in a list format. WP has list article like this aplenty - dense, easily digestible information on similar topics, allowing quick and easy comparison and scanning.

As for your link, one of the bold highlights is "explains subjects in greater detail than a dictionary". Another of the three definitions of 'encyclopaedia' your link provides says "with data on and discussion of each subject identified" (my emphasis). So that's two out of three definitions that quite strongly indicate non-brief articles - your linked article is wrong from it's own source material, and hasn't made the case that dictionary-like brevity is suitable for an encyclopaedia.

By your definition, this "(almost) one-sentence" article should be removed from WP, no?

What, are you trying to 'catch me out' here? Do you think that's a good quality article? It's a stub, it's not what WP wants to encourage, and it's more like a dictionary definition than either "explaining a subject in greater detail" or "discussion of the subject". Yes, I think it's a bad article for any encyclopaedia - it's quite brief, and full of technical jargon. If you didn't already know the specific jargon, it's completely useless as a "general course of instruction" (the etymology argument from your link). And if you do know the jargon, you have a pretty good chance of working it out from the name alone; the article merely confirms the topic if you're unsure, but you don't get any more insight into it.

As 'trick questions' go, this one sucked.

Re: In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

#66
post #34

Earlier quoted context omitted.

Sometimes I appreciate what you call "info trash." For example, I assume there is a bot that turns census data into articles for every incorporated community in the US, like this: http://en.wikipedia.org/wiki/Agency,_Missouri . I still think the article is useful as is, with just the map, data sheet, and demographics, and of course many incorporations have additional human-composed information added. I could imagine…

Looking at the history of that page,[0] it appears a couple of different bots have worked on it, with human intervention. (I suspect ``Ram-Man'' is an earlier version of ``Rambot'', but I could be wrong.) I've read pages like that before, and it never once occurred to me that they were anything other than the result of sheer human bloody-mindedness. They're not `exciting', but they're very clearly written in an easil…

Ram-Man is a human account. The same person operates the Rambot bot account. You can click on their usernames on the history page to see their user pages, which usually describe these things.

Re: In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

#67
post #65
post #61

Earlier quoted context omitted.

Why isn't it appropriate for an encyclopaedia? For example, the 1910 Encyclopaedia Britannica's complete entry for "denim" is "(an abbreviation of the serge de Nimes ), the name originally given to a kind of serge. It is now applies to a stout twilled cloth made in various colours, usually of cotton, and used for overalls, &c." The entry for "Gimli" is "In Scandinavian mythology, the great hall of heaven whither the…

Firstly, just because you can find brief entries in other encyclopaedias doesn't mean that it's good form. Secondly, I didn't demand complete excision with the kind of frothy fervor you're implying. I said a better format for these "X is type Y, discovered by Z, listed in Q" is collating them all in a list format. WP has list article like this aplenty - dense, easily digestible information on similar topics, allowing…

Trick question? I'm showing that my question - "Why isn't it appropriate for an encyclopaedia?" - is meaningful, by giving counter-examples from three encyclopedias. This suggests that your definition is not aligned with how the term is used in practice.

I ask that you clarify your reasoning.

You say my linked-to reference "hasn't made the case that dictionary-like brevity is suitable for an encyclopaedia". The link isn't trying to make that delineation between the two. It's arguing (and I agree) that a dictionary is a type of encyclopedia, not that they are two different things. You mentioned some quotes, in bold. The author later comments on those exact same quotes (with bold translated to italics):

> These definitions show that whereas dictionary is defined by words alone: “reference work that lists words, usually in alphabetical order, and gives their meanings and often other information such as pronunciations, etymologies, and variant spellings“, encyclopedia is defined either as synonymous to dictionary: “the term is often interchanged with the word “dictionary,” as in the present work” or by a larger extension than dictionary: “explains subjects in greater detail than a dictionary”. There is thus no conflict between dictionary and encyclopedia. They are either synonymous or only have different extensions (i.e., encyclopedia including dictionary, but covering a larger set of phenomena).

I checked with the OED, at http://www.oed.com/viewdictionaryentry/Entry/52325 . It concurs, since its definition 1b. for dictionary is (italics mine):

> In extended use: a book of information or reference on any subject in which the entries are arranged alphabetically; an alphabetical encyclopedia

Yes, I'm saying that the article for "gumboil" in WP is not a stub, does not need to be longer than it is, and very much like what WP should support. While I agree with you in that the older print definition of the term is easier to understand than what WP has, that's at most one more line, and more likely solved by rewriting.

BTW, I also looked up Gimlé in WP. That's three sentences long, so a full two sentences longer than the 'Gimli' entry in Encyclopaedia Britannica.

Why must everything require more than a few lines to fit into your concept of an encyclopedia? Certainly Gimlé doesn't fit in a dictionary, so where else would it go?

Re: In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

#68
post #45
post #42

Earlier quoted context omitted.

It does apply, regardless of the source of the article. Wikipedia doesn't like stub articles.

Most articles started as stubs. At the moment many language versions of Wikipedia have different opinions about stubs. e.g. german wikipedia don't allow stubs anymore and the admins delete & reverts more pages every day than new ones are created. It's maybe a cultural problem as such admins identify themself with 'their articles' and don't allow any changes.

The question of course is in how to determine when a short article is a stub. Some short entries are sufficiently complete for the purposes of WP, whose English guidelines say "there are some subjects about which very little can be written."

Consider http://en.wikipedia.org/wiki/Muati

> Muati is an obscure local god in the Sumerian pantheon. He is associated in some texts with the mythical island paradise of Dilmun, and becomes syncretised with Nabu.

That's unlikely to get much longer. For one, the "Dictionary of the Old Testament: Wisdom, Poetry & Writings ..." says "Muati, a god about whom we know very little."

Re: In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

#69
post #49
post #17

Earlier quoted context omitted.

> "and not really something you couldn't find in any number of other resources." The same should be able to be said about everything on Wikipedia, since Wikipedia is not supposed to have original research and should have a source for everything. The "Why?" would then be "Because it is better to search for [obscure butterfly] and find a short list of fact than to search for that butterfly and find nothing at all."

> The same should be able to be said about everything on Wikipedia I don't think so. Well-written encyclopedic entries are in far shorter supply than bare lists of facts. I guess the fundamental difference of opinion is between those who feel Wikipedia is an encyclopedia, and those who feel it's a dumping ground for human knowledge. Note that I'm not taking sides, just trying to explain the root causes for the differ…

Bots could be concatenating two or threes different sets to create one stub per butterfly.

Or bots could be taking something in a weird set of scans, OCRing that, and then putting it in a stub. This would be troubling unless there was a human checking the quality of the OCR.

There is plenty of stuff that is public domain and not online in a useful form.

Re: In Sweden, Sverker Johansson and His 'Bot' Have Created 2.7M Wikipedia Articles

#70

Why is he doing this in Swedish and two versions of Filipino? He apparently speaks English, so I assume he could adapt his code to do English entries as well.

Ebglish WP has more people a ailable to give him a hard time for what he's doing.

Bots sometimes have trouble on EN.Wikipedia - see for example the megabytes of discussion generated by the fair-use image bot which was following existing WP rules and helping WP follow the law.

Post reply on HN