Live data from Hacker News

Wikipedia: Database Download

en.wikipedia.org

81–90 of 106 posts

Re: Wikipedia: Database Download

#81
post #73
post #37

Earlier quoted context omitted.

Someone suggested to me to apply a filter that serves .md or txt to bots/ai scrapers instead of the regular website, seems smart if it works but i hate it when i get captchas and this could end up similarly detecting non-bots as bots maybe a view full website link loaded on js so bots dont see it idk

I would love to see most sites serve me markdown. I'd happily install a browser extension to mask me as a a AI bot scraper if it means I can just get the text without all the noise.

lol

me too tbh

someone pointed out you can enable by default reader mode on safar under settings but even then not all website’s pages are seeved as reader mode enabled pages

Re: Wikipedia: Database Download

#82
post #73
post #37

Earlier quoted context omitted.

Someone suggested to me to apply a filter that serves .md or txt to bots/ai scrapers instead of the regular website, seems smart if it works but i hate it when i get captchas and this could end up similarly detecting non-bots as bots maybe a view full website link loaded on js so bots dont see it idk

I would love to see most sites serve me markdown. I'd happily install a browser extension to mask me as a a AI bot scraper if it means I can just get the text without all the noise.

someone built a service for ai bots called pure.md its been a godsend to curl websites as markdown on the occasional where it doesnt work first time and works great for occasional use with the free tier

Re: Wikipedia: Database Download

#83

I wonder how easy it would be to make a practically indestructible, everlasting Wikipedia reader. Something using solar for power, with a rugged and water-resistant enclosure, made of extremely high-quality components that won't break for hundreds of years at least. Maybe add an IRDA port for good measure, to make it possible to transfer all the data out somewhat quickly. You could make hundreds of these and put them…

You could even make it radiation tolerant by printing it.

Then figure out how and where to store the over one thousand volumes that have 1200 pages each.

Re: Wikipedia: Database Download

#84

I wonder how easy it would be to make a practically indestructible, everlasting Wikipedia reader. Something using solar for power, with a rugged and water-resistant enclosure, made of extremely high-quality components that won't break for hundreds of years at least. Maybe add an IRDA port for good measure, to make it possible to transfer all the data out somewhat quickly. You could make hundreds of these and put them…

You got me imagining a project where the entire wiki db gets laser-etched on thin stone tablets/metal plates

microfische is probably the best option here.

Re: Wikipedia: Database Download

#85
post #83

Earlier quoted context omitted.

You could even make it radiation tolerant by printing it.

Then figure out how and where to store the over one thousand volumes that have 1200 pages each.

From "Plenty of Room at the Bottom"[1]:

  What would happen if I print all this down at the scale we have been
  discussing? How much space would it take?  It would take, of course, the
  area of about a million pinheads ...  All of the information which all of
  mankind has every recorded in books can be carried around in a pamphlet
  in your hand — and not written in code, but a simple reproduction of
  the original pictures, engravings, and everything else on a small scale
  without loss of resolution.
Need a good magnifying glass, though (:

[1] https://web.pa.msu.edu/people/yang/RFeynman_plentySpace.pdf

Re: Wikipedia: Database Download

#86
post #18

Might also be worthwhile to download a pre ~2023 dump, because Low-background steel.

The last kiwix zim dump of English Wikipedia before the release of ChatGPT is from May 2022. The Internet Archive still preserves the torrent[0]. To host this, or browse it locally you can use `kiwix-serve -p 8888 wikipedia_en_all_maxi_2022-05.zim` from kiwix-tools[1].

[0]: https://web.archive.org/web/20221007114937/https://download....>

[1]: https://github.com/kiwix/kiwix-tools>

Re: Wikipedia: Database Download

#87

I wonder how easy it would be to make a practically indestructible, everlasting Wikipedia reader. Something using solar for power, with a rugged and water-resistant enclosure, made of extremely high-quality components that won't break for hundreds of years at least. Maybe add an IRDA port for good measure, to make it possible to transfer all the data out somewhat quickly. You could make hundreds of these and put them…

https://en.wikipedia.org/wiki/WikiReader (remember Openmoko Neo 1973 and FreeRunner?)

Re: Wikipedia: Database Download

#88
post #62

Earlier quoted context omitted.

Will the 2025 zim be available as well?

I am wondering the same thing. I have Jan 2021, Jan 2024... I want to keep a snapshot each year and I wonder why a new one hasn't been generated. I haven't looked for documentation on creating my own zim file.

I looked into it once, I think the script or system that built the larger dumps broke and no one fixed it. I started working on it but other stuff got in the way.

Re: Wikipedia: Database Download

#89

I wonder how easy it would be to make a practically indestructible, everlasting Wikipedia reader. Something using solar for power, with a rugged and water-resistant enclosure, made of extremely high-quality components that won't break for hundreds of years at least. Maybe add an IRDA port for good measure, to make it possible to transfer all the data out somewhat quickly. You could make hundreds of these and put them…

Some thoughts about making it possible for individual humans to access Wikipedia, robustly to calamities that are within the sphere of human agency.

Seems like you would want it to be stored digitally. Ideally, people would have the ability to access it remotely, in case their local copy is somehow corrupted. For that, you would need a physical network by which the data can be transmitted. Economies of scale would seem to suggest that there would be one or a few entities that would “serve” the content to individuals who request it. Of course, you would want those individuals to be able to access this information without having detailed technical knowledge and ability. I guess they would have pre-packaged software “browsers” they could use to access the network.

In order to maintain this arrangement, you would want enough political stability to allow for the physical upkeep of this infrastructure, including human infrastructure (feeding the engineers who make it all possible). In order to make it worthwhile, you would need people who want to access the information too. I suspect political stability, a sufficient abundance of the necessities for human life, and the political will to make sure that everyone’s needs are met so that they can safely be curious about the world would help here too.

All of this requires sources of power. I suspect that a combination of nuclear power, solar/batteries, and geothermal energy would be sufficient and would avoid the problem of running out of fossil fuels at some point in the future. The nice side-effect here of reducing the impact of calamities exacerbated by the greenhouse effect.

For the information to continue being relevant, you would have to update it with new knowledge, and correct inaccuracies. How best to accomplish this? Well, I guess you would need a systematic way to interrogate the causes behind the various effects we observe in the world. I would propose a system where people create hypotheses, and perform experiments that exclude the influence of as many factors as possible external to the phenomenon being studied. People would then share their findings, and I guess would critique each other’s arguments in a sort of “peer review” to try to come to a consensus. You would have to feed and provide for these people at a certain basic level to make sure they are comfortable and safe enough to continue doing this work. I guess you would want to encourage the value systems compatible with this method of interrogating the world.

Just my 2 cents.

Re: Wikipedia: Database Download

#90

Earlier quoted context omitted.

> who needs to download the whole Wikipedia Anyone who wants to have access while off-line, for whatever reason. This can be as simple as saving costs via more complicated as accessing content from regions with spotty and/or expensive connectivity (you're on a ship out of reach of shore-based mobile networks, you do not have access to Starlink or something similar, you're deep in the jungle, deep underground, etc) to…

Genuine question, can you provide multiple explicit examples of such bias? I heard a lot of people railing against bias in Wikipedia, but no one provides any blatant examples of it.

A genuine answer, how about looking up some studies on this subject? Not those done by Wikipedia of course, they claim to be politically neutral after all.

Here's a few, from https://www.allsides.com/blog/wikipedia-biased

Six studies, including two from Harvard researchers, have found a left-wing bias at Wikipedia:

A 2024 analysis [1] by researcher David Rozado that used AllSides Media Bias Ratings [2] found Wikipedia associates right-of-center public figures with more negative sentiment than left-wing figures, and tends to associate left-leaning news organizations with more positive sentiment than right-leaning ones.

A Harvard study [3] found Wikipedia articles are more left-wing than Encyclopedia Britannica.

Another paper [4] from the same Harvard researchers found left-wing editors are more active and partisan on the site.

A 2018 analysis [5] found top-cited news outlets on Wikipedia are mainly left-wing.

Another analysis [6] using AllSides Media Bias Ratings found that pages on American politicians cite mostly left-wing news outlets.

American academics found [7] conservative editors are 6 times more likely to be sanctioned in Wikipedia policy enforcement.

There are far more sources out there.

If I show examples of biased pages - the one on Antifa is a good example - this will just devolve into a quibble about this or that sentence.

[1] https://davidrozado.substack.com/p/is-wikipedia-politically-...

[2] https://www.allsides.com/media-bias/ratings

[3] https://www.semanticscholar.org/paper/Do-Experts-or-Collecti...

[4] https://www.hbs.edu/faculty/Publication%20Files/17-028_e7788...

[5] https://archive.md/v4TFn

[6] https://archive.is/dDr7X

[7] https://thecritic.co.uk/the-left-wing-bias-of-wikipedia/

Post reply on HN