41–50 of 200 posts
On the other hand, with Wikipedia, you can just read and search everything.
(reason: trying to cross-reference my tons of downloaded games my HDD - for which i only have titles as i never bothered to do any further categorization over the years aside than the place i got them from - with wikipedia articles - assuming they have one - to organize them in genres, some info, etc and after some experimentation it turns out an LLM - specifically a quantized Mistral Small 3.2 - can make some sense of the chaos while being fast enough to run from scripts via a custom llama.cpp program)
Wouldn’t Wikipedia compress a lot more than llms? Are these uncompressed sizes?
Yes, they're uncompressed. For reference, `enwiki-20250620-pages-articles-multistream.xml.bz2` is 25,176,364,573 bytes; you could get that lower with better compression. You can do partial reads from multistream bz2, though, which is handy.
And 57 GB to 25 GB would be pretty bad compression. You can expect a compression ratio of at least 3 on natural English text.
Earlier quoted context omitted.
FYI: non-Western countries exist.
People who are from those countries that can nag on HN and know whant HN is are most likely still better off than most of their fellow countrymen.
Earlier quoted context omitted.
FYI: non-Western countries exist.
Eh, even just “countries that are not the US” would be a correct statement. US tech salaries are just in an entire different ballpark to what most companies outside the US can offer. I’m in Canada, I make good money (as far as Canadian salaries go), but nowhere near “buy an expensive laptop whenever” money.
Try telling a plumber that $2,000 for a laptop is a financial burden for a software engineer.
Earlier quoted context omitted.
Eh, even just “countries that are not the US” would be a correct statement. US tech salaries are just in an entire different ballpark to what most companies outside the US can offer. I’m in Canada, I make good money (as far as Canadian salaries go), but nowhere near “buy an expensive laptop whenever” money.
It's not uncommon for professionals to spend many thousands of dollars on the tools and equipment they need for their trade. Try telling a plumber that $2,000 for a laptop is a financial burden for a software engineer.
Earlier quoted context omitted.
Imagine taking the whole Web, removing spam, duplicates, bad explanations It will be the free new Wikipedia+ to learn anything in the best way possible, with the best graphs, interactive widgets, etc What LLMs have for free but humans for some reason don’t In some places it is possible to use copyrighted materials to educate if not directly for profit
Love it when Silicon Valley reinvents encyclopedias
One important distinction is that the strength of LLMs isn't just in storing or retrieving knowledge like Wikipedia, it’s in comprehension. LLMs will return faulty or imprecise information at times, but what they can do is understand vague or poorly formed questions and help guide a user toward an answer. They can explain complex ideas in simpler terms, adapt responses based on the user's level of understanding, and…
I find LLMs with the search functionality to be weak because they blab on too much when they should be giving me more outgoing links I can use to find more information.
There are 341 languages in there and 205GB of data, with English alone making up 24GB! My perspective on Simple English Wikipedia (from the OP), it's decent but the content tends to be shallow and imprecise.
0: https://omarkama.li/blog/wikipedia-monthly-fresh-clean-dumps...
One important distinction is that the strength of LLMs isn't just in storing or retrieving knowledge like Wikipedia, it’s in comprehension. LLMs will return faulty or imprecise information at times, but what they can do is understand vague or poorly formed questions and help guide a user toward an answer. They can explain complex ideas in simpler terms, adapt responses based on the user's level of understanding, and…
In a 'rebooting society' doomsday scenario you're assuming that our language and understanding would persist. An LLM would essentially be a blackbox that you cannot understand or decipher, and would be doubly prone to hallucinations and issues when interacting with it using a language it was not trained on. Wikipedia is something you could gradually untangle, especially if the downloaded version also contained associ…
fun to imagine whether images help in this scenario