Live data from Hacker News

Newsgroups and the Internet Archive: I Made a Difference

screengod.blogspot.com

31–40 of 65 posts

Re: Newsgroups and the Internet Archive: I Made a Difference

#31
post #26

> After a month I had most newsgroups - excepting binaries - and it came to 800GB. > Trying to index THAT lot was impossible Stupid question, but why would indexing 800GB of newsgroup postings be impossible?

Clucene was way too slow for body text, more than 1GB. I had my own header parser in C++ (though you can do that in Python easily).

I'm trying again on that 800GB with KISS DB (append-only hashtable), and Elasticsearch. Doesn't matter if GPL because it's a website.

Re: Newsgroups and the Internet Archive: I Made a Difference

#33
post #9
post #5

Earlier quoted context omitted.

Google Groups was great when they first acquired all those archives! It just got progressively worse and worse over time, and I've never understood why. (Stagnation I could understand, but how did the same searches get less effective?)

I get that Google essentially walked back from an at least implied role as an information archivist. But I still don't really get why they so completely abandoned things like Google Groups given the truly minuscule resources to maintain them at some low level.

Somebody (and most likely a team of somebodies) needs to own it. If nobody at the top is willing to champion the project and carve out a budget and people, then they need to fold it.

Re: Newsgroups and the Internet Archive: I Made a Difference

#34
post #20

Earlier quoted context omitted.

What seems to have happened was that Google shifted focus from sorting the web to delivering the "zeitgeist" of the present. Perhaps because more people were likely to search for whatever was on the news cycle than some arbitrary piece of info. Never mind that Google's earnings comes from ads, not search results.

Google's original mission statement from 1998 was to “organize the world’s information and make it universally accessible and useful.” Things like Usenet archives are clearly part of that. That is still apparently their mission statement but I agree the reality seems more along the lines of "deliver the best search results for users' immediate needs" [while making as much money from that as humanly possible]

It's sad. The original mission implied nothing was going to get lost and abandoned. So much for that.

Re: Newsgroups and the Internet Archive: I Made a Difference

#35
post #16

Earlier quoted context omitted.

”You searched for Stockholm Syndrome Research. Here’s a bunch of Stockholm travel blogs! They’re just missing two of your keywords, but they’re really popular with other visitors”

And the people who simply accept this have...

... gone to Sweden on vacation.

More seriously, I think a better analogy is putting a frog in cold water and turning on the stove's burner. Things are much worse but because it happened so slowly, we simply have a "new normal". The change from one day to the next on this time-scale is undiscernable.

Re: Newsgroups and the Internet Archive: I Made a Difference

#37

Usenet article periodically get promoted to the front page. I wonder what that says about the age demographics of HN, given that Usenet hasn't been significant in maybe 20 years (and even then it was a niche). Is the younger generation here? And if not here, where?

I'm kind of offended that you seem to think young people can't be aware of things that happened before their time.

EDIT: To elaborate a bit, my expectation would be that the sort of young person using HN leans more historically interested than normal. Is more likely to appreciate the value of things like Internet archive, etc. Usenet is a huge part of digital history, and the fact that it's not available even though archives were kept is something of a tragedy.

Re: Newsgroups and the Internet Archive: I Made a Difference

#38
post #35

Earlier quoted context omitted.

And the people who simply accept this have...

... gone to Sweden on vacation. More seriously, I think a better analogy is putting a frog in cold water and turning on the stove's burner. Things are much worse but because it happened so slowly, we simply have a "new normal". The change from one day to the next on this time-scale is undiscernable.

If you put a frog in cold water and gradually heat the water, the frog will jump out (https://en.wikipedia.org/wiki/Boiling_frog). The premise of that analogy is widely considered to be completely false.

Re: Newsgroups and the Internet Archive: I Made a Difference

#39

Usenet article periodically get promoted to the front page. I wonder what that says about the age demographics of HN, given that Usenet hasn't been significant in maybe 20 years (and even then it was a niche). Is the younger generation here? And if not here, where?

I'm kind of offended that you seem to think young people can't be aware of things that happened before their time. EDIT: To elaborate a bit, my expectation would be that the sort of young person using HN leans more historically interested than normal. Is more likely to appreciate the value of things like Internet archive, etc. Usenet is a huge part of digital history, and the fact that it's not available even though…

I'm offended that you're so easily offended

Re: Newsgroups and the Internet Archive: I Made a Difference

#40
post #23
post #16

Earlier quoted context omitted.

”You searched for Stockholm Syndrome Research. Here’s a bunch of Stockholm travel blogs! They’re just missing two of your keywords, but they’re really popular with other visitors”

I think is possible to have the verbatim option active all the time, but I don't remember how.

Disclaimer: I have not tested these add-ons myself.

Firefox: https://addons.mozilla.org/en-US/firefox/addon/google-1/

Chrome: https://chrome.google.com/webstore/detail/verbatim-search/oc...

Post reply on HN