41–50 of 59 posts
Earlier quoted context omitted.
Good (Edit) point! It's good that the web accepts dead links by design, we can't expect perfection from our distributed information, but it seems that the bitrot of information is too high compared to the information storage technologies available. 2 spinning rust drive can store the library of congress. ~ 2,000 drives would store the web (1). How many millions of these drives get manufactured per year? Our technolog…
2000 drives x $100 = $200,000. Double that for backup, $400,000. Admin, maintenance, let's say total, $1M/year. So, Wikipedia could end its own deadlink problem (IF the reference sources would agree. But stuff goes missing at Wayback because people don't agree to their pages being backed-up. Copyright, whatever. So it's like Global Heating, the tech is there, but people just can't agree. So 'pirate' backer-uppers go…
Internet Archive Scholar - https://news.ycombinator.com/item?id=26419782 - March 2021 (3 comments)
Internet Archive Scholar: Search Millions of Research Papers - https://news.ycombinator.com/item?id=26401568 - March 2021 (47 comments)
Earlier quoted context omitted.
Some people use this tool (of mine) for saving web content from either bookmarks or just everything you browse: https://github.com/crisdosyago/Diskernet There's also plent¥ of other similar tools: - https://github.com/ArchiveBox/ArchiveBox - https://github.com/gildas-lormeau/SingleFile
Beware the very strange and bad license for Diskernet, which is "Polyform Strict License 1.0.0"
After a little testing, this looks like a good information source, although the combination of Google Scholar and sci-hub is probably still the best option, i.e. I couldn't find anything with Internet Scholar that wasn't available with the other options, and the quality of results on searchs is somewhat higher with Google Scholar (this may be because Google Scholar utilizes citation count as a search parameter, which…
Indeed scholar.archive.org does not currently use citation count in search rankings. We have a decent citation graph, which we are working to expose in scholar (it is visible in fatcat.wiki today). Would probably only ever use citation count as a weak boost in search rankings (eg, "any citations at all", "more than 25 citations" as boosts, nothing beyond that), don't want to create too strong a feedback loop influencing future citations.
scholar.archive.org specifically was partially funded by the Mellon Foundation (and partially through donations and other service revenue). IA overall has diverse funding, including grants and service revenue from the USA (Library of Congress, IMLS, etc); other national governments (paid crawl services); foundation grants; universities and libraries (crawl, preservation, and digitization services); and of course general donations. The last category of course has the fewest strings and lets us pursue new projects which might be hard to get traditional funding for. Remember that the whole premise of web archiving was considered radical and quixotic at the beginning!
(source: I work at IA on scholar)
Earlier quoted context omitted.
Beware the very strange and bad license for Diskernet, which is "Polyform Strict License 1.0.0"
That Polyform licence looks truly awful. Lack of mention of how I could use it while working.
The readme clearly links where to buy a license for that.
A huge percentage of the operating budget is from small donors. The funding is preposterously small compared to other public-service public interest such as Wikipedia.
A lot of us take it for granted and assume there is e.g. support from FAANG companies proportionate to the degree they lean on it.
This is 100% NOT THE CASE.
Please advocate for recurring institutional donations from your firm. The audience reading this has a lot of voice in a lot of organizations who could without a though sign up to make annual 10K, 100K, 1M donations...
...and essentially, none do.
Please help change that!!!
Some days, nearly half the links I click are dead so I've found myself relying on the waybackmachine more and more over the past few months. It's really shocking just how fast digital obsolescence reared its ugly head. Of course angelcites etc. were a clear early blow, but nowadays... I've started saving the html (including the css seems like too much overhead, and often it's incomplete or relies on downloads still -…
Some people use this tool (of mine) for saving web content from either bookmarks or just everything you browse: https://github.com/crisdosyago/Diskernet There's also plent¥ of other similar tools: - https://github.com/ArchiveBox/ArchiveBox - https://github.com/gildas-lormeau/SingleFile
I wonder if this includes anything from Sci-Hub or are they unrelated.
PSA the Internet Archive is a 501c3 non-profit (library!) and survives on donations and grants. A huge percentage of the operating budget is from small donors. The funding is preposterously small compared to other public-service public interest such as Wikipedia. A lot of us take it for granted and assume there is e.g. support from FAANG companies proportionate to the degree they lean on it. This is 100% NOT THE CASE…