Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to circumvent Sci-Hub ISP block

fragile-credences.github.io

Post reply on HN
Print view View original

71–80 of 196 posts

Re: How to circumvent Sci-Hub ISP block

#71

Post by dredmorbius »

jandrese wrote: post #48
posterboy wrote: post #36

Earlier quoted context omitted.

a plain old website or a publishing house with distribution services and syndication attached, but for a sane price. "a whole lot of tiny files" severely underestimates the scale at work. Libgen's coverage is relatively shallow, and pdf books tend to be huge, at least for older material. Scihub piggy backs on the publishers, so that's your reference. syndication , syndicate , quite apt don't you think? Libraries that…

In Internet scale it's not a lot of data. Most people who think they have big data don't. Estimates I've seen put the total Scihub cache at 85 million articles totaling 77TB. That's a single 2U server with room to spare. The hardest part is indexing and search, but it's a pretty small search space by Internet standards.

The entire Library of Congress books collection is on the order of 40 million items.

At 5 MB per book, this works out to about 200 TB of disk storage.

At about $12/TB, hosting the entire LoC collection would cost roughly $2,400 presently, with prices halving about every three years.

Re: How to circumvent Sci-Hub ISP block

#72

Post by dredmorbius »

HWR_14 wrote: post #65
jandrese wrote: post #48

Earlier quoted context omitted.

In Internet scale it's not a lot of data. Most people who think they have big data don't. Estimates I've seen put the total Scihub cache at 85 million articles totaling 77TB. That's a single 2U server with room to spare. The hardest part is indexing and search, but it's a pretty small search space by Internet standards.

It still amazes me that 77TB is considered "small". Isn't that still in the $500-$1,000 range of non-redundant storage? Or if hosted on AWS, isn't that almost $1,900 a month if no one accesses it? I know it's not Big Data(tm) big data, but it is a lot of data for something that can generate no revenue.

For an institution, it's a rounding error.

AWS is not the cheapest bulk-storage hosting possible.

Re: How to circumvent Sci-Hub ISP block

#73

Post by jagged-chisel »

o8r3oFTZPE wrote: post #67
contravariant wrote: post #31

Earlier quoted context omitted.

There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way. If you can fit PDFs somewhere into that you'd be golden. Or just use IPFS I suppose.

"There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way." https://github.com/bittorrent/sqltorrent

this is the one: https://phiresky.github.io/blog/2021/hosting-sqlite-database...

HN submission: https://news.ycombinator.com/item?id=27016630

Re: How to circumvent Sci-Hub ISP block

#74

Post by andyxor »

jandrese wrote: post #48
posterboy wrote: post #36

Earlier quoted context omitted.

a plain old website or a publishing house with distribution services and syndication attached, but for a sane price. "a whole lot of tiny files" severely underestimates the scale at work. Libgen's coverage is relatively shallow, and pdf books tend to be huge, at least for older material. Scihub piggy backs on the publishers, so that's your reference. syndication , syndicate , quite apt don't you think? Libraries that…

In Internet scale it's not a lot of data. Most people who think they have big data don't. Estimates I've seen put the total Scihub cache at 85 million articles totaling 77TB. That's a single 2U server with room to spare. The hardest part is indexing and search, but it's a pretty small search space by Internet standards.

The entire archive actually fits in a small desktop NAS (e.g. QNAP or Synology) with a few 14-18TB drives, you don't even need a server rack.

There is existing index in sql format distributed by libgen: https://www.reddit.com/r/scihub/comments/nh5dbu/a_brief_intr..., it is around 30GB uncompressed.

Those 851 torrents uncompressed would probably take half a petabyte of storage, but I guess for serving pdfs you could extract individual files on demand from zip archive and (optionally) cache them. So the scihub "mirror" could run on a workstation or even laptop with 32-64GB memory connected to 100TB NAS over 1GBE, serving pdfs over VPN and using unlimited traffic plan. The whole setup including workstation, NAS and drives would cost $5-7K.

it's not a very difficult project and can be done DIY style, if you exclude the proxy part (which downloads papers using donated credentials). Of course it would still be as risky as running Scihub itself which has $15M lawsuit pending against it.

Re: How to circumvent Sci-Hub ISP block

#75

Post by lowkey »

Azrael3000 wrote: post #32
tmkadamcz wrote: post #10

By the way, Sci-Hub has stopped adding new articles to the database for a few months now (background: https://www.reddit.com/r/scihub/comments/mk46x4/scihub_v_els... ). It would be great to develop a truly decentralised solution. Having a database of individual torrent links for each paper might be a start.

Thanks for the background link. I did not know about that and it's a good incentive to donate them some money for the legal battle. TL;DR of the link: No more uploads to support a court case in India which SciHub might win and thus establish a legal basis for operation in the biggest democracy.

Sci-hub is such a great example of a clear and compelling use-case for Bitcoin. Bitcoin is censorship-resistant money that doesn't rely on countries, laws, central bankers or politicians. The US dollar cannot be used for purposes not aligned with the US government. Sometimes ideas that the US Government doesn't agree with can be useful (e.g. Wikileaks, Sci-hub.)

When I hear complaints that Bitcoin has no use except for speculation, I think of Sci-hub, Wikileaks and other organizations that may be bad for the interests of the US government but may be good for mankind.

Re: How to circumvent Sci-Hub ISP block

#76

Post by dredmorbius »

A few years back, frustrated with increasaed DNS blocking of Sci-Hub, I wrote a quick DNSMasq hack (haq?) to return Sci-Hub IPs for any "sci-hub." possible. The shins-n-grits factor of surfing "scihub.elsevier.com" were palpable.

https://old.reddit.com/r/Scholar/comments/7m3uin/meta_if_you...

As others have mentioned, Sci-Hub also maintains a Tor presence, and you can access the Onion link using the Tor browser (provided you can install that on your desktop or device).

https://scihub22266oqcxt.onion

Re: How to circumvent Sci-Hub ISP block

#77

Post by pbhjpbhj »

HWR_14 wrote: post #65
jandrese wrote: post #48

Earlier quoted context omitted.

In Internet scale it's not a lot of data. Most people who think they have big data don't. Estimates I've seen put the total Scihub cache at 85 million articles totaling 77TB. That's a single 2U server with room to spare. The hardest part is indexing and search, but it's a pretty small search space by Internet standards.

It still amazes me that 77TB is considered "small". Isn't that still in the $500-$1,000 range of non-redundant storage? Or if hosted on AWS, isn't that almost $1,900 a month if no one accesses it? I know it's not Big Data(tm) big data, but it is a lot of data for something that can generate no revenue.

I'm prepared to accept "does generate no revenue" but "can generate no revenue" ...?

Perhaps some sort of MTurk or captcha-like tasks per access? Patr[e]ons? Donation drives? Micro-payments? Something else??

Re: How to circumvent Sci-Hub ISP block

#78

Post by tmkadamcz »

jandrese wrote: post #26
tmkadamcz wrote: post #10

By the way, Sci-Hub has stopped adding new articles to the database for a few months now (background: https://www.reddit.com/r/scihub/comments/mk46x4/scihub_v_els... ). It would be great to develop a truly decentralised solution. Having a database of individual torrent links for each paper might be a start.

Millions of individual torrents is not a great solution. Keeping them all seeded is basically impossible unless they run a seed for each one, at which point they might as well just host the files. Plus you'll never get the economy of scale that makes BitTorrent really shine. When you have a whole lot of tiny files that people will generally only want one or two of there isn't much better than a plain old website. A t…

There are already torrents of the archives. But supposing scihub was taken down it's pretty non trivial to get from the archive back to a working site with search functionality. For one thing, none of Sci-Hub's code is available.

Re: How to circumvent Sci-Hub ISP block

#79

Post by divbzero »

Mediterraneo10 wrote: post #63
tomrod wrote: post #57

Earlier quoted context omitted.

Yes, due to allocation of property rights. Cease to supply this system with the fruits of your research labors.

> Cease to supply this system with the fruits of your research labors. Historically academics have felt forced to support this system, because for-profit journals are the high-prestige ones they must publish in in order to get tenure. This has changed for certain fields, but it isn’t as simple as just suggesting that one publish elsewhere.

It’s up to not only academics who publish articles, but also organizations that issue grants and tenure. Public policies to adjust their definitions of “prestige” or “quality” would help.

Re: How to circumvent Sci-Hub ISP block

#80

Post by smichel17 »

HWR_14 wrote: post #65
jandrese wrote: post #48

Earlier quoted context omitted.

In Internet scale it's not a lot of data. Most people who think they have big data don't. Estimates I've seen put the total Scihub cache at 85 million articles totaling 77TB. That's a single 2U server with room to spare. The hardest part is indexing and search, but it's a pretty small search space by Internet standards.

It still amazes me that 77TB is considered "small". Isn't that still in the $500-$1,000 range of non-redundant storage? Or if hosted on AWS, isn't that almost $1,900 a month if no one accesses it? I know it's not Big Data(tm) big data, but it is a lot of data for something that can generate no revenue.

> Isn't that still in the $500-$1,000 range of non-redundant storage?

Sure. Let's add redundancy and bump by an order of magnitude to give some headroom -- $5-10k is a totally reasonable amount to fundraise for this sort of application. If it were legal, I'm sure any number of universities would happily shoulder that cost. It's miniscule compared to what they're paying Elsevier each year.

Post reply on HN