Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to become a pirate archivist

annas-blog.org

Post reply on HN
Print view View original

51–60 of 103 posts

Re: How to become a pirate archivist

#51

Post by genewitch »

MontyCarloHall wrote: post #23
jancsika wrote: post #15

I'm curious how Sci-hub's approach compares to the What.cd/Redacted approach. IIUC Sci-hub has scooped up science docs through a good enough UX that it was able to leverage the goodwill of science folks to upload docs (plus whatever other methods it has used to scoop up docs), and it uses a public blitzkrieg-style distribution mechanism. I.e., I guess if one had a big enough harddrive and a fast enough internet conne…

Sci-Hub was interesting because until 2021, it could automatically add any article not in its database by querying proxy servers set up at universities that actually subscribed to the journals, which would then download the PDF from the journal’s website and forward it to Sci-Hub. (This was the approach most academics took in the pre-Sci-Hub days; they’d email friends at other universities and ask them if they had ac…

ohhhh, so that's why, when my people got their PhD and Master degrees from universities suddenly sci-hub stopped working correctly. I put out the word that it was possible to get any study ever and people asked me this year for a bunch of studies that, prior to this year, should have been the definition of ease.

Months later, I'm still waiting on sci-hub or anyone to get access to the studies.

The real WTF is science publishing. never-mind reproducibility of the studies, just getting the study in the first place is a predicament. at least genesis still works for 85% of requests i get.

Re: How to become a pirate archivist

#52

Post by genewitch »

pdntspa wrote: post #26
MontyCarloHall wrote: post #23

Earlier quoted context omitted.

Sci-Hub was interesting because until 2021, it could automatically add any article not in its database by querying proxy servers set up at universities that actually subscribed to the journals, which would then download the PDF from the journal’s website and forward it to Sci-Hub. (This was the approach most academics took in the pre-Sci-Hub days; they’d email friends at other universities and ask them if they had ac…

The Pirate Bay has never hosted any content; yet many, many cases have been decided against them. Why would that work for sci-hub?

Courts can make dumb decisions, either by bias or bad arguments. Proper lawyer-ing can cover for bad decisions some of the time, and perhaps losing some case and winning others is more important.

realistically everyone wants to know where they stand. There's certain "arranged" understandings when it comes to science (and all academic) publishing, and navigating that to provide a uniform interface to the actual documents is the "hard part" - although realistically, a file is a file, and distribution should be "easy".

recently i needed to check what domains i had purchased through various domain name providers (namecheap, godaddy), and i nearly lost access to my godaddy account because of the TOTP/verification stages. I consider cellphone numbers ephemeral, as i consider all phone numbers. I've been told that a dozen numbers in twice as many years is too many, but i refuse to get led around by the nose by cellular providers. So TOTP only works if i have access to the - few - phone numbers i had access to when i signed up. Several services (including some mentioned) used to have voice TOTP, but no longer do, and use an SMS bridge. Some want me to have an app on a phone *with a(n) (e)SIM*. my email service is attached to a domain name that i very nearly lost access to management for. I was complaining about an old gmail address i had and i decided to look at my overall domain health and found it lacking.

someone who is - forgive this - "balls deep" in a degree just wants whatever they publish to work for their advisors and graders. Obviously research assistants and research students in a masters or doctorate program are held to a slightly higher standard of "publish or perish" because of the way grants are written and assigned. It's all garbage, and has been for at least 15 years.

sorry for my meandering, but i felt these two annoyances are related.

Re: How to become a pirate archivist

#53

Post by rakoo »

userbinator wrote: post #8
pilimi_anna wrote: post #6

Earlier quoted context omitted.

Did not know that detail. Will add to the post, thank you.

And from your addendum: So, use random usernames on the computers you use for this stuff, in case you misconfigure something. ...or a username that is so common as to be meaningless, like "user", "Administrator", or even "root".

You're actually supposed to use a specific username for each process. That's what permissions are for.

Re: How to become a pirate archivist

#54

Post by trevyn »

chhs wrote: post #46
imhoguy wrote: post #36

As an active hoarder I think there is problem with "6. Distribution: Packaging it up in torrents, announcing it somewhere, getting people to spread it.". I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. This way we could make torrents live much longer, ev…

I did work on a proof of concept program to accomplish this for my own content library. It would scan a directory to find files and compare them with locally stored metadata. For v2 torrents this is trivial to do via a "pieces root" lookup, for v1 torrents it involves basically checking that each piece matches, and since pieces may not align with the file then it's not possible to guarantee that it's the same file wi…

>One problem that I've yet to build a solution for is finding the metadata to use for the lookup phase.

I think BEP 51 followed by BEP 9 is all you need.

Re: How to become a pirate archivist

#55

Post by genewitch »

loeg wrote: post #24
jancsika wrote: post #15

I'm curious how Sci-hub's approach compares to the What.cd/Redacted approach. IIUC Sci-hub has scooped up science docs through a good enough UX that it was able to leverage the goodwill of science folks to upload docs (plus whatever other methods it has used to scoop up docs), and it uses a public blitzkrieg-style distribution mechanism. I.e., I guess if one had a big enough harddrive and a fast enough internet conne…

> Yet the tracker was apparently already nuked off the internet as What.cd and reappeared later as Redacted. You imply there is some continuity in the operation between these two trackers, but I don’t believe that’s the case. What.cd shut down. Subsequently, redacted (passtheheadphones) and apollo started, appealing to the same userbase. Neither of those trackers were privy to what.cd’s databases.

in the sets of what.cd and waffles.fm and Redacted, approximate the intersection of the sets.

I like music a lot. I have a lot of vinyl and weird CDs, too. However, i can't be assed to rip to whatever draconian style-guide some of these private trackers want. So it's a matter of being a member of several servers and finding something that either isn't listed or seeded and "filling" or creating a torrent with some other tracker's set of files.

this is rewarding the wrong behavior. If i remember some song i heard in 1996, i should just be able to get it. It would be nice if all of the people who were involved in the creation and publication of the song got rewarded, somehow, but that's just not how art works in capitalism. I say this as someone who has personally released 11 CDs and a further 6 CDs in collaboration, of music. I haven't been paid a penny or more for anything i've ever produced in "art". I don't consider this a downside. People who know me and know i write music appreciate my music. People who don't know me will miss out. That's all there is to it.

Re: How to become a pirate archivist

#56

Post by chhs »

trevyn wrote: post #54
chhs wrote: post #46

Earlier quoted context omitted.

I did work on a proof of concept program to accomplish this for my own content library. It would scan a directory to find files and compare them with locally stored metadata. For v2 torrents this is trivial to do via a "pieces root" lookup, for v1 torrents it involves basically checking that each piece matches, and since pieces may not align with the file then it's not possible to guarantee that it's the same file wi…

>One problem that I've yet to build a solution for is finding the metadata to use for the lookup phase. I think BEP 51 followed by BEP 9 is all you need.

This is how I was originally achieving this. As I said, it's very slow. I don't think it would be a good solution on its own because it would require that every client be constantly sampling all DHT nodes, and downloading all metadata and indexing it for a potential future lookup. It's a huge amount of additional load on the DHT.

I think a better solution would be some way for clients to query the DHT for a specific "pieces root", but I don't know if all clients publishing "pieces root" for the torrents they know about would also be a good idea. Some kind of distributed metadata database where clients can query would be ideal.

Re: How to become a pirate archivist

#57

Post by genewitch »

mdaniel wrote: post #12

I'm not in the pirate archivist space, but sections 3 and 5 are relevant to my interests. I've had great luck with ZAP ( https://github.com/zaproxy/zaproxy#readme ) glued to a copy of Firefox (because it allows monkeying with the _browser_'s proxy without having to alter the system one as other browsers do) for archiving all content seen while surfing around a site. It even achieves the stated goal of preserving the…

This is extremely relevant to my interests. I will try ZAP with a VM that has the explicit purpose of mirroring all content i view "online" within that VM.

there's the web archival projects - that i cannot remember right now - that have some sort of proxy front end, but realistically, it should be possible to record the "content" portion of all web interactions, without relying on such dalliances as OCR and screengrabs or even OBS studio or a screen recorder.

Sometimes i go on a deep dive of some concept, and when i am done i feel i have a decent enough understanding to explain the concept to an adult, and sometimes i do a deep enough dive to explain to a 6 year old. I'd like to archive the entire "session" that got me there. Ideally as plaintext, but never have i wanted video documentation. I only ever use video to prove to someone that their service is acting up, since audio/visual desktop captures can do that, without cheating and provably.

Re: How to become a pirate archivist

#58

Post by genewitch »

est wrote: post #13
ynno wrote: post #3

I think Alexandra Elbakyan actually did not want to be revealed as the librarian behind Sci-Hub, it was her poor opsec that led to her being identified. Basically her servers were set up to emit detailed error messages from PHP, including full path of faulting source file, which was under directory /home/ringo-ring, which could be traced to a username she had online on an unrelated site, attached to her real name. Be…

> which was under directory /home/ringo-ring, which could be traced to a username Ha, my home dir is always called "me" or "and". Try google that.

i like how people are trying to correlate your HN account with - ostensibly - wild /home/ directories. This is why i've recently moved off of this username, going forward. 20 years as genewitch has attached a lot of bad "OPSEC" to this username, and coupled with the fact that i have federal licenses means that people can just google my entire life story.

my last name, without any other information, has one tenth of the bits of information that "genewitch" does.

Re: How to become a pirate archivist

#59

Post by est »

O__________O wrote: post #19
est wrote: post #13

Earlier quoted context omitted.

> which was under directory /home/ringo-ring, which could be traced to a username Ha, my home dir is always called "me" or "and". Try google that.

Electron star - seems oddly specific, though maybe not spelling it same way you do.

yeah that's a regrettable mistake. It was registered in early days, OPSEC wasn't considered then. The Internet was supposed to connect people not hunting people.

Re: How to become a pirate archivist

#60

Post by genewitch »

incompatible wrote: post #33
yamrzou wrote: post #2

«That secrecy, however, comes with a psychological cost. Most people love being recognized for the work that they do, and yet you cannot take any credit for this in real life.» Feels like the anonymous torrent seeder who keeps seeding a file for years just for the sake of keeping it alive. It's not easy, but some people seem to be able to derive full pleasure from accomplishing the task itself, whether recognition ha…

It's probably not that uncommon that people working for companies, governments, or criminal organizations can't talk about their work in public. One group I remember in particular are mathematicians working for the NSA, etc., who are not permitted to publish their research, then they watch as other mathematicians rediscover their work and get the credit.

Is this really a bad thing, though? I mean, personally, recognition seems important. I've withheld patenting certain inventions that became commercial products a half dozen times in my life.

That some specific instance of a discovery or whatever becomes the mainstream version is, well, it's irrelevant. Who discovered calculus? It doesn't actually matter because calculus works without some belief system and worship. Traffic routing algorithms? yes, if the person is alive and kicking, being able to lay claim to some algorithm or novel solution is a CV bullet point, but, and i say this with the utmost respect: most people are one hit wonders. If they can ride that "fame" to higher pay or respect, cool. But in the grand scheme, it's irrelevant. Ideas should be spread far and wide, so that people who have a greater understanding can explain the ideas to those without an understanding.

Capitalism is the problem.

Post reply on HN