Viewing profile — dataslap

21 Gigs in total for everything, ignoring errors (files too big / broken links)

I'm really happy their share it though. The collection is just amazing - it's a good idea for "what books to buy for a personal library when you start making decent money". Definit…

I'm at 300 right now, 9.5 gigs. I think some are skipped cause the download times out (can redownload manually I suppose)

P.S. it will take hours or days =))) <3 <3 <3

Please use responsibly. Recommended to delay by a few days - it seems like there is an initial surge going on right now. Increase download delay to 20 sec - 1 minute. Be respectful…

I think someone else is messing with it - got slower for me too.

It's slow cause things are quite large there (just saw a 2 gig book). I'm using delays, don't worry - hence "in a few hours" <3 <3 <3

<3

Uploading a scrapy crawler that downloads PDF books to github in a few hours, gonna post the link here

some "ethical" measures may do the trick to. scrapy has a setting to integrate delays + you can use fake headers. Some sites are pretty persistent with their cookies (include cooki…

depends on the task. For example they have a decent file/image downloading middleware.

scrapy has a pretty decent parser too