Crawl a website with scrapy and store extracted results with MongoDB

#1

Post by BaltoRouberol » Mon, Apr 23, 2012, 1:21 PM UTC

Crawl a website with scrapy and store extracted results with MongoDB
isbullsh.it

Re: Crawl a website with scrapy and store extracted results with MongoDB

#2

Post by hack_edu » Mon, Apr 23, 2012, 2:38 PM UTC

I really want to read. Topic is right down my alley.

Unfortunately, the page is literally broken and unreadable on Android ICS with Chrome :(

Re: Crawl a website with scrapy and store extracted results with MongoDB

#3

Post by mumphster » Mon, Apr 23, 2012, 2:50 PM UTC

hack_edu wrote: post #2

I really want to read. Topic is right down my alley. Unfortunately, the page is literally broken and unreadable on Android ICS with Chrome :(

Same using mobile safari

Re: Crawl a website with scrapy and store extracted results with MongoDB

#4

Post by jordanmessina » Mon, Apr 23, 2012, 2:52 PM UTC

mumphster wrote: post #3

hack_edu wrote: post #2

I really want to read. Topic is right down my alley. Unfortunately, the page is literally broken and unreadable on Android ICS with Chrome :(

Same using mobile safari

Same using chrome on windows unless I resize the browser and make the width about 1000px

Re: Crawl a website with scrapy and store extracted results with MongoDB

#5

Post by [deleted] » Mon, Apr 23, 2012, 3:14 PM UTC

jordanmessina wrote: post #4

mumphster wrote: post #3

Earlier quoted context omitted.

Same using mobile safari

Same using chrome on windows unless I resize the browser and make the width about 1000px

[deleted]

Re: Crawl a website with scrapy and store extracted results with MongoDB

#6

Post by noinput » Mon, Apr 23, 2012, 3:16 PM UTC

hack_edu wrote: post #2

I really want to read. Topic is right down my alley. Unfortunately, the page is literally broken and unreadable on Android ICS with Chrome :(

http://www.readability.com/mobile/articles/yfhqwo0t cleans it right up.

Re: Crawl a website with scrapy and store extracted results with MongoDB

#7

Post by martius » Mon, Apr 23, 2012, 3:35 PM UTC

hack_edu wrote: post #2

I really want to read. Topic is right down my alley. Unfortunately, the page is literally broken and unreadable on Android ICS with Chrome :(

Hi all, thank you for reporting !

This is a known issue and I'm working on it! I'll try to push a "responsive" version today or tomorrow.

Re: Crawl a website with scrapy and store extracted results with MongoDB

#8

Post by martius » Mon, Apr 23, 2012, 3:37 PM UTC

jordanmessina wrote: post #4

mumphster wrote: post #3

Earlier quoted context omitted.

Same using mobile safari

Same using chrome on windows unless I resize the browser and make the width about 1000px

Yes, it's true for any viewport with a width of less than 940px. I'll do my best to fix this too.

Re: Crawl a website with scrapy and store extracted results with MongoDB

#9

Post by jat1 » Mon, Apr 23, 2012, 3:46 PM UTC

Also check this out for a pretty good discussion on scraping http://pyvideo.org/video/609/web-scraping-reliably-and-effic...

Re: Crawl a website with scrapy and store extracted results with MongoDB

#10

Post by BaltoRouberol » Mon, Apr 23, 2012, 3:59 PM UTC

jat1 wrote: post #9

Also check this out for a pretty good discussion on scraping http://pyvideo.org/video/609/web-scraping-reliably-and-effic...

Yeah, I actually learnt scraping from Asheesh :) He's awesome.