Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I only use an iFrame to crawl and scrape content

airovic.com

Post reply on HN
Print view View original

21–30 of 53 posts

Re: I only use an iFrame to crawl and scrape content

#21

Post by ChrisSD »

I've done something similar in Firefox with scratchpad. The main reason is simply convenience. I don't need to switch to a different workflow, I merely bring up scratchpad (I often already have a window open with some utility functions) and can start hacking away immediately.

Sadly scratchpad is going away soon. Fortunately the console now has a multiline mode, unfortunately it's not as convenient for this use.

Re: I only use an iFrame to crawl and scrape content

#22

Post by hugs »

This is how the very first version of Selenium worked. The application under test was in an iframe, and the test controller was in the parent page. The Selenium "Remote Control" protocol was later added where the controller would phone home to a listening web server for commands to relay to the iframe (basically, AJAX before it had a name. It all mostly worked for the most common test cases, but we abandoned this approach for similar reasons mentioned in the article -- the edge case limitations became more and more frustrating over time. Ultimately, we merged with the WebDriver project, which was implemented in a more native way, avoiding all the limitations of automation-via-iframe.

Re: I only use an iFrame to crawl and scrape content

#23

Post by elierotenberg »

For this kind of tasks I usually create a private Firefox extension which gives me access to extended browser capabilities and the ability to lift some security-related restrictions. I run it in a sandboxed browser, much like I would do with something like Selenium or Puppeter, but I have much more options to hand-tune the automation.

Re: I only use an iFrame to crawl and scrape content

#24

Post by onlyrealcuzzo »

nodelessness wrote: post #6
captn3m0 wrote: post #3

Won't the same origin policy kick in the moment I try to read the content of an iframe that isn't on the same origin as my website? Or is this meant to run on the dev console on the target website? In which case, the iFrame and the Airovic website doesn't make sense (the electron app mentioned does sure, but it doesn't exist)

That was my thinking as well. You cannot access another website's content opened via iframe via Javascript at all.

I think you first visit the website, then you inject an iframe onto the page you're currently on, and then inside that iframe you can scrape any content on that website.

That's at least what it looked like from his examples.

Re: I only use an iFrame to crawl and scrape content

#25

Post by hugs »

pcr910303 wrote: post #17

Okay, the way I see this is that using headless tools like puppeteer or selenium is tedious; just trying to... er scrape my HN account's favorites (AFAIK no API) becomes a task when you have to automate login. Just typing in and pressing the button is much easier than automating the task, so that's why the iframe is something useful. You can interact with the content (without code).

The irony is that the iframe approach is exactly how the first version of Selenium worked. It's a cool hack, but we abandoned that approach over time because automating iframes couldn't cover all automation use cases.

Re: I only use an iFrame to crawl and scrape content

#26

Post by hayksaakian »

ausjke wrote: post #7

do not understand why iframe is a must here, why can I just scrape the whole page directly? still learning web scraping using scrapy.

I think the basic utility is that you keep your parent-frame JavaScript context.

Normally if you click a link with jQuery, you lose the current context after the next page loads.

By controlling it inside an iframe it's more convenient

Re: I only use an iFrame to crawl and scrape content

#27

Post by elorant »

pcr910303 wrote: post #17

Okay, the way I see this is that using headless tools like puppeteer or selenium is tedious; just trying to... er scrape my HN account's favorites (AFAIK no API) becomes a task when you have to automate login. Just typing in and pressing the button is much easier than automating the task, so that's why the iframe is something useful. You can interact with the content (without code).

Selenium allows you to load different browser profiles. So you create a profile manually, login to the target site and then everytime you load the profile automatically the cookies are set and you're logged-in. Works like a charm.

Re: I only use an iFrame to crawl and scrape content

#28

Post by hugs »

fenwick67 wrote: post #16

Yeesh, everyone is so critical here. It's just a blog post about how somebody does occasional one-off scraping across multiple pages using browser devtools. Yes of course injecting an iframe into a third-party site with devtools isn't going to replace Selenium. But it's a clever little hack in a pinch. No need to get upset.

Go back far enough in time (2004), and injecting an iframe into a third-party site is Selenium. I agree, using an iframe is a fun hack!

Re: I only use an iFrame to crawl and scrape content

#29

Post by natzar »

oefrha wrote: post #10

I can do everything listed in benefits with puppeteer, while I can’t even make sense of what iframe is supposed to achieve here, or how it’s even gonna load (anyone with a shred of sense would set X-Frame-Options to SAMEORIGIN, subject to exceptions). The airovic.com site doesn’t work and hilariously attempts to load two seemingly important scripts from localhost... I’m very confused about this submission, and even m…

Hi! I'm the submitter.

I just submitted the article to recieve some feedback. I was working a lot on the tool and the article, but needed to check if I was into something.

I fixed the errors already. it is woking.

Post reply on HN