Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I only use an iFrame to crawl and scrape content

airovic.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: I only use an iFrame to crawl and scrape content

#31

Post by natzar »

amelius wrote: post #18
javajosh wrote: post #11

Earlier quoted context omitted.

He's traversing the site using the injected iframe. That is, there is no top-level navigation event, only an iframe navigation event. Then he's gathering information from the iframe hosted DOM and combining it in the context of the starting, main page. I think it's kinda clever.

Injecting an iframe into websites could trigger an assertion error, because the iframe isn't supposed to be there.

Hi, that's right. Some sites protect html injection. Twitter protects their site very well, but if you use http tunneling they can't do nothing as you can modify X- headers.

Re: I only use an iFrame to crawl and scrape content

#32

Post by natzar »

captn3m0 wrote: post #3

Won't the same origin policy kick in the moment I try to read the content of an iframe that isn't on the same origin as my website? Or is this meant to run on the dev console on the target website? In which case, the iFrame and the Airovic website doesn't make sense (the electron app mentioned does sure, but it doesn't exist)

If you use an iframe yes, that is the problem.

But the airovic.com tool uses and external script, in the same server that provides http tunneling from same server, so I can modify headers and display everything on the iframe because it is under same domain.

Thanks for your comment!

Re: I only use an iFrame to crawl and scrape content

#33

Post by natzar »

Wow, I just posted this, went to take a nap, got back and 84 points ¿?!

The site is working now, it was retrieving some localhost scripts.

I was just trying to get some feedback and to check if that document was interesting, because I was spending a lot of time on it.

For what I see, it seems Selenium does exactly the same, but I would choose this iframe solution (small-medium projects) anyway.

It's a super small tool that do the job.

Please, let me know if you can fully use airovic.com

Re: I only use an iFrame to crawl and scrape content

#35

Post by hinkley »

hugs wrote: post #22

This is how the very first version of Selenium worked. The application under test was in an iframe, and the test controller was in the parent page. The Selenium "Remote Control" protocol was later added where the controller would phone home to a listening web server for commands to relay to the iframe (basically, AJAX before it had a name. It all mostly worked for the most common test cases, but we abandoned this app…

I believe karma still works this way.

Re: I only use an iFrame to crawl and scrape content

#36

Post by asdfman123 »

This sounds hilariously n00by because it's VB and Internet Explorer, but creating an Internet Explorer instance through VB in, say, Excel and then dumping data into Excel was great because I had full control over my IE instance.

Okay, I'll stop speaking now and revealing the fact I started my career as a data guy at a giant corporation instead of a software engineer.

Re: I only use an iFrame to crawl and scrape content

#37

Post by 3xblah »

hugs wrote: post #22

This is how the very first version of Selenium worked. The application under test was in an iframe, and the test controller was in the parent page. The Selenium "Remote Control" protocol was later added where the controller would phone home to a listening web server for commands to relay to the iframe (basically, AJAX before it had a name. It all mostly worked for the most common test cases, but we abandoned this app…

"It mostly worked for the most common test cases, but we abandoned this approach for similar reasons mentioned in the article -- the edge case limitations became more and more frustrating over time."

The article only mentions malformed URLs and browser run-time errors. Were there any other "edge case limitations" that became intolerable.

Re: I only use an iFrame to crawl and scrape content

#38

Post by Fauntleroy »

3xblah wrote: post #37
hugs wrote: post #22

This is how the very first version of Selenium worked. The application under test was in an iframe, and the test controller was in the parent page. The Selenium "Remote Control" protocol was later added where the controller would phone home to a listening web server for commands to relay to the iframe (basically, AJAX before it had a name. It all mostly worked for the most common test cases, but we abandoned this app…

"It mostly worked for the most common test cases, but we abandoned this approach for similar reasons mentioned in the article -- the edge case limitations became more and more frustrating over time." The article only mentions malformed URLs and browser run-time errors. Were there any other "edge case limitations" that became intolerable.

Some websites will detect the iframe and send nothing instead of the requested page. I happen to know from personal experience that YouTube is one of the sites that does this.

Re: I only use an iFrame to crawl and scrape content

#39

Post by chii »

Fauntleroy wrote: post #38
3xblah wrote: post #37

Earlier quoted context omitted.

"It mostly worked for the most common test cases, but we abandoned this approach for similar reasons mentioned in the article -- the edge case limitations became more and more frustrating over time." The article only mentions malformed URLs and browser run-time errors. Were there any other "edge case limitations" that became intolerable.

Some websites will detect the iframe and send nothing instead of the requested page. I happen to know from personal experience that YouTube is one of the sites that does this.

> YouTube is one of the sites that does this.

that's because they have a proper iframe-embed url: https://www.youtube.com/embed/quyj70RogxI (instead of https://www.youtube.com/watch?v=quyj70RogxI )

Post reply on HN