Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I only use an iFrame to crawl and scrape content

airovic.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: I only use an iFrame to crawl and scrape content

#11

Post by javajosh »

oefrha wrote: post #10

I can do everything listed in benefits with puppeteer, while I can’t even make sense of what iframe is supposed to achieve here, or how it’s even gonna load (anyone with a shred of sense would set X-Frame-Options to SAMEORIGIN, subject to exceptions). The airovic.com site doesn’t work and hilariously attempts to load two seemingly important scripts from localhost... I’m very confused about this submission, and even m…

He's traversing the site using the injected iframe. That is, there is no top-level navigation event, only an iframe navigation event. Then he's gathering information from the iframe hosted DOM and combining it in the context of the starting, main page.

I think it's kinda clever.

Re: I only use an iFrame to crawl and scrape content

#12

Post by oefrha »

javajosh wrote: post #11
oefrha wrote: post #10

I can do everything listed in benefits with puppeteer, while I can’t even make sense of what iframe is supposed to achieve here, or how it’s even gonna load (anyone with a shred of sense would set X-Frame-Options to SAMEORIGIN, subject to exceptions). The airovic.com site doesn’t work and hilariously attempts to load two seemingly important scripts from localhost... I’m very confused about this submission, and even m…

He's traversing the site using the injected iframe. That is, there is no top-level navigation event, only an iframe navigation event. Then he's gathering information from the iframe hosted DOM and combining it in the context of the starting, main page. I think it's kinda clever.

Okay, I guess the scripts are supposed to run in the same origin, not from some third party, say airovic.com. I edited my original post.

It’s still more limited than puppeteer though.

Re: I only use an iFrame to crawl and scrape content

#14

Post by heyplanet »

lucideer wrote: post #8
heyplanet wrote: post #2

Is the only reason for the iframe so that it is possible to keep a state in the top frame while loading different pages? Because otherwise - since you use the dev tools to inject the iframe - you don't really need the iframe. You can just run it as a "snippet" in Chromium or from the multi-line-code-editor in Firefox. Both have the problem that it all has to be a single file. It would be much nicer if one could impor…

> It would be much nicer if one could import modules. Is there some reason es modules wouldn't work here? Just a snippet that inserts a tag of type=module

Yes, the reason is that most sites these days serve a content security policy which only allows code from whitelisted origins.

Re: I only use an iFrame to crawl and scrape content

#15

Post by ravenstine »

Depending on the nature of the content being scraped, you can use the `sandbox` attribute to the iFrame to prevent scripts from running.

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/if...

This was useful for a brief period when I ran a news aggregator that used iFrames to display content from other news websites. Adding the sandbox attribute prevented scripts, ads, modals, etc.

For the purpose of scraping, unless you're always on the same domain(or running a proxy to add CORS), I don't see how an iFrame is better than either a web extension or a backend script using Puppeteer.

Re: I only use an iFrame to crawl and scrape content

#16

Post by fenwick67 »

Yeesh, everyone is so critical here. It's just a blog post about how somebody does occasional one-off scraping across multiple pages using browser devtools.

Yes of course injecting an iframe into a third-party site with devtools isn't going to replace Selenium. But it's a clever little hack in a pinch. No need to get upset.

Re: I only use an iFrame to crawl and scrape content

#17

Post by pcr910303 »

Okay, the way I see this is that using headless tools like puppeteer or selenium is tedious; just trying to... er scrape my HN account's favorites (AFAIK no API) becomes a task when you have to automate login.

Just typing in and pressing the button is much easier than automating the task, so that's why the iframe is something useful. You can interact with the content (without code).

Re: I only use an iFrame to crawl and scrape content

#18

Post by amelius »

javajosh wrote: post #11
oefrha wrote: post #10

I can do everything listed in benefits with puppeteer, while I can’t even make sense of what iframe is supposed to achieve here, or how it’s even gonna load (anyone with a shred of sense would set X-Frame-Options to SAMEORIGIN, subject to exceptions). The airovic.com site doesn’t work and hilariously attempts to load two seemingly important scripts from localhost... I’m very confused about this submission, and even m…

He's traversing the site using the injected iframe. That is, there is no top-level navigation event, only an iframe navigation event. Then he's gathering information from the iframe hosted DOM and combining it in the context of the starting, main page. I think it's kinda clever.

Injecting an iframe into websites could trigger an assertion error, because the iframe isn't supposed to be there.

Re: I only use an iFrame to crawl and scrape content

#20

Post by oefrha »

pcr910303 wrote: post #17

Okay, the way I see this is that using headless tools like puppeteer or selenium is tedious; just trying to... er scrape my HN account's favorites (AFAIK no API) becomes a task when you have to automate login. Just typing in and pressing the button is much easier than automating the task, so that's why the iframe is something useful. You can interact with the content (without code).

You can use non-headless puppeteer to manually log in (or perform any other manual actions, or use the dev console in the puppeteer-controlled browser) and only automate parts you want to automate.
Post reply on HN