I only use an iFrame to crawl and scrape content
1–10 of 53 posts
Re: I only use an iFrame to crawl and scrape content
#2Is the only reason for the iframe so that it is possible to keep a state in the top frame while loading different pages?
Because otherwise - since you use the dev tools to inject the iframe - you don't really need the iframe. You can just run it as a "snippet" in Chromium or from the multi-line-code-editor in Firefox.
Both have the problem that it all has to be a single file. It would be much nicer if one could import modules.
Re: I only use an iFrame to crawl and scrape content
#3Won't the same origin policy kick in the moment I try to read the content of an iframe that isn't on the same origin as my website?
Or is this meant to run on the dev console on the target website? In which case, the iFrame and the Airovic website doesn't make sense (the electron app mentioned does sure, but it doesn't exist)
Re: I only use an iFrame to crawl and scrape content
#4Is the only reason for the iframe so that it is possible to keep a state in the top frame while loading different pages? Because otherwise - since you use the dev tools to inject the iframe - you don't really need the iframe. You can just run it as a "snippet" in Chromium or from the multi-line-code-editor in Firefox. Both have the problem that it all has to be a single file. It would be much nicer if one could impor…
>Both have the problem that it all has to be a single file. It would be much nicer if one could import modules.
Isn't this a solved problem in javascript land? Just use a compiler/minifier and your module oriented js code is in a single file as a build artifact.
Re: I only use an iFrame to crawl and scrape content
#5Is the only reason for the iframe so that it is possible to keep a state in the top frame while loading different pages? Because otherwise - since you use the dev tools to inject the iframe - you don't really need the iframe. You can just run it as a "snippet" in Chromium or from the multi-line-code-editor in Firefox. Both have the problem that it all has to be a single file. It would be much nicer if one could impor…
>Both have the problem that it all has to be a single file. It would be much nicer if one could import modules. Isn't this a solved problem in javascript land? Just use a compiler/minifier and your module oriented js code is in a single file as a build artifact.
Then every time you want to make a change to your code, you have to go to your original codebase, make the change, start the compiler, copy the output and paste it into your dev tools ...
Re: I only use an iFrame to crawl and scrape content
#6Won't the same origin policy kick in the moment I try to read the content of an iframe that isn't on the same origin as my website? Or is this meant to run on the dev console on the target website? In which case, the iFrame and the Airovic website doesn't make sense (the electron app mentioned does sure, but it doesn't exist)
That was my thinking as well. You cannot access another website's content opened via iframe via Javascript at all.
Re: I only use an iFrame to crawl and scrape content
#7do not understand why iframe is a must here, why can I just scrape the whole page directly? still learning web scraping using scrapy.
Re: I only use an iFrame to crawl and scrape content
#8Is the only reason for the iframe so that it is possible to keep a state in the top frame while loading different pages? Because otherwise - since you use the dev tools to inject the iframe - you don't really need the iframe. You can just run it as a "snippet" in Chromium or from the multi-line-code-editor in Firefox. Both have the problem that it all has to be a single file. It would be much nicer if one could impor…
> It would be much nicer if one could import modules.
Is there some reason es modules wouldn't work here? Just a snippet that inserts a tag of type=module
Re: I only use an iFrame to crawl and scrape content
#9Is the only reason for the iframe so that it is possible to keep a state in the top frame while loading different pages? Because otherwise - since you use the dev tools to inject the iframe - you don't really need the iframe. You can just run it as a "snippet" in Chromium or from the multi-line-code-editor in Firefox. Both have the problem that it all has to be a single file. It would be much nicer if one could impor…
I think having this on a production website means you can crawl the web using your user's CPUs and network connections, making it harder for people to stop you from harvesting their data.
Re: I only use an iFrame to crawl and scrape content
#10I can do everything listed in benefits with puppeteer, while I can’t even make sense of what iframe is supposed to achieve here, or how it’s even gonna load (anyone with a shred of sense would set X-Frame-Options to SAMEORIGIN, subject to exceptions). The airovic.com site doesn’t work and hilariously attempts to load two seemingly important scripts from localhost...
I’m very confused about this submission, and even more confused about how it managed to almost top the front page.
Edit: Having read the code samples, it seems the code snippets are supposed be run from the same origin in the dev console. A quick and dirty way to interactively scrape without navigation, I guess? Still not sure what the “all together: Airovic.com” is supposed to mean, and definitely more limited than puppeteer.
Edit2: To be fair to the author, they did say
> You cannot bypass their protections without using a HTTP Tunneling component.
Which I didn’t see until just now. This is a pretty big caveat though, should probably be more upfront...