[deleted]
I only use an iFrame to crawl and scrape content
41–50 of 53 posts
Re: I only use an iFrame to crawl and scrape content
#42 $iframe.contents().find('.result-row').each(function(){
data.push({
title: $(this).find('.result-title').text(),
img: $(this).find('img').attr("src"),
price: $(this).find('.result-price:first').text()
});
// And everything starts running when you set first iframe's target url
$iframe.prop("src", "https://newyork.craigslist.org/d/apts-housing-for-rent/search/apa");
title:
img:
price:
image:
title:
price
n=0;while true;do test $n -le 3000||break;
curl https://newyork.craigslist.org/d/apts-housing-for-rent/search/apa?s=$n|sed -n '
/result-title hdrlnk/{s/.*\">/title: /;s//price: /;s/]*|https://images.craigslist.org/&_600x450.jpg|g;s/,/, /g;
s/1://g;s/>//;s/.*data-ids=/image: /;/^image: /p;}'
n=$((n+120));doneRe: I only use an iFrame to crawl and scrape content
#43Earlier quoted context omitted.
Injecting an iframe into websites could trigger an assertion error, because the iframe isn't supposed to be there.
Hi, that's right. Some sites protect html injection. Twitter protects their site very well, but if you use http tunneling they can't do nothing as you can modify X- headers.
Interesting, could you explain what you mean by http tunneling and how it can bypass protection against html injection?
Re: I only use an iFrame to crawl and scrape content
#44Okay, the way I see this is that using headless tools like puppeteer or selenium is tedious; just trying to... er scrape my HN account's favorites (AFAIK no API) becomes a task when you have to automate login. Just typing in and pressing the button is much easier than automating the task, so that's why the iframe is something useful. You can interact with the content (without code).
Is login required to access favorites page; isnt the page public, e.g.,
https://news.ycombinator.com/favorites?id=pcr910303
In the case login was required, using tmux it's easy to automate login to HN with a text-only browser such as links and saving the desired text/html, etc. to a file. Takes me about 500 characters of script to non-interactively log in, grab some text and log out
Re: I only use an iFrame to crawl and scrape content
#45This sounds hilariously n00by because it's VB and Internet Explorer, but creating an Internet Explorer instance through VB in, say, Excel and then dumping data into Excel was great because I had full control over my IE instance. Okay, I'll stop speaking now and revealing the fact I started my career as a data guy at a giant corporation instead of a software engineer.
Haha, loser! Looo sseeerr
Re: I only use an iFrame to crawl and scrape content
#46Working for a big tech company, stuff like this infuriates me.
It’s exactly why we’re currently pushing for the ability to disable developer tools, we want it added to Chrome and other browsers. I should be able to, as a web site owner, not allow any kind of developer tool usage.
Users do not own our product and have no right to go poking around like this!
Re: I only use an iFrame to crawl and scrape content
#47Working for a big tech company, stuff like this infuriates me. It’s exactly why we’re currently pushing for the ability to disable developer tools, we want it added to Chrome and other browsers. I should be able to, as a web site owner, not allow any kind of developer tool usage. Users do not own our product and have no right to go poking around like this!
> Users do not own our product and have no right to go poking around like this!
I feel just the same about websites and apps poking around in my stuff.
Re: I only use an iFrame to crawl and scrape content
#48Working for a big tech company, stuff like this infuriates me. It’s exactly why we’re currently pushing for the ability to disable developer tools, we want it added to Chrome and other browsers. I should be able to, as a web site owner, not allow any kind of developer tool usage. Users do not own our product and have no right to go poking around like this!
You don't own my computer either and thus you don't get to choose what I can and can't do with it. The prospect of disabling dev tools just so your business can have it easier is laughable.
Re: I only use an iFrame to crawl and scrape content
#49Working for a big tech company, stuff like this infuriates me. It’s exactly why we’re currently pushing for the ability to disable developer tools, we want it added to Chrome and other browsers. I should be able to, as a web site owner, not allow any kind of developer tool usage. Users do not own our product and have no right to go poking around like this!
Make an app then. You/your company made the choice to use web technologies and take advantage of their benefits.
Users own their computers and their browser (user agent) is for serving them. Not you.
You have no right to be telling a users computer exactly what to do. Do that on your own servers.
The state of tracking and telemetry is insane enough already with chrome gearing up to cut the legs out from ad blocking.
Plus, even if you're lucky enough to have your wishes with the browser, it doesn't affect anyone serious anyway. They will scrape outside of chrome, as they already do and have always done.
Re: I only use an iFrame to crawl and scrape content
#50Working for a big tech company, stuff like this infuriates me. It’s exactly why we’re currently pushing for the ability to disable developer tools, we want it added to Chrome and other browsers. I should be able to, as a web site owner, not allow any kind of developer tool usage. Users do not own our product and have no right to go poking around like this!
This sounds to me like « people who watch our movie shouldn’t be allowed to rewind! They don’t own the movie! »