91–100 of 121 posts
Earlier quoted context omitted.
I'm not sure about the comparison itself... I've tried DDG several times, I search for technical things in generic ways a lot. DDG almost never gives me what I want in the first page. Google almost always does.
Same here. It’s hard to blame DDG though- Google’s search index of Stack Overflow is better than SO’s own.
For dynamic sure puppeteer if you have too but my god the exceptions and stack traces need some work.
But most websites don't enact protections because it's generally not worth the opportunity cost. So you really just scrape with your LOC.
If you move money or can't code then mozenda.
Earlier quoted context omitted.
I'm not sure about the comparison itself... I've tried DDG several times, I search for technical things in generic ways a lot. DDG almost never gives me what I want in the first page. Google almost always does.
Same here. It’s hard to blame DDG though- Google’s search index of Stack Overflow is better than SO’s own.
Earlier quoted context omitted.
I understand that, but I think there are good reasons why we shouldn't always act in the narrowest sense of our self interest (provided we have enough financial wiggle room). A search monopoly is not good for website owners. It makes us very dependent on the whims of that monopolist. If you block all crawlers that don't already have a large market share and send back a lot of traffic, you're killing any possibility f…
> It makes us very dependent on the whims of that monopolist. Very true. Only allow Google and you are helping them to build their monopoly. And if they have full monopoly they do what they want - including asking you money to be included in the search results.
Since people are asking "why would you do such a thing" or insinuating that scraping need only be to compete somehow with Google, I'll present a use I've found quite interesting, that doesn't seek to replicate or replace Google search, and which hasn't been readily attainable other than by scraping Google search results, in part. The tool I've used (crude, but reasonably effective) has applied numerous attempts to wo…
The ArchiveTeam[1] have a simple VM image that anyone can use to schedule and coordinate large site archival jobs that might already address some of teh issues.
Might be tricky to find people willing to provide resources, but with even a smallish group it might work out. May need to consider abuse and run multiple queries and compare results, which might add to the overall request cost.
I was playing around with the idea of using Tor to get around IP blocks. I played around a bit with code but the Tor binary dependency was a bit much for my use case. Curious to know if anyone else tried this?
Everyone else has the same idea which is why it often makes sense to block Tor outright.
There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.
Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…
Since people are asking "why would you do such a thing" or insinuating that scraping need only be to compete somehow with Google, I'll present a use I've found quite interesting, that doesn't seek to replicate or replace Google search, and which hasn't been readily attainable other than by scraping Google search results, in part. The tool I've used (crude, but reasonably effective) has applied numerous attempts to wo…
Have you considered some sort of "crowdsourcing" / voluntary botnet type approach? The ArchiveTeam[1] have a simple VM image that anyone can use to schedule and coordinate large site archival jobs that might already address some of teh issues. Might be tricky to find people willing to provide resources, but with even a smallish group it might work out. May need to consider abuse and run multiple queries and compare r…
My approach is sufficiently fluid that this would mean pushing pretty crude code to a bunch of hosts frequently and on a irregular basis. The runs themselves are fairly ad hoc.
Being able to directly query a corpus (IA, DDG, Bing, etc.) is another option.
Search across large corpora remains fairly expensive, I can understand hesitency here.
Nonstandardisation of search APIs across sites is another frustration.
For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…
Use DDG. It's fine for most things. Use Google as a fallback.
To do this from DDG, prepend your search using !g to search Google