Live data from Hacker News

Dorking: the use of search engines to find very specific data

alec.fyi

101–110 of 207 posts

Re: Dorking: the use of search engines to find very specific data

#101
post #75

Earlier quoted context omitted.

Worth mentioning that even if you put a term in double quotes, Google still tries to be too clever - you are not guaranteed to get results that contain your quotes search term :/

This must be a recent change? It's been driving me nuts lately. I have to resort to adding a lot of negated search terms to compensate but it's still sub optimal.

No, this has been the case for a long time, years anyway. I don't know if it goes back quite as far as when they removed the '+' operator tho.

But bejesus, this drives me nuts! If I know the double quotes function even exists, then Google should know I actually want to use it as intended - it shouldn't decide "yeah, but maybe you'd like these irrelevant results too!"

Re: Dorking: the use of search engines to find very specific data

#102

All I want is the ability to search for symbols. Symbolhound.com is the only site I've heard that will support that, but it leaves a lot to be desired.

It’s strange to me that more domain-specific search engines haven’t been created. There must be value in a programmer-specific search engine for instance. Or why aren’t there search engines that specialise in news, social media, Q&A websites or events, to give a few examples.

Re: Dorking: the use of search engines to find very specific data

#103

Earlier quoted context omitted.

https://support.google.com/websearch/answer/2466433?hl=en but it's not complete. My favourite is actually the "range" operator. I don't need it often, but when combined with the exact match quotation marks, it's great. For example, here's a search for Sony bluetooth headphones available on Amazon.ca for between CA$100 and $150: https://www.google.com/search?rls=en&q=site%3Aamazon.ca+%22C... The range operator also wo…

Is there actually a page that says "too precise to be human" or are you just assuming this is what triggered the reCAPTCHA? If there is such a page, can you give an example query that would trigger it?

It mostly happens using "site:" queries which I use frequently to limit things to local websites (by domain) or for searching sites that have poor search engines (Amazon, for example). It rarely happens the first query, but often by the third or fourth modification or by the third or fourth page of results you visit, it will show a reCAPTCHA if it doesn't have enough "randomness" or doesn't think you're actually browsing Google and third-party sites the way others commonly do. (Robots are more likely to use search operators, for example, and more likely to pretend to be iPhones so they don't have to move the mouse, etc.)

My earlier query triggered it. Without a query, I can make the following text show up by going to https://www.google.com/sorry/index which when a relevant query is attached to the URL, it shows a reCAPTCHA for the search query, and also shows your IP address, etc.

> About this page

> Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot. Why did this happen?

If you click the link "Why did this happen?" it says:

> This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the Terms of Service[1]. The block will expire shortly after those requests stop. In the meantime, solving the above CAPTCHA will let you continue to use our services.

> This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests. If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible. Learn more[2]

> Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly.

[1]: https://www.google.com/policies/terms/ [2]: https://support.google.com/websearch/answer/86640

The annoying part is that my account has never been whitelisted based on good behaviour. Instead, I end up seeing such reCAPTCHAs thousands of times a year, to the point where I stop counting them. Roughly half the time I'll answer the reCAPTCHA and the other half of the time, I'll close the tab and go do something else. Cloudflare site loading captchas are even worse, though. They delay the site by 5 seconds while they "check my browser", and then show an hCAPTCHA to solve, even when I'm already signed in with the first-party site. Very annoying, though the captcha is often easier to solve than Google's. The Cloudflare block often on streaming media websites. Ironically, Cloudflare's captchas have never prevented me from using commonly available Python scripts to watch streaming flash videos in VLC, they only block my web browsing...

I can only assume that Safari's excellent ad blocking and tracking prevention is causing my browsing traffic to stand out compared to others', enough that it prompts these CAPTCHAs more frequently.

Re: Dorking: the use of search engines to find very specific data

#104
post #45

Dorking is not that easy to do, Google is very easy on assuming you are being malicious on certain queries, try one too many and you'll hit their dreaded captcha that is impossible to pass.

That really angers me, and I've tripped it more times than I can count, usually by searching for very specific things. Coworkers have also run into it multiple times (before everyone started working from home, we would exclaim "Fuck you, Google!" and raise a middle finger to the screen, which was a cue to everyone else to help). The fact that they think you're "not human" when you use a search engine for its intended…

I’m curious about this. Can you give an example of the kind of query you are talking about where Google assumes you are a bot and not a human?

Re: Dorking: the use of search engines to find very specific data

#105
Worth pointing out if you do some of these crafted operator searches quite quickly, you'll end up getting blocked or having to complete a captcha. I haven't done so in a while so I'm not sure what their current behaviour is.

Main reason being there's plenty data mining, e.g. looking for "powered by wordpress" and vulnerable versions, and generally all kinds of data mining that involve very specific requests for information, likely queries that aren't creating revenue, either.

Re: Dorking: the use of search engines to find very specific data

#106
post #75

Earlier quoted context omitted.

Worth mentioning that even if you put a term in double quotes, Google still tries to be too clever - you are not guaranteed to get results that contain your quotes search term :/

As a workaround and under search tools one can enable the “verbatim” option.

AFAICT, the verbatim option gives the same results as if I'd quoted my search term?

Re: Dorking: the use of search engines to find very specific data

#107
post #46

Earlier quoted context omitted.

It still work but some file type never return anything, I have the same problem with epub, pretty sure it's some google's shenanigan about books piracy. https://support.google.com/webmasters/answer/35287?hl=en

Maybe Google doesn’t index epub at all? I think I never saw one in search results.

Well, I may have become crazy but i have vivid memory using it in the past, and some websites even refer to this specific query ( https://ebookfriendly.com/google-search-tips-books/ )

Re: Dorking: the use of search engines to find very specific data

#108
post #67

Earlier quoted context omitted.

In this case, depreciated is incorrect. Removal has already happened, the "period of time" is already over.

Removing APIs is not a great practice though. Look at AWS, they version their APIs, they don't just remove them, and removing them should be unnecessary if your underlying tech isn't brittle and badly written. "Depreciated" is a far better term to use, with a far better outcome in my opinion. Companies that remove old versions of APIs and break existing client apps (that possibly can't be udpated) really suck.

The “public APIs form an immutable, irrevocable contract” argument means that an api layer with these tenants is always going to be a source of technical debt. Get it right the first time or fight an ever growing compatibility matainance war - even when your instrumentation is saying that old apis aren’t being used, just published, seems like a footgun

Re: Dorking: the use of search engines to find very specific data

#109

I live two towns over from Dorking. https://en.wikipedia.org/wiki/Dorking

I grew up in Dorking, but this is the first time (that I can remember...) that I actually read its wikipedia article.

TIL: No one knows why 'Dorking' is called 'Dorking', but there's a English Place Names Society which since the 1920's has researched the origins of town names in England, and is considered [0] to be "the established national body on the subject".

[0] https://epns.nottingham.ac.uk/

Re: Dorking: the use of search engines to find very specific data

#110

Earlier quoted context omitted.

As a workaround and under search tools one can enable the “verbatim” option.

AFAICT, the verbatim option gives the same results as if I'd quoted my search term?

In my experience it depends on the number of results, and the results are more accurate with verbatim.
Post reply on HN