Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How did the internet discover my subdomain?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 322 posts

Ask HN: How did the internet discover my subdomain?

#1

Post by govideo »

I have a domain that is not live. As expected, loading the domain returns: Error 1016.

However...I have a subdomain with a not obvious name, like: userfileupload.sampledomain.com

This subdomain IS LIVE but has NOT been publicized/posted anywhere. It's a custom URL for authenticated users to upload media with presigned url to my Cloudflare r2 bucket.

I am using CloudFlare for my DNS.

How did the internet find my subdomain? Some sample user agents are: "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_7; en-us) AppleWebKit/534.20.8 (KHTML, like Gecko) Version/5.1 Safari/534.20.8", "Mozilla/5.0 (Linux; Android 9; Redmi Note 5 Pro) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.89 Mobile Safari/537.36",

The bots are GET requests which are failing, as designed, but I'm wondering how the bots even knew the subdomain existed?!

Re: Ask HN: How did the internet discover my subdomain?

#4

Post by daggersandscars »

DNS query type AXFR allows for subdomain querying. There are security restrictions around who can do it on what DNS servers. Given the number of places online one can run a subdomain query, I suspect it's mostly a matter of paying the right fees to the right DNS provider.

Re: Ask HN: How did the internet discover my subdomain?

#6

Post by govideo »

codingdave wrote: post #2

If it is on DNS, it is discoverable. Even if it were not, the message you pasted says outright that they scan the entire IP space, so they could be hitting your server's IP without having a clue there is a subdomain serving your stuff from it.

Ahh yeah, my internet network knowledge was never super strong, and now is rusty to boot. Thanks for your note.

Re: Ask HN: How did the internet discover my subdomain?

#7

Post by thisisgvrt »

govideo wrote: post #5
Kikawala wrote: post #3

Is it available under HTTPS? Then it's probably in a Certificate Transparency log.

Yes, https via cloudflare's automatic https. Thanks for the info.

Automated agents can tail the certificate log to discover new domains as the certs are issued. But if you want to explore subdomains manually, https://crt.sh/ is a nice tool.

Re: Ask HN: How did the internet discover my subdomain?

#8

Post by snailmailman »

govideo wrote: post #5
Kikawala wrote: post #3

Is it available under HTTPS? Then it's probably in a Certificate Transparency log.

Yes, https via cloudflare's automatic https. Thanks for the info.

Yeah this is a surprisingly little known fact- all certs being logged means all subdomain names get logged.

Wildcard certs can hide the subdomains, but then your cert works on all subdomains. This could be an issue if the certs get compromised.

Usually there isn’t sensitive information in subdomain names, but i suspect it often accidentally leaks information about infrastructure setups. "vaultwarden.example.com" existing tells you someone is probably running a vaultwarden instance, even if it’s not publicly accessible.

The same kind of info can leak via dns records too, I think?

Re: Ask HN: How did the internet discover my subdomain?

#9

Post by paulnpace »

codingdave wrote: post #2

If it is on DNS, it is discoverable. Even if it were not, the message you pasted says outright that they scan the entire IP space, so they could be hitting your server's IP without having a clue there is a subdomain serving your stuff from it.

Shouldn't the web server only respond to a configred domain, else 404?
Post reply on HN