Enumerate all the subdomains for a domain name
merklemap.com
Enumerate all the subdomains for a domain name
1–10 of 25 posts
Re: Enumerate all the subdomains for a domain name
#2Is this just searching certificate transparency logs?
Re: Enumerate all the subdomains for a domain name
#3Is this just searching certificate transparency logs?
I think it's a mix of different sources. Certainly, some of my subdomains there never had an SSL certificate.
Re: Enumerate all the subdomains for a domain name
#4Is this just searching certificate transparency logs?
I'd imagine it's a combination of
- CT log monitoring (https://github.com/CaliDog/CertStream-Server)
- Mass-Scanning across ipv4 on 80/443 at the least?
- Brute-forcing subdomains on wildcards with large DNS wordlist (like something from assetnote: https://wordlists-cdn.assetnote.io/data/manual/best-dns-word...)
- Scraping/extracting subdomains/domains from JS
But I've never attempted to enumerate subdomains on this scale before, so I could be missing something obvious
Re: Enumerate all the subdomains for a domain name
#5Have you considered adding a monitoring feature where a user can enter a domain to be monitored and then be notified if a "similar" domain comes across the ingestion pipeline.
This would be useful for early detection of potential impersonations/typo-squatting domains typically used for phishing/scams.
Something as simple as a configurable levenshtein distance/jaro-winkler similarity check across CN and SAN of all new certs maybe? (user can configure with threshold to control how "noisy" they want their feed).
Re: Enumerate all the subdomains for a domain name
#6Have you considered adding a monitoring feature where a user can enter a domain to be monitored and then be notified if a "similar" domain comes across the ingestion pipeline. This would be useful for early detection of potential impersonations/typo-squatting domains typically used for phishing/scams. Something as simple as a configurable levenshtein distance/jaro-winkler similarity check across CN and SAN of all new…
For sure, it was on my todo list :)
Re: Enumerate all the subdomains for a domain name
#7Is this just searching certificate transparency logs?
Well, CT logs are a data dump, they are not searchable, ingesting all that data near-real time and making it searchable in a useful and fast way (especially with wildcards) is actually quite challenging!
Re: Enumerate all the subdomains for a domain name
#8Have you considered adding a monitoring feature where a user can enter a domain to be monitored and then be notified if a "similar" domain comes across the ingestion pipeline. This would be useful for early detection of potential impersonations/typo-squatting domains typically used for phishing/scams. Something as simple as a configurable levenshtein distance/jaro-winkler similarity check across CN and SAN of all new…
For sure, it was on my todo list :)
Awesome, I will keep my eye on this for sure, I've spent the past few months tinkering with ingesting CT logs for bug bounty automation.
Curious if you're running your own CertStream server, or just continuously polling known CT logs with your own implementation.
Re: Enumerate all the subdomains for a domain name
#9Earlier quoted context omitted.
For sure, it was on my todo list :)
Awesome, I will keep my eye on this for sure, I've spent the past few months tinkering with ingesting CT logs for bug bounty automation. Curious if you're running your own CertStream server, or just continuously polling known CT logs with your own implementation.
I am not using certstream as we'd lose data on the first network error. The way it's designed is more "Rsync for ct logs" than something like a stream => storage system.
Btw, you can get our feed like that:
curl -N 'https://api.merklemap.com/live-domains?no_throttle=true'Re: Enumerate all the subdomains for a domain name
#10Earlier quoted context omitted.
For sure, it was on my todo list :)
Awesome, I will keep my eye on this for sure, I've spent the past few months tinkering with ingesting CT logs for bug bounty automation. Curious if you're running your own CertStream server, or just continuously polling known CT logs with your own implementation.
I also noticed you are ingesting/storing flowers-to-the-world.com certs, not sure what stage of optimization you are at but blacklisting/ignoring these certs in my ingestion pipeline helped with avoiding storing unnecessary data
I'm not sure but I believe that's used by Google internally for testing purposes.
For example if you search google, it returns 120k+ results, and these useless results are at the front.