Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Enumerate all the subdomains for a domain name

merklemap.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Enumerate all the subdomains for a domain name

#11

Post by Eikon »

yup_sto wrote: post #10
yup_sto wrote: post #8

Earlier quoted context omitted.

Awesome, I will keep my eye on this for sure, I've spent the past few months tinkering with ingesting CT logs for bug bounty automation. Curious if you're running your own CertStream server, or just continuously polling known CT logs with your own implementation.

I also noticed you are ingesting/storing flowers-to-the-world.com certs, not sure what stage of optimization you are at but blacklisting/ignoring these certs in my ingestion pipeline helped with avoiding storing unnecessary data I'm not sure but I believe that's used by Google internally for testing purposes. For example if you search google, it returns 120k+ results, and these useless results are at the front.

> I also noticed you are ingesting/storing flowers-to-the-world.com certs, not sure what stage of optimization you are at but blacklisting/ignoring these certs in my ingestion pipeline helped with avoiding storing unnecessary data

The goal is to have something exhaustive so I'll keep them. But you are right that I probably should not put them at front. Not sure how important it is though as these results shouldn't match many queries.

Re: Enumerate all the subdomains for a domain name

#12

Post by yup_sto »

Eikon wrote: post #11
yup_sto wrote: post #10

Earlier quoted context omitted.

I also noticed you are ingesting/storing flowers-to-the-world.com certs, not sure what stage of optimization you are at but blacklisting/ignoring these certs in my ingestion pipeline helped with avoiding storing unnecessary data I'm not sure but I believe that's used by Google internally for testing purposes. For example if you search google, it returns 120k+ results, and these useless results are at the front.

> I also noticed you are ingesting/storing flowers-to-the-world.com certs, not sure what stage of optimization you are at but blacklisting/ignoring these certs in my ingestion pipeline helped with avoiding storing unnecessary data The goal is to have something exhaustive so I'll keep them. But you are right that I probably should not put them at front. Not sure how important it is though as these results shouldn't ma…

Exhaustive/Robust is the way for sure.

Minimizing storage was a priority for me since it's just a small side-project/automation.

I've looked for information on what the hell the `flowers-to-the-world` entries are that pop and have found nothing, curious what's going on there.

Re: Enumerate all the subdomains for a domain name

#13

Post by Eikon »

yup_sto wrote: post #12
Eikon wrote: post #11

Earlier quoted context omitted.

> I also noticed you are ingesting/storing flowers-to-the-world.com certs, not sure what stage of optimization you are at but blacklisting/ignoring these certs in my ingestion pipeline helped with avoiding storing unnecessary data The goal is to have something exhaustive so I'll keep them. But you are right that I probably should not put them at front. Not sure how important it is though as these results shouldn't ma…

Exhaustive/Robust is the way for sure. Minimizing storage was a priority for me since it's just a small side-project/automation. I've looked for information on what the hell the `flowers-to-the-world` entries are that pop and have found nothing, curious what's going on there.

It's actually a google thing!

I found that back then when I wondered the same: https://medium.com/@hadfieldp/hey-ryan-c0fee84b5c39

Re: Enumerate all the subdomains for a domain name

#14

Post by yup_sto »

Eikon wrote: post #13
yup_sto wrote: post #12

Earlier quoted context omitted.

Exhaustive/Robust is the way for sure. Minimizing storage was a priority for me since it's just a small side-project/automation. I've looked for information on what the hell the `flowers-to-the-world` entries are that pop and have found nothing, curious what's going on there.

It's actually a google thing! I found that back then when I wondered the same: https://medium.com/@hadfieldp/hey-ryan-c0fee84b5c39

Ahhhh, that tracks, cheers mate.

Re: Enumerate all the subdomains for a domain name

#15

Post by stavros »

Eikon wrote: post #7
Kab1r wrote: post #2

Is this just searching certificate transparency logs?

Well, CT logs are a data dump, they are not searchable, ingesting all that data near-real time and making it searchable in a useful and fast way (especially with wildcards) is actually quite challenging!

Where does one ingest them from?

Re: Enumerate all the subdomains for a domain name

#16

Post by supriyo-biswas »

stavros wrote: post #15
Eikon wrote: post #7

Earlier quoted context omitted.

Well, CT logs are a data dump, they are not searchable, ingesting all that data near-real time and making it searchable in a useful and fast way (especially with wildcards) is actually quite challenging!

Where does one ingest them from?

https://github.com/google/certificate-transparency-community...

Re: Enumerate all the subdomains for a domain name

#18

Post by 1970-01-01 »

yup_sto wrote: post #5

Have you considered adding a monitoring feature where a user can enter a domain to be monitored and then be notified if a "similar" domain comes across the ingestion pipeline. This would be useful for early detection of potential impersonations/typo-squatting domains typically used for phishing/scams. Something as simple as a configurable levenshtein distance/jaro-winkler similarity check across CN and SAN of all new…

Try https://dnstwister.report/

Re: Enumerate all the subdomains for a domain name

#20

Post by yup_sto »

1970-01-01 wrote: post #18
yup_sto wrote: post #5

Have you considered adding a monitoring feature where a user can enter a domain to be monitored and then be notified if a "similar" domain comes across the ingestion pipeline. This would be useful for early detection of potential impersonations/typo-squatting domains typically used for phishing/scams. Something as simple as a configurable levenshtein distance/jaro-winkler similarity check across CN and SAN of all new…

Try https://dnstwister.report/

[dead]
Post reply on HN