Live data from Hacker News

Show HN: Tiny Web Crawler

github.com

Post reply on HN

Print view View original

1–2 of 2 posts

Show HN: Tiny Web Crawler

Post by l1feh4ck » Tue, Nov 06, 2018, 4:54 AM UTC

Show HN: Tiny Web Crawler
github.com

Re: Show HN: Tiny Web Crawler

Post by photonios » Tue, Nov 06, 2018, 10:53 AM UTC

It's not entirely correct to remove query parameters from the URL. Query parameters are part of the URL and can make a website show entirely different content.

The Googlebot for example considers these two URL's different pages:

    /mypage?name=name1
    /mypage?name=name2

Post reply on HN