A search engine in 80 lines of Python

Here is a recommendation engine in

  def build_recommendations(logs: List[List[str]], window_size: int = 10, 
  max_recommendations_per_url: int = 50) -> dict:
      recommendations = {}
      for session in logs:
          for i in range(0, len(session)-1):
              url_id = session[i] # reference element
              window = session[i+1 : i+window_size] # sliding window
              recommendations[url_id] = recommendations.get(url_id, {})
              for pos, next_link in enumerate(window):
                  weight = window_size - pos # elements in the window get decreasing weight proportional to their distance from the reference element
                  recommendations[url_id][next_link] = recommendations[url_id].get(next_link, 0)
                  recommendations[url_id][next_link] += weight
    
      for url_id, link_recommendations in recommendations.items():
          # sort and truncate the recommendations
          recommendations[url_id] = dict(sorted(link_recommendations.items(), key=lambda item: item[1], reverse=True)[:max_recommendations_per_url])

      return recommendations

  recommendations = build_recommendations(your_logs)
  print(list(recommendations[some_url_id].keys())) # gives an ordered list of the recommended url_ids for the given url_id

- some RSS feeds are protected by cloudflare. It is true however that it is not necessary for majority of blogs. If you would like to do more then selenium would be a way to solve "cloudflare" protected links

- sometimes even selenium headless is not enough and full blown browser in selenium is necessary to fool it's protection

- sometimes even that is not enough

- then I started to wonder, why some RSS feeds are so well protected by cloudflare, but who am I to judge?

- sometimes it is beneficial to cover user agent. I feel bad for setting my user agent to chrome, but again, why RSS feeds are so well protected?

- you cannot parse, read entire Internet, therefore you always need to think about compromises. For example I have narrowed area of my searches in one of my projects to domains only. Now I can find most of the common domains, and I sort them by their "importance"

- RSS links do change. There need to be automated means to disable some feeds automatically to prevent checking inactive domains

- I do not see any configurable timeout for reading a page, but I am not familiar with aiohttp. Some pages might waste your time

- I hate that some RSS feeds are not configured properly. Some sites do not provide a valid meta "link" with "application/rss+xml". Some RSS feeds have naive titles like "Home", or no title at all. Such a waste of opportunity

My RSS feed parser, link archiver, web crawler: https://github.com/rumca-js/Django-link-archive. Especially interesting could be file rsshistory/webtools.py. It is not advanced programming craft, but it got the job done.

Additionally, in other project I have collected around 2378 of personal sites. I collect domains in https://github.com/rumca-js/Internet-Places-Database/tree/ma... . These files are JSONs. All personal sites have tag "personal".

Most of the things are collected from:

https://nownownow.com/

https://searchmysite.net/

I wanted also to process domains from https://downloads.marginalia.nu/, but haven't got time to read structure of the files