Very nice project. Thanks for sharing it here. I saw you used beautifulsoup to scrape, but HN has a public API which works great and it doesn't need any token or auth. I recently wrote a minimal wrapper for that: https://github.com/mfiro/hnconnector
Really appreciate it Super cool wrapper, love the methods that make it super easy to get the content What would be the best way to for example, get the latest Ask HN: who's hiring page and all it's top level comments? Thank you
ask_stories = client.get_askstories()
then you have the ids and you can sort them reversed to catch the latest stories.
client.get_item() will give you the story's details, in which you'll find item['kids'], which are the IDs for the top-level comments. You can then loop through these comment IDs to get more info on each comment.
Another option is to use Algolia's API for HN. But that's not yet supported in my wrapper.