111–118 of 118 posts
Earlier quoted context omitted.
I'm a bit curious about this 'back button hijacking', since the bug has now been fixed I'm afraid I'm not able to test. What was the unintended behaviour?
If you were entering deephn.org in your browser address bar, you couldn't use the browser back button to return to the previous page. That was a bug caused by the navigation code within our dynamic HTML page. It is fixed now.
The effort definetely gets my upvote. What did you use to crawl the pages and how long did it take? Curious about your experience doing it and crawler integration with Seekstorm. Is is possible to easilly expand the index with embeddings (vectors) and perform semantic search in parallel? Your pricing indicates that hosting similar index as this demo would cost $500/month. Wondering what kind of infrastructure is supp…
Crawling speed is between 100...1000 pages per second. We crawled about 4 million linked unique web pages The pricing for an Index like DeepHN would be $99/month: While we are indexing 30 million Hacker news posts, for DeepHN we are combining a single HN story with all its comments and its linked webpage into a single SeekSorm document. So that we index just 4 million SeekStorm documents. Yes, it would be possible to…
Stupid question, but you were crawling news.ycombinator.com, right?
Its robots.txt (https://news.ycombinator.com/robots.txt) contains
Crawl-delay: 30
(I'm not being accusatory. I'm both curious about web crawling in general and have personally been archiving the front page and "new" every 60 seconds or so... (Obviously there's no reason for me to retrieve them more often, but my curiosity persists.))
It’s fun to see Bitcoin posts from 2008-2010. I wonder how many bought and are now retired because of them. Seeing recommendations to use MtGox is pretty funny too https://news.ycombinator.com/item?id=1998144
Earlier post 2009: https://news.ycombinator.com/item?id=599852 "Well this is an exceptionally cute idea, but there is absolutely no way that anyone is going to have any faith in this currency." Written by co-founder/CEO of Pachyderm and first employee of RethinkDB none the less.
Earlier quoted context omitted.
Crawling speed is between 100...1000 pages per second. We crawled about 4 million linked unique web pages The pricing for an Index like DeepHN would be $99/month: While we are indexing 30 million Hacker news posts, for DeepHN we are combining a single HN story with all its comments and its linked webpage into a single SeekSorm document. So that we index just 4 million SeekStorm documents. Yes, it would be possible to…
Crawling speed is between 100...1000 pages per second. Stupid question, but you were crawling news.ycombinator.com, right? Its robots.txt ( https://news.ycombinator.com/robots.txt ) contains Crawl-delay: 30 Why did you not follow that? (I'm not being accusatory. I'm both curious about web crawling in general and have personally been archiving the front page and "new" every 60 seconds or so... (Obviously there's no re…
No, for retrieving the Hacker News Posts we were using the public Hacker News API, which returns the posts in JSON format: https://github.com/HackerNews/API
The crawling speed of 100...1000 pages per second refers to crawling the external pages linked from Hacker news posts. As they are from different domains we can achieve a high crawling speed while being a polite crawler with a low crawling rate per domain.
Apologies for being nitpicky about the already fixed back button While viewing a specific drilled-down result, each time the back button is pressed, it seems to navigate to the previous character in the incremental search. In other words, if the search term is 'C#', navigating back seems to highlight 'C' in the result detail page.
That is related to our instant search, where you don't need to hit the return key to search. This causes the problem, that we need to identify when a query ends and the next query starts, to create the correct entries for the back button. We do this via timing - if the pause between key presses is too long, a new, separate entry is assumed. As the DeepHN is a single dynamic HTML page, we need to implement and manage the browser navigation logic ourselves.
Can we exclude certain terms in search with a "-" e.g. "-oculus"?
Earlier quoted context omitted.
>> When trying to go back after clicking the original HN link, there's no response. The original HN link is opened in a new browser page/tab. The search results should remain unchanged in the previous tab. So you don't use the back button, but the previous browser tab. >> Clicking on what I understand are tags (hashtags) in a given post has no effect. Search for "google", go to the first result. there is a hashtag "o…
> Please post an example where its doesn't work, so that we can fix it. Note: I'm trying this in a mobile browser (Firefox). - Search for a word 'Decentralized' - pick the second result: 'A decentralized web would give power back to the people online' - On the result's (preview?) page, click the tag 'Decentralization' I get no response, nothing updates, no new tab opens. Not sure what is the intended action (I'd assu…
But because you are still on the preview page you don't immediately see this as it's done in the background. But once you close the preview page (with the big arrow on the top/left of the preview page) you will see the updated result list and number.
Of course, this is a design flaw, we should automatically close the preview window once you select a tag to immediately show you the updated result list. In the desktop version, this is no issue, as there result list and preview window are always simultaneously visible.