Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Add search to your Jekyll blog

jzhang.io

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Add search to your Jekyll blog

#11

Post by StavrosK »

hedgehog wrote: post #10
StavrosK wrote: post #5

This got me thinking: The obvious way to do full-text search in the post body would be to build an index of {"word": [post_id, post_id, post_id]} of posts it appears in. However, this could be huge. Does anyone know if there's a technique that uses a list of posts and a bloom filter that contains all the words in that post? I.e. you iterate over all the posts and check the bloom filter for membership of all the terms…

Very nice. You will save some space and allow for some typos if you stem and soundex before insertion. Also you can save space and improve the run time somewhat if rather than many separate bloom filters you build one large one where each item is post ID + word. If you do that you can also insert each word bare so you get O(1) empty result sets, helpful if you're updating the results with every keystroke in a search…

Huh, very nice idea! That should, indeed save a ton of space and be much simpler when searching! I'll try that now, thank you.

EDIT: Hmm, turns out it's pretty much the same size, which makes sense, I guess: http://nbviewer.ipython.org/gist/skorokithakis/0abbfebced25f...

Re: Add search to your Jekyll blog

#12

Post by comice »

swanson wrote: post #3

Here's another approach: Make a templated JSON file that mimics a database-backend. When you generate the site (or just push to a GitHub Pages repo), the JSON file will be populated with the "search index" of your site. Something like this: https://github.com/swanson/lagom/blob/master/site.json Then you can use whatever Javascript you'd like to search through your pages by pulling down the JSON file into memory and s…

pretty much the approach taken with lunr.js (or you can store the search index itself, which is also json. http://lunrjs.com/

Re: Add search to your Jekyll blog

#13

Post by hedgehog »

StavrosK wrote: post #11
hedgehog wrote: post #10

Earlier quoted context omitted.

Very nice. You will save some space and allow for some typos if you stem and soundex before insertion. Also you can save space and improve the run time somewhat if rather than many separate bloom filters you build one large one where each item is post ID + word. If you do that you can also insert each word bare so you get O(1) empty result sets, helpful if you're updating the results with every keystroke in a search…

Huh, very nice idea! That should, indeed save a ton of space and be much simpler when searching! I'll try that now, thank you. EDIT: Hmm, turns out it's pretty much the same size, which makes sense, I guess: http://nbviewer.ipython.org/gist/skorokithakis/0abbfebced25f...

The space savings for the same error rate should be small (I think the likelihood of false positives for a given load goes down slightly with size of the filter) but the benefit in lookup time should be significant for multiword searches. Thinking about it more though if you're doing live search you'll already have computed the results for the first word by the time you are given a second so maybe it doesn't matter.

Re: Add search to your Jekyll blog

#14

Post by StavrosK »

hedgehog wrote: post #13
StavrosK wrote: post #11

Earlier quoted context omitted.

Huh, very nice idea! That should, indeed save a ton of space and be much simpler when searching! I'll try that now, thank you. EDIT: Hmm, turns out it's pretty much the same size, which makes sense, I guess: http://nbviewer.ipython.org/gist/skorokithakis/0abbfebced25f...

The space savings for the same error rate should be small (I think the likelihood of false positives for a given load goes down slightly with size of the filter) but the benefit in lookup time should be significant for multiword searches. Thinking about it more though if you're doing live search you'll already have computed the results for the first word by the time you are given a second so maybe it doesn't matter.

I think it'll be faster to do multiple filters, because the one-filter way requires hashing and comparing N times while the multiple-filter way requires hashing once and comparing N times.

Re: Add search to your Jekyll blog

#16

Post by jameszhang »

paukiatwee wrote: post #15

My Jekyll blog using lunr js for fulltext search with JSON file as backend. Check it out here. http://dreamand.me/web/fulltext-search-at-jekyll-site/ and lunr js https://github.com/olivernn/lunr.js

That's neat, but is something wrong? I get this error on your site: https://cloudup.com/cti8Egu8fQS

Re: Add search to your Jekyll blog

#17

Post by jameszhang »

suter wrote: post #4

We implemented search on our help site ( http://help.simpletax.ca ) using the excellent lunr.js ( http://lunrjs.com ) + this Jekyll plug-in ( https://github.com/slashdotdash/jekyll-lunr-js-search ). The search index is updated when you build your Jekyll site, so it's a piece of cake to maintain and gives you full text search. I don't know how well it would scale, but if you are in the hundreds of posts, you should be…

That's awesome, especially for a help site. My company does the same thing with our documentations.

Re: Add search to your Jekyll blog

#18

Post by hedgehog »

StavrosK wrote: post #14
hedgehog wrote: post #13

Earlier quoted context omitted.

The space savings for the same error rate should be small (I think the likelihood of false positives for a given load goes down slightly with size of the filter) but the benefit in lookup time should be significant for multiword searches. Thinking about it more though if you're doing live search you'll already have computed the results for the first word by the time you are given a second so maybe it doesn't matter.

I think it'll be faster to do multiple filters, because the one-filter way requires hashing and comparing N times while the multiple-filter way requires hashing once and comparing N times.

Oops, you are correct.
Post reply on HN