Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bible Semantic Search

share.streamlit.io

Post reply on HN
Print view View original

1–10 of 93 posts

Re: Bible Semantic Search

#2

Post by brink »

Very cool! I could see myself really using this. I just wish it wasn't so dependent on SaaS - it tends to make things unreliable, and can disappear at any time.

Re: Bible Semantic Search

#3

Post by JasonFruit »

This is pretty good! I entered, "Every human thought is only evil," and got, among others, the verse I hoped for, Genesis 6:5: "And God saw that the wickedness of man was great in the earth, and that every imagination of the thoughts of his heart was only evil continually." "David cried" got me some unexpected results for "the Son of David", but also the verse about David and Jonathan I expected, and a couple others that were a very good fit.

…and it's gone.

Re: Bible Semantic Search

#7

Post by gk1 »

Credit goes to Chris Lee: https://www.chrisleeportfolio.com

Explanation when you click “Wat this?”:

> This is a Streamlit app I prototyped for performing semantic search on the King James Bible. It conducts full text search as well as semantic search, which is useful for surfacing passages that are similar in meaning to the query, even if the passages don't explicitly contain the query keyword(s). Suppose you wanted to bring up all verses that reference the infamous snake that tempted Eve. In a traditional keyword search system, searching for 'snake' wouldn't yield any results because the KJV uses the term 'serpent'. A semantic search system would take that 'snake' query and retrieve the relevant verses that contain 'serpent' as well as similar verses like ones about reptiles.

> Under the hood, I've generated vector embeddings of every verse in the Bible using SBERT (https://www.sbert.net/), and stored those embeddings in a vector database called Pinecone (https://www.pinecone.io). Every time you submit a query, it's converted to its vector representation using SBERT. That query vector is then sent to Pinecone, which performs an Approximate Nearest Neighbor (https://www.pinecone.io/learn/what-is-similarity-search/) search, retrieving the top n verses that are the most semantically similar to our query. The verses returned are ranked in order of most to least similar.

(Full disclosure: I work for Pinecone, but I have no connection to this demo.)

Re: Bible Semantic Search

#8

Post by eatonphil »

blueberrychpstx wrote: post #6

Quoted post unavailable.

Tried to use it and got:

    AttributeError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app).
    Traceback:
    File "/home/appuser/venv/lib/python3.8/site-packages/streamlit/scriptrunner/script_runner.py", line 475, in _run_script
        exec(code, module.__dict__)
    File "/app/bible-semantic-search/app.py", line 11, in 
        index = PineconeIndex("qa-index")
    File "/app/bible-semantic-search/pinecone_index.py", line 16, in __init__
        self.index = self.connect_to_index(index_name)
    File "/app/bible-semantic-search/pinecone_index.py", line 30, in connect_to_index
        index = pinecone.Index(index_name)
    File "/home/appuser/venv/lib/python3.8/site-packages/pinecone/index.py", line 34, in __init__
        openapi_client_config.api_key = openapi_client_config.api_key or {}

Re: Bible Semantic Search

#9

Post by gk1 »

eatonphil wrote: post #8
blueberrychpstx wrote: post #6

Quoted post unavailable.

Tried to use it and got: AttributeError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app). Traceback: File "/home/appuser/venv/lib/python3.8/site-packages/streamlit/scriptrunner/script_runner.py", line 475, in _run_script exec(code, modu…

Worked for me. Streamlit might be overloaded by HN traffic.

Re: Bible Semantic Search

#10

Post by lalaland1125 »

SBERT isn't trained on this type of archaic English and you can see it failing. It needs to be fine tuned or you should use a modern Bible translation.

A clear example is the query: "homosexuality"

This returns:

> James 2:3 - And ye have respect to him that weareth the gay clothing, and say unto him, Sit thou here in a good place; and say to the poor, Stand thou there, or sit here under my footstool

It's clearly seeing "gay" but is unaware that the meaning has changed.

A classic issue when applying a ML model out of domain.

Post reply on HN