While serving a mission for my church, a fellow member bought me a copy of “Strongs Exhaustive Concordance of the Bible”. I found it really fascinating to flip through correlated terms and be able to draw conceptual lines between them. It was the first time I’d ever come across something like that. It strikes me as something that is relatively rare simply because what the Bible is, in terms of popularity and commonal…
Bible Semantic Search
41–50 of 93 posts
Re: Bible Semantic Search
#42Earlier quoted context omitted.
Couldn't they search in a modern translation and just refer back to the ancient one? Seems like the way bible verses are split up should remove the ability of translation to move anything around much.
Ah, that's a good idea. I was considering just fully replacing the KJV with the NIV, but for people who want the KJV, having a mapping between the two versions would work. Do the verses between the two versions map each other exactly? Like they cover the same exact thing, just written in a different style?
To do this properly, you have to map versifications, because some systems use different conventions. As the most significant and pervasive example, some systems treat the title of psalms as verse one (French generally does this), while others treat it as separate (verse zero, effectively; English generally does this), so if you just reuse numbers across such transitions you’ll get consistent off-by-one errors. There are quite a few other similar off-by-one errors, and occasionally more, throughout.
https://wiki.crosswire.org/Alternate_Versification is a decent starting point for looking into the topic, with SWORD’s canon_.h files fairly tolerable for showing the number of verses in each chapter. Unfortunately, SWORD has never gone as far as doing proper mapping* between versifications for some reason—they have some basic mapping somewhere or other, but I can’t remember offhand where or what it is, as I only briefly looked into it five years or so ago.
The most significant differences occur when you switch languages (there are quite a few differences if you switch from English to French or to many Indic languages), but there may be some differences between translations within a language too, e.g. SWORD’s NRSV versification has an extra verse in 3 John and Revelation 12 compared to its KJV versification.
Re: Bible Semantic Search
#43SBERT isn't trained on this type of archaic English and you can see it failing. It needs to be fine tuned or you should use a modern Bible translation. A clear example is the query: "homosexuality" This returns: > James 2:3 - And ye have respect to him that weareth the gay clothing, and say unto him, Sit thou here in a good place; and say to the poor, Stand thou there, or sit here under my footstool It's clearly seei…
Hey, creator of the Bible Semantic Search app here. I 100% agree with you. I hacked together this prototype for the purposes of learning the Pinecone API rather than fine-tuning a language model, but I'm still pleasantly surprised by the quality of the search results despite all the shortcomings you mentioned. The results aren't great, but they're decent. I'm surprised how well SBERT works out of the box considering…
I recommend the WEB as a public domain translation. The likes of the NIV and ESV will be troublesome for licensing reasons. The WEB isn’t my favourite translation (I grew up with the RSV and prefer the RSV or ESV), a bit clumsier than the ASV from which it is mostly springs (with changes both missing and extraneous), but it’s public domain.
Re: Bible Semantic Search
#44Earlier quoted context omitted.
Couldn't they search in a modern translation and just refer back to the ancient one? Seems like the way bible verses are split up should remove the ability of translation to move anything around much.
Ah, that's a good idea. I was considering just fully replacing the KJV with the NIV, but for people who want the KJV, having a mapping between the two versions would work. Do the verses between the two versions map each other exactly? Like they cover the same exact thing, just written in a different style?
Just remember the NIV translation of The Bible is property of Harper Collins, who is also amusingly the publisher of many tabloids and The Satanic Bible.
I used to distribute a BibleGateway-ripped copy of NIV as a plugin for the open source GnomeSword project and managed to upset both Harper Collins and the open source scholar community. It was absurd and I refused to stop. I dared them to sue me for sharing the Bible as open source... the headlines would write themselves.
FWIIW they never took the bait, but expect empty threat Cease and Desist letters from the Harper Collins legal team.
Re: Bible Semantic Search
#45No post body was provided.
Re: Bible Semantic Search
#46Why with a 400 year-old translation though?
Re: Bible Semantic Search
#47Hmm. Searching “baby” returns nothing but references to Babylon. Not sure why this was my first and only search, less clear why it is doing partial text matching.
The app performs full text search as well as semantic search. The full text search results are presented first, so if you scroll down to the bottom of the page you should see the semantically related results under the header "Semantic Search Results"
Re: Bible Semantic Search
#48Why with a 400 year-old translation though?
I figured it was the quintessential version of the Bible ¯\_(ツ)_/¯. I'm not religious so cut me some slack haha. Looking back, I should've used a version using modern English.
Re: Bible Semantic Search
#49Credit goes to Chris Lee: https://www.chrisleeportfolio.com Explanation when you click “Wat this?”: > This is a Streamlit app I prototyped for performing semantic search on the King James Bible. It conducts full text search as well as semantic search, which is useful for surfacing passages that are similar in meaning to the query, even if the passages don't explicitly contain the query keyword(s). Suppose you wanted…
How does pinecone compare to pgvector? https://github.com/pgvector/pgvector
In terms of performance, I'm not sure, although I'm willing to bet on Pinecone as they have their own proprietary indexing algorithm. I can speak in terms of developer experience though, as I did use Pinecone to build this app. One obvious but important difference is that with pgvector, you need to spin up and self host a postgres server on your own. Pinecone provides a batteries-included, fully managed experience. Also, it seems that a lot of important operations in pgvector are conducted with SQL statements. With Pinecone, you don't need to deal with SQL or with any ORMs.
In terms of this project, I would not have chosen pgvector. I don't want to deal with the PITA that comes with manually setting up and self hosting a vector database. When I'm building a demo or a prototype, I care about speed of development, which lets me more effectively explore the possibility space of whatever I'm building. I'm not a database admin, so when I deploy the project, I don't want to do database admin tasks. The ease of use of Pinecone's API lets me move fast, and it was very intuitive to learn. One downside of Pinecone is that although they have a generous free tier, their next highest tier is $50/month (and that's the low end of that tier). This is unfortunate for solo devs like myself who are likely to graduate from the free tier and would be willing to pay a bit more for a higher tier, but find the $50/month plan to be overkill. I think they're trying to target startups with that $50/month plan.
Re: Bible Semantic Search
#50Hmm. I entered "kingdom of heaven" - and it returned a ProtocolError
Seems kind of appropriate. Obviously it is easier for a camel to thread the eye of a needle than to get a semantic search on the Kingdom of Heaven.