Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MiniLM-L6-v2 maps paragraphs to 384 dimension vector for clustering or search

huggingface.co

Post reply on HN
Print view View original

1–7 of 7 posts

Re: MiniLM-L6-v2 maps paragraphs to 384 dimension vector for clustering or search

#2

Post by PaulHoule »

I got clustering up and running with that in about ten minutes with k-means from scikit learn and wow, all the articles about Ukraine got clustered, all the sports, all the deep learning, all the reinforcement learning. It is way faster than my LDA clustering engine and I think gets better results. See sbert.net for a library that puts it on your fingertips.

(With DBSCAN it shows great promise for clustering news articles about the same topic..)

Re: MiniLM-L6-v2 maps paragraphs to 384 dimension vector for clustering or search

#3

Post by rahimnathwani »

Yesterday I read about this embedding model that can be run locally.

I read about it here: https://news.ycombinator.com/item?id=35236275

But it's been mentioned on HN a few times in the past 3 months:

https://news.ycombinator.com/item?id=35055808

https://news.ycombinator.com/item?id=34691307

https://news.ycombinator.com/item?id=34034940

https://news.ycombinator.com/item?id=34007942

It can be used as am embedding model in llama_index (fka gpt_index). Here's a code snippet from one of their users:

  from langchain.embeddings.huggingface import HuggingFaceEmbeddings
  from llama_index import LangchainEmbedding

  def get_embed_model():
      model_name = "sentence-transformers/all-MiniLM-L6-v2"
      embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name=model_name))
      return embed_model

  get_embed_model()

  index = GPTSimpleVectorIndex(documents, embed_model=embed_model)

Re: MiniLM-L6-v2 maps paragraphs to 384 dimension vector for clustering or search

#4

Post by sharemywin »

PaulHoule wrote: post #2

I got clustering up and running with that in about ten minutes with k-means from scikit learn and wow, all the articles about Ukraine got clustered, all the sports, all the deep learning, all the reinforcement learning. It is way faster than my LDA clustering engine and I think gets better results. See sbert.net for a library that puts it on your fingertips. (With DBSCAN it shows great promise for clustering news art…

found this on their site thanks for the reference.

https://www.sbert.net/docs/pretrained_models.html

Re: MiniLM-L6-v2 maps paragraphs to 384 dimension vector for clustering or search

#6

Post by PaulHoule »

sharemywin wrote: post #5

anybody have thoughts on this: https://engineering.fb.com/2017/03/29/data-infrastructure/fa... for doing the similarity search? I've also be thinking about looking into this: https://milvus.io/

Faiss is great.

Re: MiniLM-L6-v2 maps paragraphs to 384 dimension vector for clustering or search

#7

Post by monatis »

sharemywin wrote: post #5

anybody have thoughts on this: https://engineering.fb.com/2017/03/29/data-infrastructure/fa... for doing the similarity search? I've also be thinking about looking into this: https://milvus.io/

Similarity search becomes a part of many technical stacks. Qdrant can be deployed as a service for fast, lightweight and filterable vector search. It's opensource, and you can also get up and running instantly and for free wit the cloud offering. You can try here https://qdrant.tech
Post reply on HN