Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shapeshift: Semantically map JSON objects using key-level vector embeddings

github.com

Post reply on HN
Print view View original

31–37 of 37 posts

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#31

Post by anamexis »

magicalhippo wrote: post #30
anamexis wrote: post #24

Earlier quoted context omitted.

We do something very similar with embeddings in our product. Users import files that they have to match to a dynamically-defined target schema. The embedding matching provides suggested matches to the user that are generally very accurate, so they don't have to go through and manually match up "telephone" to "phone number" etc. It even works across languages.

I've got some similar use-cases. So, do I understand correctly that you take the source keyword and generate an embedding vector of it, then compare it using dot-product similarity or something to the embedded vectors of the target keywords?

Exactly, although we use cosine similarity.

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#32

Post by magicalhippo »

anamexis wrote: post #31
magicalhippo wrote: post #30

Earlier quoted context omitted.

I've got some similar use-cases. So, do I understand correctly that you take the source keyword and generate an embedding vector of it, then compare it using dot-product similarity or something to the embedded vectors of the target keywords?

Exactly, although we use cosine similarity.

Perfect. And yeah that's what I meant, so used to just normalizing vectors so dot product = cosine.

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#33

Post by riku_iki »

leobg wrote: post #29

The example could be handled with no machine learning at all. Just use a bag of words comparison with a subword tokenizer. And if you do need embeddings (to map synonyms/topics), fastText is faster, cheaper and runs locally. For hard cases, you can feed the source/target schemas to gpt-4o once to create a map - and then apply that one map to all instances.

> fastText is faster, cheaper and runs locally

the question is if quality will be acceptable

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#34

Post by riku_iki »

anamexis wrote: post #23
WhatsName wrote: post #2

Maybe I'm not the target audience, but here are simple questions to the author or potential users: What about anything more complex like date of birth to age or the other way round? Also since we will inevitably incur costs, why not let a llm write a transformation rule for us?

It's not using an LLM, it's just comparing embeddings (which are waaay cheaper)

but embeddings came from somewhere (LLM?).

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#35

Post by flysand7 »

riku_iki wrote: post #33
leobg wrote: post #29

The example could be handled with no machine learning at all. Just use a bag of words comparison with a subword tokenizer. And if you do need embeddings (to map synonyms/topics), fastText is faster, cheaper and runs locally. For hard cases, you can feed the source/target schemas to gpt-4o once to create a map - and then apply that one map to all instances.

> fastText is faster, cheaper and runs locally the question is if quality will be acceptable

The question if machine learning algorithm's produced embeddings will have the acceptable quality too. With a library I presume that the quality is at least predictable. I personally have less trust in machine learning though

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#36

Post by riku_iki »

flysand7 wrote: post #35
riku_iki wrote: post #33

Earlier quoted context omitted.

> fastText is faster, cheaper and runs locally the question is if quality will be acceptable

The question if machine learning algorithm's produced embeddings will have the acceptable quality too. With a library I presume that the quality is at least predictable. I personally have less trust in machine learning though

> The question if machine learning algorithm's produced embeddings will have the acceptable quality too

there are tons of benchmarks and results which demonstrated that embeddings from language models are superior to word2vec in (almost) all scenarios.

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#37

Post by srean »

leobg wrote: post #29

The example could be handled with no machine learning at all. Just use a bag of words comparison with a subword tokenizer. And if you do need embeddings (to map synonyms/topics), fastText is faster, cheaper and runs locally. For hard cases, you can feed the source/target schemas to gpt-4o once to create a map - and then apply that one map to all instances.

BTW Bag of words models were once considered ML not too long ago.
Post reply on HN