Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shapeshift: Semantically map JSON objects using key-level vector embeddings

github.com

Post reply on HN
Print view View original

21–30 of 37 posts

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#21

Post by PaulHoule »

simonw wrote: post #9

This is the code that does the work: https://github.com/rectanglehq/Shapeshift/blob/d954dab2a866c... There are a few ways this could be made a less expensive to run: 1. Cache those embeddings somewhere. You're only embedding simple strings like "name" and "address" - no need to do that work more than once in an entire lifetime of running the tool. 2. As suggested here https://news.ycombinator.com/item?id=40973028 cha…

I was involved in an attempt to do this kind of thing with CNN neural networks just around the time BERT came out that was mostly successful and actually we did great projects for companies in the beverages, telecom, aviation and consumer goods space.

It worked because it also had a conventional data-processing pipeline that revolved around JSON documents.

For (2) it seems a system like that should be able to generate a script in Python, a codesigned DSL or some other language to do the conversion.

One interesting thing about the product I worked on was that it functioned as a profiler by looking at one cell at a time, so if there is some field that has "Gruff Rhys" or "范冰冰" it could tell that was probably somebody's name, all the better if it can also see the field label is something like "Full Name" or "姓名". I'd contrast that to more conventional column-based profilers who might noticed that a certain field only has the values "true" and "false" throughout the whole column and would probably have some rule that would determine it was a boolean field.

One thing that system could do is recognize private data inside unstructured data. Where I work for instance we have

https://www.spirion.com/sensitive-data-discovery

which scans text and other files and it warns if it sees something like a lot of personal data, like an Excel spreadsheet full of names, addresses and phone numbers -- even if I just made them up as test data.

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#22

Post by btown »

simonw wrote: post #9

This is the code that does the work: https://github.com/rectanglehq/Shapeshift/blob/d954dab2a866c... There are a few ways this could be made a less expensive to run: 1. Cache those embeddings somewhere. You're only embedding simple strings like "name" and "address" - no need to do that work more than once in an entire lifetime of running the tool. 2. As suggested here https://news.ycombinator.com/item?id=40973028 cha…

> returns a reusable data structure mapping input keys to output keys

IMO this use case is exactly what Copilot is for. Write a comment including one example each of input and output, and tab-complete in your language of choice to have it create a rewriter for you.

One benefit (and danger) is that it will look at the values, not just the keys, and also may generate arbitrary code that can e.g. adapt a firstName and lastName to a fullName. But that's why you have a human being triggering and auditing this for subtle bugs, and putting it through code review and source control, right?

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#23

Post by anamexis »

WhatsName wrote: post #2

Maybe I'm not the target audience, but here are simple questions to the author or potential users: What about anything more complex like date of birth to age or the other way round? Also since we will inevitably incur costs, why not let a llm write a transformation rule for us?

It's not using an LLM, it's just comparing embeddings (which are waaay cheaper)

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#24

Post by anamexis »

yetanotherjosh wrote: post #16

So this identifies keys from source and target objects that are fuzzy synonyms and copies the values over. What is a real world use case for this? Add the fact that it's fuzzy and won't always work, so would require a great deal of extra effort in QA/testing (harder than just mapping the keys programmatically), and I'm puzzled.

We do something very similar with embeddings in our product. Users import files that they have to match to a dynamically-defined target schema. The embedding matching provides suggested matches to the user that are generally very accurate, so they don't have to go through and manually match up "telephone" to "phone number" etc. It even works across languages.

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#26

Post by momojo »

anamexis wrote: post #24
yetanotherjosh wrote: post #16

So this identifies keys from source and target objects that are fuzzy synonyms and copies the values over. What is a real world use case for this? Add the fact that it's fuzzy and won't always work, so would require a great deal of extra effort in QA/testing (harder than just mapping the keys programmatically), and I'm puzzled.

We do something very similar with embeddings in our product. Users import files that they have to match to a dynamically-defined target schema. The embedding matching provides suggested matches to the user that are generally very accurate, so they don't have to go through and manually match up "telephone" to "phone number" etc. It even works across languages.

How much time dos this save your users? Is this QOL? Or more of a "our product wouldn't work without this feature" kind of thing?

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#27

Post by anamexis »

momojo wrote: post #26
anamexis wrote: post #24

Earlier quoted context omitted.

We do something very similar with embeddings in our product. Users import files that they have to match to a dynamically-defined target schema. The embedding matching provides suggested matches to the user that are generally very accurate, so they don't have to go through and manually match up "telephone" to "phone number" etc. It even works across languages.

How much time dos this save your users? Is this QOL? Or more of a "our product wouldn't work without this feature" kind of thing?

Quite a bit of time. The product would still work without the feature, but it is a major feature. It bypasses lots of wading through dropdowns (potentially dozens for a single session)

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#28

Post by slantedview »

marvinkennis wrote: post #10
simonw wrote: post #9

This is the code that does the work: https://github.com/rectanglehq/Shapeshift/blob/d954dab2a866c... There are a few ways this could be made a less expensive to run: 1. Cache those embeddings somewhere. You're only embedding simple strings like "name" and "address" - no need to do that work more than once in an entire lifetime of running the tool. 2. As suggested here https://news.ycombinator.com/item?id=40973028 cha…

Thanks for the suggestions! Will implement these. Caching is a great idea.

In general, you might cross reference with other object mapping libraries (including in other languages) to get ideas on how they approach this problem. Caching mappings is just one common strategy.

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#29

Post by leobg »

The example could be handled with no machine learning at all. Just use a bag of words comparison with a subword tokenizer. And if you do need embeddings (to map synonyms/topics), fastText is faster, cheaper and runs locally. For hard cases, you can feed the source/target schemas to gpt-4o once to create a map - and then apply that one map to all instances.

Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings

#30

Post by magicalhippo »

anamexis wrote: post #24
yetanotherjosh wrote: post #16

So this identifies keys from source and target objects that are fuzzy synonyms and copies the values over. What is a real world use case for this? Add the fact that it's fuzzy and won't always work, so would require a great deal of extra effort in QA/testing (harder than just mapping the keys programmatically), and I'm puzzled.

We do something very similar with embeddings in our product. Users import files that they have to match to a dynamically-defined target schema. The embedding matching provides suggested matches to the user that are generally very accurate, so they don't have to go through and manually match up "telephone" to "phone number" etc. It even works across languages.

I've got some similar use-cases. So, do I understand correctly that you take the source keyword and generate an embedding vector of it, then compare it using dot-product similarity or something to the embedded vectors of the target keywords?
Post reply on HN