Since LLMs are bad at the null hypothesis (in this case, when a key does not exist in the source JSON), how does this prevent hallucinating transformations for missing keys?
Shapeshift: Semantically map JSON objects using key-level vector embeddings
11–20 of 37 posts
Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings
#12This is the code that does the work: https://github.com/rectanglehq/Shapeshift/blob/d954dab2a866c... There are a few ways this could be made a less expensive to run: 1. Cache those embeddings somewhere. You're only embedding simple strings like "name" and "address" - no need to do that work more than once in an entire lifetime of running the tool. 2. As suggested here https://news.ycombinator.com/item?id=40973028 cha…
Watch out with the array mode though, according to OpenAI docs it technically can return the results in any order and you must sort them by index to be sure you have the right associations. I’ve never seen them out of order in practice, but it’d be entirely in-character for them to suddenly change that sporadically and without warning, and now your entire vectordb may or may not be nondeterministically ruined.
Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings
#13This is the code that does the work: https://github.com/rectanglehq/Shapeshift/blob/d954dab2a866c... There are a few ways this could be made a less expensive to run: 1. Cache those embeddings somewhere. You're only embedding simple strings like "name" and "address" - no need to do that work more than once in an entire lifetime of running the tool. 2. As suggested here https://news.ycombinator.com/item?id=40973028 cha…
Watch out with the array mode though, according to OpenAI docs it technically can return the results in any order and you must sort them by index to be sure you have the right associations. I’ve never seen them out of order in practice, but it’d be entirely in-character for them to suddenly change that sporadically and without warning, and now your entire vectordb may or may not be nondeterministically ruined.
Yikes!
Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings
#14What’d be really great is a codegen aspect. A non-negligible part of any data munching operation is “this input object has fields X, Y, Z and we need an output object with fields X, f(X), Y, f(Y,Z)”. This is something and LLM has a decent chance at being really quite good at.
Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings
#15This task in the most general form is better done with question answering prompt than embeds. How do you solve "Full Name" -> "First Name", "Last Name" with embeds? QA is the right level of abstraction for schema conversion tasks. And it's simple, just put the source JSON + target JSON schema in the prompt and ask for value extraction.
Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings
#16So this identifies keys from source and target objects that are fuzzy synonyms and copies the values over. What is a real world use case for this? Add the fact that it's fuzzy and won't always work, so would require a great deal of extra effort in QA/testing (harder than just mapping the keys programmatically), and I'm puzzled.
Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings
#17Since LLMs are bad at the null hypothesis (in this case, when a key does not exist in the source JSON), how does this prevent hallucinating transformations for missing keys?
[deleted]
Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings
#18Created a Rust version using devin.ai. (untested)
Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings
#19Since LLMs are bad at the null hypothesis (in this case, when a key does not exist in the source JSON), how does this prevent hallucinating transformations for missing keys?
This isn't using an LLM, it simply checks for similarity between keys using vector embeddings
Re: Shapeshift: Semantically map JSON objects using key-level vector embeddings
#20Put together a quick version with an LLM, using Substrate: https://www.val.town/v/substrate/shapeshift
I've turned the target object into a JSON schema, but you could probably generate that JSON schema pretty reliably using a codegen LLM.