Shapeshift: Semantically map JSON objects using key-level vector embeddings

Shapeshift: Semantically map JSON objects using key-level vector embeddings
github.com

If instead of transforming a single object it would output a table of src_field->dst_field, it could potentially be a useful first pass in some ETL development.

There are a few ways this could be made a less expensive to run:

1. Cache those embeddings somewhere. You're only embedding simple strings like "name" and "address" - no need to do that work more than once in an entire lifetime of running the tool.

2. As suggested here https://news.ycombinator.com/item?id=40973028 change the design of the tool so instead of doing the work it returns a reusable data structure mapping input keys to output keys, so you only have to run it once and can then use that generated data structure to apply the transformations on large amounts of data in the future.

3. Since so many of the keys are going to have predictable names ("name", "address" etc) you could even pre-calculate embeddings for the 1,000 most common keys across all three embedding providers and ship those as part of the package.

Also: in https://github.com/rectanglehq/Shapeshift/blob/d954dab2a866c... you're using Promise.map() to run multiple embeddings through the OpenAI API at once, which risks tripping their rate-limit. You should be able to pass the text as an array in a single call instead, something like this:

        const response = await this.openai!.embeddings.create({
          model: this.embeddingModel,
          input: texts,
          encoding_format: "float",
        });
        return response.data.map(item => item.embedding);