Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Velvet – Store OpenAI requests in your own DB

usevelvet.com

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Show HN: Velvet – Store OpenAI requests in your own DB

#21

Post by pedrosorio »

OutOfHere wrote: post #12
DeveloperErrata wrote: post #9

Seems neat - I'm not sure if you do anything like this but one thing that would be useful with RAG apps (esp at big scales) is vector based search over cache contents. What I mean is that, users can phrase the same question (which has the same answer) in tons of different ways. If I could pass a raw user query into your cache and get back the end result for a previously computed query (even if the current phrasing is…

That would totally destroy the user experience. Users change their query so they can get a refined result, not so they get the same tired result.

Even across users it’s a terrible idea.

Even in the simplest of applications where all you’re doing is passing “last user query” + “retrieved articles” into openAI (and nothing else that is different between users, like previous queries or user data that may be necessary to answer), this will be a bad experience in many cases.

Queries A and B may have similar embeddings (similar topic) and it may be correct to retrieve the same articles for context (which you could cache), but they can still be different questions with different correct answers.

Re: Show HN: Velvet – Store OpenAI requests in your own DB

#24

Post by knowaveragejoe »

OutOfHere wrote: post #6

A cache is better when it's local rather than on the web. And I certainly don't need to pay anyone to cache local request responses.

How would one achieve something similarly locally, short of just running a proxy and stuffing the request/response pairs into a DB? I'm sure it wouldn't be too terribly hard to write something, but I figure something open source already exists for OpenAI-compatible APIs.

Re: Show HN: Velvet – Store OpenAI requests in your own DB

#25

Post by knowaveragejoe »

elawler24 wrote: post #7
hiatus wrote: post #3

This seems to require sharing our data we provide to OpenAI with yet another party. I don't see any zero-retention offering.

The self-serve version is hosted (it’s easy to try locally), but we offer managed deployments where you bring your own DB. In this case your data is 100% yours, in your PostgreSQL. That’s how Find AI uses Velvet.

Where is this mentioned? Is there a github(etc) somewhere that someone can use this without using the hosted version?

Re: Show HN: Velvet – Store OpenAI requests in your own DB

#26

Post by w-ll »

knowaveragejoe wrote: post #24
OutOfHere wrote: post #6

A cache is better when it's local rather than on the web. And I certainly don't need to pay anyone to cache local request responses.

How would one achieve something similarly locally, short of just running a proxy and stuffing the request/response pairs into a DB? I'm sure it wouldn't be too terribly hard to write something, but I figure something open source already exists for OpenAI-compatible APIs.

Recently did this workflow.

Started with nginx proxy with rules to cache base on url/params. Wanted more control over it and explored lua/redis apis, and opted to build a app to do be a little more smart for what i wanted. Extra ec2 cost is negligible compared to cache savings.

Re: Show HN: Velvet – Store OpenAI requests in your own DB

#28

Post by doubleorseven »

w-ll wrote: post #26
knowaveragejoe wrote: post #24

Earlier quoted context omitted.

How would one achieve something similarly locally, short of just running a proxy and stuffing the request/response pairs into a DB? I'm sure it wouldn't be too terribly hard to write something, but I figure something open source already exists for OpenAI-compatible APIs.

Recently did this workflow. Started with nginx proxy with rules to cache base on url/params. Wanted more control over it and explored lua/redis apis, and opted to build a app to do be a little more smart for what i wanted. Extra ec2 cost is negligible compared to cache savings.

Yes! It's amazing how many things you can do with lua in nginx. I had a server that served static websites where the files and the certificates for each website were stored in a bucket. Over 20k websites with 220ms overhead if the certificate wasn't cached.
Post reply on HN