71–80 of 124 posts
Earlier quoted context omitted.
I don’t think OpenAI train on data processed via the API, unless there’s an exception specifically for this.
Can you truly trust them though?
But seriously what an amazing write up, plus animations, analysis etc etc. Bravo.
It was also ironic to see AWS failing quite a few use cases here. Stuff to think about.
"There was one surprise when I revisited costs: OpenAI charges an unusually low $0.0001 / 1M tokens for batch inference on their latest embedding model. Even conservatively assuming I had 1 billion crawled pages, each with 1K tokens (abnormally long), it would only cost $100 to generate embeddings for all of them. By comparison, running my own inference, even with cheap Runpod spot GPUs, would cost on the order of 10…
A vector-only search engine will fail for a lot of common use cases where the keywords do matter. I tried searching for `garbanzo bean stew` and got totally irrelevant bean recipes.
"There was one surprise when I revisited costs: OpenAI charges an unusually low $0.0001 / 1M tokens for batch inference on their latest embedding model. Even conservatively assuming I had 1 billion crawled pages, each with 1K tokens (abnormally long), it would only cost $100 to generate embeddings for all of them. By comparison, running my own inference, even with cheap Runpod spot GPUs, would cost on the order of 10…
Is this the drug dealer scheme? Get you hooked later jack up prices? After all, the alternative would be regenerating all your embeddings no?
The title should be “10x engineer creates Google in their spare time” But seriously what an amazing write up, plus animations, analysis etc etc. Bravo. It was also ironic to see AWS failing quite a few use cases here. Stuff to think about.
> SQS had very low concurrent rate limits that could not keep up with the throughput of thousands of workers across the pipeline.
I could not find this perhaps the author meant Lambda limits?
> services like S3 have quite low rate limits — there are hard limits, but also dynamic per-account/bucket quotas
You have virtually unlimited throughput with prefix partitions