This is super cool man!
Thanks! We would appreciate any feedback or feature requests!
Show HN: PromptTools – open-source tools for evaluating LLMs and vector DBs
21–25 of 25 posts
Re: Show HN: PromptTools – open-source tools for evaluating LLMs and vector DBs
#22I like that it's not limited to single prompts and allows to have chat messages.
It would be great if `OpenAIChatExperiment` could also handle OpenAI's function calling.
Re: Show HN: PromptTools – open-source tools for evaluating LLMs and vector DBs
#23I like that it's not limited to single prompts and allows to have chat messages. It would be great if `OpenAIChatExperiment` could also handle OpenAI's function calling.
Good catch! We're looking to add function calling support very soon, and have an open issue for it on our GitHub. If you want to raise a PR and add it, we'll help you land it and get it merged
Re: Show HN: PromptTools – open-source tools for evaluating LLMs and vector DBs
#24I'll put in a friendly request for a Dockerfile in the repo.
I've been trying out AI tools as test cases for our supply chain security platform and had to cobble a Dockerfile together to get this running easily. Really cool tool overall!
Across 200+ transitive dependencies in prompttool, risk prioritization can remove 97% of security investigation in my quick test, and I most of these came from a thick base image. I'd love one curated from y'all.
Re: Show HN: PromptTools – open-source tools for evaluating LLMs and vector DBs
#25I'd like to see support for qdrant.
We've actually been in contact with the qdrant team about adding it to our roadmap! Andre (CEO) was asking for an integration. If you want to work on the PR, we'd be happy to work with you and get that merged in
Qdrant here! We're already working on that :D