Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: FiddleCube – Generate Q&A to test your LLM

github.com

Post reply on HN
Print view View original

1–10 of 19 posts

Show HN: FiddleCube – Generate Q&A to test your LLM

#1

Post by kaushik92 »

Convert your vector embeddings into a set of questions and their ideal responses. Use this dataset to test your LLM and catch failures caused by prompt or RAG updates.

Get started in 3 lines of code:

```

pip3 install fiddlecube

```

```

from fiddlecube import FiddleCube

fc = FiddleCube(api_key="") dataset = fc.generate( [ "The cat did not want to be petted.", "The cat was not happy with the owner's behavior.", ], 10, ) dataset

```

Generate your API key: https://dashboard.fiddlecube.ai/api-key

# Ideal QnA datasets for testing, eval and training LLMs

Testing, evaluation or training LLMs requires an ideal QnA dataset aka the golden dataset.

This dataset needs to be diverse, covering a wide range of queries with accurate responses.

Creating such a dataset takes significant manual effort.

As the prompt or RAG contexts are updated, which is nearly all the time for early applications, the dataset needs to be updated to match.

# FiddleCube generates ideal QnA from vector embeddings

- The questions cover the entire RAG knowledge corpus.

- Complex reasoning, safety alignment and 5 other question types are generated.

- Filtered for correctness, context relevance and style.

- Auto-updated with prompt and RAG updates.

Show HN: FiddleCube – Generate Q&A to test your LLM
github.com

Re: Show HN: FiddleCube – Generate Q&A to test your LLM

#5

Post by kaushik92 »

johnsutor wrote: post #4

How does this differ from Ragas? https://docs.ragas.io/en/latest/index.html

Ragas is an eval tool which needs ground truths and queries for evaluation. FiddleCube generates the queries and the ground truth needed for eval in Ragas, LangSmith or an eval tool of choice.

We incorporate user prompts to generate the outputs and provide diagnostics and feedback for improvement, rather than eval metrics. So you can plug your low scored queries provided by Ragas, your prompt and context. FiddleCube can provide the root cause and the ideal response.

This is an alternative to manual auditing and testing, where an auditor works on curating the ideal dataset.

Re: Show HN: FiddleCube – Generate Q&A to test your LLM

#6

Post by cruxcode »

neha_n wrote: post #3
cruxcode wrote: post #2

Can it generate HTML as part of prompt?

Can you elaborate on the use case a bit? HTML as a part of the prompt for what kind of use case.

I am scraping some information from a list of company's website. I would like to create a evaluation set for my agent.

Re: Show HN: FiddleCube – Generate Q&A to test your LLM

#10

Post by michalf6 »

kaushik92 wrote: post #5
johnsutor wrote: post #4

How does this differ from Ragas? https://docs.ragas.io/en/latest/index.html

Ragas is an eval tool which needs ground truths and queries for evaluation. FiddleCube generates the queries and the ground truth needed for eval in Ragas, LangSmith or an eval tool of choice. We incorporate user prompts to generate the outputs and provide diagnostics and feedback for improvement, rather than eval metrics. So you can plug your low scored queries provided by Ragas, your prompt and context. FiddleCube…

Ragas also has a feature to generate ground truths and queries: https://docs.ragas.io/en/latest/getstarted/testset_generatio... Although simply prompting an LLM with chunks of source documents might work better / cheaper - ragas tends to explode with retries in my experience.
Post reply on HN