Why is promptfoo so popular?

#1

Post by dabs » Sat, Jul 27, 2024, 3:55 AM UTC

Re: Why is promptfoo so popular?

#2

Post by dabs » Sat, Jul 27, 2024, 3:55 AM UTC

I have seen frameworks in other spaces attempt the configuration driven approach and have seen it always fall out of favor vs. programmatic approaches. I am reading the docs right now and I feel like it's so much work to try to learn the syntax of their YAML specification vs. using something like LangFuse. The fact that you can write code as an array element in a YAML low key makes me angry.

Yet this is used in Shopify, Microsoft, etc. supposedly. Are there people here who have chosen promptfoo over other alternatives? Why?

Re: Why is promptfoo so popular?

#3

Post by dabs » Sat, Jul 27, 2024, 4:08 AM UTC

dabs wrote: post #2

I have seen frameworks in other spaces attempt the configuration driven approach and have seen it always fall out of favor vs. programmatic approaches. I am reading the docs right now and I feel like it's so much work to try to learn the syntax of their YAML specification vs. using something like LangFuse. The fact that you can write code as an array element in a YAML low key makes me angry. Yet this is used in Shopi…

Seriously, this is a statement you can make in promptfoo configuration:

```

    assert:

      - type: equals

        value: 'Hello, World!'

```

We are redefining the equality operator in YAML. Someone please explain.

Re: Why is promptfoo so popular?

#4

Post by haldunanil » Sat, Jul 27, 2024, 5:31 PM UTC

I can't speak to why it's popular, but I will say that the reason I chose it for one of my projects was because I pretty much had to as there were no better alternatives.

Most of the other tools like RAGAS and DeepEval are:

- Written in Python (vs. JavaScript like promptfoo), whereas our codebase is in Rails

- Get too into the weeds of AI, whereas I just want to write simple tests for my webapp not stare at a bunch of math equations

- Are hosted SaaS solutions (vs. open source like promptfoo) and I don't want to deal with the 6 month enterprise software sales cycle just to add a couple tests

So we basically ended up with promptfoo. That said, it felt really awkward to use promptfoo and I don't really love it. I'd love a super simple way to add some integration tests for an LLM so I can just keep iterating