The danger of the mediocre success when testing startup hypotheses (2023)

It was incredibly damaging and would now be a red flag for me when looking at future workplaces.

It was a viscous cycle where the sales team promised all kinds of customisations to land deals. The software team would be lumped with the responsibility of delivering many of these while not getting time to refine the core products.

The software despite being critical for ultimately delivering also was still seen as a "cost centre" because on paper the credits were given away for pennies to land the "juicy" consultancy with it.

If any deal was tight, the sales team would just give away more software credits / promise more software customisation to land it.

The result was the consultant side of the business acted like they were responsible for all the revenue despite the software delivering a huge bulk.

On paper the software didn't bring in much because we had a sales team effectively giving away our software.

I've written previously about the dangers of courting a tiny fish from a big pond which is related. The one-sided nature of the emerging relationship where to the small supplier the contract means everything, but to the mega-corp it's actually just "that thing that Bob from Sub-division 3 uses" and doesn't actually have mind-share, because very little software is actually ever rolled out "org-wide". Mega-corps rarely behave in such a synchronised manner and chances are you're not really selling to "MegaCorp", you're just selling to Bob. And when he moves on, there's a good chance that no-one else has even heard of you. If you're lucky your invoices will still get paid for a while after despite no-one now even using your system, if unlucky you'll have to fight twice as hard now to get paid for the outstanding invoices, despite the undercut price you decided to charge Bob because you thought you were getting your foot in the door at MegaCorp.

I have encountered this "mediocre success" many times in AI solutions due to lack of problem definition. For instance, now with LLMs is very easy to write a prompt that gives you the output you want in 5 or 6 examples you have in mind. The problem is to build up your testing scenario from there, and gather as much data as possible until you make it representative of your use cases.

That is the only way to actually test your prompts, RAG strategies, and so on, instead of buying the last CoT-like prompt trend.

Like say, having a side project or hobby. If you try something and find you both don't enjoy it and fail miserably at it, that's probably a sign it's not the right thing to be doing. If you try and succeed beyond your wildest dreams, then you'd continue.

But if you're in the middle and become moderately successful, then what (as the article says)? Do you see that as a sign you've made the right decision and should continue? Or a sign that you'd be better of trying something else?

It's all too easy to take it as a sign that "things will work out if I keep going down this path" and plow weeks/months/years into something that may not pay off in the end, or be best suited to your talents and interests.