> delivering results 50x faster at 50x less cost What about quality of results? Are you measuring that too? Did you do so for the traditional reference practice? Using what sort of methodology? How did it your technique compare in quality? What kind of errors was it most likely to make? What techniques have you devised for spotting those errors? Are they the same kind of errors that users would experience when outsou…
I am sure most people never asked these questions to a human doing this research.
Congrats on the launch. This looks awesome. I'm actually working with a number of companies who are exploring this space. Many of them are in the current YC batch. We're helping to provide the core business data, then we're exploring how we can leverage our scraping infrastructure in other ways to bring costs down. I'm open to chat if you're interested: michael (a t) bigpicture.io
> delivering results 50x faster at 50x less cost What about quality of results? Are you measuring that too? Did you do so for the traditional reference practice? Using what sort of methodology? How did it your technique compare in quality? What kind of errors was it most likely to make? What techniques have you devised for spotting those errors? Are they the same kind of errors that users would experience when outsou…
Yeah when it comes to anything involving numbers there is absolutely no room for hallucination. It must be 100% accurate 100% of the time. No exceptions.
> delivering results 50x faster at 50x less cost What about quality of results? Are you measuring that too? Did you do so for the traditional reference practice? Using what sort of methodology? How did it your technique compare in quality? What kind of errors was it most likely to make? What techniques have you devised for spotting those errors? Are they the same kind of errors that users would experience when outsou…
I am sure most people never asked these questions to a human doing this research.
Most B2B focused AI tools will do 10x better if they pretend to be a normal human run company but just have the AI at the back.
Their clients want to know that the research report was written by a real person and not a bot.
But that doesn’t mean it actually has to be written by a real person and not a bot.
This is so cool, congrats on the launch! I have a friend who looked into doing something similar but they couldn't figure out a way to get the cost low enough. This was like a year ago so I'd guess it's much cheaper now and you could do something like fine-tuning a smaller domain specific model on GPT-4 outputs. Any ballparks on pricing / cost? What models are ya'll using?
Thanks! That's fascinating, do you remember what kind of tasks they served? Using GPT-4 and GPT-3.5 currently, and costs can be $1.50+ per request right now (have been benefiting from YC cloud credits!). Definitely steep at the moment but we expect costs to come down at least 10x over time. Not super clear on pricing yet (only a few weeks post-launch)
I think it was similar... an agent finds a bunch of info on a private/public company to evaluate investment (automating some associate analyst work). TBH I'm not sure where they ended up but I know they had an interesting distribution channel lined up. Happy to connect if you want.
> $1.50
Thanks for sharing! I think they were ~10x that but hadn't done a ton of optimization yet. To me, having a swag at cost makes these tool demos a lot more interesting because you can start figuring out what types of businesses you can/can't build with them.
> delivering results 50x faster at 50x less cost What about quality of results? Are you measuring that too? Did you do so for the traditional reference practice? Using what sort of methodology? How did it your technique compare in quality? What kind of errors was it most likely to make? What techniques have you devised for spotting those errors? Are they the same kind of errors that users would experience when outsou…
This is a great point. We completely agree that high-quality results is essential for adoption. It's basically table stakes for any tool like this to be useful. We've had several versions of this tool that weren't quite "good enough" and never saw any real use. Our latest version seems to meet the first quality threshold for actual work use. Our method of evaluating quality is not super systematic right now. For this…
I wonder if it's more about convincing yourself that it faithfully follows the same workflow an analyst would follow. It's always possible to miss stuff, so the best a person or a machine can do is be demonstrably methodical, it sounds like... and that is easier to test. Unless there is really some magic tacit step that human analyst perform to get better answers.