31–40 of 69 posts
Earlier quoted context omitted.
Thank you!! You've hit the nail on the head and our thoughts exactly. One anecdote is we offer video demos and one potential customer just said, "I don't need to see it. I know what Copilot does." We do offer pilots for customers after fine-tuning on their codebase!
Lol if Swatcoder is available they could be great at devrel
https://github.blog/2023-02-14-github-copilot-for-business-i...
Actual Copilot for Enterprise: https://github.blog/2023-02-14-github-copilot-for-business-i...
How are you different?
Have you gotten feedback from folks using it on how it compares to Copilot in terms of usefulness? What's the 'order of magnitude' difference?
A few questions that might help an enterprise customer: How big is your base model? Where did you find more datasets (maybe just a hint would be sufficient)? Are you using SantaCoder [3]? Anything you can say about your fine-tuning that makes it special? Totally on board with you that HumanEval/MBPP are not great benchmarks for real world, and do you have a suggested alternative to help me see the value?
The calculus for an enterprise customer might be: "We could fine tune a 6B model on our internal code and internal benchmarks (say with a month of work, a few thousand in compute, 2 people on task), but I'd rather buy an off-the-shelf solution like codecomplete.ai. They give us XYZ benefits." Articulate the XYZ for a technical decision maker who will be your target audience.
* [1] https://huggingface.co/datasets/bigcode/the-stack
Always astonishes me how negative hacker news can be whenever people try to launch stuff. This is cool!
Codeium already has this enterprise option, with self-hosting, fine-tuning, and even things like search on top of autocomplete and way more IDEs: https://codeium.com/ How are you different?
Congrats on the launch. I think you should share some technical details for a more substantial pitch. You are using the OSS BigCode effort and "The Stack" [1, 2] (as you say in another comment), which is great. A few questions that might help an enterprise customer: How big is your base model? Where did you find more datasets (maybe just a hint would be sufficient)? Are you using SantaCoder [3]? Anything you can say…
I will expand a bit on fine-tuning. It's really hard to get this right, and the iteration speed is slow. Of course these companies can build their own, but we want to save them a lot of headache.
So far, we haven't found any off-the-shelf open source base model that works super well for code completions. We've augmented models with a huge amount of data in order to see our current performance, and we ran into a lot of pain along the way.