Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Copilot loses an average of $20 per user per month

thurrott.com

Post reply on HN
Print view View original

61–70 of 214 posts

Re: GitHub Copilot loses an average of $20 per user per month

#61

Post by refulgentis »

bhouston wrote: post #43
insanitybit wrote: post #42

Earlier quoted context omitted.

I'd probably pay ~2-3x, idk about 5x. I'd consider it though. Hopefully they can cut the costs and meet in the middle.

At some point, the open source models you can run locally are going to start to get competitive. I suspect the cost problem came from upgrading to ChatGPT 4 over ChatGPT 3 - right there their costs more than 5x.

Did they? I'm surprised to hear that. Hard to tell for autocomplete of course. For conversational answers, inference speed is _far_ higher than I'm used to from GPT-4, and lower quality. As in, it matches GPT 3.5/ChatGPT.

Re: GitHub Copilot loses an average of $20 per user per month

#62

Post by ackbar03 »

I have a small cloud based Ai image processing service running that's still running (we did it before it was cool). The processing is currently done locally on a computer in my partners closet, but scaling it to that point, whilst ensuring it runs smoothly for multiple users with reasonable processing times was no joke.

Re: GitHub Copilot loses an average of $20 per user per month

#63

Post by _5ygc »

bryanlarsen wrote: post #49

AI is such a huge expectations dichotomy. For those of us used to the continual disappointment that pre-LLM AI was, the current crop of LLM's are amazing, mind blowing things. We start raving about them, so other people take a look expecting that modern LLM's are the greatest thing since sliced bread. They're not quite that, so we get HN comments complaining that AI sucks.

I think the problem is that they aren’t mind-blowing, but an improvement.

I resent some LLM implementations on principle, but decided to give these code helpers a try. What I found was they’re reasonably bad, and I kept telling them the solution doesn’t work, only to be presented with a little tweak.

So I don’t see the point of outsourcing my thinking, I’d rather remain intelligent and do the search/try/tweak on my own, instead of pretending a half-assed LLM is genius.

That doesn’t mean they don’t have good use cases, or aren’t an improvement on previous tech. But we definitely should stop calling them mind-blowing. Jaron Lanier had long ago predicted we’d willingly downplay human intelligence to pretend AI was… I.

Re: GitHub Copilot loses an average of $20 per user per month

#64

Post by visarga »

js8 wrote: post #24
grogenaut wrote: post #8

Earlier quoted context omitted.

I've used some prototypes that tried it. One of the things it does it regurgitate the old patterns you want to stop using and not the new stuff you do. Basically it's a copy pasta tech debt generator.

That's a problem with all LLMs. They will average what they are trained on, without any reflection what is good and what is not. They shouldn't be called intelligent (AI), they're information meatgrinders.

They are great information meat grinders, the best. /s

Prompting allows you to direct the model into a different way than the training data. If it were not so, LLMs would never solve problems that were not explicitly in their training set.

Re: GitHub Copilot loses an average of $20 per user per month

#65

Post by ntoshev »

tandav wrote: post #16

Does anyone know vscode extension that can use OpenAI API to perform code completions like Copilot? There are several ChatGPT-like UIs that you can self host and pay only for API and not for ChatGPT plus. For example, I'm using https://github.com/Yidadaa/ChatGPT-Next-Web . It would be nice to use Copilot in the same way.

You can modify this one to do that - or write your own using the same approach:

https://github.com/danielgross/localpilot

Re: GitHub Copilot loses an average of $20 per user per month

#66

Post by visarga »

benterix wrote: post #28
hesdeadjim wrote: post #5

The moment I can train Copilot on my codebase and improve its predictions, I’d gladly pay more than $20. Even now, I find that I can’t live without it simply as a slightly more intelligent auto-complete.

I have mixed feeling on this. On the one hand, sometimes it's able to guess correctly and save me a couple of seconds, up to half a minute I think. On the other hand, very often it's completely wrong, so I needed to constantly verify the suggestions and I found out it interrupted my flow and in the end I'd code faster without it.

Maybe you just need shorter predictions, like one single line or 3-4 words

Re: GitHub Copilot loses an average of $20 per user per month

#67

Post by visarga »

bloopernova wrote: post #7
hesdeadjim wrote: post #5

The moment I can train Copilot on my codebase and improve its predictions, I’d gladly pay more than $20. Even now, I find that I can’t live without it simply as a slightly more intelligent auto-complete.

Yeah, I really, really want to be able to configure copilot to explicitly include directories to scan in addition to its built-in corpus of knowledge. That would increase its worth to me a lot.

That's what I expected Copilot 2.0 to be, what a bummer.

Re: GitHub Copilot loses an average of $20 per user per month

#68

Post by baq »

insanitybit wrote: post #47
noodlesUK wrote: post #20

I feel like the centralisation of server resources for something like copilot doesn’t really make sense. Many (most) professional developers are working on beefy laptops. If ever there were a case to run these models client side, a software developer’s laptop is probably the ideal place. What are the specs that are needed to run inference on these models?

I don't think most dev laptops have 8GB+ of GPU memory, which (based on ollama requirements) seems like it's on the low-mid end of the requirements. I've tried experimenting with some local models and: a) They're much slower on my 6GB laptop GPU b) The seem to not be as good, functionally b) I can't make use of larger models I haven't done more than just some experimentation but I can see how this would make sense to…

MacBooks especially pros are in a very privileged position due to their memory architecture. With some dedicated hardware it may be reasonable to run, perhaps even train, useful LLMs on device.

I guess we’ll see in a year or two. This must be on everyone’s radar now, Apple won’t be the odd man out.

Re: GitHub Copilot loses an average of $20 per user per month

#69

Post by finnthehuman »

bryanlarsen wrote: post #49

AI is such a huge expectations dichotomy. For those of us used to the continual disappointment that pre-LLM AI was, the current crop of LLM's are amazing, mind blowing things. We start raving about them, so other people take a look expecting that modern LLM's are the greatest thing since sliced bread. They're not quite that, so we get HN comments complaining that AI sucks.

> We start raving about them, so other people take a look expecting…

If that’s what you want to call it.

I see hypemen overpromising and product underdelivering. And when pressed about specifics, attempts to drown queries in jargon or an ass-covering retreat to treating it like it’s just a tech demo not intended to be used for anything ever.

Re: GitHub Copilot loses an average of $20 per user per month

#70

Post by rgbrgb »

swexbe wrote: post #9

They're trying to corner the market and betting on the running costs of these models going down substantially in the future.

I think it’s a good bet based on watching inference speed of llama.cpp consistently improving and model ability / size on a similar trajectory. I expect there’s similar room for optimization (probably more) with hosted models. If you don’t care about code/model privacy (I do but it seems like most don’t yet at least) there are even more batching/caching tricks to engineer.
Post reply on HN