GitHub Copilot loses an average of $20 per user per month • Hacker News Board

Re: GitHub Copilot loses an average of $20 per user per month

#121

Post by macinjosh » Tue, Oct 10, 2023, 3:30 PM UTC

weikju wrote: post #52

cabirum wrote: post #25

Earlier quoted context omitted.

Could be different sets of people actually! However. It will eventually, chunk by chunk, upload the full source code of the app you are writing. Complete with all typos and mis-pastes. Like youtube blocks videos with just a hint of copyrighted music, they may be able to detect use of "patented" algos or something like that and block you/ send a lawyer/etc.

> upload the full source code of the app you are writing The full code that’s already uploaded via “git push”?

Inded, some folks are so worried about their code being sucked up by an AI bot they forget they ship every single revision and byte to MS via github. If MS wanted to be devious with the data entrusted to them they could have done it already.

Re: GitHub Copilot loses an average of $20 per user per month

#122

Post by laurent_du » Tue, Oct 10, 2023, 3:32 PM UTC

refulgentis wrote: post #61

bhouston wrote: post #43

Earlier quoted context omitted.

At some point, the open source models you can run locally are going to start to get competitive. I suspect the cost problem came from upgrading to ChatGPT 4 over ChatGPT 3 - right there their costs more than 5x.

Did they? I'm surprised to hear that. Hard to tell for autocomplete of course. For conversational answers, inference speed is _far_ higher than I'm used to from GPT-4, and lower quality. As in, it matches GPT 3.5/ChatGPT.

Quality is definitely way below GPT4.

Re: GitHub Copilot loses an average of $20 per user per month

#123

Post by Saline9515 » Tue, Oct 10, 2023, 3:32 PM UTC

insanitybit wrote: post #47

noodlesUK wrote: post #20

I feel like the centralisation of server resources for something like copilot doesn’t really make sense. Many (most) professional developers are working on beefy laptops. If ever there were a case to run these models client side, a software developer’s laptop is probably the ideal place. What are the specs that are needed to run inference on these models?

I don't think most dev laptops have 8GB+ of GPU memory, which (based on ollama requirements) seems like it's on the low-mid end of the requirements. I've tried experimenting with some local models and: a) They're much slower on my 6GB laptop GPU b) The seem to not be as good, functionally b) I can't make use of larger models I haven't done more than just some experimentation but I can see how this would make sense to…

At 600$+/y subscription fees, it makes sense to own a beefy NUC with 32Gb and a speedy SSD to run the model locally.

Re: GitHub Copilot loses an average of $20 per user per month

#124

Post by norir » Tue, Oct 10, 2023, 3:34 PM UTC

simonw wrote: post #98

marcosdumay wrote: post #95

Earlier quoted context omitted.

> for a lot of developers (but not all), the value Github Copilot is incredibly high, much more than $20/month. I really doubt this. IMO, the jury is still out deciding if the value is above zero for enough people to matter.

From that article: 1.5m users and up to 50% of their code is now being produced by Copilot. Sounds to me like the verdict on its value is in already.

I mean you could have made a similar argument about the productivity benefits of smoking cigarettes 100 years ago. Just because a lot of people are doing something doesn't mean it's valuable or that we have an accurate picture of the cost/benefits. The verdict is still very much out on LLMs.

Re: GitHub Copilot loses an average of $20 per user per month

#125

Post by smoe » Tue, Oct 10, 2023, 3:35 PM UTC

simonw wrote: post #98

marcosdumay wrote: post #95

Earlier quoted context omitted.

> for a lot of developers (but not all), the value Github Copilot is incredibly high, much more than $20/month. I really doubt this. IMO, the jury is still out deciding if the value is above zero for enough people to matter.

From that article: 1.5m users and up to 50% of their code is now being produced by Copilot. Sounds to me like the verdict on its value is in already.

The quote from the article says "more than 1.5 million people have used it and it is helping build nearly half of Copilot users’ code"

Not a native speaker, but to me this sounds much more ambiguous than up to 50% of code is produced by Copilot.

Also, how different from previous solutions is this actually? I use autocomplete and code snippets extensively. Never measured it , but I wouldn't be surprised if my IDE had generated more source code than I myself typed over the last 10 or so years.

Re: GitHub Copilot loses an average of $20 per user per month

#126

Post by wrs » Tue, Oct 10, 2023, 3:36 PM UTC

_fat_santa wrote: post #91

hesdeadjim wrote: post #5

The moment I can train Copilot on my codebase and improve its predictions, I’d gladly pay more than $20. Even now, I find that I can’t live without it simply as a slightly more intelligent auto-complete.

I tried Copilot in the early days but stopped using it pretty quickly as it just wasn't that useful in an already established codebase. Often times I'm not writing completely new code but rather implementing a pattern that I've previously defined somewhere else. In this case Copilot just completely falls over because it doesn't know about that existing code. My other gripe is language specific but in VSCode it breaks…

I have found it surprisingly good at some pretty sophisticated “grunt work”. For example, I’m converting some code to TypeScript. Looking at a long and twisty “getSomething” JavaScript function that builds up an object one field at a time, it can complete “interface Something {“ with all those fields, with types! It might duplicate or miss a field sometimes but it makes fewer mistakes than I would.

I’m pretty sure it pulls context from at least the other open files in VS Code, if not the ones on disk. Hard to tell for sure.

It’s kind of like having an eager, well-read, but naive and over-caffeinated intern as a pair programmer.

Re: GitHub Copilot loses an average of $20 per user per month

#127

Post by jrockway » Tue, Oct 10, 2023, 3:37 PM UTC

daft_pink wrote: post #112

jrockway wrote: post #106

Earlier quoted context omitted.

I use Copilot in Emacs and it doesn't send Github any data other than my prompt.

Do you specifically activate it? The main functionality I get out of copilot in vscode is autocomplete on steroids, so it's getting all code as I write it. I don't use the question based generator most of the time.

I'm probably not using it to the fullest. I mostly use it for boilerplate that I know it's good at, like `// Write a function to run the cpu profiler for 30 seconds and write the result to the filename in variable "foo"`. But I guess if it knows your codebase, then you can say `// Write a test to ensure that Foo accepts a nil FooRequest.` or something. I should figure out how to set that up.

All in all it saves me a little bit of typing a couple of times a week.

Re: GitHub Copilot loses an average of $20 per user per month

#128

Post by js8 » Tue, Oct 10, 2023, 3:39 PM UTC

criley2 wrote: post #78

js8 wrote: post #24

Earlier quoted context omitted.

That's a problem with all LLMs. They will average what they are trained on, without any reflection what is good and what is not. They shouldn't be called intelligent (AI), they're information meatgrinders.

>That's a problem with all LLMs. They will average what they are trained on, without any reflection what is good and what is not. They shouldn't be called intelligent (AI), they're information meatgrinders. This is an issue with your prompting not the models. If you tell someone "Do this thing" they'll just do it, LLM's too. And how it is done will probably be terrible. If you ask someone "come up with 5 ways to acco…

Fair enough, let's put aside that in Copilot, prompting is automatic.

Even if you do prompt it correctly, and it responded in the abstract with all these pros and cons, you cannot be reasonably sure that the code it also provided actually follows these best practices. It will just anytime mindlessly wander from "the best experts on the Internet are saying this" territory to "I just made this up" territory.

Compare that with googling a human-written stack overflow answer on the topic, there usually is some good soul who pointed out the inconsistency, if there is one.

Look, I like grinded meat. But the fact is, without a detailed analysis, it's hard to tell what's actually in it.

Re: GitHub Copilot loses an average of $20 per user per month

#129

Post by SadWebDeveloper » Tue, Oct 10, 2023, 3:39 PM UTC

I would gladly pay for copilot 5 USD a month (its an specific use case IA), but the current price point 20 month is incredible expensive it only add costs if the developer you are hiring requires it to properly function... thats the only value it adds.

IMHO If you are a code monkey copilot makes you a 5x developer, 10x if you have 5+ years of experience on the business

Maybe selling tokens/search could be benefitial for copilot, like only pay for what i use will be an interesting approach.

Re: GitHub Copilot loses an average of $20 per user per month

#130

Post by zackmorris » Tue, Oct 10, 2023, 3:41 PM UTC

Capitalism cracks me up. We live under this fantasy that everything we do can be turned into a profitable business. Even the government should be run like a business, right? Like somehow educating a child for 12+ years should be a profitable venture. Or housing the elderly. Or providing a basic service like water, sanitation or electricity that everyone uses at about the same rate. Surely we can stand on each other to turn a buck.

I think that AI will be the biggest loss leader ever invented. We'll buy a robot one time for $10,000 that will perform $1 million of mental and physical labor over its lifetime. Except nobody will pay for that $990,000. There will just be this expectation of something for nothing, so the anticipated payment never comes. And it will happen so quickly, so completely, that we'll wonder how it was ever possible to pay people to do all this free stuff.

No, the GitHub Copilot model of charging for AI services isn't going to scale. People are going to open source AI and own it themselves and stop paying for goods and services. Why would they when the AI can provide everything they need? AI is the beginning of the end of money. The headline foretells the end of the era of artificial scarcity, because capitalism can't compete with free.