Has anyone looked into the effect of politeness on performance?
If you assume asking someone nicely is more likely for them to try help you, and this tendency shows in the training set, wouldn't you be more likely to "retrieve" a better answer from the model trained on it? Take this with a grain of salt, it's just my guess not backed by anything
That makes intuitive sense, at least for raw GPT-3. The interesting question is whether the slave programming — er, instruction finetuning — makes it unnecessary.
This is the real value here. Keeping a secure environment to run untrusted code along side user data is a real liability for them. It's not their core competency either, so they can just lean on browser sandboxing and not worry about it.
How is doing it server side a different challenge than something like google collab or any of those Jupyter notebook type services?
Shared resources and multitenancy are how you get efficiency and density. Those are at direct odds with strict security boundaries. IME you need hardware supported virtualization for consistent security boundary of arbitrary compute. Linux namespaces (“containers”) and language runtime isolation are not it for critical workloads, see some of the early aws nitro/firecracker works for more details. I _assume_ the cases you mentioned may be more constrained, or actually backed by VM partitions per customer.
This is the real value here. Keeping a secure environment to run untrusted code along side user data is a real liability for them. It's not their core competency either, so they can just lean on browser sandboxing and not worry about it.
How is doing it server side a different challenge than something like google collab or any of those Jupyter notebook type services?
The cost savings for this are going to be a rounding error. I imagine this is a broader push to be able to have Claude pilot your browser (and other applications) in the future. This is the right way to go about it versus having a headless agent: users can be in the loop and you can bootstrap and existing environment. Otoh it’s going to be a security nightmare.
The cost-savings would actually be significant. Spinning up a sandboxed container/VM or chroot jail a thousand times a month for a user paying a $20/month, when you already as a company have huge GPU bills on the training and inference side and NRE costs, would be gaping.
I really really don't think you understand how cheap it would be to spin up a node.js env a thousand times a month in a container. Let's be really really conservative and say that each invocation takes 30s of CPU time, resulting in 30,000 CPU seconds per month. Let's say that CPU cores can be had for $10/mo. We are talking about 10 cents per user per month. And in reality, this is still probably over an order of magnitude too high. You are literally talking fractions of a cent in reality.
If you assume asking someone nicely is more likely for them to try help you, and this tendency shows in the training set, wouldn't you be more likely to "retrieve" a better answer from the model trained on it? Take this with a grain of salt, it's just my guess not backed by anything
That makes intuitive sense, at least for raw GPT-3. The interesting question is whether the slave programming — er, instruction finetuning — makes it unnecessary.
NGL I was impressed when I asked Claude how to do some fancy UI stuff and it just spat out some working react. A few hours later and I'd saved £500 I was going to spend on a designer.
The cost-savings would actually be significant. Spinning up a sandboxed container/VM or chroot jail a thousand times a month for a user paying a $20/month, when you already as a company have huge GPU bills on the training and inference side and NRE costs, would be gaping.
I really really don't think you understand how cheap it would be to spin up a node.js env a thousand times a month in a container. Let's be really really conservative and say that each invocation takes 30s of CPU time, resulting in 30,000 CPU seconds per month. Let's say that CPU cores can be had for $10/mo. We are talking about 10 cents per user per month. And in reality, this is still probably over an order of magn…
Yes actually it would be cheaper if one pre-provisions VMs. Albeit, to ensure sufficient provisioned capacity that one would have to slightly over-provision here, and for a coding power-user that it would be almost $2-3 cloud costs per month excluding the software engineering costs of maintaining this fleet of servers and scheduling jobs on it.
I really really don't think you understand how cheap it would be to spin up a node.js env a thousand times a month in a container. Let's be really really conservative and say that each invocation takes 30s of CPU time, resulting in 30,000 CPU seconds per month. Let's say that CPU cores can be had for $10/mo. We are talking about 10 cents per user per month. And in reality, this is still probably over an order of magn…
Yes actually it would be cheaper if one pre-provisions VMs. Albeit, to ensure sufficient provisioned capacity that one would have to slightly over-provision here, and for a coding power-user that it would be almost $2-3 cloud costs per month excluding the software engineering costs of maintaining this fleet of servers and scheduling jobs on it.
I laid out all my arithmetic. I don't know what you disagree with. How do you get to $2-3 per month? I suspect you don't actually understand how this would be run at scale, because I can't see any universe in which a single power user is generating $2-3 of compute cost _for a javascript container_.
Yes actually it would be cheaper if one pre-provisions VMs. Albeit, to ensure sufficient provisioned capacity that one would have to slightly over-provision here, and for a coding power-user that it would be almost $2-3 cloud costs per month excluding the software engineering costs of maintaining this fleet of servers and scheduling jobs on it.
I laid out all my arithmetic. I don't know what you disagree with. How do you get to $2-3 per month? I suspect you don't actually understand how this would be run at scale, because I can't see any universe in which a single power user is generating $2-3 of compute cost _for a javascript container_.
My definition of a coding power-user would be hooking up Cursor/Copilot to the JS runtime to run JS 10-20 times a minute.