71–78 of 78 posts
Earlier quoted context omitted.
Then there will also be a push for cheaper, freer energy as well.
Yeah, with all the AI needs we’ll just have to transition to fusion energy.
Not even for the environmental aspects, just purely from energy demands, at any spot on the Earth where a data center could go.
Earlier quoted context omitted.
This is actually how Jeff Dean, Chief Scientist at Google recommends doing thought exercises to challenge assumptions (watch is recent YX Startup School talk). The end goal isn’t “128GB of RAM”, the goal is “useful effective software”. If for whatever reason RAM was an immovable constraint, a different tech tree would emerge.
The tech tree where we develop tech to have tons of RAM is better and leads to more opportunities than the tech tree where everything is built for only a handful of GB of RAM. We may never have had LLMs if we didn’t go down the first path.
Currently navigating this situation at my employer. We went from virtually unlimited token spend per software engineer, to $150 per month due the recent change in billing terms from GitHub Copilot. Rationed over a month, about $7.50 per day. Basically, a couple bad apples spoiled the bunch (contractors using Opus to center divs). We're 15 days into this new policy and its going ~okay~. Engineers adapt as they do, and…
I know some people are doing 10-agent parallel harness stuff, but that cannot possibly be the norm, and I find it hard to believe any company is expecting that level of output.
Earlier quoted context omitted.
The tech tree where we develop tech to have tons of RAM is better and leads to more opportunities than the tech tree where everything is built for only a handful of GB of RAM. We may never have had LLMs if we didn’t go down the first path.
Or maybe we’d have LLMs that fit in a handful of GBs (and we technically already do). It’s hard to tell without completely going down that parallel universe and simulating all the milestones in it, but it’s a fun thought experiment.
Earlier quoted context omitted.
Yeah but energy is still a cost, and local inference without batch and multiplexing for many users (like an office would be) is even less optimal. I'm not sure I think it's just pushing the problem forward
Then there will also be a push for cheaper, freer energy as well.
On-prem AI is another solution. I’m surprised more companies aren’t leaning in this direction due to security/IP concerns. My employer is planning to spend several million on local AI hardware.
Earlier quoted context omitted.
Set a limit, train people to use tokens efficiently. For anyone getting high value from it, raise their limit.
How do you figure out they are "getting high value from it" What if they feel like they are getting value from it, but are actually not?
That's part of the basic job description of both of them. Explain how your work relates to your company's goals both at a large scale and at a localized team level.