If you want to play with this stuff without spending a lot of money, what are your best options?
You should write an agent
301–310 of 409 posts
Re: You should write an agent
#302If you want to play with this stuff without spending a lot of money, what are your best options?
Most cloud providers, like Azure have free credits at the start. On azure you can deploy your own model and pay with the free credits.
Re: You should write an agent
#303If you want to play with this stuff without spending a lot of money, what are your best options?
You can just stick a tenner in OpenAI though and it won't charge anymore than the credit you've put in
Re: You should write an agent
#304Earlier quoted context omitted.
If you can get a specialized agent to work in its domain at 10% parameters of a foundation model, you can feasibly run locally, which opens up e.g. offline use cases. Personally I’d absolutely buy an LLM in a box which I could connect to my home assistant via usb.
Can you (or someone else) explain how to do that? How much does it typically cost to create a specialized agents that uses a local model? I thought it was expensive?
An agent is just a program which invokes a model in a loop, adding resources like files to the context etc. It's easy to write such a program and it costs nothing, all the compute cost is in the LLM call. What parent was referring to most likely is fine-tuning a smaller model which can run locally, specialized for whatever task. Since it's fine-tuned for that particular task, the hope is that it will be able to perform as well as a general purpose frontier model at a fraction of the compute cost (and locally, hence privately as well).
Re: You should write an agent
#305If you want to play with this stuff without spending a lot of money, what are your best options?
and sorry, forgot you can also run local models aswell :)
Re: You should write an agent
#306Absolutely, especially the part about just rolling your own alternative to Claude Code - build your own lightsaber. Having your coding agent improve itself is a pretty magical experience. And then you can trivially swap in whatever model you want (Cerebras is crazy fast, for example, which makes a big difference for these many-turn tool call conversations with big lumps of context, though gpt-oss 120b is obviously no…
What are you using for transcription? I tried Whisper, but it's slow and not great. I tried the gpt audio models, but they're trained to refuse to transcribe things. I tried Google's models and they were terrible. I ended up using one of Mistral's models, which is alright and very fast except sometimes it will respond to the text instead of transcribing it. So I'll occasionally end up with pages of LLM rambling paste…
Handy is free, open-source and local model only. Supports Parakeet: https://github.com/cjpais/Handy
Re: You should write an agent
#307Why write an agent when you can just ask the LLM to write one?
Re: You should write an agent
#308Earlier quoted context omitted.
What are you using for transcription? I tried Whisper, but it's slow and not great. I tried the gpt audio models, but they're trained to refuse to transcribe things. I tried Google's models and they were terrible. I ended up using one of Mistral's models, which is alright and very fast except sometimes it will respond to the text instead of transcribing it. So I'll occasionally end up with pages of LLM rambling paste…
I recently bought a mint-condition Alf phone, in the shape of Gordon Shumway of TV's "Alf", out of the back of an old auto shop in the south suburbs of Chicago, and naturally did the most obvious thing, which was to make a Gordon Shumway phone that has conversations in the voice of Gordon Shumway (sampled from Youtube and synthesized with ElevenLabs). I use https://github.com/etalab-ia/faster-whisper-server (I think?…
Been meaning to build something very similar! What hardware did you use? I'm assuming that a Pi or similar won't cut it
Re: You should write an agent
#309Earlier quoted context omitted.
Sorry, I assumed a service offering Virtual machines shares python code with the intent to get people to run that python on their infra.
Yes. You've caught on to our devious plan. To do anything I suggested in this post, you'd have to use a computer. By spending compute cycles, you'd be driving scarcity of compute. By the inexorable law of supply and demand, this would drive the price of compute cycles up, allowing us to profit. We would have gotten away with it, if it wasn't for you.
Scooby Doobie Doooo!
Re: You should write an agent
#310Earlier quoted context omitted.
what's the point of specialized agents when you just have one universal agent that can do anything e.g. Claude
Composing multiple smaller agents allows you to build more complex pipelines, which is a lot easier than getting a single monolithic agent to switch between contexts for different tasks. I also get some insight into how the agent performs (e.g via langfuse) because it’s less of a black box. To use an example: I could write an elaborate prompt to fetch requirements, browse a website, generate E2E test cases, and compi…
Plus I'd say that the smaller context or more specific context is the important thing there.
Even the biggest models seem to have attention problems if you've got a huge context. Even though they support these long contexts it's kinda like a puppy distracted by a dozen toys around the room rather than a human going through a checklist of things.
So I try to give the puppy just one toy at a time.