[flagged]
I burned all my tokens researching how to save tokens
171–180 of 237 posts
Re: I burned all my tokens researching how to save tokens
#172[flagged]
Re: I burned all my tokens researching how to save tokens
#173Earlier quoted context omitted.
Cache sharing is not possible. The numbers in the cache are completely specific to the model.
Currently. I'm sure that you could make a system where the cache values are a superset C of e.g. models A and B where C is probably bigger than max(A,B) but smaller than A+B
Reddit post so take with a grain of salt but this does seem possible. But unclear whether this is actually a viable architecture https://www.reddit.com/r/LocalLLaMA/comments/1t8s83r/nvidia_...
Re: I burned all my tokens researching how to save tokens
#174Earlier quoted context omitted.
You can reduce hallucinations with appropriate grounding checks. You can’t really get rid of them though, so feedback loops with verification are essential.
Error percentages compound. Having an LLM do something with a 5% hallucination rate, then having an LLM (same or different) with a 5% error rate check it means there's a 10% chance of error, not .25%. Prompting can't save you here; it's the fundamental math.
Your fundamental math is wrong
Re: I burned all my tokens researching how to save tokens
#175It sometimes feels to me that cloud AI providers have convinced people that cloud AI is worth it because of all the ways people have been able to use cloud AI to write blog posts about using cloud AI to make cloud AI more efficient for something that they haven't shipped yet and aren't really ready to talk about.
People busy building shippable things are generally too busy to blog about it. It is selection bias in its most classical meaning.
Yes, it's the Youtube phenomenon - often when looking for domain-specific videos on Youtube I tend to find videos by professional Youtubers not videos by professional domain specialists (overlap is very rare but occasionally you can find a domain specialist Youtuber who applies their domain knowledge to making Youtube videos).
Re: I burned all my tokens researching how to save tokens
#176Earlier quoted context omitted.
Error percentages compound. Having an LLM do something with a 5% hallucination rate, then having an LLM (same or different) with a 5% error rate check it means there's a 10% chance of error, not .25%. Prompting can't save you here; it's the fundamental math.
Your fundamental math is wrong
no ur wrong
Re: I burned all my tokens researching how to save tokens
#177Earlier quoted context omitted.
Trust me there are plenty of us using cloud AI to actually ship stuff. We just aren't writing blog posts about it.
what did you ship?
Re: I burned all my tokens researching how to save tokens
#178Earlier quoted context omitted.
So far you have provided no evidence of achieving more results with less effort. You've only said that you can write more lines of code (effectively) and that your coworkers are happier
I have to wonder what sort of evidence you would accept? Because it seems to me that the original question was answered clearly enough and now the goalposts have shifted to demanding evidence that the answer wasn't falsified.
Well you answered how it is useful by saying it writes more lines of code and that seems to be the only way it's useful. But we all know that lines of code aren't productivity.
Re: I burned all my tokens researching how to save tokens
#179TFA says "no hallucinations" but you can't fix hallucinations with rules or other models. I know I'm screaming into the void but whatever.
I hear you! I read the article and thought the same thing because I've actually built a surprisingly similar pipeline myself, but it culminates with Human review. I've got a trivia app (Hedge Calibrated Trivia) whose whole content pipeline is built on a similar architecture of models evaluating models. I didn't want the same trivia that every other app has and I naively imagined that it would be easy to ask an LLM to…
Very similar experience. The review UI might actually be the most important part of the whole pipeline. Does the confidence ranking match what you end up rejecting?
Re: I burned all my tokens researching how to save tokens
#180TFA says "no hallucinations" but you can't fix hallucinations with rules or other models. I know I'm screaming into the void but whatever.
I can't remember who it was I spoke to recently, but it was some startup for doing code reviews. So I asked how they're handling larger diffs and especially when a code change is related to something requiring knowing a lot of context. Will it hallucinate things that make the review low-quality, etc. I was told "no it's been given guardrails to prevent hallucinations. it doesn't hallucinate." but I was thinking, that…
This is a good/fast/cheap tradeoff, but in the form of accurate/precise/creative, or something in that ballpark anyway.
You can constrain the AI to be more accurate by having it a) equivocate (lose precision) or b) turning the temperature to effectively zero, thus losing "intelligence" and creativity.
There's no such thing as a free lunch. The models perform best out of the box, and everything you add into context makes them perform worse generally, but more likely to do what you want. Every token is a burden, but without those constraints the generation won't do anything.
So by that standard, hallucination is something completely core - you can't get rid of it without losing what makes them useful. You can make different tradeoffs, or use a very smart model in a way that is completely locked down (or adversarially verified, same result) which results in far more tokens used than if you just let the model do it's thing and validated for hallucination independently, outside of the generation loop.