That 200k context needs some proper testing. GPT-4-Turbo advertises 128k but the quality of output there goes down significantly after ~32k tokens.
Read the article, it's addressed with charts.
Claude 2.1
131–140 of 339 posts
Re: Claude 2.1
#132Earlier quoted context omitted.
Luckily, unlike OpenAI, Anthropic lets you prefill Claude's response which means zero refusals.
Can you give an example in how Anthropic and OpenAI differ in that?
From Anthropic's docs: https://docs.anthropic.com/claude/docs/configuring-gpt-promp...
In OpenAI's case their "\n\nAssistant:" equivalent is added server side with no option to prefill the response.
Re: Claude 2.1
#133The documentation for system prompts is setting off all of my prompt injection red flags: https://docs.anthropic.com/claude/docs/how-to-use-system-pro... I guess that design is at least honest: OpenAI field the system prompt in a separate fragment of JSON, but it all gets concatenated back together (with some magic delimiter tokens) when it's fed to the underlying model.
Relatedly, I checked and OpenAI deleted all references to their ChatML spec from their GitHub repo. This is what it said in an earlier commit: https://github.com/openai/openai-python/blob/2942bf4bb635b1e...
Something I never understood about ChatML: were those "" things reserved sequences of text that mapped to specific integer tokens, but were not things you could include in your own text that you submitted to their API (or if you did try they would be tokenized differently)?
Re: Claude 2.1
#134Earlier quoted context omitted.
I gather the pricing is $8 for a million input tokens [1] so if your language's manual is the size of a typical paperback novel, that'd be about $0.8 per question. And presumably you get to pay that if you ask any follow-up questions too. Sounds like a kinda expensive way of doing things, to me. [1] https://www-files.anthropic.com/production/images/model_pric...
From my perspective it sounds pretty cheap if we get to the answers immediately.
Have you tried it? GPT4 fails as often as it succeeds at coding questions I ask so I'm not going to shell out that kind of money to take my chances.
Re: Claude 2.1
#135Earlier quoted context omitted.
That can't be, because I can ask it a simple question that an answer is maybe 1 sentence, and it repeats the question then provides a whole novel. So ton of tokens.
GPT still writes like a highschooler trying to hit a high word count :(
You can ask it to be very concise.
I added it to my custom instructions and it has helped a lot.
Re: Claude 2.1
#136I was excited about Claude 2 for a few days but quickly determined that it’s much, much worse than GPT4 and haven’t used it much since. There really isn’t much point in using a worse LLM. And the bigger context window is irrelevant if the answers are bad despite that. I’ll give this new one a try but I doubt it will be better than the newly revamped GPT4.
Howdy! Please give 2.1 a try and let me know what you think. You can see the benchmark data in the appendix of our updated 2.1 model card here: https://www-files.anthropic.com/production/images/ModelCardC...
Re: Claude 2.1
#137Earlier quoted context omitted.
idk we're just "have more kids" simulators and we do pretty good at programming as a side-task
Someone doesn't get good at programming with low quality learning sources. Also, a poor comparison because models are not people - might as well complain about how NPCs in games behave because they fail at problems real people can solve.
We are both substrate that has been aggressively optimized for a task with a lot of side benefits. "NPC"s are not optimized at all, they are coded using symbolic rules/deterministic behavior.
Re: Claude 2.1
#138Earlier quoted context omitted.
Relatedly, I checked and OpenAI deleted all references to their ChatML spec from their GitHub repo. This is what it said in an earlier commit: https://github.com/openai/openai-python/blob/2942bf4bb635b1e...
Something I never understood about ChatML: were those " " things reserved sequences of text that mapped to specific integer tokens, but were not things you could include in your own text that you submitted to their API (or if you did try they would be tokenized differently)?
ChatGPT presumably adds them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken
In theory they could be added in normal input but it's possible OpenAI has safeguards against it.
Re: Claude 2.1
#1391. A 200k context is bittersweet with that 70k->195k error rate jump. Kudos on that midsection error reduction, though! 2. I wish Claude had fewer refusals (as erroneously claimed in the title). Until Anthropic stops heavily censoring Claude, the model is borderline useless. I just don't have time, energy, or inclination to fight my tools. I decide how to use my tools, not the other way 'round. Until Anthropic stops…
I don't know what you're doing with your LLM, but I've only ever had one refusal and I've been working a lot with Claude since it's in bedrock
I hear a lot of complaints about refusals but rarely any examples of said refusals, likely because they are embarrassing.
Is it fair to assume that I won't get refusals for code generation and RAG on documentation?
Re: Claude 2.1
#140Earlier quoted context omitted.
Yeah but to be honest been a pain last days to get gpt 4 to write full pieces of code for more the 10-15 lines. Have to re-ask many times and at some point it forgets my initial specifications.
Earlier in the year I had ChatGPT 4 write a large, complicated C program. It did so remarkably well, and most of the code worked without further tweaking. Today I have the same experience. The thing fills in placeholder comments to skip over more difficult regions of the code, and routinely forgets what we were doing. Aside all the recent OpenAI drama, I've been displeased as a paying customer that their products rou…
My understanding is they reduced the number of ensembles feeding gpt4 so they could support more customers. I want to say they cut it from 16 to 8. Take that with a grain of salt, that comes through the rumor telephone.
Are you prompting it with instructions about how it should behave at the start of a chat, or just using the defaults? You can get better results by starting a chat with "you are an expert X developer, with experience in xyz and write full and complete programs" and tweak as needed.