Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why XML tags are so fundamental to Claude

glthr.com

Post reply on HN
Print view View original

111–120 of 160 posts

Re: Why XML tags are so fundamental to Claude

#111

Post by thesz »

krackers wrote: post #108
thesz wrote: post #65

Earlier quoted context omitted.

Transformers are able to recognize balanced brackets grammar at 97% success rate: https://openreview.net/pdf?id=kaILSVAspn This is 3% or infinitely far away from the perfect tech. The perfect tech is the stack.

This is very interesting since there is another notable paper which shows LLMs can recognize and generate CFGs https://arxiv.org/abs/2305.13673 and of course a^n b^n is also classic CFG, so it's not clear why one paper had positive results while the other hand negative.

Dyck grammar (balanced brackets) are not an a^nb^n, there are several kinds of brackets.

I cannot find probability of success in paper you linked. Is it 100%? I believe it is less than 100%, because LLMs are intrinsically probabilistic machines.

Re: Why XML tags are so fundamental to Claude

#112

Post by krackers »

thesz wrote: post #111
krackers wrote: post #108

Earlier quoted context omitted.

This is very interesting since there is another notable paper which shows LLMs can recognize and generate CFGs https://arxiv.org/abs/2305.13673 and of course a^n b^n is also classic CFG, so it's not clear why one paper had positive results while the other hand negative.

Dyck grammar (balanced brackets) are not an a^nb^n, there are several kinds of brackets. I cannot find probability of success in paper you linked. Is it 100%? I believe it is less than 100%, because LLMs are intrinsically probabilistic machines.

Figure 12 shows probabilities I think, it actually does seem to be 100% at temperature 0.1 for certain pretraining runs.

Re: Why XML tags are so fundamental to Claude

#113

Post by Bratmon »

RadiozRadioz wrote: post #39

> a contrast between Claude’s modern approach [...] XML, a technology dating back to 1998 Are we really at the point where some people see XML as a spooky old technology? The phrasing dotted around this article makes me feel that way. I find this quite strange.

XML is as old now as the PDP-11 was when XML came out.

Re: Why XML tags are so fundamental to Claude

#114

Post by mpyne »

kid64 wrote: post #28

The thesis here seems to be that delimiters provide important context for Claude, and for that putpose we should use XML. The article even references English's built-in delimiter, the quotation mark, which is reprented as a token for Claude, part of its training data. So are we sure the lesson isn't simply to leverage delimiters, such as quotation marks, in prompts, period? The article doesn't identify any way in whi…

I had a lot of success with Claude just by telling it I would throw a relevant snippet in a pair. That's not even XML, nor has it been really needed in my experience.

Even simple --- separators is usually enough to get good results, it just needs to be reasonably clear which items are distinct from each other.

Re: Why XML tags are so fundamental to Claude

#115

Post by thesz »

krackers wrote: post #112
thesz wrote: post #111

Earlier quoted context omitted.

Dyck grammar (balanced brackets) are not an a^nb^n, there are several kinds of brackets. I cannot find probability of success in paper you linked. Is it 100%? I believe it is less than 100%, because LLMs are intrinsically probabilistic machines.

Figure 12 shows probabilities I think, it actually does seem to be 100% at temperature 0.1 for certain pretraining runs.

  > it actually does seem to be 100%
For all Dyck grammar sequences, infinitely many of them? ;)

Re: Why XML tags are so fundamental to Claude

#116

Post by thesz »

krackers wrote: post #112
thesz wrote: post #111

Earlier quoted context omitted.

Dyck grammar (balanced brackets) are not an a^nb^n, there are several kinds of brackets. I cannot find probability of success in paper you linked. Is it 100%? I believe it is less than 100%, because LLMs are intrinsically probabilistic machines.

Figure 12 shows probabilities I think, it actually does seem to be 100% at temperature 0.1 for certain pretraining runs.

And this Figure 12 is not about Dyck/balanced-brackets grammar. This figure is about something not properly described in the paper.

Re: Why XML tags are so fundamental to Claude

#117

Post by krackers »

thesz wrote: post #115
krackers wrote: post #112

Earlier quoted context omitted.

Figure 12 shows probabilities I think, it actually does seem to be 100% at temperature 0.1 for certain pretraining runs.

> it actually does seem to be 100% For all Dyck grammar sequences, infinitely many of them? ;)

Well they used strings of > Dyck/balanced-brackets grammar

Yes, it's not the Dyck grammar but another CFG they created, they call it the "cfg3" family.

Of course I agree the stack (/pushdown automaton) is the simpler and perfectly optimal structure for this task, but I think it's unfair to say that LLMs _cannot_ recognize or generate CFGs.

(Then again I know you didn't make any such broad refutation of that sort, I mostly wanted to bring up that paper to show that it is possible for them to at least "grok" certain CFGs with low enough error ratio that they must have internalized the underlying grammar [and in fact I believe the paper goes on to apply interprability methods to actually trace the circuits with which it encodes the inductive grammar, which puts to rest any notion of them simply "parroting" the data]). But these were "synthetic" LLMs specifically trained for that grammar, these results probably don't apply in practice to your chatGPT that was trained mostly on human text.

Re: Why XML tags are so fundamental to Claude

#118

Post by treebeard901 »

actionfromafar wrote: post #73
hbarka wrote: post #53

Earlier quoted context omitted.

If you think XML is old tech, wait until you hear of EDI, still powering Walmart and Amazon logistics. XML came in like a wrecking ball with its self-documenting promise designed to replace that cryptic pesky payload called EDI. XML promised to solve world hunger. It spawned SOAP, XML over RPC, DOM, DTD, the heyday was beautiful and Microsoft was leading the charge. C# was also right around this time. Consulting firm…

EDI is XML now.

It all brings back nightmares from migrating the older style EDI for healthcare data for what was HL7 XML at the time. XML is widely used still for all kinds of stuff. On some level if JSON was allowed to evolve the same way, eventually you would just wind up with something like XML.

Re: Why XML tags are so fundamental to Claude

#119

Post by RHSeeger »

array_key_first wrote: post #106
coldtea wrote: post #99

Earlier quoted context omitted.

> It's not the hot new thing but when has hype ever mattered for getting shit done? People who wanted to "get shit done" had much better alternatives. XML grew out of hype, corporate management forcing it, and bundling to all kinds of third party products and formats just so they can tick the "have this hot new format support" box.

XML is perfectly fine. What are these alternatives? YAML is just bad. JSON is harder to read for deeply nested structures. TOML and the like don't have enough features.

XML is pretty fantastic for a lot of things that JSON is not up to the task for. And YAML ... has it's own, special issues.

Re: Why XML tags are so fundamental to Claude

#120

Post by vbezhenar »

girvo wrote: post #77
coldtea wrote: post #44

Earlier quoted context omitted.

XML has been "spooky old technology" for over a decade now. It's heyday was something like 2002. Nobody dares advertise the XML capabilities of their product (which back then everybody did), nobody considers it either hot new thing (like back then) or mature - just obsolete enterprise shit. It's about as popular now as J2EE, except to people that think "10 years ago" means 1999.

I kind of miss SOAP. Ahead of its time? Probably not, but I built some cool things on top of it

Right now I'm writing adapter so people could call one SOAP service using simpler interfaces. That involves implementing WS-Security with non-standard algorithms, that also involves dealing with things like XML escaped into a string and embedded inside another XML.

Let's say I hope for the day I'll miss SOAP. Right now I have too much of it.

Post reply on HN