Scoring 96 percentile among humans taking the exam without moving goal posts would have been science fiction two years ago. Now it’s suddenly not good enough and the fact a computer program can score decent among passing lawyers and first time test takers is something to sneer at. The fact I can talk to the computer and it responds to me idiomatically and understands my semantic intent well enough to be nearly indist…
> The fact I can talk to the computer and it responds to me idiomatically and understands my semantic intent well enough to be nearly indistinguishable from a human being is breath taking That's called a programming language. It's nothing new.
It's a programming language except the programming part, and the language part.
It is difficult to comment without sounding obnoxious, but having taken the bar exam, I found the exam simple. Surprisingly simple. I think it was the single most over hyped experience of my life. I was fed all this insecurity and walked into the convention center expecting to participate in the biggest intellectual challenge in my life. Instead, it was endless multiple choice questions and a couple contrived scenari…
> It may also be surprising to some to understand that legal writing is prized for its degree of formalism. It aims to remove all connotation from a message so as to minimize misunderstanding, much like clean code. > The more your argument is structured, akin to a computer program, the better. You certainly make legal writing sound like a flavor of technical writing. Simplicity, clarity, structure. Is this an accurat…
recently I read a US law trade magazine article on a particular term used in US Federal employment law.. the article was about 12 pages long.. by the second page, they were using circular references, and switching between two phrases that used the same words but had different word order, contexts and therefore meanings, without clearly saying when they switched. By the third or fourth page I was done with that exercise. As a coder and reader of English literature, there was no question at all that the terms where being "churned" as a sleight of hand, directly in writing. One theory about a reason that they did that in an article that claimed to explain the terms, was to setup confusion and misdirection as it is actually practiced in law involving unskilled laymen, and then "solving" the problems by the end of the article.
I took a sample CA bar exam for fun, as a non-lawyer who has never set foot in law school. Maybe the sample exam was tougher than the real thing, but I found it surprisingly difficult. A lot of the correct answers to questions were non-obvious -- they weren't based on straightforward logic, nor were they based on moral reasoning, and there was no place for "natural law" -- so to answer questions properly you had to h…
Obviously you need subject knowledge, that should be implicit? Keep in mind even today[1] ( in California and few other states) you don't need to go law school to write the Bar exam and practice law, various forms of apprenticeship under a judge or lawyer are allowed You also don't need to write the exam to practice many aspects of the legal profession. The exam is never meant to be a high bar of quality or selection…
> Obviously you need subject knowledge, that should be implicit?
Well, in a lot of the so-called soft sciences, you can easily beat a test without subject knowledge. I had figured that the bar exam might be something like that -- but it's more akin to something like biology, where there are a lot of arcane and counterintuitive little rules that have emerged over time. And you need to know those, or you're sunk. You can't guess your way past them, because the best-looking guesses tend to be the wrong ones.
(For what it's worth, I realize that this mostly has to do with the Common Law's reverence of precedent-as-binding, and that continental Civil Law systems don't suffer as much from it. But I suppose those continental systems have other problems of their own.)
I like that it immediately assumed the US, even though nothing in your question suggested it. I love that all LLMs have a strong US centric bias. Btw I'm not personally a lawyer, but I've heard that GPT is especially prone to mixing laws across the borders - for example you ask a law question in language X, and get a response that uses a law from a country Y - and it's extremally convincing doing that (unless you're…
I mean, to be fair, if you're speaking English to it, the most likely possibility is that you're inside the US: https://en.wikipedia.org/wiki/List_of_countries_by_English-s... I know there's a lot of complaints about things being US-centric, but the US is a very large country.
Well, except the number of English speakers outside the US is much larger than inside the US (as per the wikipedia page you point to) by 5 to 1. Granted many folks are speaking it as their 2nd (or nth) language. But when you take into account the limited set of languages supported by ChatGPT one could reasonably assume English-speaking (typing) users of ChatGPT are from outside the U.S. as non-U.S. folks are in the majority of 'folks for whom english would be their first option when interacting with ChatGPT'. Even if you only count India, Nigeria and Pakistan.
Though of course OpenAI can tell (frequently, roughly) where folks are coming from geographically and could (does?) take that into account.
Honestly, this is giving the bar exam (and GPT-4) too much credit. The bar tests memorization because it's challenging for humans and easy to score objectively. But memorization isn't that important in legal practice; analysis is. LLMs are superhuman at memorization but terrible at analysis.
You clearly don't know anything about the bar. One half of your score is split between 6 essay questions, and reviewing two cases to then follow instructions from a theoretical lead attorney.
I’m licensed in multiple states, including California.
The essay questions also test memorization. They don’t require any difficult analysis - just superficial issue-spotting and reciting the correct elements.
If the bar exam were not a memorization test, it would be open book!
Scoring 96 percentile among humans taking the exam without moving goal posts would have been science fiction two years ago. Now it’s suddenly not good enough and the fact a computer program can score decent among passing lawyers and first time test takers is something to sneer at. The fact I can talk to the computer and it responds to me idiomatically and understands my semantic intent well enough to be nearly indist…
On any topic that I understand well, LLM output is garbage: it requires more energy to fix it than to solve the original problem to begin with. Are we sure these exams are not present in the training data? (ability to recall information is not impressive for a computer) Still I'm terrible at many many tasks e.g., drawing from description and the models widen significantly types of problems that I can even try (where…
On what topics you understand well does GOT-4o or Claude Opus produce garbage?
Scoring 96 percentile among humans taking the exam without moving goal posts would have been science fiction two years ago. Now it’s suddenly not good enough and the fact a computer program can score decent among passing lawyers and first time test takers is something to sneer at. The fact I can talk to the computer and it responds to me idiomatically and understands my semantic intent well enough to be nearly indist…
The nerds aren't jaded, they are worried. I'd be too if my job needed nothing more than a keyboard to be completed. There are a lot of people here who need to squeeze another 20-40 years out of a keyboard job.
Similar comments were made that microwaves will eliminate cooking.
At the end of the day (a) LLMs aren't accurate enough for many use cases and (b) there is far more to knowledge worker's jobs than simply generating text.
Scoring 96 percentile among humans taking the exam without moving goal posts would have been science fiction two years ago. Now it’s suddenly not good enough and the fact a computer program can score decent among passing lawyers and first time test takers is something to sneer at. The fact I can talk to the computer and it responds to me idiomatically and understands my semantic intent well enough to be nearly indist…
On any topic that I understand well, LLM output is garbage: it requires more energy to fix it than to solve the original problem to begin with. Are we sure these exams are not present in the training data? (ability to recall information is not impressive for a computer) Still I'm terrible at many many tasks e.g., drawing from description and the models widen significantly types of problems that I can even try (where…
It depends on the topic (and the LLM - ChatGPT-4 equivalent at least, any model equivalent to 3.5 or earlier is just a toy in comparison) - but I've had plenty of success using it as a productivity enhancing tool for programming and AWS infrastructure, both to generate very useful code and as an alternative to Google for finding answers or at least a direction to answers. But I only use it where I'm confident I can vet the answers it provides.
On any topic that I understand well, LLM output is garbage: it requires more energy to fix it than to solve the original problem to begin with. Are we sure these exams are not present in the training data? (ability to recall information is not impressive for a computer) Still I'm terrible at many many tasks e.g., drawing from description and the models widen significantly types of problems that I can even try (where…
On what topics you understand well does GOT-4o or Claude Opus produce garbage?
I do run into the issue where the longer the conversation goes the more inaccurate the information.
But a common situation is that with code generation it will fail to understand the context of where the code belongs and so it's a function that will compile but makes no sense.
On what topics you understand well does GOT-4o or Claude Opus produce garbage?
I do run into the issue where the longer the conversation goes the more inaccurate the information. But a common situation is that with code generation it will fail to understand the context of where the code belongs and so it's a function that will compile but makes no sense.
Yeah. I often springboard into a new context by having the LLM compose the next prompt based on the discussion and restart the context. Remarkably effective if you ask it to incorporate “prompt engineering” terms from research.