Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4o's Memory Breakthrough – Needle in a Needlestack

nian.llmonpy.ai

Post reply on HN
Print view View original

111–120 of 256 posts

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#111

Post by semi-extrinsic »

Eisenstein wrote: post #28
youssefabdelm wrote: post #9

Someone needs to come up with a "synthesis from haystack" test that tests not just retrieval but depth of understanding, connections, abstractions across diverse information. When a person reads a book, they have an "overall intuition" about it. We need some way to quantify this. Needle in haystack tests feel like a simple test that doesn't go far enough.

My idea is to buy to a unpublished novel or screenplay with a detailed, internally consistent world built in to it and a cast of characters that have well crafted motivations and then ask it to continue writing from an arbitrary post-mid-point by creating a new plot line that combines two characters that haven't yet met in the story. If it understands the context it should be able to write a new part of the story and…

Just use memes. People generate new high-quality niche memes so fast it's impossible for the LLMs to keep up.

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#112

Post by sschueller »

We are all so majorly f*d.

The general public does not know nor understand this limitation. At the same time OpenAI is selling this a a tutor for your kids. Next it will be used to test those same kids.

Who is going to prevent this from being used to pick military targets (EU law has an exemption for military of course) or make surgery decisions?

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#113

Post by Arn_Thor »

sschueller wrote: post #112

We are all so majorly f*d. The general public does not know nor understand this limitation. At the same time OpenAI is selling this a a tutor for your kids. Next it will be used to test those same kids. Who is going to prevent this from being used to pick military targets (EU law has an exemption for military of course) or make surgery decisions?

If any regulator acts it will be the EU. The action, if it comes, will of course be very late, possibly years from now, when the horse has long left the stable.

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#114

Post by sschueller »

Arn_Thor wrote: post #113
sschueller wrote: post #112

We are all so majorly f*d. The general public does not know nor understand this limitation. At the same time OpenAI is selling this a a tutor for your kids. Next it will be used to test those same kids. Who is going to prevent this from being used to pick military targets (EU law has an exemption for military of course) or make surgery decisions?

If any regulator acts it will be the EU. The action, if it comes, will of course be very late, possibly years from now, when the horse has long left the stable.

My only hope for the EU government is that they put and AI in charge and it accidentally becomes sentient...

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#115

Post by rolandog »

sschueller wrote: post #112

We are all so majorly f*d. The general public does not know nor understand this limitation. At the same time OpenAI is selling this a a tutor for your kids. Next it will be used to test those same kids. Who is going to prevent this from being used to pick military targets (EU law has an exemption for military of course) or make surgery decisions?

> or make surgery decisions?

  Analyzing surgical field...
  Identified: open chest cavity, exposed internal organs
  Organs appear gooey, gelatinous, translucent pink
  Comparing to database of aquatic lifeforms...
  93% visual match found:
  Psychrolutes marcidus, common name "blobfish"
  Conclusion: Blobfish discovered inhabiting patient's thoracic cavity
  Recommended action: Attempt to safely extract blobfish without damaging organs

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#116

Post by hehdhdjehehegwv »

sschueller wrote: post #112

We are all so majorly f*d. The general public does not know nor understand this limitation. At the same time OpenAI is selling this a a tutor for your kids. Next it will be used to test those same kids. Who is going to prevent this from being used to pick military targets (EU law has an exemption for military of course) or make surgery decisions?

Right now insurance companies make those decisions based on how your life affects the profit/loss statement at the end of the quarter. (In the USA).

So it can’t really be worse if there’s just a RNG in a box. It may be better.

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#117

Post by KeplerBoy »

bckr wrote: post #99
bearjaws wrote: post #81

I just used it to compare two smaller legal documents and it completely hallucinated that items were present in one and not the other. It did this on three discrete sections of the agreements. Using ctrl-f I was able to see that they were identical in one another. Obviously this is a single sample but saying 90% seems unlikely. They were around ~80k tokens total.

Yeah I asked for an estimate of the percentage of the US population that lives in the DMV area (DC, Maryland, Virginia) and it was off by 50% of the actual answer, which I only realized when I realized I shouldn’t trust its estimate for anything important

Those models still can't reliably do arithmetic, so how could it possibly know that number unless it's a commonly repeated fact?

Also: would you expect random people to fare any better?

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#118

Post by demilich »

throwthrowuknow wrote: post #46

This is a very promising development. It would be wise for everyone to go back and revise old experiments that failed now that this capability is unlocked. It should also make RAG even more powerful now that you can load a lot more information into the context and have it be useful.

Agreed

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#119

Post by histories »

sschueller wrote: post #112

We are all so majorly f*d. The general public does not know nor understand this limitation. At the same time OpenAI is selling this a a tutor for your kids. Next it will be used to test those same kids. Who is going to prevent this from being used to pick military targets (EU law has an exemption for military of course) or make surgery decisions?

> OpenAI is selling this a a tutor for your kids.

The Diamond Age.

Re: GPT-4o's Memory Breakthrough – Needle in a Needlestack

#120

Post by GuardianCaveman »

sschueller wrote: post #112

We are all so majorly f*d. The general public does not know nor understand this limitation. At the same time OpenAI is selling this a a tutor for your kids. Next it will be used to test those same kids. Who is going to prevent this from being used to pick military targets (EU law has an exemption for military of course) or make surgery decisions?

I was in a counter-intelligence unit briefly and there was a mathemtician who spoke to us about the work they were doing to pick targets with the idea that if you can only out one person, who would be the most disruptive. You have all these interconnected but mostly isolated terrorist cells that don't know about each other except through a few people who may not be high up in the command but who are critical for the continuing cohesive existence of that terrorist group of cells and logistics etc.

So the military already was using math to pick targets, this is just the next logical step, albeit, scary as hell step.

Post reply on HN