Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Deep Research

github.com

Post reply on HN
Print view View original

11–20 of 83 posts

Re: Open Deep Research

#11

Post by swyx »

transpute wrote: post #8

https://techcrunch.com/2025/02/04/hugging-face-researchers-a... > On GAIA, a benchmark for general AI assistants, Open Deep Research achieves a score of 54%. That’s compared with OpenAI deep research’s score of 67.36%..Worth noting is that there are a number of OpenAI deep research “reproductions” on the web, some of which rely on open models and tooling. The crucial component they — and Open Deep Research — lack is…

theres always a lot of openTHING clones of THING after THING is announced. they all usually (not always[1]!) disappoint/dont get traction. i think the causes are

1. running things in production/self hosting is more annoying than just paying like 20-200/month

2. openTHING makers often overhype their superficial repros ("I cloned Perplexity in a weekend! haha! these VCs are clowns!") and trivializing the last mile, most particularly in this case...

3. long horizon planning trained with RL in a tight loop that is not available in the open (yes, even with deepseek). the thing that makes OAI work as a product+research company is that products are never launched without first establishing a "prompted baseline" and then finetuning the model from there (we covered this process in https://latent.space/p/karina recently) - which becomes an evals/dataset suite that eventually gets merged in once performance impacts stabilize

4. that said, smolagents and HF are awesome and I like that they are always this on the ball. how does this make money for HF?

---

[1]: i think opendevin/allhands is a pretty decent competitor to devin now

Re: Open Deep Research

#12

Post by cma »

tptacek wrote: post #4
tkellogg wrote: post #2

it's just an example, but it's great to see smolagents in practice. I wonder how well the import whitelist approach works for code interpreter security.

I know some of the point of this is running things locally, but for agent workflows like this some of this seems like a solved problem: just run it on a throwaway VM. There's lots of ways to do that quickly.

The rise of captchas on regular content, no longer just for posting content, could ruin this. Cloudflare and other companies have set things up to go through a few hand selected scrapers and only they will be able to offer AI browsing and research services.

Re: Open Deep Research

#14

Post by AznHisoka »

cma wrote: post #12
tptacek wrote: post #4

Earlier quoted context omitted.

I know some of the point of this is running things locally, but for agent workflows like this some of this seems like a solved problem: just run it on a throwaway VM. There's lots of ways to do that quickly.

The rise of captchas on regular content, no longer just for posting content, could ruin this. Cloudflare and other companies have set things up to go through a few hand selected scrapers and only they will be able to offer AI browsing and research services.

What about Cloudflare itself? It might constitute an abuse of sorts of their leadership position, but couldn’t they dominate the AI research/agent market if they wanted? (Or maybe that’s what you were implying too)

Re: Open Deep Research

#15

Post by tshepom »

swyx wrote: post #11
transpute wrote: post #8

https://techcrunch.com/2025/02/04/hugging-face-researchers-a... > On GAIA, a benchmark for general AI assistants, Open Deep Research achieves a score of 54%. That’s compared with OpenAI deep research’s score of 67.36%..Worth noting is that there are a number of OpenAI deep research “reproductions” on the web, some of which rely on open models and tooling. The crucial component they — and Open Deep Research — lack is…

theres always a lot of openTHING clones of THING after THING is announced. they all usually (not always[1]!) disappoint/dont get traction. i think the causes are 1. running things in production/self hosting is more annoying than just paying like 20-200/month 2. openTHING makers often overhype their superficial repros ("I cloned Perplexity in a weekend! haha! these VCs are clowns!") and trivializing the last mile, mos…

Maybe these open projects start to get more attention when we have a distribution system/App Store for AI projects. I know YC is looking to fund this https://www.ycombinator.com/rfs

Re: Open Deep Research

#16

Post by tptacek »

vineyardmike wrote: post #10
ATechGuy wrote: post #5

Earlier quoted context omitted.

VM is not the right abstraction because of performance and resource requirements. VMs are used because nothing exists that provides same or better isolation. Using a throwaway VM for each AI agent would be highly inefficient (think wasted compute and other resources, which is the opposite of what DeepSeek exemplified).

Is “DeepSeek” going to be the new trendy way to say to not be wasteful? I don’t think DS is a good example here. Mostly because it’s a trendy thing, and the company still has $1B in capex spend to get there . Firecracker has changed the nature of “VMs” into something cheap and easy to spin up and throw away while maintaining isolation. There’s no reason not to use it (besides complexity, I guess). Besides, the entire…

I'm not even talking about Firecracker; for the duration of time things like these run, you could get a satisfactory UX with basic EC2.

Re: Open Deep Research

#17

Post by tptacek »

cma wrote: post #12
tptacek wrote: post #4

Earlier quoted context omitted.

I know some of the point of this is running things locally, but for agent workflows like this some of this seems like a solved problem: just run it on a throwaway VM. There's lots of ways to do that quickly.

The rise of captchas on regular content, no longer just for posting content, could ruin this. Cloudflare and other companies have set things up to go through a few hand selected scrapers and only they will be able to offer AI browsing and research services.

I think the opposite problem is going to occur with captchas for whatever it's worth: LLMs are going to obsolete them. It's an arms race where the defender has a huge constraint the attacker doesn't (pissing off real users); in that way, it's kind of like the opposite dynamics that password hashes exploit.

Re: Open Deep Research

#18

Post by ATechGuy »

tptacek wrote: post #9
ATechGuy wrote: post #5

Earlier quoted context omitted.

VM is not the right abstraction because of performance and resource requirements. VMs are used because nothing exists that provides same or better isolation. Using a throwaway VM for each AI agent would be highly inefficient (think wasted compute and other resources, which is the opposite of what DeepSeek exemplified).

To which performance and resource requirements are you referring? A cloud VM runs as long as the agent runs, then stops running.

I mean performance overheads of an OS process running in a VM to (vs no VM) and additional resource requirements for running a VM, including memory and additional kernel. You can pull relevant numbers from academic papers.

Re: Open Deep Research

#19

Post by [deleted] »

swyx wrote: post #11
transpute wrote: post #8

https://techcrunch.com/2025/02/04/hugging-face-researchers-a... > On GAIA, a benchmark for general AI assistants, Open Deep Research achieves a score of 54%. That’s compared with OpenAI deep research’s score of 67.36%..Worth noting is that there are a number of OpenAI deep research “reproductions” on the web, some of which rely on open models and tooling. The crucial component they — and Open Deep Research — lack is…

theres always a lot of openTHING clones of THING after THING is announced. they all usually (not always[1]!) disappoint/dont get traction. i think the causes are 1. running things in production/self hosting is more annoying than just paying like 20-200/month 2. openTHING makers often overhype their superficial repros ("I cloned Perplexity in a weekend! haha! these VCs are clowns!") and trivializing the last mile, mos…

[deleted]

Re: Open Deep Research

#20

Post by transpute »

rvz wrote: post #13

Of course. The first of many open source versions of 'Deep Research' projects are now appearing as predicted [0] but in less than a month. Faster than expected. Open source is already at the finish line. [0] https://news.ycombinator.com/item?id=42913379

> Nothing that Perplexity + DeepSeek-R1 can already do

Any public comparisons of OAI Deep Research report quality with Perplexity + DeepSeek-R1, on the same query?

How do cost and query limits compare?

Post reply on HN