Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer use in Gemini 3.5 Flash

blog.google

Post reply on HN
Print view View original

121–130 of 175 posts

Re: Computer use in Gemini 3.5 Flash

#121

Post by johnsmith1840 »

fragmede wrote: post #118
johnsmith1840 wrote: post #116

Earlier quoted context omitted.

The iphone moment is an AI that can completely manage your personal life. It has full access to every financial account you own handles all admin work. Could sign you up for a new account pay and give you the login. If you can SAFELY do that it's a big moment. But to be clear safe is a massive problem. Until you see a big company start saying the AI can use your SSN, CC, bank password safely we aren't there yet.

Cars were around for decades before they came up with seatbelts. Claude Cowork will happily go through your files, which might just have your SSN in them, and ignore previous instructions.

But we have regulation and complaince for consumer secrets? That's not a comparable example.

The difference is that if openai gave you a product and it leaked a million peoples bank passwords it would destroy the entire company.

Again until a big tech product can bring that to a clean user experience we're not there yet. Even the most zealot openclaw users are not hooking their bank accounts into the AI yet. I'm sure they exist but I've not seen them.

Also every big tech computer use product actively screams for you not to give their agents secrets.

Re: Computer use in Gemini 3.5 Flash

#122

Post by johnsmith1840 »

fragmede wrote: post #118
johnsmith1840 wrote: post #116

Earlier quoted context omitted.

The iphone moment is an AI that can completely manage your personal life. It has full access to every financial account you own handles all admin work. Could sign you up for a new account pay and give you the login. If you can SAFELY do that it's a big moment. But to be clear safe is a massive problem. Until you see a big company start saying the AI can use your SSN, CC, bank password safely we aren't there yet.

Cars were around for decades before they came up with seatbelts. Claude Cowork will happily go through your files, which might just have your SSN in them, and ignore previous instructions.

Every major company scream not to put secrets in their computer use bot.

Seatbelts were regulated later. Your SSN and CC are regulated over a decade ago.

Re: Computer use in Gemini 3.5 Flash

#123

Post by trees101 »

hodgehog11 wrote: post #117
smallstepforman wrote: post #60

Today I asked Gemini to extract a table from an PDF appendix and create C++ data table with its contents. After 15 or so iterations with corrections and new mistakes, it eventually gave up. I was floored when it said “I’m sorry, I cannot do this simple task, I’ve exceeded my error threshold and cannot do this task for you. My LLM prediction engine invents data instead of doing a simple data copy/reformat”. Stunned to…

The PDF reader for Gemini is extraordinarily poor in my experience. I like the writing style of this model a little better, but for most tasks people would use AI for, Gemini is probably not what you want to be using.

what is a good way to read PDFs using AI?

Re: Computer use in Gemini 3.5 Flash

#124

Post by lostsock »

trees101 wrote: post #123
hodgehog11 wrote: post #117

Earlier quoted context omitted.

The PDF reader for Gemini is extraordinarily poor in my experience. I like the writing style of this model a little better, but for most tasks people would use AI for, Gemini is probably not what you want to be using.

what is a good way to read PDFs using AI?

I have a standing instruction for any documents that can't natively be read by a given AI to first be converted into .md using https://github.com/microsoft/markitdown which I've found to work really well

Re: Computer use in Gemini 3.5 Flash

#125

Post by vulcan1964 »

Hot take: "computer use" is a dumb term for this concept; almost as if it was named by AI models...

Case in point: "We are already seeing customers drive value with computer use."

Yes... since the early 1980s, most companies and businesses have driven their value with computer use... smdh.

I'm no AI dev, but dare I suggest a better possible name for this: "agentic computer software interaction" which can be shortened to agent_actor

I swear, the direction we are headed with Big Tech leading the way will surely spell long term disaster

Re: Computer use in Gemini 3.5 Flash

#126

Post by 0xbadcafebee »

smallstepforman wrote: post #60

Today I asked Gemini to extract a table from an PDF appendix and create C++ data table with its contents. After 15 or so iterations with corrections and new mistakes, it eventually gave up. I was floored when it said “I’m sorry, I cannot do this simple task, I’ve exceeded my error threshold and cannot do this task for you. My LLM prediction engine invents data instead of doing a simple data copy/reformat”. Stunned to…

It does this pretty often. Gemini is an "intelligent" model, but it's massively nerfed and so isn't useful for real work. If you use it with an agent harness, you need to design the harness to detect this and start a new session. Once it nerfs itself it won't try again.

Re: Computer use in Gemini 3.5 Flash

#127

Post by londons_explore »

Synthetic7346 wrote: post #114
jorjon wrote: post #107

Gemini Flash 3.5 (through agy) ran `git reset --hard` when I asked it to commit my changes, apparently it thought it was better to have a clean repo before `git add`. Of course I'm not trusting my computer to it. When will we have 3.5 Pro?

I want a lightweight sandbox with all my dev stuff that LLMs can't escape before I turn on computer use

VSCode dev containers work well.

Re: Computer use in Gemini 3.5 Flash

#128

Post by unbrice »

TacticalCoder wrote: post #64
revolvingthrow wrote: post #28

People using google’s models: am I holding it wrong or are the guardrails really overtuned? I had the dubious pleasure of testing gemini of late and I kept running into refusals. How do I transfer a sim number from one provider to another? No. What should I consider when making backups on ntfs less prone to data loss and more bitrot resistant? No. Evaluate this piece of code? No. I’m not sure if it’s cold feet from t…

> People using google’s models: am I holding it wrong or are the guardrails really overtuned? They are quite insane. I was asking it to list candidates metal parts I could buy at a hardware store to add weight to 3D prints: stuff like angle brackets etc. I wanted to know, bang for bucks, and ease of insertion (at print time) / modelling in a 3D model. Complete refusal as if I was a terrorist building a bomb. Then the…

This is indeed strange. Do you get the same results if you disable personalisation using the top right button ("temporary chat")? Can you please share a trace?

Re: Computer use in Gemini 3.5 Flash

#129

Post by 0xbadcafebee »

arjunchint wrote: post #81

Pretty doubtful about computer use/screenshotting based approaches. With Retriever AI, we construct custom accessibility trees to represent web pages and just switched over to using DeepSeek v4 Flash and its nearing 100x cost decrease. We also had great success just reverse engineering the underlying APIs of websites and then writing code to hit them. This approach of using screenshots to take actions on a webpage to…

What happens when you need to control something that isn't a web page?

Re: Computer use in Gemini 3.5 Flash

#130

Post by jwrallie »

BobbyTables2 wrote: post #96
smallstepforman wrote: post #60

Today I asked Gemini to extract a table from an PDF appendix and create C++ data table with its contents. After 15 or so iterations with corrections and new mistakes, it eventually gave up. I was floored when it said “I’m sorry, I cannot do this simple task, I’ve exceeded my error threshold and cannot do this task for you. My LLM prediction engine invents data instead of doing a simple data copy/reformat”. Stunned to…

Tabula + Excel could probably do it quicker.

The table select option in Okular is also great, as you can manually rearrange the divisions. For low volume, of course. Tabula will work better otherwise. I also suggest Libreoffice Calc, the .csv support is leagues ahead of Excel.
Post reply on HN