Live data from Hacker News

Computer use in Gemini 3.5 Flash

blog.google

131–140 of 175 posts

Re: Computer use in Gemini 3.5 Flash

#131

Earlier quoted context omitted.

The PDF reader for Gemini is extraordinarily poor in my experience. I like the writing style of this model a little better, but for most tasks people would use AI for, Gemini is probably not what you want to be using.

what is a good way to read PDFs using AI?

In my experience it really depends on what sort of pdfs you are trying to extract (ie what the content is).

For regular pdfs that have been produced in a “normal way” (ie using latex or a modern application with a “save to pdf” function) will contain the text and for those I’ve had a lot of success on general pdfs using pypdf.

“Image” Pdfs that have been produced via a scan so don’t actually contain a text transcript require actual OCR. At the moment my personal rag pipeline is doing this using a local Gemma4 model (you could use something else).

Either way I do an audit post-ingest where I select a random set of pages and also get the local gemma model to try those same set and compare. The symptoms to look out for here will depend a lot on what you’re trying to extract but I’m extracting maths mostly so I get the model to check extraction of symbols, equations etc. One thing I have consistently found useful is to look for “mojibake” (scrambled text caused by decoding in an unintended character encoding) as this almost always catches pdfs that have just extracted as pure garbage. I added this step because I was ingesting a lot of old maths pdfs which have specialist notation that wasn’t always getting correctly ingested and as they were image pdfs it was coming in as pure garbage. So the fix here is to use a specialist ocr service (I have been using “mathpix” which has been great and isn’t too expensive if you don’t want to do too much).

The other thing that can cause problems is things like tables (eg if you were trying to ingest a lot of pdfs like financials of companies etc). Those can cause problems for both the ocr and the pure text extraction methods. I don’t have a current recommendation for that because I haven’t done it recently enough and the state of the art has moved a lot. It’s something to be aware of that will require special treatment though.

Re: Computer use in Gemini 3.5 Flash

#132
My work requires me to use `agy cli` (Google AI Ultra) for development, and it's been incredibly frustrating. I strongly dislike the Gemini models because they consistently fail to grasp basic instructions. I also can't use the Claude models included in the AI Ultra plan because the agy cli wrapper makes the experience completely unusable. I'd rather use the free plan on OpenCode than deal with this Gemini setup.

Re: Computer use in Gemini 3.5 Flash

#133

Pretty doubtful about computer use/screenshotting based approaches. With Retriever AI, we construct custom accessibility trees to represent web pages and just switched over to using DeepSeek v4 Flash and its nearing 100x cost decrease. We also had great success just reverse engineering the underlying APIs of websites and then writing code to hit them. This approach of using screenshots to take actions on a webpage to…

What happens when you need to control something that isn't a web page?

Honestly with Fable I think anyone is going to be able to reverse engineer a desktop app and get the coding agent to automate it.

The Codex computer use functionality actually uses OS level accessibility trees, so thats also possible without screenshots.

Re: Computer use in Gemini 3.5 Flash

#134
post #86

Pretty doubtful about computer use/screenshotting based approaches. With Retriever AI, we construct custom accessibility trees to represent web pages and just switched over to using DeepSeek v4 Flash and its nearing 100x cost decrease. We also had great success just reverse engineering the underlying APIs of websites and then writing code to hit them. This approach of using screenshots to take actions on a webpage to…

Reverse engineering APIs is just a recipe to get blocked sooner. Good luck!

its been working fine on LinkedIn/IG, the trick is to make the requests from the main world of the website itself.

Re: Computer use in Gemini 3.5 Flash

#135

My work requires me to use `agy cli` (Google AI Ultra) for development, and it's been incredibly frustrating. I strongly dislike the Gemini models because they consistently fail to grasp basic instructions. I also can't use the Claude models included in the AI Ultra plan because the agy cli wrapper makes the experience completely unusable. I'd rather use the free plan on OpenCode than deal with this Gemini setup.

They can’t follow instructions at all. They are a year behind Claude.

Re: Computer use in Gemini 3.5 Flash

#136

Earlier quoted context omitted.

The PDF reader for Gemini is extraordinarily poor in my experience. I like the writing style of this model a little better, but for most tasks people would use AI for, Gemini is probably not what you want to be using.

what is a good way to read PDFs using AI?

Doing a preprocess using some pdf extraction and ocr tool and then feeding that to the big model is usually way more stable.

Re: Computer use in Gemini 3.5 Flash

#137
post #29
post #16

Earlier quoted context omitted.

> Computer use is such a terrible idea. It's slow, insecure, error prone, expensive. And yet having an agent able yo use a computer on your behalf is really useful. Recently I gave a Nix OS vm to my hermes agent and it has been a good experience. I don't really care if destroy the machine I can just rollback to an earlier version, and for any meaningful data he creates for me I make sure he creates a repo, commit and…

> And yet having an agent able yo use a computer on your behalf is really useful. I honestly cannot think of a single use case

Have you ever done something tedious on a computer?

Re: Computer use in Gemini 3.5 Flash

#138
post #80

Today I asked Gemini to extract a table from an PDF appendix and create C++ data table with its contents. After 15 or so iterations with corrections and new mistakes, it eventually gave up. I was floored when it said “I’m sorry, I cannot do this simple task, I’ve exceeded my error threshold and cannot do this task for you. My LLM prediction engine invents data instead of doing a simple data copy/reformat”. Stunned to…

That's interesting because my experience has been almost the opposite. A few months ago I tested Gemini on converting screenshots of tables from PDF files into CSV. I tried it on several different tables and it got every one right. It consistently outperformed ChatGPT.

Tangentially related question. Has anyone analyzed if the content that is being converted could break the model.

So let's say you have a super dull pdf ( or even a scan ) that has the same line over and over again, could this get the model into one of those loops that just keep spewing nonsense.

And thinking that further, could someone prompt inject a model with a handwritten note that only gets "activated" once it's in the context?

Re: Computer use in Gemini 3.5 Flash

#139

Computer use is such a terrible idea. It's slow, insecure, error prone, expensive. I guess if you're trying to get people to tokenmaxx it may look like a valid strategy, but ain't no way this will be delightful to users. I think it's a symptom of just not understanding how LLMs should interface with the OS because we're still in their early days. Eventually there'll be an iPhone moment for the ergonomics of LLM usage…

Tens of millions of users every day rely on Robotic Process Automation. It’s glue that hold companies together.

Re: Computer use in Gemini 3.5 Flash

#140
post #84
post #7

Earlier quoted context omitted.

It highlights the Gemini models blue since that's what the article is about. The bar heights seem consistent with the values.

They should be sorting the models by performance on the horizontal axis.

Why would they make their own product look worse?
Post reply on HN