Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro Preview

developers.googleblog.com

Post reply on HN
Print view View original

491–500 of 728 posts

Re: Gemini 2.5 Pro Preview

#491

Post by andybak »

sjhatfield wrote: post #442
jmward01 wrote: post #415

Google's models are pretty good, but their API(s) and guarantees aren't. We were just told today that 'quota doesn't guarantee capacity' so basically on-demand isn't prod capable. Add to that that there isn't a second vendor source like Anthropic and OpenAI have and Google's reliability makes it a hard sell to use them unless you can back up the calls with a different model family all together.

And gemini-1.5-pro is months from depreciation and there is no production alternative. 2.0 does not pass our benchmarks and in a regulated industry we need time to move to a new modek

("deprecation" not "depreciation")

Re: Gemini 2.5 Pro Preview

#492

Post by danpalmer »

artdigital wrote: post #449

Gemini 2.5 pro is great, but also VERY expensive with non opaque cost insights Just recently a lot of people (me included) got hit with a surprise bill, with some racking up $500 in cost for normal use I certainly got burnt and removed my API key from my tools to not accidentally use it again Example: https://x.com/pashmerepat/status/1918084120514900395?s=46

From the linked tweet the author seems to be using Gemini through another layer called OpenRouter - it seems quite possible that the issue around lack of clarity of billing/caching could be from that extra layer of indirection.

Re: Gemini 2.5 Pro Preview

#493

Post by xyzzy123 »

ssalazar wrote: post #454
jstummbillig wrote: post #314

Earlier quoted context omitted.

> no amount of prompting will get current models to approach abstraction and architecture the way a person does I find this sentiment increasingly worrisome. It's entirely clear that every last human will be beaten on code design in the upcoming years (I am not going to argue if it's 1 or 5 years away, who cares?) I wished people would just stop holding on to what amounts to nothing, and think and talk more about wha…

I code with multiple LLMs every day and build products that use LLM tech under the hood. I dont think we're anywhere near LLMs being good at code design. Existing models make _tons_ of basic mistakes and require supervision even for relatively simple coding tasks in popular languages, and its worse for languages and frameworks that are less represented in public sources of training data. I am _frequently_ having to t…

I can't point to any evidence. Also I can't think of what direct evidence I could present that would be convincing, short of an actual demonstration? I would like to try to justify my intuition though:

Seems like the key question is: should we expect AI programming performance to scale well as more compute and specialised training is thrown at it? I don't see why not, it seems an almost ideal problem domain?

* Short and direct feedback loops

* Relatively easy to "ground" the LLM by running code

* Self-play / RL should be possible (it seems likely that you could also optimise for aesthetics of solutions based on common human preferences)

* Obvious economic value (based on the multi-billion dollar valuations of vscode forks)

All these things point to programming being "solved" much sooner than say, chemistry.

Re: Gemini 2.5 Pro Preview

#494

Post by sampullman »

sweezyjeezy wrote: post #421
DanHulton wrote: post #351

Earlier quoted context omitted.

> It's entirely clear that every last human will be beaten on code design in the upcoming years Citation needed. In fact, I think this pretty clearly hits the "extraordinary claims require extraordinary evidence" bar.

I would argue that what LLMs are capable of doing right now is already pretty extraordinary, and would fulfil your extraordinary evidence request. To turn it on its head - given the rather astonishing success of the recent LLM training approaches, what evidence do you have that these models are going to plateau short of your own abilities?

I agree that they can do extraordinary things already, but have a different impression of the trajectory. I don't think it's possible for me to provide hard evidence, but between GPT2 and 3.5 I felt that there was an incredible improvement, and probably would have agreed with you at that time.

GPT4 was another big improvement, and was the first time I found it useful for non-trivial queries. 4o was nice, and there was decent bump with the reasoning models, especially for coding. However, since o1 it's felt a lot more like optimization than systematic improvement, and I don't see a way for current reasoning models to advance to the point of designing and implementing medium+ coding projects without the assistance of a human.

Like the other commenter mention, I'm sure it will happen eventually with architectural improvements, but I wouldn't bet on 1-5 years.

Re: Gemini 2.5 Pro Preview

#495

Post by cloudking »

simonw wrote: post #416

Here's a summary of the 394 comments on this post created using the new gemini-2.5-pro-preview-05-06. It looks very good to me - well grouped, nicely formatted. https://gist.github.com/simonw/7ef3d77c8aeeaf1bfe9cc6fd68760... 30,408 input, 8,535 output = 12.336 cents. 8,500 is a very long output! Finally a model that obeys my instructions to "go long" when summarizing Hacker News threads. Here's the script I used: htt…

Pelican on a bicycle?

Re: Gemini 2.5 Pro Preview

#496

Post by cma »

danpalmer wrote: post #492
artdigital wrote: post #449

Gemini 2.5 pro is great, but also VERY expensive with non opaque cost insights Just recently a lot of people (me included) got hit with a surprise bill, with some racking up $500 in cost for normal use I certainly got burnt and removed my API key from my tools to not accidentally use it again Example: https://x.com/pashmerepat/status/1918084120514900395?s=46

From the linked tweet the author seems to be using Gemini through another layer called OpenRouter - it seems quite possible that the issue around lack of clarity of billing/caching could be from that extra layer of indirection.

OpenRouter lets you fund a wallet and spend no more than that. Google will let it go out of control and they purposely delay the billing console by up to 24 hours so if you don't track it all yourself you can get hit big, especially if it is a coding error that uses up to the rate limits.

Re: Gemini 2.5 Pro Preview

#497

Post by ookblah »

segphault wrote: post #18

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

the future is probably something that looks pretty "inefficient" to us but a non-factor for a machine. i sometimes think a lot of our code structure is just for our own maintenance and conceptualization (DRY, SRP), but if you throw enough compute adn context at a problem im sure none of this even matters (as much).

at least for 90% of the CRUD apps out there, you can def abstract away the entire base framework of getting, listing, and updating records. i guess the problem is validating that data for use in other more complex workflows.

Re: Gemini 2.5 Pro Preview

#499

Post by jackthetab »

fragmede wrote: post #466
ssalazar wrote: post #454

Earlier quoted context omitted.

I code with multiple LLMs every day and build products that use LLM tech under the hood. I dont think we're anywhere near LLMs being good at code design. Existing models make _tons_ of basic mistakes and require supervision even for relatively simple coding tasks in popular languages, and its worse for languages and frameworks that are less represented in public sources of training data. I am _frequently_ having to t…

https://chatgpt.com/c/681aa95f-fa80-8009-84db-79febce49562 it becomes a question of how much you believe it's all just training data, and how much you believe the LLM's got pieces that are composable. I've given the question on the link as an interview questions and had humans been unable to give as through an answer (which I chose to believe is due to specialization on elsewhere in the stack). So we're already at a…

Unable to load conversation 681aa95f-fa80-8009-84db-79febce49562

Re: Gemini 2.5 Pro Preview

#500

Post by robinei »

tastysandwich wrote: post #431
segphault wrote: post #18

My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…

Re hallucinating APIs that don't exist - I find this with Golang sometimes. I wonder if it's because the training data doesn't just consist of all the docs and source code, but potentially feature proposals that never made it into the language. Regexes are another area where I can't get much help from LLMs. If it's something common like a phone number, that's fine. But anything novel it seems to have trouble. It will…

Since it's trained on a vast a mount of code (probably all publicly accessible Go code and more), it's seen a vast amount of different bespoke APIs for doing all kinds of things. I'm sure some of that will leak into the output from time to time. And to some extent can generalize, so it may just invent APIs.
Post reply on HN