Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Nova

aws.amazon.com

Post reply on HN
Print view View original

141–150 of 157 posts

Re: Amazon Nova

#141

Post by jmward01 »

plumeria wrote: post #119
jmward01 wrote: post #57

No audio support: The models are currently trained to process and understand video content solely based on the visual information in the video. They do not possess the capability to analyze or comprehend any audio components that are present in the video. This is blowing my mind. gemini-1.5-flash accidentally knows how to transcribe amazingly well but it is -very- hard to figure out how to use it well and now Amazon…

Is Gemini better than Whisper for transcribing?

'better' is always a loaded term with ASR. Gemini 1.5 flash can transcribe for 0.01/hour of audio and gives strong results. If you want timing and speaker info you need to use the previous version and a -lot- of tweaking of the prompt or else it will hallucinate the timing info. Give it a try. It may be a lot better for your use case.

Re: Amazon Nova

#142

Post by int_19h »

marcosdumay wrote: post #139
meta_x_ai wrote: post #76

Earlier quoted context omitted.

None of those matters (except multimodal). If you are running a business, the only thing that matters is a) How does it perform on my set of evals b) What is the cost/latency of serving it to my consumers. It shouldn't matter to me how many parameters, corpus it is trained on, whether it's LLM or Transformer or something else

> How does it perform on my set of evals What kinds of eval? Personally, I have no idea what kind of data you can throw at a "foundation model" and what kind of response you will get. The only thing it says is that there's machine learning involved... Once you get enough context to understand it's not a spin-off of a TV series.

"Foundation model" is not Amazon lingo, though, but pretty standard industry term at this point. If you're doing any sort of AI in prod, you know what it means.

Re: Amazon Nova

#143

Post by int_19h »

SparkyMcUnicorn wrote: post #51
kajecounterhack wrote: post #4

Earlier quoted context omitted.

TL;DR comparison of models vs frontier models on public benchmarks here https://imgur.com/a/CKMIhmm

This doesn't include all the benchmarks. The one that really stands out is GroundUI-1K, where it beats the competition by 46%. Nova Pro looks like it could be a SOTA-comparable model at a lower price point.

Benchmarks are way too easy to game. There's no shortage of models that "beat GPT-4" according to some benchmark or another, that are obviously nowhere even close when you try them on novel tasks.

Re: Amazon Nova

#144

Post by int_19h »

xnx wrote: post #108
mrg3_2013 wrote: post #95

DOA When marketing talks about price delta and not quality of the output, it is DOA. For LLMs, quality is a more important metric and Nova would always try to play catch with the leaderboard forever.

Maybe. The major models seem to be about tied in terms of quality right now, so cost and ease of use (e.g. you already have an AWS account set up for billing) could be a differentiator.

The major models are not tied in terms of quality. GPT-4 and GPT-o1 still beat everyone else by a significant margin on tasks that require in-depth reasoning. There's a reason why people just don't go for the cheapest option, whatever the benchmarks say.

Re: Amazon Nova

#145

Post by int_19h »

smt88 wrote: post #37
jklinger410 wrote: post #36

It's really amusing how bad Amazon is at writing and designing UI. For a company of their size and scope it's practically unforgivable. But they always get away with it.

You say they "get away with it," but it makes more sense to conclude that UI design has a lot lower ROI than we assume it does as users.

It makes more sense to conclude that if you have market dominance, you can get away with a lot, especially since we see this time and again in other matters, not just UI.

Re: Amazon Nova

#146

Post by xnx »

int_19h wrote: post #144
xnx wrote: post #108

Earlier quoted context omitted.

Maybe. The major models seem to be about tied in terms of quality right now, so cost and ease of use (e.g. you already have an AWS account set up for billing) could be a differentiator.

The major models are not tied in terms of quality. GPT-4 and GPT-o1 still beat everyone else by a significant margin on tasks that require in-depth reasoning. There's a reason why people just don't go for the cheapest option, whatever the benchmarks say.

> GPT-4 and GPT-o1 still beat everyone else by a significant margin on tasks that require in-depth reasoning

I haven't seen examples of this. Do you know where I could find some?

Re: Amazon Nova

#147

Post by int_19h »

xnx wrote: post #146
int_19h wrote: post #144

Earlier quoted context omitted.

The major models are not tied in terms of quality. GPT-4 and GPT-o1 still beat everyone else by a significant margin on tasks that require in-depth reasoning. There's a reason why people just don't go for the cheapest option, whatever the benchmarks say.

> GPT-4 and GPT-o1 still beat everyone else by a significant margin on tasks that require in-depth reasoning I haven't seen examples of this. Do you know where I could find some?

Here's a fairly simple test that I throw at any model that claims to be "GPT-4 level": https://news.ycombinator.com/item?id=42262661

For more complicated stuff, I did some experiments using LLMs to drive high-level AI decisions in video games. Basically, it gets a data schema and a question like "what do you do next?", and can query the schema to retrieve the info that it thinks it needs to give the best answer to that. GPT-4 and GPT-o1 especially are consistently the best performers there, both in terms of richness of queries they produce, and how they make use of them.

There's also a bunch of interesting examples along the same lines here: https://github.com/cpldcpu/MisguidedAttention. Although I should note that even top OpenAI models have troubles with much of this stuff.

https://github.com/fairydreaming/farel-bench is another interesting benchmark because it's so simple, and yet look at the number disparity in that last column! It's easy to scale, too.

Unfortunately, we're still at the point in this game where even seemingly trivial and unrelated minor changes in the prompt (e.g. slightly rewording it, and even capitalization in some cases) can have large effect on quality of output, which IMO is a tell-tale sign when the model is really operating in a "stochastic parrot" mode more so than any kind of actual reasoning. Thus benchmarks can be used as a way to screen out the poorly performing models, but they cannot reliably predict how well a model will actually do what you need it to do.

Re: Amazon Nova

#148

Post by attentive »

SparkyMcUnicorn wrote: post #51
kajecounterhack wrote: post #4

Earlier quoted context omitted.

TL;DR comparison of models vs frontier models on public benchmarks here https://imgur.com/a/CKMIhmm

This doesn't include all the benchmarks. The one that really stands out is GroundUI-1K, where it beats the competition by 46%. Nova Pro looks like it could be a SOTA-comparable model at a lower price point.

on https://aider.chat/docs/leaderboards/ Nova Pro is on par with Yi Coder 9B Chat. Which is not very inspiring.

Re: Amazon Nova

#149

Post by attentive »

HarHarVeryFunny wrote: post #11

Since Amazon are building their own frontier models, what's the point of their relationship with Anthropic ?

Bedrock might be the best way to consume sonnet 3.5. So people setup and use bedrock while might try other hosted models like Nova.

On the other hand, not many are going to onboard bedrock if they don't have SOTA models in the offering.

Re: Amazon Nova

#150

Post by OJFord »

weitendorf wrote: post #111
OJFord wrote: post #101

Earlier quoted context omitted.

Your 14 steps appear to be 'create an IAM user'..?

AWS does not use the exact same authn/authz/identity model or terminology as other providers, and for people familiar with other models, it's pretty non-trivial to adapt to. I recently posted a rant about this to https://www.reddit.com/r/aws/comments/1geczoz/the_aws_iam_id... Personally I am more familiar with directly using API keys or auth tokens than AWS's IAM users (which are more similar to what I'd call "servic…

If you're looking for a generative AI model API only, I think Nova is not for you. If you want to build that capability into your cloud application, it uses exactly the model you expect and have, and you just add a new policy/role/whatever for whatever piece of it's going to use Nova.
Post reply on HN