Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

61–70 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#61

Post by ivan_gammel »

m101 wrote: post #56

So if we assume this is the future, the useful life of many semiconductors will fall substantially. What part of the semiconductor supply chain would have pricing power in a world of producing many more different designs? Perhaps mask manufacturers?

It might be not that bad. “Good enough” open-weight models are almost there, the focus may shift to agentic workflows and effective prompting. The lifecycle of a model chip will be comparable to smartphones, getting longer and longer, with orchestration software being responsible for faster innovation cycles.

Re: How Taalas “prints” LLM onto a chip?

#62

Post by alcasa »

rustybolt wrote: post #9

Note that this doesn't answer the question in the title, it merely asks it.

Frankly the most critical question is if they can really take shortcuts on DV etc, which are the main reasons nobody else tapes out new chips for every model. Note that their current architecture only allows some LORA-Adapter based fine-tuning, even a model with an updated cutoff date would require new masks etc. Which is kind of insane, but props to them if they can make it work.

From some announcements 2 years ago, it seems like they missed their initial schedule by a year, if that's indicative of anything.

For their hardware to make sense a couple of things would need to be true: 1. A model is good enough for a given usecase that there is no need to update/change it for 3-5 years. Note they need to redo their HW-Pipeline if even the weights change. 2. This application is also highly latency-sensitive and benefits from power efficiency. 3. That application is large enough in scale to warrant doing all this instead of running on last-gen hardware.

Maybe some edge-computing and non-civilian use-cases might fit that, but given the lifespan of models, I wonder if most companies wouldn't consider something like this too high-risk.

But maybe some non-text applications, like TTS, audio/video gen, might actually be a good fit.

Re: How Taalas “prints” LLM onto a chip?

#63

Post by brainless »

lm28469 wrote: post #59
brainless wrote: post #55

Earlier quoted context omitted.

New GPUs come out all the time. New phones come out (if you count all the manufacturers) all the time. We do not need to always buy the new one. Current open weight models < 20B are already capable of being useful. With even 1K tokens/second, they would change what it means to interact with them or for models to interact with the computer.

hm yeah I guess if they stick to shitty models it works out, I was talking about the models people use to actually do things instead of shitposting from openclaw and getting reminders about their next dentist appointment.

The trick with small models is what you ask them to do. I am working on a data extraction app (from emails and files) that works entirely local. I applied for Taalas API because it would be awesome fit.

dwata: Entirely Local Financial Data Extraction from Emails Using Ministral 3 3B with Ollama: https://youtu.be/LVT-jYlvM18

https://github.com/brainless/dwata

Re: How Taalas “prints” LLM onto a chip?

#64

Post by lancebeet »

lm28469 wrote: post #50

Who's going to pay for custom chips when they shit out new models every two weeks and their deluded CEOs keep promising AGI in two release cycles?

You obviously don't believe that AGI is coming in two release cycles, and you also don't seem to have much faith in the new models containing massive improvements over the last ones. So the answer to who is going to pay for these custom chips seems to be you.

Re: How Taalas “prints” LLM onto a chip?

#65

Post by fennecbutt »

zupa-hu wrote: post #47
roncesvalles wrote: post #20

Earlier quoted context omitted.

That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.

This would be a hell of a hot power bank. It uses about as much power as my oven. So probably more like inside a huge cooling device outside the house. Or integrated into the heating system of the house. (Still compelling!)

*the whole server uses 2.2kw or whatever, not a single board. I think that was for 8 boards or something.

Re: How Taalas “prints” LLM onto a chip?

#66

Post by coppsilgold »

How feasible would it be to integrate a neural video codec into the SoC/GPU silicon?

There would be model size constraints and what quality they can achieve under those constraints.

Would be interesting if it didn't make sense to develop traditional video codecs anymore.

The current videolatents networks (part of the generative AI model for video) don't optimize just for compression. And you probably wouldn't want variable size input in an actual video codec anyway.

Re: How Taalas “prints” LLM onto a chip?

#67

Post by briansm »

I wonder if you could use the same technique (RAM models as ROM) for something like Whisper Speech-to-text, where the models are much smaller (around a Gigabyte) for a super-efficient single-chip speech recognition solution with tons of context knowledge.

Re: How Taalas “prints” LLM onto a chip?

#68

Post by NitpickLawyer »

beAroundHere wrote: post #6
owenpalmer wrote: post #5

> Kinda like a CD-ROM/Game cartridge, or a printed book, it only holds one model and cannot be rewritten. Imagine a slot on your computer where you physically pop out and replace the chip with different models, sort of like a Nintendo DS.

That's the kind of hardware am rooting for. Since it'll encourage Open weighs models, and would be much more private. Infact, I was thinking, if robots of future could have such slots, where they can use different models, depending on the task they're given. Like a Hardware MoE.

> Since it'll encourage Open weighs models

Is this accurate? I don't know enough about hardware, but perhaps someone could clarify: how hard would it be to reverse engineer this to "leak" the model weights? Is it even possible?

There are some labs that sell access to their models (mistral, cohere, etc) without having their models open. I could see a world where more companies can do this if this turns out to be a viable way. Even to end customers, if reverse engineering is deemed impossible. You could have a device that does most of the inference locally and only "call home" when stumped (think alexa with local processing for intent detection and cloud processing for the rest, but better).

Re: How Taalas “prints” LLM onto a chip?

#69

Post by kilroy123 »

owenpalmer wrote: post #5

> Kinda like a CD-ROM/Game cartridge, or a printed book, it only holds one model and cannot be rewritten. Imagine a slot on your computer where you physically pop out and replace the chip with different models, sort of like a Nintendo DS.

This is what I've been wanting! Just like those eGPUs you would plug into your Mac. You would have a big model or device capable of running a top-tier model under your desk. All local, completely private.

Re: How Taalas “prints” LLM onto a chip?

#70

Post by lm28469 »

NinjaTrance wrote: post #60
lm28469 wrote: post #50

Who's going to pay for custom chips when they shit out new models every two weeks and their deluded CEOs keep promising AGI in two release cycles?

To run Llama 3.1 8B locally, you would need a GPU with a minimum of 16 GB of VRAM, such as an NVIDIA RTX 3090. Talas promises a 10x higher throughtput, being 10x cheaper and using 10x less electricity. Looks like a good value proposition.

What do you do with 8b models ? They can't even reliably create a .txt file or do any kind of tool calling
Post reply on HN