Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why AI Infrastructure Startups Are Insanely Hard to Build

nextword.substack.com

Post reply on HN
Print view View original

111–120 of 190 posts

Re: Why AI Infrastructure Startups Are Insanely Hard to Build

#111

Post by hobs »

ankit219 wrote: post #67

Great article. I am not going to name names, but over the last one year, whenever there is a concept that became popular in Gen AI, thousands of startups pivoted to doing that. Many come from software background where the expectation was that if the code works on one dataset, it would work for everything. You can see this with 1/ Prompt engineering 2/ RAG 3/ and now, after Apple's WWDC, it's adapters. Enterprises I h…

Pretty much this, 18 months ago my CEO told me we HAD to get into this space, and I told him that basically our money came from our private product and that the only way our big enterprise customers were going to play game with us was either ironclad agreements that went all the way to openai, or more likely a completely single tenant system, which would cost far more than they were willing to pay.

Of course they went with both, and as far as I can tell both are a major disaster post layoffs :)

Re: Why AI Infrastructure Startups Are Insanely Hard to Build

#112

Post by cootsnuck »

BrunoJo wrote: post #8

Good tips, especially the point about narrowing the scope. At https://Lemonfox.ai we started with a LLM, image and speech-to-text API. Now we are only focusing on the speech-to-text API as the other areas are already very crowded and there's a lack of innovation in the speech-to-text space.

> Now we are only focusing on the speech-to-text API as the other areas are already very crowded and there's a lack of innovation in the speech-to-text space.

I'm legitimately wondering how your hosted Whisper API for $0.17/hr is supposed to compete with groq's exact same API that costs $0.03/hr.

You may be about to find out how crowded all of the AI infra spaces are.

I strongly recommend narrowing your scope far beyond modality. If you've been working with this tech and getting familiar with it then you already have valuable expertise. Pivot now or panic later. If you want to stay in the speech space find what markets are being underserved with speech AI related solutions. Are there pain points there that can be solved by a STT API? If so, build those solutions. You can't compete at the infra layer and I'm not sure why you would want to try if you don't already have something unique about your offering beyond hosting open source models. It's never good if your competition is potentially just a single developer in a company standing up your entire service internally in a week.

If you are determined to stay in the AI infra space then you'll need to be tackling a hard problem that companies want solved. Maybe take a look at fine-tuning models. Hard problem and maybe there's a hunger for it. (It's a risky one to tackle too though since it's very possible general/foundational models will maintain a grip on "good enough".)

Re: Why AI Infrastructure Startups Are Insanely Hard to Build

#113

Post by iknownthing »

siquick wrote: post #57
dvt wrote: post #33

Earlier quoted context omitted.

In my humble opinion, a chat interface (API or not) does not a product make. Not to mention that Llama is free and competitive with both (same with Mistral, heck the 7B model works great on my RTX 3080). If you started a company that blew up because you made a badass product (and let's say you used ChatGPT under the hood), you would just eventually train and deploy your own model because an LLM is not a product .

Uploading a a photo/file to a server does not a product make yet you referenced Dropbox and Instagram and that’s what they started out as. The UX of the AI applications is the moat and the infrastructure providers behind those applications is pretty much always OpenAI and Anthropic at the moment because running your own open source LLMs (which are inferior out of the box) at scale is not cheap or easy to do it right…

Agreed. The LLMs have effectively been democratized. As much as people deride "LLM wrappers", the quality of the wrapper and how creatively they use the LLM api is the differentiator.

Re: Why AI Infrastructure Startups Are Insanely Hard to Build

#114

Post by altdataseller »

BobbyJo wrote: post #100
swalsh wrote: post #31

I'm not sure I'm exactly at the edge of things, but I have 2 companies trying to setup regular meetings with me to be a beta customer. Both have promised I can help define a new product, but when I list my real problems... they aren't in the mission. Everyone wants to solve RAG (that's easy, don't need help) or they want to give me a gui I don't need, or wrap open source software like vllm. Or "solve privacy" (which…

> Want to solve a real problem, help me create custom benchmarks, clean my data, get my small parameter model to reason better etc. I recently started a company with a friend if mine to do exactly this. Ive worked at a few AI startups over the last 8 years, and the problem everyone tackles independently (and poorly) is the long tail of dealing with input data that isn't great. You build a demo with sample data that w…

What are some examples of data theyre trying to use as input?

Re: Why AI Infrastructure Startups Are Insanely Hard to Build

#115

Post by iknownthing »

nextworddev wrote: post #49
dvt wrote: post #18

Infra has always been a tarpit idea. Google didn't start out as an "infra" company, and neither did Amazon or Facebook. In fact, the few "infra companies" that did start back then (companies like Godaddy) are minuscule compared to the aforementioned. VC pouring money in LLM infra is legitimately crazy to me. It's clear as day that there will be winners of this AI cycle, but, as always, they will be companies that pro…

Author here. I think the tarpit extends to most chatgpt wrappers as well, which is why I called out pivoting prematurely to application layer is a futile exercise.

So AI infrastructure startups are tarpits and so are the wrappers? Is it just tarpits all the way down?

Re: Why AI Infrastructure Startups Are Insanely Hard to Build

#117

Post by csomar »

amarcheschi wrote: post #90
rapsey wrote: post #32

Earlier quoted context omitted.

> VC pouring money in LLM infra is legitimately crazy to me. VC business model is throwing money at the wall and seeing what sticks. They love congratulating themselves on how smart they are but at the end of the day their overall returns trail S&P 500. They are salespeople and their job is to sell themselves to private capital on how smart and connected they are.

Does their return trail s&p 500? I guess there are some of them that are successful and some that aren't, but to be honest I've only heard about vc as people who on average have a very very high return. Now, it is very possible that's just survivor bias, but I would need to search for some data

Most of the smallish funds do not publish any data. They are sales people who are selling the idea of "higher risk higher return". The sales is their alpha not the investments.

Re: Why AI Infrastructure Startups Are Insanely Hard to Build

#118

Post by [deleted] »

jdross wrote: post #21
mlsu wrote: post #10

Earlier quoted context omitted.

I would almost invert that statement. Sorry if this comes off ranty, but what exactly are people doing in the "AI space" currently that isn't "undifferentiated spam/chatbot" being sold to non-techies who heard about AI on NPR? What are real people using "AI" for that is so insanely valuable today? How much "company Y: same product with a chat window, sparks emoji" do we all need before this thing levels out and we al…

personally? - writing and refactoring code. probably 50 times a day now - improving documentation across the company - summarizing meetings automatically with follow ups - drafting most legal work before a lawyer edits (saved 70% on legal bills) - entity extraction and data cleanup for my users

[deleted]

Re: Why AI Infrastructure Startups Are Insanely Hard to Build

#119

Post by jerf »

hobs wrote: post #111
ankit219 wrote: post #67

Great article. I am not going to name names, but over the last one year, whenever there is a concept that became popular in Gen AI, thousands of startups pivoted to doing that. Many come from software background where the expectation was that if the code works on one dataset, it would work for everything. You can see this with 1/ Prompt engineering 2/ RAG 3/ and now, after Apple's WWDC, it's adapters. Enterprises I h…

Pretty much this, 18 months ago my CEO told me we HAD to get into this space, and I told him that basically our money came from our private product and that the only way our big enterprise customers were going to play game with us was either ironclad agreements that went all the way to openai, or more likely a completely single tenant system, which would cost far more than they were willing to pay. Of course they wen…

I fully expect in somewhere around 3-6 months the dam will burst and we're going to start hearing more and more about all the teams out there that are pouring tens of millions of dollars into AI and all they have to show for it is a worse version of whatever it is they were doing.

To placate the AI fans, that's not because AI isn't interesting, it's because that's how these hype cycles always go. I remember when everything had to be XML'd. XML has its uses, but a lot of money was wasted jamming it everywhere because XML Was Cool. AI has its uses, but it is still an engineering tool; it has a grain, it has things it is good at, it has things it can't just wave a magic wand and improve, the demarcation between those two things is very, very complicated, and people are being actively discouraged from thinking about those lines right now.

But there really isn't any skipping the Trough of Disillusionment on your way to the Plateau of Productivity.

Re: Why AI Infrastructure Startups Are Insanely Hard to Build

#120

Post by brummm »

ankit219 wrote: post #67

Great article. I am not going to name names, but over the last one year, whenever there is a concept that became popular in Gen AI, thousands of startups pivoted to doing that. Many come from software background where the expectation was that if the code works on one dataset, it would work for everything. You can see this with 1/ Prompt engineering 2/ RAG 3/ and now, after Apple's WWDC, it's adapters. Enterprises I h…

Random question, but what are Apple adapters? Kind of hard to google it, lol
Post reply on HN