Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

After the Bubble

tbray.org

Post reply on HN
Print view View original

71–80 of 86 posts

Re: After the Bubble

#71

Post by keeda »

evanelias wrote: post #67
keeda wrote: post #64

Earlier quoted context omitted.

1. Being a VP in these companies does not imply they have an understanding of financing, accounting or data-center economics unless their purview covered or was very close to the teams procuring and running the infrastructure. 2. That level of seniority does, on the other hand, expose them to a lot of the shenanigans going on in those companies, which could credibly lead them to develop a "big tech bad" mindset.

So on the one side, we have a famous widely-respected 70-year-old software engineer with a lengthy wikipedia bio and history of industry-impactful accomplishments. His statements on depreciation here are aligned with things that have been discussed on HN numerous times from my recollection; here are a couple discussions that I found quickly: https://news.ycombinator.com/item?id=29005669 (search "depreciation"), https…

How do his accomplishments (numerous as they may be) matter if they are only tangentially related to the topic he's discussing? The fact that his take aligns with many others' does not help if they are all outsiders ruminating on hearsay and innuendo about tightly guarded, non-public numbers. He may well simply be echoing the talking points he has heard.

I mean, per the top comment in this thread, he cites an article -- the only source with official, concrete numbers -- that seems to contradict his thesis about depreciation: https://news.ycombinator.com/item?id=46208221

I'm no expert on hardware depreciation, but the more I've dug into this, the more I'm convinced people are just echoing a narrative that they don't really understand. Somewhat like stochastic parrots, you could say ;-)

My only goal here is to get a real sense of the depreciation story. Partially out of financial interest, but more because I'm really curious about how this will impact the adoption of AI.

Re: After the Bubble

#72

Post by hackernews_7364 »

evanelias wrote: post #70
hackernews_7364 wrote: post #69

Earlier quoted context omitted.

There are many legitimate concerns about the financial implications of these huge investments in AI. In fact the podcast that he references is great at providing _informed_ and _nuanced_ observations about all of this - Paul Kedrosky is great. BUT (my point) Is that the article is terrible at reflecting all of that and makes wrong and misleading comments about it. The idea that companies depreciating assets is them "…

Those statements are quite clearly simplifications, provided for readers who lack accounting experience (e.g. the majority of readers, since it is an engineering-focused blog). That's entirely reasonable in context. I don't know why you're laser-focusing on these two phrases as some kind of smoking gun that discredits the entire piece. On the "management gets to pick your depreciation period" one in particular, despi…

I don't think they are simplifications, I think they are misleading.

Notice that I never disagreed with his underlying take. Everyone should have concerns about investments in the magnitude of 1 or 2 pp of GDP that serve novel and unproven business models. He even cites a few people that I love to read - Matt Levine. I just think that the way he is representing things is grossly misleading.

If I had little financial knowledge my take away from reading his article would be (and _this_ is a simplification for comedic purposes): all of these big tech companies are all "cooking the books", and they all know that these investments are all bad and they just don't care (for some reason)... and they just hide all of these costs because they don't have money... And this is not a fair representation.

I think we are too forgiving of these type of "simplifications", if you think they are reasonable, ok. I just shared my take, probably I should have stuck with just the observations on the content and left out the subtle ad hominem, so that's a fair point.

Re: After the Bubble

#74

Post by kevin061 »

keeda wrote: post #68
dragonwriter wrote: post #59

Earlier quoted context omitted.

> The dotcom bubble was a bunch of Silicon Valley types buying fancy domain names and getting showered in money before they even released anything remotely useful. The AI bubble involves a lot of that, too. > AI companies are releasing useful things right this second, even if they still require human oversight, they are also able to significantly accelerate many tasks. So were the Googles and other leading firms in t…

I feel like this bubble actually has two bubbles happening: 1. The infra build out bubble: this is mostly the hypescalers and Nvidia. 2. The AI company valuation bubble: this includes the hyperscalers, pure-play AI companies like OpenAI and Anthropic, and the swarm of startups that are either vaporware or just wrappers on top of the same set of APIs. There will probably be a pop in (2), especially the random startups…

OK, that is interesting. Separating infra from AI valuation. I can see what you mean though because stock prices are volatile and unpredictable but a datacenter will remain in place even if its owner goes bankrupt.

However, I think the AI datacenter craze is definitely going to experience a shift. GPU chips get obsolete really fast, especially now that we are moving into specialised neural chips. All those datacenters with thousands of GPUs will be outcompeted by datacenters with 1/4th the power demand and 1/10th the physical footprint due to improved efficiency within a few years. And if indeed the valuation collapses and investors pull out of these companies, where are these datacenters supposed to go? Would you but a datacenter chock full of obsolete chips?

Re: After the Bubble

#75

Post by evanelias »

keeda wrote: post #71
evanelias wrote: post #67

Earlier quoted context omitted.

So on the one side, we have a famous widely-respected 70-year-old software engineer with a lengthy wikipedia bio and history of industry-impactful accomplishments. His statements on depreciation here are aligned with things that have been discussed on HN numerous times from my recollection; here are a couple discussions that I found quickly: https://news.ycombinator.com/item?id=29005669 (search "depreciation"), https…

How do his accomplishments (numerous as they may be) matter if they are only tangentially related to the topic he's discussing? The fact that his take aligns with many others' does not help if they are all outsiders ruminating on hearsay and innuendo about tightly guarded, non-public numbers. He may well simply be echoing the talking points he has heard. I mean, per the top comment in this thread, he cites an article…

From what I've seen, folks at that level have massive amounts of insider knowledge, a huge network of well-informed connections at every big tech company, and a stock portfolio in the 8 to 9 figure range. Personally I would strongly doubt he's simply aligning with "outsiders ruminating on hearsay and innuendo", but I guess believe what you'd like!

Re: After the Bubble

#76

Post by evanelias »

jsnell wrote: post #22

> Nobody who is doing this is willing to come clean with hard numbers but there are data points, for example from Meta and (very unofficially) Google. The Meta link does not support the point. It's actually implying a MTBF of over 5 years at 90% utilizization even if you assume there's no bathtub curve. Pretty sure that lines up with the depreciation period. The Google link is even worse. It links to https://www.toms…

> It's actually implying a MTBF of over 5 years [...] Pretty sure that lines up with the depreciation period.

You're assuming this is normal, for the MTBF to line up with the depreciation schedule. But the MTBF of data center hardware is usually quite a bit longer than the depreciation schedule right? If I recall correctly, for servers it's typically double or triple, roughly. Maybe less for GPUs, I'm not directly familiar, but a quick web search suggests these periods shouldn't line up for GPUs either.

Re: After the Bubble

#77

Post by keeda »

kevin061 wrote: post #74
keeda wrote: post #68

Earlier quoted context omitted.

I feel like this bubble actually has two bubbles happening: 1. The infra build out bubble: this is mostly the hypescalers and Nvidia. 2. The AI company valuation bubble: this includes the hyperscalers, pure-play AI companies like OpenAI and Anthropic, and the swarm of startups that are either vaporware or just wrappers on top of the same set of APIs. There will probably be a pop in (2), especially the random startups…

OK, that is interesting. Separating infra from AI valuation. I can see what you mean though because stock prices are volatile and unpredictable but a datacenter will remain in place even if its owner goes bankrupt. However, I think the AI datacenter craze is definitely going to experience a shift. GPU chips get obsolete really fast, especially now that we are moving into specialised neural chips. All those datacenter…

Right, the obsolence rate of GPUs is one of the primary drivers of the depreciation shenanigans aspect of the bubble.

However, I've come across a number of articles that paint a very different picture. E.g. this one is from someone in the GPU farm industry and is clearly going to be biased, but by the same token seems to be more knowledgeable. They claim that the demand is so high that even 9-year old generations still get booked like hot cakes: https://www.whitefiber.com/blog/understanding-gpu-lifecycle

Re: After the Bubble

#78

Post by brazukadev »

keeda wrote: post #71
evanelias wrote: post #67

Earlier quoted context omitted.

So on the one side, we have a famous widely-respected 70-year-old software engineer with a lengthy wikipedia bio and history of industry-impactful accomplishments. His statements on depreciation here are aligned with things that have been discussed on HN numerous times from my recollection; here are a couple discussions that I found quickly: https://news.ycombinator.com/item?id=29005669 (search "depreciation"), https…

How do his accomplishments (numerous as they may be) matter if they are only tangentially related to the topic he's discussing? The fact that his take aligns with many others' does not help if they are all outsiders ruminating on hearsay and innuendo about tightly guarded, non-public numbers. He may well simply be echoing the talking points he has heard. I mean, per the top comment in this thread, he cites an article…

> My only goal here is to get a real sense of the depreciation story

It doesn't look like your goal is to get a real sense or at least your strategy is really poor as you have an opinion already and wants to confirm it

Re: After the Bubble

#79

Post by spwa4 »

littlestymaar wrote: post #63
spwa4 wrote: post #61

Earlier quoted context omitted.

Google is using nVidia GPUs. More than that, I'd expect Google to still be something like 90% on nVidia GPUs. You can't really check of course. Maybe I'm an idiot and it's 50%. But you can see how that works: go to colab.research.google.com. Type in some code ... "!nvidia-smi" for instance. Click on the down arrow next to "connect", and select change runtime type. 3 out of 5 GPU options are nVidia GPUs. Frankly, unle…

> Frankly, unless you rewrite your models you don't really have a choice but using nVidia GPUs, thanks to, ironically, Facebook (authors of pytorch). There is pytorch/XLA automatic translation to TPU but it doesn't work for "big" models. And as a point of advice: you want stuff to work on TPUs? I don't understand what you mean, most models aren't anywhere near big in terms of code complexity, once you have the effici…

> I don't understand what you mean, most models aren't anywhere near big in terms of code complexity, once you have the efficient primitives to build on (like you have an efficient hardware-accerated matmul, backprop, flash attention, etc.) these models are in the sub-thousand LoC territory and you can even vibe-convert from one environment to another.

You're right but that doesn't work. Transformers won't perform well without an endless series of tricks. So endless you can't write that series of tricks. You can't initialize the network correctly when starting from scratch. You can't do the basic training that makes the models good (ie. the trillions of tokens). Flash attention, well that's 2022, it's cuda assembly, and only works on nVidia. Now there's 6 versions of flash attention, all of which are written in Cuda Assembly. It's also only fast on nvidia.

So what do you do? Well you, as they say "start with a backbone". That used to always be a llama model, but Qwen is making serious inroads.

The scary part is that this is what you do for everything now. After all, llama and Qwen are text transformers. They answer "where is Paris?". They don't do text-speech, speech recognition, object tracking, classification, time series, image-in or out, OCR, ... and yet all SOTA approaches to all of these can be only slightly inaccurately described as "llama/qwen with a different encoder at the start".

That even has the big advantage that mixing becomes easy. All encoders produce a stream of tokens. The same tokens. So you can "just" have a text encoder, a sound encoder, an image encoder, a time series encoder and just concatenate (it's not quite that simple, but ...) the tokens together. That actually works!

So you need llama or Qwen to work, not just the inference but the training and finetuning, with all the tricks, not just flash attention, half of which are written in cuda assembly, because that's what you start from. Speech recognition? SOTA is taking sounds -> "encoding" into phonemes -> have Qwen correct it. Of course, you prefer to run the literal exact training code from ... well from either Facebook or Alibaba, with as little modifications as possible, which of course means nvidia.

Re: After the Bubble

#80

Post by bryanlarsen »

LexiMax wrote: post #39
bryanlarsen wrote: post #33

Earlier quoted context omitted.

> A lot of people are acting like they don’t know it. Or they're acting like they think there's going to be significant stock price growth between now and the bubble popping. Behaviors aren't significantly different between those two scenarios.

It's always been an interesting mental exercise for me to try and measure the unknowable gap between what people say they believe and the motivated reasoning that might drive their stated beliefs. Putting your statement another way, if you and I can see the bubble, then it's almost a certainty that the average tech CEO also sees a bubble. They're just hoping that when the music stops, they won't be the one left holdi…

You can "have your cake and eat it too", as long as you're happy with a smaller portion of the cake.

I own some NVDA. I've sold a good portion of it, so I've "locked in the profit" on it. If it doubles again, I'll sell some more. If it crashes I won't be too disappointed -- I've locked in my profit, and now I own more reasonably priced NVDA shares.

Note that if you have index funds, you probably already own a surprising amount of NVDA.

Post reply on HN