231–240 of 252 posts
Earlier quoted context omitted.
No, that intuition is incorrect. Denoising models work because a lot of regions turn out to be smooth, you cannot do that "in a discrete way" if that makes sense.
Feel free to give a better explanation. I am not an expert. Clearly denoising models do work on text though.
Serious question: Why does it appear that pages from this URL very often end up on top of HN? I don't find the content particularly special compared to the average HN post. Does the algorithm prefer certain URLs?
If anything HN is getting harder for me to get stuff on these days. Most of my self submissions of my content have failed to chart over the past few months: https://news.ycombinator.com/submitted?id=simonw You can see submissions from my domain by other people here: https://news.ycombinator.com/from?site=simonwillison.net It's weird what DOES make it. I had high hopes for my piece on ChatGPT memory yesterday - https:…
Your shout that "more people should do this" is resonating with me - I have some interest in similar short form posts covering various topics of interest (even if only for my own reference), but I am not sure if I can manage this on the side.
I'm curious about the time required for this volume of content output. Do you use AI to help with writing?
Earlier quoted context omitted.
The relative unimportance of the exact SDPA attention in use in modern transformers is already known: https://arxiv.org/abs/2111.11418 The FFN, normalization, and residual connections are absolutely irreplaceable -- but attention can be replaced with almost any other layer that shares information between tokens, such as pooling, convolution, random mixing, etc.
hm, residual is what I would not expect, can you elaborate why?
Also, most blocks with a residual approximate the identity function when initialised, so tend to be well behaved.
Earlier quoted context omitted.
If anything HN is getting harder for me to get stuff on these days. Most of my self submissions of my content have failed to chart over the past few months: https://news.ycombinator.com/submitted?id=simonw You can see submissions from my domain by other people here: https://news.ycombinator.com/from?site=simonwillison.net It's weird what DOES make it. I had high hopes for my piece on ChatGPT memory yesterday - https:…
How much of your time do you spend in writing for your blog? Do you do this full time or is it more of a side gig? Your shout that "more people should do this" is resonating with me - I have some interest in similar short form posts covering various topics of interest (even if only for my own reference), but I am not sure if I can manage this on the side. I'm curious about the time required for this volume of content…
No AI for the writing itself but plenty for things like research, finding the right word, getting feedback on if what I've written makes sense.
Definitely a side gig! I make maybe $200/month off ads from it.
Earlier quoted context omitted.
> LLM can wander through the codebase by itself and do research and build a "mental model" It can't really do that due to context length limitations.
It doesn't need the entire codebase, it just needs the call map, the function signatures, etc. It doesn't have to include everything in a call - but having access to all of it means it can pick what seems relevant.
Earlier quoted context omitted.
> LLM can wander through the codebase by itself and do research and build a "mental model" It can't really do that due to context length limitations.
1k LOC is perfectly fine, I did not experience issues with Claude with most (not all) projects around ~1k LOC.
With 1k lines of code you don't need an LLM, the entire source code can fit in one intern's head.
Earlier quoted context omitted.
1k LOC is perfectly fine, I did not experience issues with Claude with most (not all) projects around ~1k LOC.
Actual projects where you'd want some LLM help start with millions of lines of code, not thousands. With 1k lines of code you don't need an LLM, the entire source code can fit in one intern's head.
Earlier quoted context omitted.
It doesn't need the entire codebase, it just needs the call map, the function signatures, etc. It doesn't have to include everything in a call - but having access to all of it means it can pick what seems relevant.
In a real project the call map and function signatures are millions of tokens themselves.