Viewing profile — 0xjunhao

Thank you! We have incorporated your suggestion.

Yes, typically users send the newest user message and the full conversation history. These combined become the prompt. Our API endpoint will try to route requests that has the same…

Yes, decoding is very I/O heavy. It has to stream in the whole of the model weights from HBM for every token decoded. However, that cost can be shared between the requests in the s…

Hi, I'm the author of this post. Writing it was a great learning experience. I gained a lot of insight into vLLM. If you have any feedback or questions, feel free to drop a comment…

FYI a literature review from a deep research AI ====== The effect of noise on sleep is multifaceted, involving various types of noise exposure, physiological mechanisms, and conseq…

IMHO the most appropriate medical decision should depend on the patient's economic situation.

I remember trying to publish something of a similar style to arxiv and getting rejected. Seems that having an abstract and references is key :)

If you have ever lost someone due to other drivers not watching the road, you will start to appreciate that these cars are at least always watching the road and won’t kill people s…

What's the difference between resign and layoff :)

Hi, I had a quick question. Would it be correct to say the following? 1. For long inputs and short outputs, the inference can be arbitrarily number of times faster, as it avoids re…

Before I became a software engineer, I was a computational physicist. My days back then were pretty much tweaking some parameters, running a job, then reading papers and checking b…

With the rise of Agentic AI, this increasingly feels like the right move, unless AWS drastically lowers their prices.

In a world of LLMs, it's great to see classic NLP works like Harper. Both definitely have their own use cases.

Maybe we should build more railways. The ground is more stable than the air.