11–20 of 22 posts
That is fun: it lends c-style block markers (curly braces) and python-style line separation (new lines). No objection.
Don't mean to be rust fanatic or whatever but anyone know of anything similar for rust?
Whatever this is doing could be wrapped up in another language.
Either way it's arguable that is even a good idea, since dealing with a regular thread in the same memory space, getting data to and from the GPU and doing computations on the GPU are all completely separate and have different latency characteristics.
Don't mean to be rust fanatic or whatever but anyone know of anything similar for rust?
That is fun: it lends c-style block markers (curly braces) and python-style line separation (new lines). No objection.
The latency point matters more than it looks imo like the GPU work isn't just async CPU work at a different speed, the cost model is completely different. In LLM inference, the hard scheduling problem is batching non-uniform requests where prompt lengths and generation lengths vary, and treating that like normal thread scheduling leads to terrible utilization. Would be curious if Eyot has anything to say about non-un…
That is fun: it lends c-style block markers (curly braces) and python-style line separation (new lines). No objection.
The latency point matters more than it looks imo like the GPU work isn't just async CPU work at a different speed, the cost model is completely different. In LLM inference, the hard scheduling problem is batching non-uniform requests where prompt lengths and generation lengths vary, and treating that like normal thread scheduling leads to terrible utilization. Would be curious if Eyot has anything to say about non-un…
Not right now, it is far too early days. I'm currently working through bugs, and missing stdlib, to get a simple backpropagation network efficient. Once I'm happy with that I'd like to move onto more complex models.
This reminds me that I'd love to see SYCL get more love. Right now, out of the computer hardware manufacturers, it seems that only Intel is putting any effort into it.