11–20 of 81 posts
Most of my and my peers PortCos run their own eval and benchmark sets, simply because they know what they need best. The reality is, capabilities have largely converged across foundation models over the last 18 months, and much of the value add is coming from the harness layer itself now. This has been the operating assumption for me and my peers, and has largely played out that way. That said, this has always been a…
I think there’s still something really off with current (frontier) models when it comes to creating “novel” stuff? Even 2004 style graphics..
Or am promoting it wrong?
> That’s the problem, these tests can’t tell you how good a model is anymore because it’s trivial for labs to optimise for exactly these tests by the next release. But they don't prove the claim. Are the models amazing at recreating Minecraft, but the second you swap the word Minecraft out with another game or a custom game, it shits the bed? That's not what I see. My feed is full of people using Astra to recreate al…
I cant select text nor click links on this page with firefox.
> That’s the problem, these tests can’t tell you how good a model is anymore because it’s trivial for labs to optimise for exactly these tests by the next release. But they don't prove the claim. Are the models amazing at recreating Minecraft, but the second you swap the word Minecraft out with another game or a custom game, it shits the bed? That's not what I see. My feed is full of people using Astra to recreate al…
> ridiculously polished detail Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump
- https://x.com/MengTo/status/2096213835460084184?s=20
- https://x.com/birdabo/status/2096101381925114327?s=20
Seems really impressive at first glance, but there's no guarantee that they are real
I keep seeing Astra make beautiful 3d stuff online, yet when I feed it some old school RuneScape assets (even tried with some very detailed guidelines) and asked it to generate some new plausible assets it failed horribly. I think there’s still something really off with current (frontier) models when it comes to creating “novel” stuff? Even 2004 style graphics.. Or am promoting it wrong?
Build a flowsheet based steady state chemical process simulator, then use it to simulate and optimize a full scale oil refinery.
1) Building a solver engine that works at this scale is not a trivial problem, and the successful ones rely more on heuristics than some categorically different solution approach.
2) Defining the engineering equations relevant to this task is relies on understanding what level of fidelity is required to answer the questions people ask of steady state process models.
3) Knowing the thermophysical properties of chemicals and crude oils is possible from the open literature, but the information is diffuse and different correlations are applicable in different situations.
4) Creating a GUI which converts a flowsheet into matrix math is non-trivial, although a sequential modular approach is a bit easier.
5) Defining large scale models in such a way that they solve robustly is as much art as science. For example, completely closed recycle loops like refrigeration systems are a nightmare for solvers, so it is often better to define them in an open-loop way.
6) Optimization involves knowing the relevant commodity prices, but more importantly how to define the constraints on the model so it doesn't just say to produce infinite gasoline.
7) Troubleshooting the inevitable convergence failures is also as much art as science. There are a large number of diagnostic techniques, but fundamentally you need to be able to relate what is happening during the solver iterations with the intent of your model because more often than not the problem is that you've asserted something impossible, redundant, or irrelevant.
Earlier quoted context omitted.
> ridiculously polished detail Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump
X.com is full of unverified posts, but here's a few: - https://x.com/MengTo/status/2096213835460084184?s=20 - https://x.com/birdabo/status/2096101381925114327?s=20 Seems really impressive at first glance, but there's no guarantee that they are real