Skills Officially Comes to Codex • Hacker News Board

Re: Skills Officially Comes to Codex

#111

Post by JamesSwift » Sat, Dec 20, 2025, 9:17 PM UTC

mikaelaast wrote: post #12

Etheryte wrote: post #9

Earlier quoted context omitted.

The modern state of the art is inherently not verifiable. Which way you give it input is really secondary to that fact. When you don't see weights or know anything else about the system, any idea of verifiability is an illusion.

Sure. Verifiability is far-fetched. But say I want to produce a statistically significant evaluation result from this – essentially testing a piece of prose. How do I go about this, short of relying on a vague LLM-as-a-judge metric? What are the parameters?

How would you evaluate it if the agent were not a fuzzy logic machine?

The issue isnt the LLM, its that verification is actually the hard part. In any case, its typically called “evals” and you can probably craft a test harness to evaluate these if you think about it hard enough

Re: Skills Officially Comes to Codex

#112

Post by wahnfrieden » Sat, Dec 20, 2025, 9:33 PM UTC

SatvikBeri wrote: post #93

hu3 wrote: post #81

Earlier quoted context omitted.

I made a small mcp script for database with 3 tools: - listTables - getTableSchema - executeQuery (blocks destructive queries like anything containing DROP, DELETE, etc..) I wouldn't trust a textual instructions to prevent LLMs from dropping a table.

That's why I give the LLM a readonly connection

This is much better than MCP, which also stuffs every session's precious context with potentially irrelevant instructions.

Re: Skills Officially Comes to Codex

#113

Post by fassssst » Sat, Dec 20, 2025, 9:49 PM UTC

tacone wrote: post #89

I don't understand how skills are different than just instructing your model to read all the front-matters from a given folder on your filesystem and then decide if they need to read the file body.

Post training :)

Re: Skills Officially Comes to Codex

#114

Post by ericflo » Sat, Dec 20, 2025, 10:02 PM UTC

chickensong wrote: post #101

ericflo wrote: post #86

People are really misunderstanding Skills, in my opinion. It's not really about the .md file. It's about the bundling of code and instructions. Skills assume a code execution environment.

You could already pre-approve an executable and just call that from your prompt. The context savings by adding/indexing metadata and dynamically loading the rest of the content as-needed is the big win here IMHO.

Yes, and! It's the combo/bundling/distribution of these things that makes this powerful.

Re: Skills Officially Comes to Codex

#115

Post by mycall » Sat, Dec 20, 2025, 10:03 PM UTC

cube2222 wrote: post #11

It's so nice that skills are becoming a standard, they are imo a much bigger deal long-term than e.g. MCP. Easy to author (at its most basic, just a markdown file), context efficient by default (only preloads yaml front-matter, can lazy load more markdown files as needed), can piggyback on top of existing tooling (for instance, instead of the GitHub MCP, you just make a skill describing how to use the `gh` cli). Comp…

So a skill is effectively use case / user story / workflow recipe caching

Re: Skills Officially Comes to Codex

#116

Post by mycall » Sat, Dec 20, 2025, 10:06 PM UTC

NitpickLawyer wrote: post #76

cube2222 wrote: post #11

It's so nice that skills are becoming a standard, they are imo a much bigger deal long-term than e.g. MCP. Easy to author (at its most basic, just a markdown file), context efficient by default (only preloads yaml front-matter, can lazy load more markdown files as needed), can piggyback on top of existing tooling (for instance, instead of the GitHub MCP, you just make a skill describing how to use the `gh` cli). Comp…

On top of everything you've described, one more advantage is that you can use the agents themselves to edit / improve / add to the skills. One easy one to do is something like "take the key points from this session and add the learnings as a skill". It works both on good sessions with new paths/functionality and on "bad" sessions where you had to hand-hold the agent. And they're pretty good at summarising and extract…

I think taking key points from a session and making a new skill is less useful than "precaching" by disseminating the key findings and updating related or affected skills, eliminating the need for a new skill (in most cases).

On the other hand, from a pure functional coding appeal, new skills that don't have leaking roles can be more atomic and efficient in the long run. Both have their pros/cons.

Re: Skills Officially Comes to Codex

#117

Post by wek » Sat, Dec 20, 2025, 10:37 PM UTC

JamesSwift wrote: post #38

hu3 wrote: post #13

Earlier quoted context omitted.

Perhaps you could help me. I'm having a hard time figuring out how could I leverage skills in a medium size web application project. It's python, PostgreSQL, Django. Thanks in advance. I wonder if skills are more useful for non crud-like projects. Maybe data science and DevOps.

Skills are the matrix scene where neo learns kungfu. Imagine they are a database of specialized knowledge that can an agent can instantly tap into _on demand_. The key here is “on demand”. Not every agent or convention needs to know kung fu. But when they do, a skill is waiting to be consumed. This basic idea is “progressive disclosure” and it composes nicely to keep context windows focused. Eg i have a metabase skil…

Nice analogy!

Re: Skills Officially Comes to Codex

#118

Post by shrx » Sat, Dec 20, 2025, 10:40 PM UTC

mellosouls wrote: post #70

How can skills be monetised by creators? Obviously they are empowering Codex and Claude etc, and many will be open source or free. But for those who have commercial resources or tools to add to the skills choice, is there documentation for doing that smoothly, or a pathway to it? I can see at least a couple of ways it might be done - skills requiring API keys or or other authentication approaches, but this adds frict…

How would you enforce DRM on a markdown file?

Re: Skills Officially Comes to Codex

#119

Post by joshka » Sat, Dec 20, 2025, 11:06 PM UTC

NitpickLawyer wrote: post #92

fizx wrote: post #87

Earlier quoted context omitted.

Add reinforcement learning to figure out which skills are actually useful, and you're really cooking.

DSPy with GEPA should work nicely, yeah. Haven't tried yet but I'll add it to my list. I think a way to share within teams is also low-hanging fruit in this space (outside of just adding them to the repo). Something more org-generic.

> DSPy with GEPA should work nicely

I think that would be a really really interesting thing to do on a bunch of different tasks involving developer tooling (e.g. git, jj, linters, etc.)

Re: Skills Officially Comes to Codex

#120

Post by joshka » Sat, Dec 20, 2025, 11:25 PM UTC

mikaelaast wrote: post #8

Are we sure that unrestricted free-form Markdown content is the best configuration format for this kind of thing? I know there is a YAML frontmatter component to this, but doesn't the free-form nature of the "body" part of these configuration files lead to an inevitably unverifiable process? I would like my agents to be inherently evaluable, and free-text instructions do not lend themselves easily to systematic evalu…

The DSPy + GEPA idea for this mentioned above[1] seems like it could be a reasonable approach for systematic evaluation of skills (not agents as a whole though). I'm going to give this a bit of a play over the holiday break to sort out a really good jj-vcs skill.

[1]: https://news.ycombinator.com/item?id=46338371