Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Skills Officially Comes to Codex

developers.openai.com

Post reply on HN
Print view View original

81–90 of 133 posts

Re: Skills Officially Comes to Codex

#81

Post by hu3 »

jonrosner wrote: post #16
hu3 wrote: post #13

Earlier quoted context omitted.

Perhaps you could help me. I'm having a hard time figuring out how could I leverage skills in a medium size web application project. It's python, PostgreSQL, Django. Thanks in advance. I wonder if skills are more useful for non crud-like projects. Maybe data science and DevOps.

you could for example create a skill to access your database for testing purposes and pass in your tables specifications so that the agent can easily retrieve data for you on the fly.

I made a small mcp script for database with 3 tools:

- listTables

- getTableSchema

- executeQuery (blocks destructive queries like anything containing DROP, DELETE, etc..)

I wouldn't trust a textual instructions to prevent LLMs from dropping a table.

Re: Skills Officially Comes to Codex

#82

Post by NitpickLawyer »

orliesaurus wrote: post #47

If there was a marketplace or directory of skills.md files that were ranked with comments, it would be a good idea for the propagating of this tech

It would be trivial to create something like this but there are a few major problems with running such a platform that I think makes it not worth while for anyone (maybe some providers will try it, but it's still tough).

- you will be getting a TON of spam. Just look at all the MCP folks, and how they're spamming everywhere with their claude-vibed mcp implementation over something trivial.

- the security implications are enormous. You'd need a way to vet stuff, moderate, keep track of things and so on. This only compounds with more traffic, so it'd probably be untenable really fast.

- there's probably 0 money in this. So you'd have to put a lot of work in maintaining a platform that attracts a lot of abuse/spam/prompt kiddies, while getting nothing in return. This might make sense to do for some companies that can justify this cost, but at that point, you'd be wondering what's in it for them. And what control do they exert on moderation/curation, etc.

I think the best we'll get in this space is from "trusted" entities (i.e. recognised coders / personalities / etc), from companies themselves (having skills in repos for known frameworks might be a thing, like it is with agents.md), and maybe from the token providers themselves.

Re: Skills Officially Comes to Codex

#83

Post by visarga »

mikaelaast wrote: post #12
Etheryte wrote: post #9

Earlier quoted context omitted.

The modern state of the art is inherently not verifiable. Which way you give it input is really secondary to that fact. When you don't see weights or know anything else about the system, any idea of verifiability is an illusion.

Sure. Verifiability is far-fetched. But say I want to produce a statistically significant evaluation result from this – essentially testing a piece of prose. How do I go about this, short of relying on a vague LLM-as-a-judge metric? What are the parameters?

You 100% need to test work done by AI, if it's code it needs to pass extensive tests, if it's just a question answered, it needs to be the common conclusion of multiple independent agents. You can trust a single AI as much as a HN or reddit comment, but you can trust a committee of 4 as a real expert.

More generally I think testing AI by using its web search, code execution and ensembling is the missing ingredient to increased usage. We need to define the opposite of AI work - what validates it. This is hard, but once done you can trust the system and it becomes cheaper to change.

Re: Skills Officially Comes to Codex

#84

Post by hu3 »

macNchz wrote: post #72
hu3 wrote: post #13

Earlier quoted context omitted.

Perhaps you could help me. I'm having a hard time figuring out how could I leverage skills in a medium size web application project. It's python, PostgreSQL, Django. Thanks in advance. I wonder if skills are more useful for non crud-like projects. Maybe data science and DevOps.

There’s nothing super special about it, it’s just handy if you have some instructions that you don’t need the AI to see all the time, but that you’d like it to have available for specific things. Maybe you have a custom auth backend that needs an annoying local proxy setup before it can be tested—you don’t need all of those instructions in the primary agents.md bloating the context on every request, a skill would let…

Thanks. I think I could use skills as "instructions I might need but I don't want to clutter AGENTS.md with them".

Re: Skills Officially Comes to Codex

#85

Post by heliumtera »

mikaelaast wrote: post #8

Are we sure that unrestricted free-form Markdown content is the best configuration format for this kind of thing? I know there is a YAML frontmatter component to this, but doesn't the free-form nature of the "body" part of these configuration files lead to an inevitably unverifiable process? I would like my agents to be inherently evaluable, and free-text instructions do not lend themselves easily to systematic evalu…

Then rename your markdown skill files to skills.md.yaml.

There you go, you're welcome.

Re: Skills Officially Comes to Codex

#87

Post by fizx »

NitpickLawyer wrote: post #76
cube2222 wrote: post #11

It's so nice that skills are becoming a standard, they are imo a much bigger deal long-term than e.g. MCP. Easy to author (at its most basic, just a markdown file), context efficient by default (only preloads yaml front-matter, can lazy load more markdown files as needed), can piggyback on top of existing tooling (for instance, instead of the GitHub MCP, you just make a skill describing how to use the `gh` cli). Comp…

On top of everything you've described, one more advantage is that you can use the agents themselves to edit / improve / add to the skills. One easy one to do is something like "take the key points from this session and add the learnings as a skill". It works both on good sessions with new paths/functionality and on "bad" sessions where you had to hand-hold the agent. And they're pretty good at summarising and extract…

Add reinforcement learning to figure out which skills are actually useful, and you're really cooking.

Re: Skills Officially Comes to Codex

#88

Post by zahlman »

colonCapitalDee wrote: post #79
zahlman wrote: post #77

Recently there was a submission ( https://news.ycombinator.com/item?id=45840088 ) breaking down how agents are basically just a loop of querying a LLM, sometimes receiving a specially-formatted (using JSON in the example) "request to use a tool", and having the main program detect, interpret and execute those requests. What do "skills" look like, generically, in this framework?

Before the first loop iteration, the harness sends a message to the LLM along the lines of. postgres Directions on how to query the pre-prod postgres db skills/postgres.md The harness then may periodically resend this notification so that the LLM doesn't "forget" that skills are available. Because the notification is only name + description + file, this is cheap r.e tokens. The harness's ability to tell the LLM "IMPO…

> The harness's ability to tell the LLM "IMPORTANT: this is a skill, so pay attention and use it when appropriate" and then periodically remind them of this is what differentiates

... And do we know how it does that? To my understanding there is still no out-of-band signaling.

Re: Skills Officially Comes to Codex

#90

Post by wincy »

jimmydoe wrote: post #69
btown wrote: post #63

Something that’s under-emphasized and vital to understand about Skills is that, by the spec, there’s no RAG on the content of Skill code or markdown - the names and descriptions in every skill’s front-matter are included verbatim in your prompt, and that’s all that’s used to choose a skill. So if you have subtle logic in a Skill that’s not mentioned in a description, or you use the skill body to describe use-cases no…

but that's same for MCP and tools, no?

A consultant started recommending the Azure devops MCP and my context window would start around 25% full. It’s really easy to accidentally explode your token usage and destroy your context windows. Before I’d use az cli calls as needed and tell the agent to use the same, which used significantly less context and was more targeted.
Post reply on HN