The YAML parser that got out of hand

In 2018, I wrote a YAML parser in Bash. It was funny, useful, and—at the centre—a bad parser design.

Frank Vumbaca helped turn the first experiment into something people could actually use. He added blocks, lists, stdin support, documentation, and years of patient fixes. Then the repository mostly sat still. It kept doing its small job while I went on to other things.

I came back eight years later intending to do a full audit, close the old issues, replace Travis, rename the default branch, and ship a respectable release. Then I asked Codex a more dangerous question: if we were willing to break the CLI, how much better could this become?

A useful stunt

The original YAML.sh translated YAML into flattened paths and values. That was a sensible shape for shell scripts: easy to print, easy to grep, and just enough structure to query a configuration file.

It was also the ceiling. Once a mapping, sequence, alias, tag, or empty collection had been flattened, important relationships were gone. Every new feature had to reconstruct meaning the parser had already thrown away. Editing YAML without mangling it was almost impossible.

I asked whether there was a better parsing design I had missed. There was. The missing idea was not a smarter regular expression. It was an intermediate representation that respected YAML.

Version 1 replaced the flattened stream with a node graph. Mappings remain mappings. Sequences remain ordered. Empty collections survive. Tags and source positions stay attached. Anchors and aliases can share identity. Queries and updates operate on nodes instead of guessing their way back from strings.

The new shape

YAML stream → node graph → expression stream → values / YAML

Permission to break it

That change broke the old CLI. I decided that was fine. Compatibility with a clever limitation was less valuable than making the premise useful.

We also stopped requiring Bash. The released program is one POSIX shell file with its AWK parser and evaluator embedded. The .sh name still fits: the shell owns the executable, arguments, files, and process behaviour; AWK does the work it is unusually good at. Calling it yaml.awk would describe an implementation detail while hiding the way people actually use it.

The goal became precise: yq-shaped configuration work with no runtime dependencies beyond /bin/sh and AWK. Not a miniature clone of every yq feature. Not a standards-compliance trophy. A useful tool for bootstrap scripts, tiny containers, old machines, and the mildly cursed recovery shell where installing the better tool is the problem you are trying to solve.

Once that premise was clear, breaking the interface was not reckless. It was the cost of making the product coherent.

“The point is to have fun.”

The estimate stopped helping

Codex began with estimates that sounded responsible and were consistently wrong. Work described as weeks kept turning into passing tests in minutes. I kept raising the goal. The parser became a query language; the query language gained updates; semantic edits learned to preserve the source around them; then multi-file updates became one preflighted transaction that could roll itself back.

Speed created its own temptation. At one point we briefly treated a large capability jump as a new major version. I stopped it. More work is not a breaking change, and velocity is not permission for version-number theatre. We corrected the release line and made the version communicate compatibility again.

The releases still moved quickly, but each one earned a bounded contract. The sequence mattered more than the spectacle.

v0.3Audit the old tool and make its limits explicit.
v1.0Replace flattened paths with a proper YAML node graph.
v1.2Turn queries into writable transformations.
v1.4Measure compatibility instead of describing it vaguely.
v1.6Add practical language depth without losing the one-file runtime.
v1.9Make repository-wide edits surgical, explainable, and transactional.

Make it real

The hard part was not producing more code. The hard part was deciding what evidence would let us believe it.

We compared YAML.sh with the official YAML Test Suite and with yq itself. We added real configuration workflows from Kubernetes, Docker Compose, GitHub Actions, GitLab CI, and deployment overlays. We generated expressions, hostile inputs, large documents, deep collections, and presentation-preserving mutations. A supported behaviour needed a test; nearby unsupported behaviour needed an explicit error instead of a confident misparse.

That distinction matters. Matching 2,620 categorized yq programs does not mean YAML.sh contains all of yq. Passing 282 expected YAML Test Suite cases does not mean every corner of the YAML specification is implemented. The numbers describe the tested contract. They do not dissolve the boundary around it.

By v1.9, that contract was large enough to be useful and specific enough to be trusted. One command could also prepare changes across a repository, refuse to write if any candidate failed, roll back a broken commit, and leave a JSON audit trail containing paths but not values.

yq programs: 2,620 / 2,620; in the categorized v1.9 corpus
YAML outcomes: 282 / 282; expected YAML Test Suite cases
generated properties: 12,000 / 12,000; grammar-guided checks
extra dependencies: 0; the runtime is POSIX sh + AWK

What the agent changed

The surprising part was not that an agent could write AWK quickly. It was how quickly we could turn a claim into a gate, use the failure to find the next missing idea, and repeat.

I kept responsibility for the premise. I pushed when the evidence made the estimates obsolete. I pulled us back when the versioning became silly, the documentation started talking too much about the old tool, the brand drifted, or CI spent macOS minutes proving that a web page still rendered. Codex carried an enormous implementation surface without getting bored by the ninety-first invalid fixture.

Neither role substitutes for the other. Unbounded generation would have produced an impressive pile. Direction without the implementation pace would have left a very good plan. The useful system was the loop between judgment, code, external comparison, and correction.

I have spent much of my career building the first path through ambiguous systems. This felt familiar, only compressed. Make the goal concrete. Build something real. Put it against evidence. Change the plan when the evidence changes. Keep the principle steady.

Where it fits

YAML.sh is not yq in disguise. yq remains the right answer when you want its complete language, codecs, ecosystem, and mature performance envelope. YAML.sh deliberately leaves out dynamic evaluation, file-loading operators, dates, many codecs, and other capabilities that would weaken its small and inspectable runtime.

It is better at a narrower thing: arriving as one readable text file in an environment that already has a shell and AWK. There is no package manager, language runtime, YAML library, or mystery binary. The security model can also be smaller because several powerful classes of behaviour simply do not exist. Its repository-wide edits preflight every file, can roll back a partial commit, and explain their paths without recording the values they changed.

That is enough of a difference to be a product, not just a stunt. Sometimes yq is exactly right. Sometimes you are writing the script that would install yq.

The constraint is the fun part

YAML in shell still sounds like a bad idea. I like that. Good constraints create a shape you would not reach by chasing generality, and a little absurdity can provide enough energy to stay with the boring work that makes an idea dependable.

The premise survived the rewrite: one file, no dependencies, real YAML work. Everything else was allowed to change.

The point was to have fun. The trick was refusing to use fun as an excuse to be sloppy.