You use invisible characters whenever you press Enter or Space. If you're referring to Tab, many of the most popular programming languages like Go and Python use them as part of their syntax.
The reason YAML was popularized is because it was a response to XML which isn't user friendly to write. It's unfortunate that the spec got so convoluted, and uses a lot of implicit behavior, but I'd rather write YAML than XML, JSON or TOML for things like configuration files. Nowadays there might be better alternatives, but YAML is the de facto standard.
It's also unfortunate that YAML got abused by people who wanted to turn it into a DSL, so we ended up with thousands of lines of Ansible playbooks, CI workflows, and Helm charts, but here we are.
The point is that "going right ahead and write your .yml files in JSON" is not valid. You'd have to restrict yourself to a subset of JSON to not get different semantics.
Python's "PyYAML" package will not decode this to the same result as a JSON decoding.
Rust's `serde_yaml` will fail on this.
I don't know about other parsers, but I'd be curious to.
The standard itself isn't well written here, IMO.
> The content of a scalar node is an opaque datum that can be presented as a series of zero or more Unicode characters.
The example here is a "quoted scalar", which can contain the escapes you see. Those escapes represent "Unicode characters", specifically,
> Escaped 16-bit Unicode character.
But "Unicode characters" is never defined by YAML.
Most implementation seem to treat them as Unicode code points, and so thus the resulting string type in almost all cases in something like [UnicodeCodePoint]; in Rust, that means no unpaired surrogates, or we can't convert it to a Rust `String`, which is roughly speaking `[USV]`. In Python, that's workable, since that's Python's `str` datatype, but that means no surrogate decoding occurs.
The grammar also further implies that it's [UnicodeCodePoint] and not [USV], and the prose never restricts unpaired surrogates. (The JSON standard strongly implies the UTF-16 decoding should happen on escaped values, though it too waffles around unpaired surrogates. Whether unpaired surrogates are accepted is variable in JSON.)
But compare with a JSON string: a JSON string decodes to a something like a [USV], so surrogate pairs are decoded to their corresponding USV.
While YAML has all sorts of issues and disadvantages compared to XML, security is certainly not one of them. XML is a crazy source of security issues by design, especially with the crazy idea of adding built-in support for URLs that parsers are expected to follow.
I have no horse in that race but just to see people talking about XML like this a quarter of a century after the first time I saw similar comments is just funny, I don't care who you are.
NIH syndrome and "inverse second system effect". In the real second system effect, the second system is more complicated because it includes everything that could possibly be perceived as missing in the first system. In the inverse second system effect the first system was perceived as too complicated, not too simple, so the second system is much simpler and doesn't do its job well.
It is tangentially related in that yaml became normal to use as a DSL within the devops world. As another post said, everything becomes a DSL eventually because people want to be "fully configurable" not realizing that is roughly the same thing as not being complete yet. But in this case the lack of direct acknowledgement of yaml as an interpreted language with an interpreter that doesn't think of itself as such and hence doesn't have a real sandbox, is what leads us to the present. People didn't use xml as a DSL as often because it was so flexible. That would be like using c++ as a DSL instead to write the interpreter for one.
I wasn't blaming yaml at all. Our mistake is thinking we are using it as a configuration file. When we are actually using it as an interpreted language. Not yaml's fault people are writing dsl interpreters unknowingly. It's just related because people who make that mistake are picking yaml. I nearly made the mistake with toml a few years ago. You could even make the mistake with complicated environment variable usage. Whenever your configuration source is flexible enough to create executable primitives it needs to be sanitized. And really that is whenever a configurable value gets used in a conditional, which is often. Especially considering that even numeric values become conditional when they are used in operations that can result in ub or even just exceptions/panics/unhandled errors. Not a yaml exclusive.
This is not the first RCE involving YAML and it won't be the last.