The prefix tuning approach was largely abandoned for LoRA, it does not change the process if you tune the prefix or some adapter layers, but it is more flexible to train the LoRAs.
The Skills concept emerged naturally when you see how coding agents use docs, CLI tools and code. Their advantage is they can be edited on the fly to incorporate new information and can learn from any feedback source - human, code execution, web search or LLMs.
KV-based "skill capsules" are very different from LoRAs / classic prefix tuning:
* A "hypernetwork" (which can be, in fact, same LLM) can build
a skill capsules _from a single example_.
You can't get LoRA or KV-prefix using just one example.
* It can be inserted at any point, as needed. I.e. if during reasoning you find that you need particular skill, you can insert it.
* They are composable, and far less likely to over-write some information, as they only affect KV cache and not weights.
Skills as used by Anthropic & OpenAI are just textual instruction. KV-based skill capsule can be a lot more compact (and thus would contribute less to context rot) and might encode information which is difficult to convey through instruction (e.g. style).
The Skills concept emerged naturally when you see how coding agents use docs, CLI tools and code. Their advantage is they can be edited on the fly to incorporate new information and can learn from any feedback source - human, code execution, web search or LLMs.