knot8 —
not a Kubernetes package manager! Annotate manifests
and expose tunables for in-place editing and merging.
The
knot8 utility allows the user to perform
in-place edits of kubernetes manifest files.
The core abstraction on top of a manifest is a named shortcut to one or more
values inside of of the set of manfest files called a “field”.
Fields allow manifest authors to define a selection of tunable knobs, a sort of
“API” that manifest authors can conveniently reference it their
setup instructions.
knot8 provides a 3-way merge functionality
allows users to painlessly {up|down}grade to another version of the manifest
while preserving the field values previously set, all the while letting the
authors to heavily refactor the shape of the manifest, moving fields freely
around.
Fields are defined in annotations and are preserved during kubernetes manifest
manipulations, including roundtrips to the API server. Each resource can have
one or more fields. The “field.knot8.io” annotation defines a
mapping between the field name and a JSONPointer relative to the manifest
where the annotation appears.
metadata:
annotations:
field.knot8.io/foo: /data/foo
knot8 implements an extension of the
JSONPointer syntax to allow to address array elements by a “primary
key”
knot8 consumes metadata from manifest
annotations, which means they can be produced by any technique that produces
valid manifests, including templating systems like Helm, ytt, jsonnet, etc.
Other tools can be applied downstream to apply further customizations and
overlays, such as kustomize, ytt, jsonnet, etc.
knot8 set
[
-f
file,...
]
{
field=value ... |
--from
file,...}
Set a
field declared in one or more manifests.
The manifest will be updated in place, preserving as much as possible from the
original formatting, including comments, whitespace, indentation.
The
value will be quoted and indented as
appropriate to the file format syntax of the location where the value ends up
to be in the manifest text.
If
value starts with “@” it's
interpeted as filename, whose content is used as
value.
--from- Read values from one or more files. The files can be simple key/value YAML
maps (like YTT values.yaml) or full blown manifests which contain knot8
field annotations. In that case, the annotations will be used to locate
the values (i.e. the file will be implicitly passed to
--scheme).
Order matters as values present in later files will override values
specified in earlier files. By default a file called
Knot8file in the current directory will
be prepended to the list of from files.
--freeze- Update the knot8.io/orig annotation with a snapshot of the current field
values. This should be used when maintaining a manifest for
publishing.
--stdout- Print the modified manifests to stdout instead of mutating them
in-place.
knot8 cat
[
-f
file,...
]
{
field=value ... |
--from
file,...}
Alias for
set
-
-stdout.
knot8
values
[
-f
file,...
]
[
-k
]
[
field
]
Print the fields defined in the selected manifests along with the current value
in a format suitable for subsequent ingestion with
set --from.
-k,
--names-only- Print only the field names and omit the value.
knot8 pull
[
-f
file,...
]
upstream
Pull and merge
upstream.
The current manifests (as defined by the
-f
flags) are replaced by the content of
upstream after merging the custom field
values present in the local manifests.
-f,
--file- Path to one or more JSON/YAML manifests. The flag can be repeated and/or
the filenames can be comma-separated. If omitted, the manifests are read
from standard input. Each YAML file can contain multiple manifests.
--schema- Path to a file containing a stream of YAML manifests containing the
out-of-band schema for the main manifest set. The same file can be used
while setting values, see --from. If a file called
Knot8file exists in the current
directory it will be implicitly used as a schema file.
--help- Show context-sensitive help.
--version- Print version information and quit.
Arrays in JSONPointer can only be addressed by numeric index:
/spec/template/spec/containers/1/env/3/value
Such pointers are hard to read and also brittle since they will be broken as
soon as the order of the array elements will change.
The symbol “~” is reserved in JSONPointer and cannot appear as a
legitimate character in field names (it must be escaped).
knot8 uses this fact to extend the
JSONPointer syntax with the “~{}” construct:
/spec/template/spec/containers/~{"name":"app"}/env/~{"name":"FOO"}/value
Instead of numerical indices, the user provides an inline “query”
JSON objects. The pointer selects the array element for which the intersection
of it and this query object yields the query object itself (in other words: if
the element has at least the same fields as the query object).
When multiple array element matches, the match is ambiguous and an error is
returned.
Sometimes the tunable fields are not just whole scalar values inside the
manifest, but nested deeper into a part of a string value. A common example
is:
apiVersion: v1
kind: ConfigMap
metadata:
name: demo
data:
foo: |
bar: baz
knot8 takes a recursive approach to the
problem. A string blob in the outer YAML file is just a string blob, but that
doesn't mean we cannot also think about it as if it was yet another structured
file for which we know the format.
knot8
implements a number of “lenses.” You can think of lenses as of
format-preserving bidirectional parsers, which yield map the source text into
a tree addressable via JSONPointer.
The pointer is split into segments and each segment is used to address one
string field. Then, the lens for the next segment is applied and the process
is repeated:
/a/b/c/~(lens1)/d/e/f/~(lens2)/g/h/~(lens3)/i
Example:
apiVersion: v1
kind: ConfigMap
metadata:
name: demo
annotations:
field.knot8.io/foo: /data/foo/~(yaml)/bar
data: \___________________/
foo: | /
bar: baz <-------------------/
knot8 currently supports the following
lenses:
- yaml
- Nested YAML file; quoting and indentation style is preserved as much as
possible. Since JSON is (not quite, but close enough in practice) a subset
of YAML, the same lens works for JSON too.
- toml
- TOML support is preliminary, but simple key = "value" lines can
addressed.
- base64
- The Base64 codec allows editing base64 encoded text bodies (e.g. in
Secrets).
- regexp
- The regexp lens is a useful escape hatch when no lenses exist for your
data type. The first path element after the lens defines a regular
expression (using the RE2 syntax), while the second path element selects
which capture group (0 for the whole match). Named capture groups are
supported. The regular expressions is applied on the whole field
contents.
- line
- Selects a whole line matching a regexp. Like awk's or sed's
"/regexp/" construct.
$ wget https://my.app/v1/app.yaml
$ kubectl apply -f app.yaml
$ knot8 set -f app.yaml foo=WOOF
$ kubectl apply -f app.yaml
$ knot8 pull -f app.yaml https://my.app/v2/app.yaml
$ kubectl apply -f app.yaml
Sometimes you want to be apply different sets of values on the same config file
and thus the in-place edit approach is not a good fit:
$ cat staging/values.yaml
foo: WOOF
$ knot8 set <app.yaml --from=staging/values.yaml | kubectl apply -f
$ kubectl apply -f https://my.app/v1/app.yaml
$ kubectl get deploy myapp -oyaml | knot8 set foo=WOOF | kubectl apply -f -
So far we've seen how knot8 can be used to update fields whose declaration lives
inside the manifest itself. This doesn't work unless the upstream author of
the manifest embraces knot8 field definitions.
The
--schema
flag allows us to define the fields in an external file, without having to
touch the original file. By default a file called
Knot8file is used as schema even if no
-fl
-schema flag is provided.
$ wget https://raw.githubusercontent.com/kubernetes/website/master/\
content/en/examples/application/deployment.yaml
$ cat >Knot8file <<EOF
appImage: bitnami/nginx:1.14.2
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
annotations:
field.knot8.io/replicas: /spec/replicas
field.knot8.io/appImage: /spec/template/spec/containers/~{"name":"nginx"}/image
EOF
$ knot8 cat -f .
Imagine you download an app manifest:
$ wget
https://my.app/v1/app.yaml
Let's take a look at the content of that manifest:
$ cat app.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: demo2
annotations:
field.knot8.io/foo: /data/foo
field.knot8.io/bar: /data/bar
knot8.io/original: |
foo: meow
bar: "1"
data:
foo: meow
bar: "1"
You can edit some of the supported fields manually or via the
set command:
$ knot8 set -f app.yaml
foo=WOOF
We can see how this command affected the manifest file:
$ cat app.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: demo2
annotations:
field.knot8.io/foo: /data/foo
field.knot8.io/bar: /data/bar
knot8.io/original: |
foo: meow
bar: "1"
data:
foo: WOOF
bar: "1"
Now imagine you want to upgrade to the v2 version of the manifest:
apiVersion: v1
kind: ConfigMap
metadata:
name: bettername
annotations:
field.knot8.io/foo: /data/fu
field.knot8.io/bar: /data/ba
knot8.io/original: |
foo: miau
bar: "42"
data:
fu: miau
ba: "42"
The
pull command will download the new
version and perform the 3-way merge:
$ knot8 pull -f app.yaml
https://my.app/v2/app.yaml
Let's see the result of the merge:
$ cat app.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: bettername
annotations:
field.knot8.io/foo: /data/fu
field.knot8.io/bar: /data/ba
knot8.io/original: |
foo: miau
bar: "42"
data:
fu: WOOF
ba: "42"
Sometimes there is no lens that works with your actual field format. When the
relevant parts of the field format can be expressed with a regular expression
you can use the "regexp" lens, where the format is expressed in-line
in the field definition itself.
For example, you can locate a docker image name inside of some configuration
file (e.g. a jsonnet file) and then use the oci lens to further parse the
image reference.
field.knot8.io/workerImageDigest: "/data/worker-ubuntu16-04.jsonnet/~(regexp)/{ name: 'container-image', value: 'docker:~1~1([^']*)/1/~(oci)/digest"
kubectl(1)
RC6901 JSONPointer
Created in 2020 as an experiment to see how far we can go without requiring to
template all the things.
Marko Mikulicic
<
mmikulicic@gmail.com>