Buildkit didn't break anything here except that it each individual build step is no longer exposed as a runnable image in docker.
That was unfortunate, but you can actually have buildkit run a command in that filesystem these days, and buildx now even exposes a DAP interface.
Buildkit is still a separate system, unlike the old builder. So you get that extra step of importing the result back.
And since it's a separate system, there are also these strange limitations. For example, I can't just cache pre-built images in an NFS directory and then just push them into the Buildkit context. There's simply no command for it. Buildkit can only pull them from a registry.
> Buldkit is far more efficient than the old model.
I understand why. I tried to debug it, and simply getting it running under a debugger is an adventure.
So far, I found that switching to podman+podman-compose is a better solution. At least my brain is good enough to understand them completely, and contribute fixes if needed.
Buildkit is integrated into dockerd the same way the old builder was.
If you want a newer Buildkit you'll need to run it separately of course.
I'm not quite sure I understand what you are trying to do with nfs there.
But you can definitely export the cache to a local filesystem and import it with cache-from.
You can also provide named contexts.
"Buildkit can only pull them from a registry" is just plain false.
> Buildkit is integrated into dockerd the same way the old builder was. If you want a newer Buildkit you'll need to run it separately of course.
I don't think that the older builder created special containers for itself?
> I'm not quite sure I understand what you are trying to do with nfs there. But you can definitely export the cache to a local filesystem and import it with cache-from.
Which is dog-slow, because it squirts the cache through a socket. I have an NFS disk that can be used to cache the data directly. This was just one of the attempts to make it go faster.
> You can also provide named contexts.
Which can only refer to images that are built inside this particular buildkit or are pullable from a repo.
This is really all I want, a way to quickly reused the previous state saved in some format in Github Cache, NFS, or other storage.
> Buildkit doesn't create special containers for itself? It's literally a service integrated into dockerd.
No, it's not. It's a utility container that is hidden from the normal "docker ps". You can see it easily when you use docker-compose with podman.
The easiest way to see it in regular Docker is to create a simple Dockerfile with 'RUN sleep 1000' at the end and start building it. Then enter the Docker host ("docker run -it --rm --privileged --pid=host justincormack/nsenter1") and do 'mount' to see the mounts.
You'll see that buildkit will have its own overlay tree ('/var/lib/docker/buildkit/containerd-overlayfs') and the executor will have its own separate branch too. However, they do share the layers. Now wait for the container to complete building and run it.
You'll see that the running container uses an entirely _different_ set of layers. There is no reuse of layers between the buildkit and the running image.
Yes, the Docker buildkit is technically a daemon that is co-located with dockerd and just runs in its own cgroup tree. But it might as well be remote, because the resulting image runs in a completely different environment.
And the way the image is transferred from buildkit is through the containerd. Which is another separate container in the "moby" namespace.
> No, it supports anything Buildkit can fetch: git, http, client dir... for that matter the client itself can shim that to be whatever it wants.
Any examples?
> You can cache to GitHub actions cache, S3, az blob, gcs, registries, or export to the client.
First, you can build the base image, with the GHA or registry cache. It works. But the `proto` stage will never use cache. The "base" image is supplied through an additional context.
If by "utility" container you mean the containers aren't managed under the same stack, that is true.
Buildkit, at least prior to docker 29, executes runc directly.
It is still using the same storage backend, though there is a shim involved to convert docker's (now deprecated) graph drivers to containerd's snapshotter interface which is what Buildkit speaks. That's why there is a different tree.
As of docker 29, containerd's storage is used by default.
I can't recall if this used containerd to execute containers or just storage.
> > No, it supports anything Buildkit can fetch: git, http, client dir... for that matter the client itself can shim that to be whatever it wants.
> Any examples?
Then you can "FROM foo" or whatever you want to do with that context.
> First, you can build the base image, with the GHA or registry cache. It works. But the `proto` stage will never use cache. The "base" image is supplied through an additional context.
What are you expecting to cache here?
Are you saying using an extra context like this is preventing it from using the cache?
Buldkit is far more efficient than the old model.