For as fundamental and important as encryption-in-transit is, it's always baffled me that there isn't a simpler, easier solution to accomplishing it on private networks. Everyone knows its important, and everyone wants to do it, but it's just such a pain in the ass and so prone to error that even some top security leaders will tell you not to bother because it's such a footgun.
We really need something to help make the process simpler, like how Let's Encrypt made public HTTPS so much easier to do for even the smallest of websites.
I would argue that if you have services then the right place to put encryption and authentication is at the service level. Building secure channels between IP addresses is all good, but do you really want to map roles/identities/privileges to specific IP addresses if those roles/identities/privileges really represent services?
What if you end up spinning more than one container for that service?
How are these containers getting the different secrets they need to identify themselves? Are you attaching IAM roles to them to get secrets from some secret store?
Said it twice before: differently complex. There are plenty of potential “solutions” to the specific scenario you’re describing, but my original comment was more “generally consider X instead of Y so you don’t have to care about Z” rather than “use X in this specific way and it will simply solve every problem with Y”.
Well, there are encrypted CNIs like Weave. I've used Calico over ZeroTier to similar effect. The network is 'encrypted' and there isn't much effort required past initial configuration.
But that's not really the issue. You still have a big plaintext network with a bunch of random stuff talking, no mutual auth and no security controls other than segmentation. That's the tricky problem that mTLS and service meshes attempt to solve.
First, I’ll respond this w.r.t. k8s CNI specifically: all inter-node traffic is encrypted, the only plaintext is localhost. If you’re worried about network snooping on localhost you’ve got bigger problems. As for security controls, that’s what Network Policies are for.
Outside of k8s (where one has greater control over how specifically e.g. Wireguard is deployed). Again, there is no plaintext outside of localhost. Wireguard is mutual auth, I’m not sure why you think it isn’t. Wireguard + firewall is security control since, well, you have mutual auth so rules can be applied per-client.
If Operating Systems had TLS built into the TCP/IP stack exposed by the kernel/system, you would never need to shim it in anywhere. You would just make a system call and use an open file descriptor/socket. One of the many programming-in-1970s-style things we still have not fixed.
But 1) kernel hackers won't implement it, 2) app devs are too possessive of their stack/codebase to just use one standard implementation/interface, and 3) security people are too paranoid to leave something "so important" up to the OS so they'd rather everyone implement it poorly/fragmentedly.
The thing is you can use Let’s Encrypt for private networks too. For example, I use a dns challenge to get a wildcard certificate for a sub domain on my personal site, but those domains only resolve in my house. The wildcard cert isn’t essential for this - you could get individual ones - but it was easier for my home lab.
I think both of those focus on ingress? I suppose you could just create your CA with cert manager and manually issue cert requests, but securing in-cluster traffic (automagically) will need some other moving piece, like a sidecar proxy that the service meshes use.
As far as a service mesh, check out Linkerd! I find Istio much harder to setup and manage. Linkerd is super simple and has always worked pretty much out of the box for me.
One thing to keep in mind is that Linkerd is pretty much strictly k8s only one while Istio and Consul Connect have first-class support for out-of-cluster services as well as e.g. Nomad. Relying on linkerd digs you waaay deeper into k8s lock-in.
This may be fully acceptable for you, but should not be glossed over.
From my experience, linkerd had the most seamless deployment to get to the most feature-complete out-of-the box experience with monitoring etc. But as it goes with these things there’s a much bigger amortized cost in terms of magic to unwind if you need to integrate it.
We really need something to help make the process simpler, like how Let's Encrypt made public HTTPS so much easier to do for even the smallest of websites.