If IRC suffices for your purposes, then Matrix, with its encryption and all, is apparently overkill.
If I were to upgrade an IRC-based community to something newer and richer, I'd go with Jabber, well-known, well-established, with a ton of various clients and several servers. Yes, it's not ideal, but it's still a massive upgrade compared to IRC, if your server supports a good list XEPs and your community members agree to use non-esoteric clients that also support them.
I guess it's not end-to-end, it's decrypted on the server.
Presumably if you want to send an encrypted message from one literal endpoint to another, you'd use some other technology. I'm prepared to bet there are enough people doing just that, too.
If I were to upgrade an IRC-based community to something newer and richer, I'd go with Jabber, well-known, well-established, with a ton of various clients and several servers. Yes, it's not ideal, but it's still a massive upgrade compared to IRC, if your server supports a good list XEPs and your community members agree to use non-esoteric clients that also support them.