> This isn’t actually unsafe unless shared across threads right?

Multiple mutable borrows do not need multiple threads to cause UB. Consider the following:

    fn main() {
        let mut v = vec![0, 1, 2];
        let e = &mut v[0];
        v.push(3);
        *e = 4;
    }

rustc refuses to compile this code due to multiple mutable borrows of `v` [0]:

    error[E0499]: cannot borrow `v` as mutable more than once at a time
     --> <source>:4:5
      |
    3 |     let e = &mut v[0];
      |                  - first mutable borrow occurs here
    4 |     v.push(3);
      |     ^ second mutable borrow occurs here
    5 |     *e = 4;
      |     ------ first borrow later used here

If rustc allowed multiple mutable borrows here, `v.push(3)` would cause the underlying vec to reallocate, which invalidates `e`, and so `*e = 4` would cause UB.

[0]: https://rust.godbolt.org/z/bsxKTWG3K