Remembering can be replaced with a piece of paper with the code written on it. That's not any more onerous than being required to carry a key around.
And a key generally requires more dexterity to operate than a keypad. I think vision issues might be a valid complaint, since you can usually operate a key by feel; harder to do that with a keypad, especially in the dark.
I don't think we know the precise nature of this guy's limitations, so it's hard to say why he has the dexterity to use a key but not a smartphone. A keypad seems to me to be more similar to using a smartphone than using a key.
And a key generally requires more dexterity to operate than a keypad. I think vision issues might be a valid complaint, since you can usually operate a key by feel; harder to do that with a keypad, especially in the dark.