And why is Mercury so hard to get there in terms of energy required? It's because its orbital velocity is so fast and you need to match that. (You can get there without matching the orbital velocity, but that won't be useful, you're either doing a non-capturable flyby or a very hard impact.)
Mercury's orbital velocity is 48 km/s. Earth's is 30. An object at infinity would be zero. Kinetic energy is proportional to velocity squared. Square those numbers and you see the energy differential between Earth's orbit and Mercury's is greater than going from Earth to infinity.
I suppose another very loosely defined notion of closeness is the degree of human habitability, in which sense Mars might be closest. Both Venus and Mercury are incredibly hostile environments, although one could argue for floating cities on Venus and a thin habitable zone at the poles of Mercury.
I understand of course that anywhere "not on Earth" is incredibly hostile to Human life, at least what we can see with present day technology. For truly habitable planets, we might have to consider other star systems and even then there's no guarantee we'll find one.
I don't know to what degree is the abundance of oxygen as a loose element a sign of life, but I'd expect it to be bound to minerals anywhere without significant plant-like life. Perhaps finding another habitable planet is the same task as finding life on another planet?
As a photon flies? Mercury.
In terms of energy required to get there? Venus.
In fact Mercury is the furthest planet in terms of energy required -- it's easier to get to Neptune than Mercury.
In time to get there assuming minimum energy and no gravity assists from other planets? Venus I think, but maybe Mars.