I don't think a bunch of physicists is the best example if you're trying to make the point that most people are smart but just in different ways. Physics requires and attracts intelligence, of course a bunch of physicists were smarter than you in some ways.
Try it with, say, a parole officer's case list, a group of high school dropouts, or people who have not touched a book in the past ten years. You'll certainly find things they know that you don't, but that's mostly going to be a matter of experience, not them being smarter in that area. No doubt the average petty thief knows more about shoplifting than I do, but I'm pretty sure I could learn quickly and become a much better shoplifter than them if I put my mind to it. And those groups will certainly contain some really smart people who just happen to have ended up in those groups, but that's going to be a small number of them.
Try it with, say, a parole officer's case list, a group of high school dropouts, or people who have not touched a book in the past ten years. You'll certainly find things they know that you don't, but that's mostly going to be a matter of experience, not them being smarter in that area. No doubt the average petty thief knows more about shoplifting than I do, but I'm pretty sure I could learn quickly and become a much better shoplifter than them if I put my mind to it. And those groups will certainly contain some really smart people who just happen to have ended up in those groups, but that's going to be a small number of them.