I've been at one place where agile/scrum worked. None of us were superstar 10x developers. What made it work was that we had a lot of trust between the developers and management. And that we knew the code-base well enough to be able to give accurate estimates (low turnover is good that way).