Harry pondered quitting royals in his 20s