On mine I swivelled the stems forward slightly rather than to the most "outward" position, then tilted the actual mirrors back at the ball joint to get the right angle. I still see my shoulder in the mirrors, but I get a good view behind without having to shift position.
I've since fitted Louis "American Oval" mirrors because the originals rusted. They sit a bit lower and suit the bike, but the view is about the same.