Hey Steve,
I personally think it's the washer and nut not being able to hold firmly onto the swing arm plate which allows it to move. On my swing arm, the bolt is 10.8mm and the slot is about 13mm, which allows about 2.2 mm of movement if the nut is unable to hold the drum assembly.
If you can find some 2mm metal the same thickness as the swing arm plate and jam it above the bolt, then tighten the nut and washer on top of it all, it should stop the drum assembly rotating up or down. Be careful not to strip any thread when jamming the metal in there, and make sure you centre the drum brake by pressing the rear brake while you tighten the nut.
If this doesn't do it, I think the swing arm slot has become too wide after being deformed, and there is simply not enough metal for the washer and nut to grip onto... so you need to get a new swing arm, or weld metal to the slot to make it narrower.
Hope that helps SectorSteve