The shoes are probably out of center, or the springs are weak, or the actuating cams are rusty and don't move freely.
I would remove the front wheel, inspect and service all the internal brake parts, especially paying attention to the springs and cams. Ensure that the brake plate is properly centered when re-installing, and set your levers parallel.
When properly adjusted, the shoes will not drag the drum, but the slightest movement of the levers will cause the shoes to touch the drum.
You should have about 1/4" free play in the cable when the brake is released. You can feel this at your control lever on the handlebar.