Steering dampener is the right answer. You bike is dealing with a lot of force pulling in two different directions. No matter how well set up it is it will pull and push depending on accelerating or braking. Look at Ural, they all have dampeners built in. That is the best example of a sidecar rig you can find, they are built that way for a reason. All that said, go over your setup again, make adjustments and see how much you can eliminate, then get a dampener. It sounds like you may be toed in a little too far, but I could be wrong. Lean out, toe in, get that right and have fun.
Samgar