I did the same some time ago, with the same observations. I slotted the pipe bracket in addition to bending it, but it still needs a whack with a dead blow hammer to get into the exhaust port. I'd think by now that they'd be made to fit, but evidently this is a rite of passage. Next time it's off I'll slot the rear pipe bracket as well, maybe I won't need the hammer after that.
My muffler bracket needed some serious bending to miss the swingarm. The muffler doesn't really grip the pipe either. Same solution, a real bolt.
And same results; the exhaust is now louder than the intake and has a proper thumper sound instead of a steam engine sort of chuffing, it seems to spin up more freely, and generally seems happier. Despite the hand-fitting required, it's a pretty nice piece of goods and well worth doing. The pancake K&N is a nice complement.