You absolutely do not want to use a center punch. You want to use a brass drift with a flat face that will not deform the face of your pin.
If your screwdriver marked up the end of that pin, it may be flared just enough to have become oversize for the hole it came out of.