Before removing mine I worried about it too.
In my work I found that vibration can do incredible things to materials.
On one of my many projects I once designed a power unit for a tank. This was a small gas turbine engine with a gearbox, controls and accessories that drove a high power alternator.
Because it was going into a tank, weight was not an issue so I made the housing out of cast stainless steel with .200 inch thick walls. To say it was "stout" would be an understatement. A direct hit with a high powered rifle wouldn't have fazed it.
Anyway, part of its testing involved being placed on the "shake table", a large steel table that could be vibrated at an infinite number of cycles per second at various magnitudes of intensity.
This took place in a room filled with many large cabinets containing the power sources for the magnetic drives that moved the table. (Before going into the room you had to remove your watches or anything that could be destroyed by intense magnetic fields).
The room was darkened with multiple strobe lights pointed towards the unit flashing away as the operator started turning up the power and frequency. It reminded me a lot of Scotty in the Engineering deck of the Enterprise starship with its low hum getting louder and higher in pitch.
As the frequency and intensity picked up with the strobe lights flashing things on the engine started to move. The CDI box on its soft mounts started wagging back and forth, the fuel pump and control started moving in a circular pattern about its mount, the turbine case stated changing from round to elliptical. As the frequency increased the movement of some things started slowing while other things started moving faster. As I watched, then IT started.
IT was the hell for stout gearbox housings walls started moving!
The front face started moving in and out while the side walls with the engine mounts on them also started moving in and out. I'm talking over 1/4 inch of movement!
Anyway, we had taken it far beyond its operating requirements and nothing broke.
After the test my good friend, the Chief of Engineering says to me, "Congratulations on the test and remember Jim, the world is made of Jello."
Anyway, getting back to the subject, if you notice there are a number of cast in cylinders that connect the fins on the cylinder to tie them together.
These are there specifically to prevent the vibrations in the engines operating range from exciting the fins and leading to metal fatigue.
Just to prove nothing bad was happening, after removing the rubber dampers I started the cold engine with my hand resting on the fins.
After running it up and down thru its normal speed ranges I didn't feel any of the fins beginning to vibrate beyond the normal vibration of a big 500 thumper. This is good.
If the fins are producing any noise at all the magnitude of the vibrations is so small I don't think there will ever be a problem with metal fatigue.