Okay, I like building stuff too!
In the spirit of innovation, I've thought about this before, and will add my 2 cents..
If you go to an AutoZone or someplace like that, you'll find sway-bar bushings and other suspension bushings made of polyurethane, which are made for sports applications.
They usually come in a kit, which may have more bushings than you need, and perhaps not all the correct size.
But, you can drill out the center holes if they are smaller, and you can turn down the outer diameter if they are large. If the center hole is too big, you can probably find a metal sleeve that you can press into it, to make it the right size to fit the shaft.
You can end up with the right size after you are done.
I would make them a little big on the O.D. and push them in tight.
I've seen bushings at AutoZone which would work in an Iron Barrel swing-arm with modifications like I described, for under twenty bucks, and you'd have extras left over.
Cheap, available, and easy to modify, and the material is made for the purpose.
I think a little bit of "give" is good, and it won't move noticeably, and should do.
I don't particularly like solid metal bushings or needle bearings because I think they are harsh, and transmit alot of vibration.
My 2 cents.