Sounds like you have the wrong/dried-out/??? bushing. Like Dave said, the ID is too tight. Odd, especially if the firewall hole is sloppy.
There is a split bushing that doesn't require disassembly and some cut the OEM style with good luck (SpecialT, iirc).
3M Trim Adhesive will hold it in place, but a PITA for the next guy to remove...