From an engineering standpoint, I would say that at a minimum consider increasing the outer diameter of your spacers (assuming you're sure about doing this). The clamping area of your bolts is very important, especially on something you're pulling or pushing so much (creating a bending moment). In the standard configuration, the bottom surface of the shifter assembly reacts this load so the bolts dont see any kind of bending or prying. As a result, their load is purely tensile. With your bolts being so long, and your spacers being so thin walled (and being split), you essentially have a prying force being distributed over a very tiny surface area, that is then putting your bolts into bending.

Something like this would certainly be a step to reducing the loading on your bolts.
Spacer.jpg

What class of bolts are you using?