Where's the slop in the swing arm? I don't have that frame, but usually the spacer fits snug between the frame, then the bearings ride in the swing arm and tighten down on the spacer. When you make torque, the bearings should seat, and if by chance the frame is out of spec, bend the flanges, and everything should be snug.
Are you sure you're making torque? It's usually the only way to have slop. Too small a spacer would bind, and too big a spacer wouldn't fit.