I used a dremel to bore out a keyway, well actually keyholes on my 6mm shaft. I used a tungsten carbide bit which is more like an endmill than a drilling bit, so the cutting face is on the side.
If you go this route just mark your holes on the shaft, then come at the shaft perpendicularly...