Going from ceramic to Nd would cut the speed of the motor by around 4x or more. Jumping the voltage up to compensate is not going to fly with the brush arcing.
I'm not sure if it would be that much. I suppose it depends on how good the magnets are. Yes, you'd have to increase the voltage to compensate, but brush arcing is not just voltage dependent. I've seen plenty of brushed motors running 90V or more as long as the current isn't too high. I did a similar thing on my old Zappy. I rewound the motor to compensate for part of it and doubled the voltage. The brushes were about the only thing I didn't have problems with.
Hmm... I suppose that's a good excuse to make it brushless.
Oh, and I missed the part about the freewheel. That makes much more sense now.