Depends. If your controller draws too much power at standby, the charger will think the batteries are still charging and damage them. If you have a lot of SLA and the controller only draws like 10mA, however, you'll probably be okay.
In some "overvolted' systems, the higher charge voltage might get real close to the maximum allowed voltage for some of the controller components, so disconnect those.
The drain of the controller just being connected (but off) is very minimal.
I have heard reports of voltage spikes passing from the AC line through a charger and blasting the controller. With a brushed controller, this resulted in a scooter spontaneously taking off at full speed with no way to stop it.