Thanks for your suggestion. I had also noticed this effect on the e-bike. By increasing the tire pressure, it was more manageable, but I could feel it pulling sometimes. Your pannier suggestion sounds great for reducing this.
However, at the moment, I think I will go with the battery hanging...