Hmm, did you happen to try all 3 phase in 8 gauge without insulation by chance?
No, because the 10g was the biggest I had at the time.
(well, I also had some stuff the size of the entire axle end, but that definitely wouldn't fit thru there).
If I was really going to do a wire upgrade to something truly FAAAAT,
I'd do something proposed by John in CR years ago: change out the side cover for a custom-made one with a MUCH larger ID bbearingg, and make a ring with separate holes for the finger-sized phase wires and the grommets that will seal their entrance, and two other separate holes for a hall sensor bundle, and a thermal/etc sensor bundle.
This ring would fit between the bearing ID and the axle shoulder, and be attached (epoxy, threads on it's ID and axle shoulder OD, some form of mounting screw, etc) to the axle itself so ti cannot rotate relative to the stator.
It's tough to do this kind of mod on a motor that has to be driven by a chain or belt on one side and have a disc rotor on the other, but it would work fine on my SB Cruiser v1.x where the disc would be on the outboard side and the inboard side would be clear on both rear wheels. (the chain drive from the pedals would go to the front wheel, up and over the headtube/fork crown like the Trucktrike does. (wish I could buy the assembly from him). ) Then the phase wries would head right along the frame to the Lebowski/HondaIMA controller inches away on each side.
An even better version of that mod (for anything that can use a single-ended axle) would replace the entire inboard side cover and axle with a unit that has a giant bearing on the outside edge and a thin ring to support it that bolts to the motor cover bolt ring area. The stub axle would go on to the outside adn be threaded there for a spinner nut like 100-year-old cars sometimes had. The cover/axle unit would have plugs integraed into them that mated with plugs on the stator module so you can unplug and remove the entire motor/wheel just by slipping it off the axle once the nut is removed. Splines or keyways on the axle would mate with the stator support core to ensure fitting only one way on there to keep alignment easy.
Not that I'll ever be able to build any of that....but it'd be an easy strong way to deal with motor mounting and power delivery.
(a more practical version of that simply uses something liek the qsmotor single-sided hubs that use standard bolt on automotive wheels, but instead of using those wheels use a custom hub that a spoked wheel can be built on).