If no self leaning is taking place with the controller, then you are getting mixed results...a true reverse and a false positive reverse denoted by a rougher start, a lack of torque, higher no load current, and typically higher max rpm due to the advanced timing of the incorrect wiring config. If it's a sine wave controller then the sonic difference would be harder to notice.
Can you measure no-load current? Can you apply the brake without motor cutoff to test for torque? Use slow short throttle pulses when testing wiring configs, because incorrect ones can cause high current and blow another controller or worse, the motor. I always use jumper wires to complete the connections as they're easier to swap and act like a fuse to prevent high currents while being very careful not to cause phase or battery wire shorts during the process.
Since you've swapped so many wires, start at the beginning with the original wiring config, and determine if that is a good smooth reverse or a false positive. If it is a false positive reverse, then swap ONLY the phase wires until you find the good forward of the 5 remaining possible combos....small throttle pulses. If you get reverse on every one, then self learning is going on.
If it's a true reverse, then swap only 2 of the hall wires, and find the right combo of phase wires of the 6 possible. To be systematic it's easiest to swap just 2 at a time choose one wire as an anchor wire (eg motor wire green...try the 2 combos of blue and yellow with motor green on controller green, controller yellow, and controller blue)
Our 3 phase motors have 6 "good" wiring configs. 3 in reverse and 3 forward. They also 6 false positives that will spin the wheel, and in every case I've seen, the false positive is in the opposite direction of "good" config while keeping the halls static.
Note that above I talk about swapping phase wires and keeping halls static (or at most one swap of halls to change direction). That's because halls are typically harder to swap around, but you can get exactly the same results swapping halls if they're easier to change around and keep the phase wires static other than one swap of only 2 to change direction.
I also keep a supply of all the colors of electrical tape to mark the wires the matching colors when I find the good wiring config for easier connection later.
Do this once and you'll have confidence to connect any bldc motor even if the wires aren't color coded. It is NEVER necessary to go through the possible 36 possible combos of hall and phase wires, and those promoting it and the table to help go through it should be ignored.
