Is it possible there is a short to or in the brake wiring? Not enough current to fully activate the brakes but enough to cause one to drag unless backed too far off.
I'm not sure if the brakes are wired in series (current flows from one brake to the other) or parallel (each has it's own power line) but a limited short giving small amount of power to the brakes in series might engage one side more than the other if there was not full power.
Also is there a fuse that would blow or a circuit breaker that will keep resetting on the brake circuit. Have seen cases where a little bit of a short such as a wire with the insulation pinched or just barely worn through will slowly trip circuit breaker which cools then resets only to trip again in a minute or two.
Weirdest short I ever saw that was not a full short was a camper with a bare 110 volt wire touching a can with a paper label. Would not trip circuit breaker but if you touched the door handle in bare feet you got zapped. Paper would not let enough current flow through can to trip but enough to give you a poke. Saw the same thing in a house, kids got small poke from
light switch, adults did not. No tripped fuse. Turned out kids stood on the heat vent under the switch and were grounded, adults stood further back due to longer arms and being taller. Not on heat vent so not grounded = no poke.
This is just SWAG but might be worth checking to see if any current is flowing through brake circuit without brakes being applied. Especially if you have a timed brake controller which can be tripped by current back feeding into brake
light circuit. Possibly even if not enough current to
light the brake
lights.
If you set the spindle nut using a torque wrench and backing off correct amount I would accept that as "good" and move on to testing other things.