That is why.
Red is Down, when orientation of the Sanwa JLF is with Connector to right.
And since you made Red be Ground, and Black be Down, you sending Signal for Down through all the Microswitches.
On the Wire Harness, the Black wire goes to all Microswitches.
You made that wire be Down.
And Red be Ground.
So only one direction would work.
And is Down.
Looking at your pictures, the wiring should be this:
With your Joystick in orientation as shown in picture (right in respect to Buttons from the inside), here is how wire:
Black = GROUND
Green = RIGHT
Yellow = LEFT
Orange = UP
Red = DOWN
But it does matter when Wire Harness is used with Joystick.
When plugged in, the wires go to specific Microswitch on Joystick PCB.
No matter which way Joystick is, the wire gets lead to the same specific Microswitch.