Need help with repairing SE stick!

Basically the wire connection/adapter that connects the board to the actual stick itself has been broken (the black one that comes with the SE stick). I have a sanwa replacement stick anyway but because the adapter is broken, I’m not sure what to do next.

What I’m thinking about doing is just soldering the wires to the board (the prongs that the blue arrow points to in my picture) but I’m not sure which colour wire goes to what. If this is possible, please bear in mind that when I received my wire kit, the cables were loose so I put them in any old order lol. From what I understand, my black(ground) wire is in the correct place though.

Can anyone help me please =)

SE stick picture by layton9071 - Photobucket

Sorry, forgot the link.

Black is GROUND.
Yellow is RIGHT.
Green is LEFT.
Orange is UP.
Red is DOWN.

Man, you had me confuse for a little.
Because you didn’t put the colors in regular order.
The order for Sanwa Wire Harness goes; Black, Green, Yellow, Orange, Red.

My answer above is for your new colors.