Asian teams had managed three draws in 17 matches against South American sides but the four-times Asian champions made their extra man count after Colombia midfielder Carlos Sanchez became the first player to get a red card in Russia.
His sending-off in the third minute, for a deliberate handball, was the second fastest in World Cup history after Uruguay’s Jose Batista was dismissed after 55 seconds against Scotland in 1986.
Shinji Kagawa scored from the resulting penalty to give Japan the lead.
Juan Quintero’s clever low free kick in the 39th minute brought Colombia’s equaliser but Yuya Osako leapt above his marker to head a corner into the net for the 73rd-minute winner.
It was sweet revenge for Japan, who were thumped 4-1 by Colombia in Brazil four years ago, but they wasted plenty of chances to kill the game off.
Kagawa went close in the 16th minute but his shot rolled past the post and Osako fired against the face of goal from a dangerous position in the box.
With Japan dominating in midfield, Colombia’s Argentine coach Jose Pekerman opted to sacrifice Juan Cuadrado around the half hour mark to send on defensive midfielder Wilmar Barrios.
That gave Colombia extra control and their all-time leading scorer and captain Radamel Falcao came more into the game.
Falcao, making his World Cup debut after missing the last tournament in Brazil due to injury, showed his hunger when he stretched full length to get his foot to an overhead cross but did not have the power to beat the goalkeeper.
Genki Haraguchi’s foul on Falcao earned Colombia a foul on the edge of the box and Quintero fired a low free kick under the leaping wall. Japan goalkeeper Eiji Kawashima reacted late and by the time he got his hands to the ball, it had crossed the line.
Pekerman introduced James Rodriguez, who did not start due to a calf problem, in the 59th minute but the Golden Boot winner in Brazil was unable to change the course of the match.