Chelsea Cornet's lobbed strike in the 78th minute sealed the victory in a thrilling match at New Douglas Park. Rangers had taken the lead twice, only to be equalized by their opponents.
Rangers went ahead through Rio Hardy and Katie Wilkinson, but Celtic levelled each time through Caitlin Hayes and then Saorise Noonan, before Cornet sealed the win. They are second, a point behind Glasgow City, who won 6-1 at Queens Park.
The visitors were handed a slice of luck for the opener after four minutes, as Hardy's strike from the edge of the box took a deflection and looped in over the wrong-footed Celtic goalkeeper Kelsey Daughterty to give them the lead.
Celtic equalised just before the break, as Hayes headed in at the back post from Emma Louise Lawton's cross.
TrendingRangers regained their lead through Katie Wilkinson's penalty just past the hour after Charlie Devlin went down in the box under a challenge from Shannon McGregor.
Celtic boss Elana Sadiku was sent off after the incident, receiving a booking for her initial response to the decision and then picking up another for sarcastically clapping referee Stacey Fullicks.
That did not deter her side from fighting back, as Noonan made it 2-2 with from a free-kick that just evaded Rangers goalkeeper Jenna Fife in the 67th minute.
But Cornet came off the bench to settle the game, latching on to a loose clearance from Daugherty, before dinking the ball over the Celtic goalkeeper's head in the final quarter of the game.