Arsenal's captain for the day, Dennis Bergkamp, was sent off for violent conduct in the 35th minute after an incident with Danny Cullip.
The ten-man Gunners grabbed a 78th minute lead through substitute Robert Pires. Stoppage time loomed when reserve centre-back Philippe Senderos handled Cullip's chip back into the box and Gray tucked away the spot kick. Once again, Arsenal had paid for failing to clear a corner.
After the game Gray said: "When we got the penalty, I made up my mind which side to put it and concentrated on hitting the target.
"We created some good chances in the first half. I could have done better with my early header. If I'd headed it towards the corner, it would probably have gone in.
"It was special to get this result at Highbury and come back after they'd scored. Not a lot of teams come back against Arsenal."
United manager Neil Warnock said: "We deserved a draw and my directors are happy because we can boost our coffers from the replay."
Gunners boss Arsene Wenger confirmed that Arsenal would appeal against Bergkamp's red card. He said: "It wasn't a sending off offence. The whole incident was started by an attack on Cesc Faberegas when he was on the ground. The players were incensed."
He added: "I've no regrets about fielding the team I did, with the Champions League game against Bayern Munich coming up on Tuesday. If it had been 11 against 11, I think we'd have won. Losing Bergkamp was crucial."
But it will be interesting to see how many of his major players Wenger will include for the replay at Bramall Lane on March 1.
Arsenal's rejigged side lacked fluency early on. But they showed great spirit in the second when Freddie Ljungberg and Jose Antonio Reyes stretched the United defence to breaking point.
Ashley Cole has flu, Edu was missing after suffering a hamstring injury against Palace while Wenger rested Jens Lehmann, Patrick Vieira and Thierry Henry. Manuel Almunia and Ljungberg returned for the Gunners.
United were without suspended defender Chris Morgan who was sent off in their fourth round shoot-out win over West Ham.
There is a history between these sides in recent FA Cup ties. Their first meeting in 1999 was declared void and replayed after Arsenal scored from a throw that should have been returned to United after an injury. The Gunners won the 'replay' 2-1.
Two years ago, Arsenal beat United 1-0 in the semi-finals. Ljungberg scored the winner after referee Graham Poll collided with United midfielder Michael Tonge and David Seaman pulled off an amazing save from Paul Peschisolido.
Today's contest was just as intense. United began with Andy Gray as a lone striker, with Andy Liddell and Tonge in wide positions. Phil Jagielka dropped into a back three, Cullip stuck tight on Bergkamp and Arsenal were clearly rattled by United's no-nonsense tackling.
United targeted Arsenal's vulnerable central defence at set-pieces. Wenger has been promised 'sufficient transfer funds' to strengthen his squad in the summer but Arsenal desperately need cover at centre back, where Senderos looked ponderous.
United worked a quick corner routine after seven minutes. Gray was unmarked to power a downward header which Almunia palmed away at the second attempt.
Leigh Bromby wasted another free header when Arsenal failed to pick him up at a free kick and then Cullip headed home a long throw but the goal was disallowed for pushing. That pressure would pay dividends for United at the death.
When Arsenal found their rhythm, they found United keeper Paddy Kenny in top form. He held shots by Fabregas and Robin Van Persie, then came rushing out, forcing Fabregas to lob too high from Reyes' throughball.
Bergkamp ended a fine move by Ljungberg and Mathieu Flamini with a shot onto the top of the net. Kenny saved low at the near post from Van Persie, then made a brilliant stop with his feet from Reyes after Fabregas' diagonal pass split the United defence.
Referee Neale Barry showed Bergkamp the red card in the 35th minute and a melee followed Cullip's foul on Fabregas. Bergkamp moved angrily towards Cullip, but seemed bemused when he was sent off. TV replays suggested that he had made no contact with the United defender, though another Arsenal player might have.
Warnock thought it might have been Reyes, but he added: "Bergkamp is so important for Arsenal, that, from our point of view, he was the right player to go off!"
Barry dished out three yellow cards in as many minutes before half time as tempers flared. Tonge complained to Barry, claiming that Emmanuel Eboue had spat at him.
Warnock tried to exploit Bergkamp's dismissal by bringing on ex-Gunner Paul Shaw to support Gray and moving Jagielka into central midfield but his side were too hasty and kept giving the ball away.
Kenny saved from Eboue, held Van Persie's weak header and then saved the Dutchman's shot on the turn. Reyes blasted a 70th-minute shot wide after a break by Ljungberg while Kenny grasped long-range free-kicks by Kolo Toure and Reyes.
He was finally beaten with 12 minutes left. Reyes' pass found its way through United's ranks to Flamini. Kenny beat out his shot - but Pires knocked in the rebound.
Fabregas wasted a good chance to seal victory, driving a cross shot wide when Ljungberg was well-placed.
It proved a costly miss. Tonge's 89th minute corner caused chaos in the home defence, Senderos raised his arms and Gray did the rest.
MAN OF THE MATCH: Paddy Kenny (Sheffield United) - Made a string of important saves as the Blades produced a spirited display.