Gray converted the spot-kick after Philippe Senderos was adjudged to have handled Danny Cullip's goal-bound flick with just seconds remaining of normal time.
Substitute Robert Pires appeared to have saved Arsenal from a replay in Yorkshire with a 78th-minute opener.
Arsene Wenger's side had struggled to overcome the loss of Dutchman Dennis Bergkamp, who was sent-off in the first half for shoving United defender Cullip in a keenly contested affair in North London. Arsenal had plenty of possession but until Pires tucked away his chance, United 'keeper Paddy Kenny had repelled all they had to offer with a string of fine saves.
The opening minutes saw plenty of bustle in midfield but Arsenal had only one real effort to show for their early possession - Cesc Fabregas testing Kenny with a left-foot drive from 18-yards. United, however, were by no means overawed by the reigning Barclays Premiership champions and should have taken the lead in the eighth minute.
Nick Montgomery progressed down the right and should have done better when presented with the chance to pull the ball back from the byline. However, the United player won a corner and when Andy Liddell delivered the ball into the penalty area, Gray's point-blank header was somehow kept out by Manuel Almunia at the near post.
It was a real scare for the Premiership side who looked distinctly uncomfortable having to contend with some fierce tackling from United's midfielders.
In the 23rd minute, Arsenal were awarded a free-kick on the edge of the penalty area when Phil Jagielka fouled van Persie but the striker wasted the chance by aiming his shot straight at United's defensive blockade.
Dennis Bergkamp went close for Arsenal in the 25th minute when he tried to curl his 18-yard effort into the top corner of the net. However, he could not keep his shot down and it just flew over the top of the crossbar. It was followed three minutes later by another van Persie shot which Kenny did well to palm away for a corner.
It was all Arsenal now as the home side finally began to turn up the heat on their Championship opponents. Kenny produced another fine stop from Jose Antonio Reyes as the Gunners searched for a nerve-settling opening goal.
Leigh Bromby then wasted a chance at the opposite end when Liddell's free-kick picked him out in oceans of space inside the Arsenal penalty area.
The big defender rose unchallenged but could not direct his header on target and the home side's momentary lapse of concentration went unpunished. The impressive Liddell then sent a right foot volley flashing just wide of Almunia's left-hand post.
Ten minutes from half-time Arsenal were sensationally reduced to ten men when Dutch maestro Bergkamp was handed a red card by referee Neale Barry.
Arsenal's players were incensed about a number of challenges by Montgomery and Cullip and in the melee that followed, Bergkamp appeared to raise his hands and push Cullip away. Barry immediately reached for the red card to the fury of Arsene Wenger and Bergkamp's team-mates.
The dismissal then sparked a flurry of bookings with United's Liddell and Paul Thirlwell added to Barry's notebook along with Fabregas for a wild challenge on Montgomery.
United made one change during the interval with former Arsenal striker Paul Shaw added to the attack in place of midfielder Thirlwell.
United had shown they were no pushovers in a frenetic first half but it was Arsenal who almost broke the deadlock in the 52nd minute when Emmanuel Eboue forced Paddy Kenny into a fine save at his near post. United struggled to make the most of their one man advantage chiefly because Arsenal remained full of commitment and running.
But without Bergkamp's guile in the final third, they lacked the creativity to break down a resolute United rearguard. In the 65th minute Arsene Wenger removed van Persie in favour of Pires in a bid to provide the required spark in attack.
The change did the trick 12 minutes from time when Pires forced the ball home after Flamini's first effort had rebounded into his path off the hands of the unfortunate Kenny, and the Frenchman made no mistake from six yards to the relief of the home fans.
But Gray sent the visiting fans wild in the last minute of normal time when Danny Cullip's hooked effort was handled by Senderos and he converted the subsequent spot-kick to earn a deserved rematch in Yorkshire in a fortnight.