Arsene Wenger saw his Highbury side fall further behind Chelsea in the Premiership race after the weekend draw with Southampton while last week Bayern Munich beat the Gunners 3-1 in the Champions League.
But the Arsenal keeper kept his nerve bringing off two saves to foil Alan Quinn and Jon Harley before Ashley Cole smashed in the winner.
United were guilty of an astonishing miss after just 39 seconds. Andy Gray found space on the right flank to pick out the unmarked Michael Tonge as he charged into the area but from ten yards out the midfielder scooped the ball high over the bar.
Arsenal found a way through in the 18th minute. Cesc Fabregas, booed by the crowd for his vicious tackle on Nick Montgomery in the first match at Highbury, picked out Arturo Lupoli. His short pass was measured for Fredrik Ljungberg who sliced the ball inches wide from 13 yards.
Patrick Vieira got the better of midfielder Paul Thirlwell five minutes before half-time before slamming in a low cross that was spectacularly cut out by keeper Paddy Kenny who snatched the ball at the second attempt with Ljungberg closing in.
United squandered another chance after just 45 seconds of the restart. In a carbon-copy move of the first-half howler, the Blades got round the back of the Arsenal back four.
This time Montgomery was the provider, his cross from the right finding Andy Liddell who swung and missed. The ball was eventually met by Harley who stabbed his shot into the turf and Almunia was able to smother the effort.
A mistake by Tonge then let in substitute Quincy Owusu-Abeyie. His angled drive was pushed away by Kenny before full-back Derek Geary almost put through his own net.
Vieira carved out the opening, threading the ball to French midfielder Mathieu Flamini whose cross-cum-shot was scooped over his own bar by Geary, the ball missing the target by inches.
Arsenal began to open up the United defence. Fabregas hit the crossbar with a stunning drive and was foiled by Kenny in the 83rd minute, the United keeper diving at the to his left to save the Frenchman's low drive.
With three minutes of normal time left, Vieira headed over from close range and Harley's fierce header was tipped over by Almunia as the game went into extra-time.
Phil Jagielka cleared a Pascal Cygan shot off the line, Kenny blocked from substitute Jeremie Aliadiere and the Blades keeper snuffed out Cole's free-kick, but it was Arsenal that eventually prevailed in the penalty shoot-out.