Two first-half goals were enough to see off their second Premier League side after they ousted Bolton in the previous round.
United, who are currently a disappointing 14th in the Championship table, came close to an eighth minute lead when youngster Luton Shelton flighted in a testing cross and Jon Stead headed down but keeper Joe Hart dropped smartly to save.
Four minutes later, the home side were in front with a little assistance from Manchester City fans.
Lee Martin did well to drive in a dangerous cross that caused problems and Michael Ball appeared to be distracted as the ball twice hit pale blue balloons that had drifted on to the pitch from the 6,000-strong travelling fans.
But Shelton kept his eye firmly on the ball and cashed in by placing his shot out of the reach of Hart.
An angry Hart then spent several seconds stamping on the remaining balloons to prevent a repeat.
A minute later City should have been level when Ball's raking cross found Emile Mpenza, but he failed to hit the target and headed over.
As City turned up the pressure for a leveller, Elano found space on the edge of the area and squeezed a low shot through a crowd of players but it grazed the post before going out.
Elano had another super chance for City when he drove a fierce free-kick from 25 yards but United keeper Paddy Kenny dived full-stretch to save.
It was the home side who struck again with a second goal after 24 minutes.
Derek Geary crossed from the right and, as City's defence dithered and failed to clear, Stead kept cool and rammed in his low shot.
Five minutes before half-time a long-range effort from midfielder Dietmar Hamann might have given Kenny a problem but it was deflected wide.
City boss Sven-Goran Eriksson brought on teenager Danny Sturridge at the start of the second half and it had immediate results.
Martin Petrov's corner had the home side in trouble and, after the ball deflected to Sturridge 12 yards out, he crashed in a left-foot shot for his first City goal.
City continued to look for an equaliser but on the break the home side went close when another impressive cross from Geary met the head of little Stephen Quinn but he directed it wide.
City hurled everything at the Blades in the dying minutes but the Steel City side held out.