But after 90 minutes and extra time failed to bring a goal, the Bramall Lane men finally won through 4-3 penalties.
The decisive penalty miss from Shrewsbury's point of view came from teenage goalkeeper Joe Hart whose spot-kick struck an upright.
Seconds earlier he had saved brilliantly from Kyle McFadzean to keep Shrewsbury's hopes alive and throughout the night had produced a string of fine saves that must have impressed watching Manchester City manager Stuart Pearce and his goalkeeping coach, the former Blackburn Rovers and England keeper Tim Flowers.
The pair were running the rule over Hart for the second time in four days, having also seen Shrewsbury' home match against Notts County last Saturday.
From the side that defeated the League Two leaders at the weekend, Shrews boss Gary Peters made a number of changes that included a recall for striker Duane Darby who was overcome a lengthy spell on the sidelines due to injury to start his first game since February.
It was Darby who went closest to an early breakthrough, bringing a fine diving save from Blades keeper Phil Barnes in the seventh minute.
Ten minutes before the break, Darby went close again but just failed to make sufficient contact with a Ben Herd cross as he slid in at the far post.
Hart excelled himself in the 69th minute denying Danny Webber with a superb reflex stop and with extra-time looming substitute Mark Stallard almost won it for Town but Barnes was able to hold his glancing header.
In extra time there were near things at both ends but neither side could muster a goal with United taking a 3-2 lead on penalties when Stallard missed with Shrewsbury's third kick.
Both sides then managed a successful strike with Neil Warnock's side finally booking a place in round three when Hart smashed his spot-kick against the woodwork.