The Blades went into the clash unbeaten but goals from Leon Best, Clinton Morrison and Martin Cranie won it for the home side, with goals from Jamie Ward and James Harper for the visitors.
Coventry made a very bright start to the clash at the Ricoh Arena and they took a 12th minute lead through Best.
The big striker took a long Sammy Clingan pass on his chest and powered a shot past Mark Bunn in the United net.
Kevin Blackwell's side had two great chances to equalise within ten minutes of going a goal down and both fell to Stephen Quinn.
On both occasions he was one-on-one with Coventry's highly-rated goalkeeper Keiren Westwood. Both times the Republic of Ireland international won the battle after making two very good saves.
The visitors did draw level just after the half-hour mark after a Coventry attack broke down, which led to a counter from United.
Ward cut in from the right and fired a sweet left-foot shot into the top corner of the net.
On-loan Chelsea midfielder Jack Cork came close to restoring Coventry's lead but he shot just over the bar from Morrison's knock-down.
Nick Montgomery had a shocker in front of goal just before half-time. Chris Morgan headed down for the midfielder and he failed to make any sort of contact on the ball just six yards from goal.
Montgomery made another mistake at the start of the second half and this time it cost his team a goal.
His slip allowed Michael McIndoe to cross unchallenged and Morrison side-footed the ball into the corner of the net past Bunn.
And the Sky Blues went 3-1 up after an hour when Cranie headed his first goal for the club from McIndoe's inswinging corner.
United pulled one back with 14 minutes left on the clock as Harper scored from close range after a shot from the edge of the area was too hot to handle for Westwood.
The visitors almost nicked a point at the death as sub Ched Evans clipped the bar with a shot from 20 yards.