The Bramall Lane promotion favourites survived an early battering against a Coventry side who had made a stuttering start to their campaign.
Midfielder Jay Tabb broke the deadlock midway through the opening half, before Sharp struck with a clinical finish after China international Sun Jihai had been red carded.
Coventry started brightly and Freddy Eastwood, who was snapped up for £1.2million in the summer, cut inside to blast a low left-foot shot which was smothered on the line by the diving Paddy Kenny.
United were saved by Kenny in the 15th minute after the impressive Aron Gunnarsson raced from midfield before delivering a stinging drive from 20 yards.
Kenny, diving to his left, could only scoop the ball away to the waiting Clinton Morrison, whose close range strike was blocked by the United keeper with the Bramall Lane defence in tatters.
Gary Speed tried his luck with a long range strike high over the target, before Coventry went ahead in the 30th minute.
Gunnarsson's long throw from the left reached Tabb 15 yards out and he finished clinically with a low angled drive beyond Kenny's reach.
United escaped three minutes after the restart when Sun tripped substitute Michael Mifsud, who was on for the ineffective Leon McKenzie.
Centre-back Elliott Ward stepped forward and blasted his spot-kick wide of Kenny's left-hand post.
There was more misery for the Sheffield side when Sun was handed a straight red card for a lunging tackle on Mifsud in the 52nd minute.
However, Sharp equalised on 56 minutes after turning quickly to stab in David Cotterill's cross.
Speed almost grabbed a sensational winner deep into time added on but his chipped shot was tipped over the bar by Keiren Westwood.