Coventry took the lead through Stern John in an eventful first half for the striker who also missed a penalty and struck a post.
Chris Morgan equalised after a scramble before Gray clipped home with five minutes to go before the break.
The Blades had nothing to show for their early possession and it was the home side should have wrapped the game up in a 15-minute first-half spell.
Stephen Hughes went close on 12 minutes when he twisted and turned on the edge of the box and then shot low - forcing Ian Bennett to tip wide.
Gary McSheffrey's cross was almost turned into the United net by defender Chris Morgan but again Bennett came to the rescue with another good save.
From the resulting flag-kick from Louis Carey, Danny Cullip was spotted clinging to Adie Williams' shirt and referee Mike Dean awarded a penalty.
Morgan antagonised the home supporters by trying to stamp down the penalty spot while McSheffrey and John debated who should take the spot-kick.
John won the argument but his penalty was weak and Bennett made an easy save down to his left.
McSheffrey and John put their differences aside on 19 minutes as the Trinidad & Tobago international atoned for his earlier miss.
McSheffrey reacted quickest to Dele Adebola's flick and squared for John who took one touch before rifling under the body of Bennett and into the back of the net.
Andy Liddell forced a smart stop out of Luke Steele as he got a head to Leigh Bromby's cross.
But it was Coventry who were on top and John almost made it two with the best piece of football in the match.
The striker controlled a fierce pass on the touchline and then ran 40 yards, beating two defenders on the way before chipping Bennett, but his effort clipped the post and went wide.
The Sky Blues' failure to take their chances came back to haunt them in the 32nd minute as the visitors equalised.
Michael Tonge crossed to the back post from a short corner and Steele failed to claim. The goalkeeper was out of position as the ball bounced around his area and Morgan turned the ball in from six yards.
With five minutes to go before the break, Sheffield United took the lead. Coventry seemed to have cleared a corner but when the ball came back towards them, the defence was all over the shop and Phil Jagielka had all the time in the world to cross for Gray who crashed home.
The home side began the second half desperate for an equaliser and it almost came straight away when John found McSheffrey with a through ball, but his effort was deflected wide.
United nearly found a third after 55 minutes when Alan Quinn crossed to the back post where Liddell was free, but his volley was well saved by Steele.
Steele was forced into action again on 64 minutes when Jon Harley shot from the edge of the area and the keeper, who is on loan from Manchester United tipped wide.
Liddell whipped in the corner and Morgan rose highest, but his header struck the post and bounced back into the hands of Steele.
Morgan had another opportunity to extend United's lead but he header over from another Liddell corner on 75 minutes.