Draco's frustration is understandable but for this time only I do understand Harry's cluelessness as wel..
I think this is the hardest time in Draco's recovery. He starts remembering things but is still processing them from a childlike perspective. As soon as he is able to see those from an adult perspective for longer than one minute things will improve.