Should you want to delete a piece of evidence that's been attached to a Task, you can do so by going to the evidence of the respective learner, hover over the evidence entry you'd like to delete, and you should see a delete icon you can click to delete the entry:
​

NOTE: You'll only be able to delete the evidence if the status of the 'In Progress' (shown in the green box above).