Remove avatar image and not replace it?

Not a huge issue but is it correct that after you upload a picture for your avatar you can't remove it and go back to the default of no avatar image? I see that I can replace it with a new image but I don't see a way to delete the one that is there and not upload a new one. This has to be possible, right?