You do. Because the full surface is acting as a stray light source. And as all rays coming from the object fill the FULL aperture (say, lens/mirror diameter), the stray light will cover the full frame/image. When the object would not be hot/bright, this stray light would be hardly visible. But...