Straylight is indeed not part of the main optical path, but it does make the image more "hazy". This lowers the contrast of the image and also the dynamic range of course is reduced.
I don't agree the lack of contrast would have been the cause of the effect though. The diffractive telescopes...