The angular profile and the orbital angular momentum of a light mode are related by Fourier transform. Any modification of the angular distribution, e. g. via diffraction off a suitably programmed spatial light modulator, influences the orbital angular momentum spectrum of the light. This holds true even at the single photon level. We observe the influence of various angular masks on the orbital angular momentum spectrum, both in the near and the far field, and describe the resulting patterns in terms of angular diffraction. If photons are entangled in their orbital angular momentum, diffraction of one photon affects the orbital angular momentum spectrum of its partner photon, and angular ghost diffraction can be measured in the coincidence counts. We highlight the role of the angular Fourier relationship for these effects.