The change of coherence and polarization of an electromagnetic beam modulated by a random anisotropic phase screen passing through any optical system is found within the framework of complex ABCD-matrix theory This means that the formalism can treat imaging and Fourier transform and free-space optical systems, as well as fractional Fourier transform systems, with finite-size limiting apertures of Gaussian transmission shape. Thus, the current paper shall be considered as a continuation, extension, and generalization of a previous work by Shirai and Wolf [J. Opt. Soc. Am. A 21, 1907 (2004)]. It will be shown that the inclusion of apertures in the optical system strongly influences not only the propagation of spatial coherence but also the degree of polarization of a propagating field. Analytical expressions of coherence and polarization propagation will be given in terms of the matrix elements for any complex optical system. © 2008 Optical Society of America.