This paper is a review of our recent work on frequency analysis of polarization imaging system. To study the effects of partial polarization and partial coherence in a polarization imaging system, we perform frequency domain analysis of the Fourier spectra of the Stokes images by using a four-dimensional frequency domain approach. We begin with the relations of the generalized Stokes parameters in the frequency domain and calculate the spectra of the Stokes images as a generalized treatment of polarization imaging system. Then, we explore the frequency response for diffraction-limited polarization imaging system in the coherent and incoherent limits. The Optical Transfer matrix (OTM) which is a transfer matrix to the spectral Stokes vector has been introduced for the diffraction-limited incoherent polarization imaging system. Some general properties associated with OTM are presented and one example of diffraction-limited OTM is provided with their comparison with the conventional Mueller matrix to reveal their intrinsic relationship.