Optical metasurfaces have shown unprecedented capabilities in the local manipulation of the light’s phase, intensity and polarization profiles, and represent a new viable technology for applications such as high-density optical storage, holography and display. Here, we demonstrate a novel metasurface platform for simultaneously encoding color and intensity information into the wavelength-dependent polarization profile of a light beam. Unlike typical metasurface devices in which images are encoded by phase or amplitude modulation, the color image here is multiplexed into several sets of polarization profiles, each corresponding to a distinct color, which further allows polarization-modulation induced additive color mixing. This unique approach features the combination of wavelength selectivity and arbitrary polarization control down to a single subwavelength pixel level. The encoding approach for polarization and color may open a new avenue for novel, effective color display elements with fine control over both brightness and contrast, and may have significant impact for high-density data storage, information security, and anti-counterfeiting.