We introduce a non-Markovianity measure for continuous-variable open quantum systems based on the idea put forward in H.-P. Breuer et al. [ Phys. Rev. Lett. 103 210401 (2009)], that is, by quantifying the flow of information from the environment back to the open system. Instead of the trace distance we use here the fidelity to assess distinguishability of quantum states. We employ our measure to evaluate non-Markovianity of two paradigmatic Gaussian channels: the purely damping channel and the quantum Brownian motion channel with Ohmic environment. We consider different classes of Gaussian states and look for pairs of states maximizing the backflow of information. For coherent states we find simple analytical solutions, whereas for squeezed states we provide both exact numerical and approximate analytical solutions in the weak coupling limit.