In this paper we continue the study of the superconformal index of four-dimensional N =2 theories of class S in the presence of surface defects. Our main result is the construction of an algebra of difference operators, whose elements are labeled by irreducible representations of A N −1. For the fully antisymmetric tensor representations these difference operators are the Hamiltonians of the elliptic Ruijsenaars-Schneider system. The structure constants of the algebra are elliptic generalizations of the Littlewood-Richardson coefficients. In the Macdonald limit, we identify the difference operators with local operators in the two-dimensional TQFT interpretation of the superconformal index. We also study the dimensional reduction to difference operators acting on the three-sphere partition function, where they characterize supersymmetric defects supported on a circle, and show that they are transformed to supersymmetric Wilson loops under mirror symmetry. Finally, we compare to the difference operators that create ’t Hooft loops in the four-dimensional N =2* theory on a four-sphere by embedding the three-dimensional theory as an S-duality domain wall.