Microrheology with optical tweezers (MOT) measurements are usually performed using optical traps that are close to isotropic across the plane being imaged, but little is known about what happens when this is not the case. In this work, we investigate the effect of anisotropic optical traps on microrheology measurements. This is an interesting problem from a fundamental physics perspective, but it also has practical ramifications because in 3D all optical traps are anisotropic due to the difference in the intensity distribution of the trapping laser along axes parallel and perpendicular to the direction of beam propagation. We find that attempting viscosity measurements with highly anisotropic optical traps will return spurious results, unless the axis with maximum variance in bead position is identified. However, for anisotropic traps with two axes of symmetry such as traps with an elliptical cross section, the analytical approach introduced in this work allows us to explore a wider range of time scales than those accessible with symmetric traps. We have also identified a threshold level of anisotropy in optical trap strength of ~30%, below which conventional methods using a single arbitrary axis can still be used to extract valuable microrheological results. We envisage that the outcomes of this study will have important practical ramifications on how all MOT measurements should be conducted and analyzed in future applications.