In this paper, we study the inverse problem of recovering the spatially varying material properties of a solid polycrystalline object from ultrasonic travel time measurements taken between pairs of points lying on the domain boundary. We consider a medium of constant density in which the orientation of the material's lattice structure varies in a piecewise constant manner, generating locally anisotropic regions in which the wave speed varies according to the incident wave direction and the material's known slowness curve. This particular problem is inspired by current challenges faced by the ultrasonic non‐destructive testing of polycrystalline solids. We model the geometry of the material using Voronoi tessellations and study two simplified inverse problems where we ignore wave refraction. In the first problem, the Voronoi geometry itself and the orientations associated to each region are unknowns. We solve this nonsmooth, nonconvex optimisation problem using a multistart non‐linear least squares method. Good reconstructions are achieved, but the method is shown to be sensitive to the addition of noise. The second problem considers the reconstruction of the orientations on a fixed square mesh. This is a smooth optimisation problem but with a much larger number of degrees of freedom. We prove that the orientations can be determined uniquely given enough boundary measurements and provide a numerical method that is more stable with respect to the addition of noise.