We present new understanding of excitonic fine structure in close-to-symmetric InAs/GaAs and InGaAs/GaAs quantum dots. We demonstrate excellent agreement between spectroscopy and many-body pseudopotential theory in the energy splittings, selection rules and polarizations of the optical emissions from doubly charged excitons. We discover a marked difference between the fine structure of the doubly negatively and doubly positively charged excitons. The features in the doubly charged emission spectra are shown to arise mainly from the lack of inversion symmetry in the underlying crystal lattice. © 2007 The American Physical Society.