We extend the classical compartmental frameworks for susceptible-infected-susceptible (SIS) and susceptible-infected-recovered (SIR) systems to include an exposed/latent class or a chronic class of infection. Using a suite of stochastic continuous-time Markov chain models we examine the impact of latent and chronic infection on the mean time to extinction of the infection. Our findings indicate that the mean time to pathogen extinction is increased for infectious diseases which cause exposed/latent infection prior to full infection and that the extinction time is increased further if these exposed individuals are also capable of transmitting the infection. A chronic infection stage can decrease or increase the mean time to pathogen extinction and in particular this depends on whether chronically infected individuals incur disease-induced mortality and whether they are able to transmit the infection. We relate our findings to specific infectious diseases that exhibit latent and chronic infectious stages and argue that infectious diseases with these characteristics may be more difficult to manage and control.