We report numerical and theoretical investigations of the formation and transition of domain patterns in a two-dimensional optical system with cosine-type nonlinearity and a feedback loop. Labyrinthine stripe domain patterns of the electric field are observed in the system, intiated from the Turing instability. The labyrinths are found to undergo a transition to domain patterns of coexisting stripes and hexagons and disordered hexagon domains on variation of the incident field intensity, a control parameter of the system. The parameter regions for these domain structures are explained through the existence and competition of stripes and hexagons in terms of their amplitude equations. Moreover, the transition from straight stripes to labyrinths is investigated by varying the feedback coupling coefficient of the system. The transition is shown to be the consequence of coexistence of and interaction between stripes and domain walls. ©2000 The American Physical Society.