Navigating the landscape of automated feedback generation techniques for programming exercises

Programming demands a variety of cognitive skills, and mastering these competencies is essential for success in computer science education. The importance of formative feedback is well acknowledged in programming education, and thus, a diverse range of techniques has been proposed to generate and enhance formative feedback for programming exercises. This article reviews state-of-the-art automated feedback generation techniques and categorizes the various approaches based on the underlying computational techniques, programming languages, the kind of programming errors they deal with, and the type of feedback they provide. It covers data-driven techniques, those which use program repair methods, machine learning-based techniques, and techniques based around the use of large language models, particularly noting the rapid uptake of the latter. The article provides a summary of key findings and challenges, alongside recommendations for future work. The findings reveal that although there exist numerous tools for automated programming feedback, many studies depend on non-public benchmarks, which limits reproducibility and independent evaluation of the tools and their datasets. Additionally, tools are not always language agnostic and in some cases involve complex configuration steps. Large language models have demonstrated transformative potential in generating feedback. However, most research has focused on introductory courses (CS1 and CS2) indicating the need to apply them in advanced fields like machine learning and image processing. Although large language models have outperformed traditional approaches, challenges related to hallucinations and incorrect responses still need to be addressed as precision is critical in a pedagogical setting. Most of the studies use proprietary models that lack transparency and customization options, emphasizing the need for further research into open LLM alternatives.