On Automated Essay Grading using Large Language Models

Automated Essay Grading (AEG), combining Automated Essay Scoring (AES) and Automated Writing Evaluation (AWE), is a time-saving solution to the challenges of manual essay evaluation. It aims to reduce the workload on educators by offering a more consistent grading approach. Inspired by ChatGPT’s impressive language comprehension and generation capabilities, this study explored the potential of various Large Language Models (LLMs) in AEG tasks. The models examined include GPT-4, GPT-3.5, PaLM, and LLaMA2. Tailored prompts were designed and their performance was assessed in conjunction with each LLM through prompt engineering. Our study shows that LLMs can achieve substantial agreement with human markers in AES, with a Quadratic Weighted Kappa (QWK) score of 0.68. In AWE, the feedback on the essay was assessed qualitatively. It achieved an agreement level score of 4.9 (out of 5) with a standard deviation of 0.05, closely aligned with human assessment. This study provided valuable insights into the effectiveness of LLMs in automated essay grading. It highlighted their potential to enhance educational assessment practices.