LitCrit: exploring intentions as a basis for automated feedback on Related Work.
Casey, Arlene Jane
Learning the skill of academic writing is critical for post-graduate (PG) students to be successful, yet many struggle to master the required standard. Feedback can play a formative role in developing these skills, but many students do not find sufficiently helpful the kinds of feedback available to them. As the Related Work section is known to be particularly difficult for PG students to master that is the focus of this thesis. To date, models of academic writing have been built on observational studies of academic articles. In contrast, we carry out a user study to explore what content experts look for in Related Work and how this differs from PG students. We claim that by understanding what experts look for in Related Work and what aspects PG students struggle with, a useful author intention model can be developed to support writing feedback for Related Work sections. Our work demonstrates reliable annotation of the model intentions. Developing on existing algorithms, designed to identify rhetorical intentions in academic writing, we build a supervised machine learning classifier, showing how features focused on Related Work sections improve recognition of content aspects. Carrying out a study to rate the quality of Related Work, we demonstrate that the model is a good proxy for predicting quality, validating the choice of intentions in our model. In addition to recognising author intentions, we automate the generation of feedback based on observations of intentions that are present and missing, taking into account areas that PG students struggle to recognise. The thesis also contributes a new prototype writing analytic tool, called LitCrit, that supports visualising the intention narrative of Related Work and presents feedback. We claim this visualisation approach changes the PG student’s perception of Related Work, and demonstrate through a user study that it does draw attention to aspects previously missed bringing PG student responses in line with experts. Finally, we explore the performance of our classifier, originally set within the Computational Linguistics discipline, to that of Computer Graphics. This shows us that while performance may be lower when care is taken to understand those features which are discipline dependent, there is scope for improvement. Also, while a discipline may have the same intentions present in a section, their structural presentation may differ impacting feature choice.