Feature image created by DALL-E AI image generator.
UPDATE: ANU has disabled this functionality as of 1 January, 2024 in Turnitin and it is no longer available for ANU staff.
From 5 April, 2023, a new feature became available in Turnitin for all staff at ANU which aims to detect if student work contains text generated by artificial intelligence (such as from ChatGPT). But how does it work, and how accurate is it? This post will share strategies for educators on how to interpret these reports and what to do if it flags student text as AI-generated.聽
Accessing the AI detection
All student assignments submitted to Turnitin will be checked automatically for AI-generated content. This will appear as part of the Feedback Studio, underneath the Originality Report.
In the image below, you can see the standard Originality Report for this piece of writing, with the AI score in the bottom right-hand corner.
This detection score is based on analysing sentence structure and writing patterns within the document to determine how much may have been by AI. It’s completely separate to the standard Turnitin Originality Report which matches text in the document against existing publications and web sources. It’s important to note that this is not displayed to students – only to teaching staff and administrators.
How accurate is it?
That is a tough question to answer!
Turnitin indicates that their detection is 98% accurate, but also highlight that all detection should be reviewed carefully and discussed with students. The AI detector comes with a built-in warning to caution against assuming that the score automatically indicates academic misconduct, as you can see in the image above.
Material provided by Turnitin explains that their strategy is to prioritise precision over detection – that is to say, they want to be very confident that something is AI written before it is flagged, which means sometimes AI writing can go undetected.
False positive and false negative results
The system can flag writing as AI-generated when it is not (a false positive) or also not flag writing as AI-generated when it is (a false negative).
This video from Turnitin explains when and how writing may be flagged as AI-written when it is not – usually due to repetitive writing style, or the use of lists, outlines, tables, code, or styles such as scripts or poetry.
Turnitin has not made their false negative rate public, where the detector does not recognise text as AI written. However, it is relevant to note that OpenAI, the team that created ChatGPT, have a detection tool that only identifies writing as AI generated 26% of the time (thus, 74% of the time, it fails to detect it).
It will be less accurate for shorter pieces of writing, as it requires a good sample size to determine if it is AI generated.
聽
Issues to consider
It has been well-publicised that it can flag innocent students, such as in this article from the Washington Post. Click here to read more.
My own testing found that it often missed detecting AI-generated content: in the sample essays I submitted, it only flagged 31% of the writing as AI-written, when it was nearly 90% written by ChatGPT.
Unlike the traditional Originality Report, it does not show where or how it is generating the matches so it can be unclear how the result was determined. It does not show directly which sentences are AI generated, but rather gives an overall indication of what they predict could be AI written.
AI tools are changing rapidly, and it’s hard for detection tools to keep up. For example, Turnitin’s model was trained on GPT-3 and GPT-3.5, but the company released GPT-4 early in March 2023. Turnitin is actively seeking feedback from educators on their tool, including in this pop-up that appears when you look at the detection score.
Several universities in the UK and North America have asked to opt-out of the AI detection feature due to the concerns listed above. You can read more about this decision from the University of British Columbia.
How to respond to AI detection in Turnitin
So what should educators do when they see student work in Turnitin flagged as written by AI? Here are a few general guidelines to consider.
- Be clear with students about your expectations for their writing and their use of tools such as ChatGPT before they submit their first assignments.
- Treat any matches with caution and use it as an opportunity to have a conversation with students, rather than as a reliable indicator of academic misconduct.
- Consider including a section in assignment rubrics for AI detection.
- Learn more about how the tool works: https://www.turnitin.com/solutions/ai-writing聽
- Stay informed about the university’s approach to AI-generated content and academic integrity. Updates to the ANU policy and procedures are coming soon!
We welcome your comments! Please feel free to use the comment feature below or contact the team for more information.