A recent study conducted at New York University Abu Dhabi has found that the artificial intelligence chatbot, ChatGPT, performed as well as or even better than students in approximately 25% of university courses. However, this superiority was mostly observed in questions that required memory recall rather than critical analysis.
To assess ChatGPT’s performance, Yasir Zaki and his team reached out to colleagues in various departments at the university, requesting assessment questions from courses such as computer science, psychology, political science, and business. Along with the questions, they also obtained real student answers. Both the chatbot’s responses and the students’ answers were then evaluated by a team of graders who were unaware of the sources.
In nine out of the 32 surveyed courses, the answers provided by ChatGPT were rated as good as or better than those of the students. In some cases, the chatbot’s responses were significantly superior. For instance, in a course called “Introduction to Public Policy,” ChatGPT achieved almost double the average score of students.
However, it is important to note that ChatGPT excelled in questions that required information recall but struggled when critical analysis was necessary. This raises concerns about the design of university assessments, as they should also evaluate students’ critical thinking skills. Thomas Lancaster from Imperial College London mentions that if ChatGPT can provide better answers, it indicates flaws in assessment design.
Interestingly, Lancaster also points out that many of the assessments vulnerable to cheating using ChatGPT could have already been prone to exploitation through existing contract cheating services. These services involve students paying professional essay writers to complete their assignments, potentially resulting in a lack of critical analysis.
In addition to the performance evaluation, Zaki and his team also conducted a survey among academics and students from the UK, US, India, Japan, and Brazil to gauge their attitudes towards ChatGPT. The results revealed that students across all countries were more open to using the chatbot than the academics expected.
Overall, the study highlights the potential of ChatGPT as a valuable tool for certain university courses and raises important questions about assessment design and the role of technology in education.
