A reader (Roger Taylor) recently asked if I would like to share their research paper titled "Assessment in the Time of ChatGPT." Given the current interest in ChatGPT, many readers may be interested in reading it.

Assessment in the Time of ChatGPT

Roger Taylor, Jeffrey Marzluft

Boston University, Boston, USA. Quincy College, Quincy, USA

Presentation

Poster & Pitch Presentation

Abstract

The recent public release of Open AI’s ChatGPT, with its near human-level writing ability, has been met with excitement and fear in the public. The potential impact of this new technology is also causing controversy in higher education. The responses have ranged from panic (Marche, 2022) to pragmatic (D’Agostino, 2023) to optimistic (Duckworth and Ungar, 2023).

To help the public better understand this complex issue, we conducted an experimental research study in which twenty college faculty members were the participants. The study occurred shortly after ChatGPT was released and none had used it.

Their task was to evaluate responses to an open-ended question about information literacy skills. More specifically, the question asked students to paraphrase a short passage on the sociology of gun control. The responses of 130 students were independently analyzed by two reviewers using a standardized rubric. A subset of the highest rated scores was selected. The same question was given to Open AI’s ChatGPT (version 3.5) and recorded.

Twenty faculty members were independently shown the original question and five responses, one of which was generated by ChatGPT. The faculty indicated which answer they believed was generated by the AI and their reasoning for their decision. A statistical analysis revealed that the faculty did not perform significantly better than chance.

There were two general strategies employed by the faculty members. The first strategy, which was relatively successful, was to identify lower-quality answers (e.g., poor word choice, sentence fragments) as coming from students, then they selected the “best” response as the one coming from the AI. The second strategy, which was relatively unsuccessful, was to choose the answer that seemed “artificial” and didn’t capture the “human” aspect.

Implications for assessment in higher education, both in the short-term and long-term, will be discussed.

Key References

D'Agostino, S. (2023) ‘ChatGPT Advice Academics Can Use Now’, Inside Higher Ed, 12 January.

• https://www.insidehighered.com/news/2023/01/12/academic-experts-offer-advicechatgpt (Accessed: 15 January 2023).

Deng, J., & Lin, Y. (2023) ‘The benefits and challenges of ChatGPT: an overview’, Frontiers in Computing and Intelligent Systems, 2(2), pp. 81–83.

• https://doi.org/10.54097/fcis.v2i2.4465

Duckworth, A. & Ungar, L. (2023), ‘Op-Ed: Don’t ban chatbots in classrooms — use them to change how we teach’, Los Angeles Times, 19 January.

• https://www.latimes.com/opinion/story/2023-01-19/chatgpt-ai-education-testing-teaching-changes (Accessed: 20 January 2023).

Marche, S. (2022) ‘The college essay is dead: nobody is prepared for how AI will transform academia’, The Atlantic, December.

• https://www.theatlantic.com/technology/archive/2022/12/chatgpt-ai-writing-college-studentessays/672371/ (Accessed: 09 January 2023).

OpenAI (2023). ChatGPT.

• https://openai.com/blog/chatgpt (Accessed: 06 January 2023).

Pham, S. T. H., & Sampson, P. M. (2022) ‘The development of artificial intelligence in education: A review in context’, Journal of Computer Assisted Learning, 38(5), pp.1408– 1421.

• https://doi.org/10.1111/jcal.12687