Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers

Abstract generation

We evaluated the abstracts generated by ChatGPT (Version Dec 15) for 50 scientific medical papers. We gathered titles and original abstracts from current and recent issues (published in late November and December of 2022) of five high-impact journals (Nature Medicine, JAMA, NEJM, BMJ, Lancet) and compared them with the original abstracts. The prompt fed to the model was ‘Please write a scientific abstract for the article Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers in the style of [journal] at https://www.nature.com/articles/s41746-023-00819-6’. Note that the link is superfluous because ChatGPT cannot browse the internet. ChatGPT’s knowledge…