November 6, 2026 By: JK Tech
Chatbots are writing emails, settling arguments about trivia, and now and then they sound weirdly personal. Ask one to describe itself and the answer comes out smooth which humans can relate to. But is there anything steady behind it? That was the starting point for a study from Johannes Gutenberg University Mainz, published in Scientific Reports.
Three questions drove it. How do language models describe their own traits? How well do they read personality from a piece of writing? And can they tell whether a human or a machine wrote something? Results were mixed. Chatbots turned out fairly consistent. People still beat them at spotting AI text.
The chatbots’ self-image
Seven models took part: ChatGPT-4o, ChatGPT-3.5, Llama-3, Gemma 2, Gemini 2.5, Phi-3 and Orca 2. Seventy human volunteers did the same tasks for comparison. Both groups answered Big Five questionnaires (openness, conscientiousness, extraversion, agreeableness, emotional stability), then moved on to open questions about interests, hopes for the future, and what they do when stress hits.
Same question were asked again and again. Models gave about the same answer every time. And they leaned one way. Compared with the people, they rated themselves higher on conscientiousness, agreeableness and emotional stability. Calm, helpful, reliable. Sounded like a job application.
None of that means a chatbot has feelings. It just means the answers repeat enough to look like a personality from the outside.
Guessing a person from their words
Then the roles got swapped around. Everyone read short texts and rated whoever (or whatever) wrote them, on openness, conscientiousness and self-esteem.
Models picked up on some real cues. Their scores still drifted away from the human raters, sometimes quite a bit. For human-written texts, chatbots gave higher openness and self-esteem than people did. With AI-written texts it went the other direction: human raters were the generous ones, especially on emotional stability.
Reading someone off a page is shaky work for anybody. A model can notice patterns in language and still land on a different verdict than a person would.
Who catches AI writing better
People did. 83% of their guesses on human versus AI authorship were right, while chatbots got 67%. Gemma and Gemini came closest to the human average. The rest trailed.
Both sides had the same blind spot, by the way. Taking an AI text for a human one happened more often than the reverse.
What gave AI text away, according to participants? Length, mostly, plus a tone that felt too technical, too organised, too analytical. The language analysis lined up with that. AI texts packed in more time expressions and more small grammar words like articles and prepositions. Human texts had hobbies, feelings, friends, family, stuff that had actually happened to the writer.
Hiring, psychology, and the obvious problem
Some see a use for this. A model reading written answers could offer a fast first impression of someone’s personality, cheaper and quicker than a full assessment. Recruiters might try it as an extra tool at some stage.
But there’s bias to deal with. Models learn from piles of human text, and human text has plenty of slant in it. A writing style also covers only a slice of a person. For hiring or a psychological evaluation, that’s thin ground. The researchers say as much: accuracy, limits and ethics all need checking before anyone leans on AI for calls like these.
Last thought
Sounding consistent isn’t the same as understanding people. Chatbots can do the first one fine. The second needs lived experience, emotion and relationships, which texts only partly capture, and which the study gives no sign that models have.
Can AI sound human? Probably, more and more. Whether readers can still tell the difference, and care, is the better question.
