Judging Partner Identity In A Turing Test

In a Turing test, a human participant interacts with two agents through a text interface. One of the agents is human, while the other is an artificial agent. After a period of time, the participant is asked to decide which of the agents is human and which is artificial. Some AI researchers would consider a machine (or software program) to be intelligent if it is indistinguishable from a human agent.

Much work has been done to improve artificial conversational agents in order to pass the Turing test, but relatively little work has been done to investigate the factors that a participant uses to determine the identity of the agents. The current work describes two such factors.

When interacting with an artificial agent, such as a chatbot, it often becomes clear quite quickly that one is talking to a machine, as the conversation does not appear to have any context. While individual sentences may appear to be grammatically correct, they sometimes appear ad hoc and do not have any relevance to the topic at hand or may be inconsistent with earlier utterances. For example, when an agent claims to be vegetarian, later in the conversation they would be unlikely to say that their favorite food is a hamburger. In the current study, we manipulated the context of Turing test transcripts to investigate its effect on human-likeness.

But even when an artificial agent produces grammatically correct sentences, the type of grammar being used could provide clues to its identity. While both humans and animals can learn linear grammar, only humans seem to be able to learn recursive grammar. Recursive complex grammar can create multiple levels of information within another unit of information (e.g. sentences within a sentence) and requires forward and backward shifts of attention. For example, the sentence [The boy the girl kisses laughs] requires the reader to bind the verb [laughs] to its subject [the boy]. In contrast, the linear construction [The girl kisses the boy who laughs] requires no such attention shifts. In this study, we manipulated the grammatical structure of Turing test transcripts to either recursive or linear form to investigate their effects on human-likeness.

In our first experiment, we presented individual sentences to a group of 53 participants. They were asked to rate each sentence on how human-like or artificial it was, on a scale from 1 to 7. We found that recursive sentences — even though they are unique to humans — were considered less humanlike than linear sentences (see figure).

Figure courtesy Roy de Kleijn

In a second experiment, we manipulated Turing test conversations from an annual Turing test (Loebner Prize) to show either correct or incorrect use of conversational context. The participants were shown a conversation between a human and another agent and were asked to rate whether this agent was likely to be human or artificial. Surprisingly, we did not find an effect of context on ratings of humanness. That is, it did not matter if the agent used earlier information in a conversation correctly in later utterances.

In conclusion, grammatical construction of sentences provides a judge with clues of identity. When an agent uses recursive grammar, it is more likely to be judged as being artificial — even though the use of recursive grammars is uniquely human. Second, it does not matter for the rating of humanness whether or not an agent uses contextual information correctly. This was a surprising finding, but, on the other hand, we do not expect humans to be perfect stores of information — indeed, this is more of a characteristic of computers. The current study did not allow us to distinguish between storing information correctly and using it correctly. There are likely to be many other factors that a judge can use to determine the identity of a conversational partner, and we will continue the search for them.

These findings are described in the article entitled The effect of context-dependent information and sentence constructions on perceived humanness of an agent in a Turing test, recently published in the journal Knowledge-Based Systems.

About The Author

Roy de Kleijn

Roy de Kleijn received his MSc. in Cognitive Neuroscience, graduating on the thesis Computational modeling of individual differences using stochastic information accumulation models, supervised by Jay McClelland at Stanford University. While working on his MSc. in Computer Science at Georgia Tech, he received his PhD. from Leiden University under the supervision of Bernhard Hommel, for the dissertation Control of complex actions in humans and robots. He is currently an assistant professor at Leiden University, lecturing on topics such as artificial intelligence, computational modeling, artificial neural networks, and cognitive psychology.

Speak Your Mind!

READ THIS NEXT

Examining Parasitoid Effects On Aphid Populations In Québec

Insect pest populations are known to be regulated, at least in part, by naturally-occurring natural enemies such as pathogens, predators, and parasitoids — the latter living at the expense of their host, like parasites, but eventually killing it. Natural enemies have the advantage to exert a prolonged pressure on pest populations. They maintain pest densities […]

Animal Classification And Chart

Animals are lifeforms within the kingdom Animalia. From there, the classification of animals gets more specific, going through various other classes and orders. Let’s take a look at the ways animals are classified. All animals are multicellular organisms, are composed of multiple cells. These cells have various forms, shapes, and functions and they combine together […]

The Carbon Cycle Explained Using Diagrams

Carbon is quite possibly the most important element required for life. Every living creature requires carbon to function and most organic molecules consist of chains of hydrogen and carbon molecules. About 18% of your body by mass is made entirely of carbon, without which you would have no DNA, cells, or even basic glucose molecules […]

Vulnerability Of Antarctic Marine Ecosystem Engineers With Skeletal Magnesium Content In A Changing World

The oceans are being impacted by climate change from rising atmospheric CO2 levels and increasing pollution (e.g. petroleum hydrocarbons and heavy metals) from human activities, leading to increases in seawater temperature and changes in ocean chemistry (e.g. pH decrease or ocean acidification). Ocean acidification and pollution threaten marine calcifiers such as corals, bryozoans, molluscs, and […]

Better Understanding The Impact Of Deoxynivalenol On Rainbow Trout

The future growth and sustainability of the aquaculture industry depends on the ability of the sector to identify economically viable and environmentally-friendly alternatives to marine-derived ingredients. In the last few years, the industry has been concentrating its efforts on finding alternative sources of protein to substitute fishmeal in aquafeeds. Consequently, many new alternatives are available, […]

Fluoxetine, A Drug Used For Depression, May Adversely Affect Bones, While Another Drug, Escitalopram, Appears To Be Safer

Clinical studies provide evidence that treatment with drugs for depression increases the risk of fracture in humans. Some of these drugs have been previously reported to reduce bone density in experimental studies, but their effect on bone biomarkers was not clear. The group of researchers at Jamia Hamdard, New Delhi, under the Pharmaceutical Medicine program […]

An Invader In The Galapagos: When History And Genetics Merge

The study of genetics deals with a broad range of topics, from the molecules responsible for heredity to the adaptation and evolution of organisms. Within this spectrum is a niche where genetics can turn into a historical science and can be used to test the historical record of a species, thereby better understanding its story. […]