Autonomous Acquisition of Natural Situated Communication

Thórisson, Kristinn R. and Nivel, Nivel and Steunebrink, Bas R. and Helgason, Helgi P. and Pezzulo, Giovanni and Sanz Bravo, Ricardo and Schmidhuber, Jürgen and Dindo, Haris and Rodríguez Hernández, Manuel and Chella, Antonio and Jonsson, Gudberg K. and Ognibene, Dimitri and Hernández Corbato, Carlos (2014). Autonomous Acquisition of Natural Situated Communication. "IADIS International Journal on Computer Science And Information Systems", v. 9 (n. 2); pp. 115-131. ISSN 1646-3692.


Title: Autonomous Acquisition of Natural Situated Communication
  • Thórisson, Kristinn R.
  • Nivel, Nivel
  • Steunebrink, Bas R.
  • Helgason, Helgi P.
  • Pezzulo, Giovanni
  • Sanz Bravo, Ricardo
  • Schmidhuber, Jürgen
  • Dindo, Haris
  • Rodríguez Hernández, Manuel
  • Chella, Antonio
  • Jonsson, Gudberg K.
  • Ognibene, Dimitri
  • Hernández Corbato, Carlos
Item Type: Article
Título de Revista/Publicación: IADIS International Journal on Computer Science And Information Systems
Date: 2014
ISSN: 1646-3692
Volume: 9
Freetext Keywords: Knowledge acquisition, natural language, situated, communication
Faculty: E.T.S.I. Industriales (UPM)
Department: Automática, Ingeniería Eléctrica y Electrónica e Informática Industrial
Creative Commons Licenses: Recognition - No derivative works - Non commercial

Full text

PDF - Requires a PDF viewer, such as GSview, Xpdf or Adobe Acrobat Reader
Download (424kB) | Preview


An important part of human intelligence, both historically and operationally, is our ability to communicate. We learn how to communicate, and maintain our communicative skills, in a society of communicators – a highly effective way to reach and maintain proficiency in this complex skill. Principles that might allow artificial agents to learn language this way are in completely known at present – the multi-dimensional nature of socio-communicative skills are beyond every machine learning framework so far proposed. Our work begins to address the challenge of proposing a way for observation-based machine learning of natural language and communication. Our framework can learn complex communicative skills with minimal up-front knowledge. The system learns by incrementally producing predictive models of causal relationships in observed data, guided by goal-inference and reasoning using forward-inverse models. We present results from two experiments where our S1 agent learns human communication by observing two humans interacting in a realtime TV-style interview, using multimodal communicative gesture and situated language to talk about recycling of various materials and objects. S1 can learn multimodal complex language and multimodal communicative acts, a vocabulary of 100 words forming natural sentences with relatively complex sentence structure, including manual deictic reference and anaphora. S1 is seeded only with high-level information about goals of the interviewer and interviewee, and a small ontology; no grammar or other information is provided to S1 a priori. The agent learns the pragmatics, semantics, and syntax of complex utterances spoken and gestures from scratch, by observing the humans compare and contrast the cost and pollution related to recycling aluminum cans, glass bottles, newspaper, plastic, and wood. After 20 hours of observation S1 can perform an unscripted TV interview with a human, in the same style, without making mistakes.

Funding Projects

FP7231453HUMANOBS STREP–Cognitive RoboticsUnspecifiedHumanoids that Learn Socio-Communicative Skills by Observation
FP7ICT - 317662NASCENCEUnspecifiedNAnoSCale Engineering for Novel Computation using Evolution

More information

Item ID: 35971
DC Identifier:
OAI Identifier:
Official URL:
Deposited by: Memoria Investigacion
Deposited on: 24 Feb 2016 17:07
Last Modified: 24 Feb 2017 15:47
  • Logo InvestigaM (UPM)
  • Logo GEOUP4
  • Logo Open Access
  • Open Access
  • Logo Sherpa/Romeo
    Check whether the anglo-saxon journal in which you have published an article allows you to also publish it under open access.
  • Logo Dulcinea
    Check whether the spanish journal in which you have published an article allows you to also publish it under open access.
  • Logo de Recolecta
  • Logo del Observatorio I+D+i UPM
  • Logo de OpenCourseWare UPM