Working Title:

Proposal for interdisciplinary project on human dolphin communication through video tracking technology. prepared by Gottfried Mayer-Kress, Department of Kinesiology, Penn State Universtity, University Park, gmk@santafe.edu, http://www.personal.psu.edu/gxm21/

Humans have evolved from apes a few million years ago and started developing one of the largest brains on the planet. Other intelligent creatures, whales and dolphins with even larger brains had already been inhabiting this planet peacefully for tens of millions of years. They have been doing this without developing hands or other means to build tools for manipulating their environment. Everyone who interacts with dolphins or whales is impressed by their intelligence. It seems their intelligence is so different from our own that they might as well come from a different planet. Many attempts have been made to find means of communication between the two smartest species on land and in the ocean. There has been amazing progress but most of it consisted in dolphins learning to understand what their human trainers ask them to do. In their "correct behavior" they can demonstrate their astonishing mental capabilities. Much of this research required ongoing "face-to-face interaction between trainer and dolphin and therefore was mostly done with captive dolphins. New electronic interfaces could create new opportunities to learn more about dolphin communication and intelligence with free ranging dolphins who could select their own schedule when and how they want to interact with their trainers.

Some of the most important research on dolphin communication has been performed by Dr. Luis Herman and his collaborators (including the four dolphins Akeakamai, Phoenix, Hiapo, Elele) at the Dolphin Institute of the Kewalo Basin Marine Mammal Laboratory in Honolulu, Hawaii. They could show that dolphins are not only able to understand a language with an alphabet of sophisticated gestures but also to generalize that symbolic language to their computer graphic abstractions: Video images of a trainer or even abstract disks and bars moving on a computer screen can be correctly interpreted by the dolphins as representing the originally learned hand gestures. Dolphins are not only able to recognize themselves on a video-screen but also correctly interpret and memorize their behavior as presented on a video screen.

Other attempts of finding a common acoustic language have been of limited success. Besides certain whistles and tail slapping to generate attention for play and other interactions there exist not many examples of dolphins spontaneously and symbolically communicating with their human partners. Part of the reason might be the limited opportunities of a shared environment either in water or on land.

Over the past couple of billion years organisms of increasing complexities have evolved and eventually lead to intelligent life on earth. Many believe that next step in evolution is a global, self-organized structure, a "Global Brain" (see [Mayer-Kress & Barczys, 1995] and references therein). It is predicted to emerge as a result of enhanced communication capabilities between a large number (10¹⁰) of intelligent individuals. The Internet has been seen as a way that brains can interchange ideas and information analogous to neurons networking in the brain. So far the only intelligent species actively participating in the Internet are humans. In the section " Extensions to Non-Human Species" in [Mayer-Kress, 1996] we proposed a computer interface that would allow other intelligent species - specifically dolphins and whales - to connect to the Internet. Perhaps an interface of the type we discuss here will help to form a truly global intelligence that will help to solve planet-wide problems of the next millennium.

 The advent of virtual reality (VR) interfaces opened up completely novel channels of communication: The interactive interfaces to fast digital computers makes it possible for participants to share a completely computer-generated synthetic environment that is sometimes called "CyberSpace". There have been a number of interesting applications of VR in many different areas of science, art, music and entertainment. An excellent example is the "A-Volve" a real-time interactive environment created by Christa Sommerer and Laurent Mignonneau. In this environment human visitors are interacting with computer generated, artificial creatures that are projected into a pool of water. A video camera tracks the hands of the human participants and generates computer input that will influence the behavior of the artificial creatures thereby allowing a direct "interaction" between the human and the artificial beings. It is important to note that no physical contact is involved merely the movement of the hand being tracked by the video camera.

A commercially available product based on the same video tracking technology is the Mandala Video Gesture Control System of the Vivid Group. It is installed in a number of different virtual environments and interactive games in museums and theme parks around the world. Mostly the players are represented in the form of their video images in a computer-generated environment such as subterranean landscapes or playing fields. Certain objects on the screen (e.g. the image of a ball) are control areas: If the participant moves his or her hand in a way that its video image intersects the screen image of the ball the computer will interpret that as a collision and animate a realistic looking movement of the ball image as response. In other words one can "touch", "move", and interact with objects on the screen by just moving in front of the video-camera. This contact free and purely optical interface seems to be ideal for animals like dolphins and whales.

We propose to install a video tracking system in a pool of the Kewalo Basin Marin Mammal Laboratory (KBMML) with an ISDN connection to the Internet. Partner stations anywhere in the world can be connected to the KBMML and human or other intelligent participants can interact in a virtual environments with Akeakamai, Phoenix, Hiapo, and/or Elele at pre-arranged times. Depending on bandwidth and goals these interactions could either be in the form of natural video-images or with the help of digital "avatars" i.e. graphic objects that represent the participants and are also controlled by them.

In an interdisciplinary collaboration we design a virtual environment that can be shared by dolphins and humans. As part of an interactive, global art installation partner stations at locations around the world can connect via Internet or direct ISDN link to the KBMML. Theme and artistic content of the design will be created by artist members of the collaboration. The activity could be symbolic manipulation of objects on the screen related to the repertoire of the dolphins. For example the dolphin could "ask" the human to "perform" certain tasks similar to the ones performed by the dolphin in the pool. For instance objects could be placed into "baskets" rewards could be symbolically given etc. Gesture based commands could be simultaneously translated into synthesized voice commands so that the dolphin could actually "speak" to humans in (synthesized) human language. This interface would avoid the problem of human players having to learn to interpret the gesture-based language.

Other activities could be based on a "painting" paradigm where humans and dolphins collaborate on a digital painting by moving icons of paint brushes across the screen. Besides visual interfaces it would also be feasible to trigger acoustic activities through video tracking: digital instruments could be displayed on the screen and played collaboratively by dolphin and human players.

In recent break-through experiments the group of Prof. Niels Birbaumer at the University of Tuebingen could demonstrate that locked-in patients, who are totally paralyzed and have lost all muscle functions, can communicate with the help of a computer by using an interface that records their brainwaves. In order to better understand the fundamentals of direct symbolic and analog communication a link between the patients in Tuebingen and the dolphins in Hawaii can be established and patterns of emergent communication can be studied. It would be interesting to see if the dolphins are able to "understand" the concept of paralyzed humans who communicate with their brain waves only. If this preliminary study yields promising results then future experiments could include brainwave interfaces for the dolphins as well and perhaps help to develop new direct interfaces to computers.

• Depending on the connection speed one can either transmit a live video (ISDN video-conferencing and/or
  application sharing) or work with avatars perhaps using a digital still images.
• Besides the optical interface that controls the position of the cursor on the screen we can also use acoustic interfaces to signify the activation of specific control areas. In the context of computer interfaces the dolphins thereby had a fully functional "mouse" with optical movement control and acoustically activated "mouse buttons".
• Besides tracking the features on the body of the dolphin like the rostrum or fin it might be more reliable to train the dolphins to move objects of different color and geometrical shape. This could be easier to track and also give the dolphins the opportunity to choose among different "brushes" or other control objects.
• Because of potential problems with reflections from the water surface the video interface could be placed outside a window of the tank or the system could be used at night with an appropriate arrangement of illumination of the dolphins and a blue background.

 G. Mayer-Kress, C. Barczys, (1995) The Global Brain as an Emergent Structure from the Worldwide Computing Network, and its Implications for Modeling , The Information Society, Vol 11 No 1, 1-28

 G. Mayer-Kress, (1996) Messy Futures and Global Brains , Predictability of Complex Dynamical Systems, Yu.A. Kravtsov, J.B. Kadtke (Eds.) Springer Verlag, Berlin