How Can Technology Augment Human Abilities?

TL;DR

Technology can augment people by combining human perception and judgment with machine capabilities such as computation, flight, remote vision, and precise interaction. Jun Rekimoto presents mobile augmented reality, multi-touch interfaces, human-drone integration, telepresence, and external-body viewing as examples of systems that enhance individual abilities, extend a person's presence, and support skill acquisition.

Transcript

Thank you very much. Uh, it's, it's very honored to be here, so thank you for hosting. Uh, so my name, uh, my name is Jun Rekimoto from the University of Tokyo and the Sony Computer Science Laboratories. So today's topic is human augmentation. So the where the technology-- how technology can augment humans. So actually, let me introduce myself. Thi... Read More

Key Insights

• NaviCam was created in 1994 as an early mobile augmented reality system that used a camera and LCD connected to a Silicon Graphics workstation. It recognized aspects of the surrounding world and displayed relevant information, anticipating later augmented reality applications on smartphones and tablets.
• Marker-based augmented reality works by recognizing a rectangular shape and measuring how its appearance is distorted in the camera image. From that distortion, the system can recover the camera position and accurately overlay a digital object without relying on sensors beyond a camera and computer.
• SmartSkin demonstrated multi-touch interaction around 2001 and 2002, when the mouse was still the dominant interface. Its central idea was that controlling computers with several fingers should feel natural because people routinely use multiple fingers when manipulating objects in the physical world.
• Multi-touch input allows users to control several points simultaneously, which supports tasks that are awkward with a mouse. Rekimoto illustrates this advantage with Bézier curves, which require four control points, and with pinching, a gesture that later became familiar on smartphones and tablets.
• Human-computer teams can outperform either participant working alone when their respective strengths are combined effectively. Rekimoto uses advanced or cyborg chess to argue that a human paired with a computer can exceed a human player and can also perform better than a computer-only player.
• Human augmentation includes three areas in Rekimoto's laboratory: augmenting an individual's physical abilities, augmenting existence through telepresence, and augmenting the acquisition of abilities. Together, these areas extend augmentation beyond intellect to physical action, remote presence, perception, and training.
• Human-drone integration combines complementary capabilities: people can understand situations, while drones can fly. When drone movement responds to a person's body and the drone's camera view is returned to that person, the person can feel as though they are flying and control the drone physically.
• External-body viewing can support physical training by showing a person from an outside perspective. A drone can follow someone who is jogging, while a specialized robot can observe a swimmer, providing visual information about movement or form that may otherwise be difficult for the person or coach to see.

Questions & Answers

Q: What is human augmentation in human-computer interaction?

Human augmentation is the use of technology to extend or enhance human capabilities. Rekimoto describes augmentation across three areas: improving an individual's physical abilities through wearable or processing devices, extending a person's existence through telepresence, and strengthening the acquisition of skills. His broader goal is to combine human intelligence, perception, and physical action with useful machine capabilities.

Q: How did NaviCam anticipate smartphone augmented reality?

NaviCam, created in 1994, combined a camera and LCD with a large Silicon Graphics workstation operating behind the scenes. When pointed at the real world, it recognized aspects of the surrounding situation and displayed additional information over the user's view. Rekimoto connects this basic approach to later smartphone and tablet applications that use cameras and sensors for augmented reality.

Q: How does marker-based augmented reality determine camera position?

Marker-based augmented reality uses computer vision to recognize a simple rectangular shape and analyze how that rectangle is distorted in the camera image. The observed distortion allows the system to recover the camera's position relative to the marker. It can then overlay a digital object accurately using only a camera and computer, without requiring additional sensors.

Q: Why is multi-touch interaction more natural than using a mouse?

Multi-touch interaction reflects how people normally use several fingers to manipulate physical objects. A mouse controls only one point at a time, which can make simultaneous manipulation difficult. SmartSkin allowed more than one finger to control multiple points together, supporting interactions such as pinching and the adjustment of the four control points required by a Bézier curve.

Q: Why can a human-computer team outperform either one alone?

A well-designed human-computer team can combine different strengths instead of treating human and machine intelligence only as competitors. Rekimoto points to advanced or cyborg chess, where a human working with a computer can perform better than a human alone and even better than a computer-only player. The result depends on creating an effective relationship between both participants.

Q: What are the three main forms of human augmentation?

Rekimoto's laboratory organizes human augmentation into three main forms. The first augments an individual through wearable systems or processing devices that enhance physical ability. The second augments existence through telepresence, freeing a person from one location. The third augments ability acquisition by using technological systems to improve how people learn and develop skills.

Q: How does human-drone integration make drone control easier?

Human-drone integration links a person's movement with a drone's behavior and sends the drone camera's visual information back to the person. This arrangement can make the person feel as if they are the drone and are flying. Rekimoto argues that using physical abilities in this immersive relationship can be easier than controlling the aircraft through a conventional remote commander.

Q: How can an external-body view improve sports training?

An external-body view lets a person see their own movement from outside, which can reveal physical form that is difficult to observe during activity. Rekimoto describes a drone flying around someone who is jogging and a robot providing an outside view of a swimmer. Such visual feedback can help the participant, and potentially a connected coach, examine technique during training.

Summary & Key Takeaways

• Jun Rekimoto traces his interest in human augmentation from early inspiration by science fiction, technology, gadgets, and Ivan Sutherland's head-mounted display. His 1994 NaviCam anticipated mobile augmented reality by using a camera, an LCD, and a workstation to recognize real-world situations and place relevant digital information over the user's view.

• SmartSkin explored multi-touch interaction when the mouse remained the dominant computer interface. Rekimoto argues that using several fingers is natural because people already manipulate physical objects that way. The system enabled simultaneous control of multiple points, including four control points for a Bézier curve and an early realization of the pinching gesture.

• Human augmentation can target individuals, presence, or the acquisition of abilities. Human-drone integration combines human situational understanding with a drone's capacity to fly, while immersive visual feedback makes control feel more direct. External-body viewing uses drones or swimming robots to show people their own movement and help them examine physical form during training.