Fujitsu Develops User Interface Technology that Converts Entire Rooms into Digital Spaces

OREANDA-NEWS. July 28, 2015. Fujitsu Laboratories Ltd., Fujitsu Design Limited, and Fujitsu Social Science Laboratory Limited today announced that they will commence field trials on Fujitsu Laboratories-developed user interface (UI) technology that converts an entire room into a digital space, and on a co-creation support application developed by Fujitsu Social Science Laboratory. The trial will be conducted from August at the HAB-YU platform⁽¹⁾, a venue developed by Fujitsu Design to facilitate new value-creation initiatives.

Using the new space-digitalizing UI technology liberates people from the small screens of smart devices, tablets, and PCs, and projects a virtual window system over a large shared space, such as walls and tables, letting people display and share information with simple operations. For example, participants in a workshop can connect their smart devices to the large-format displays in the meeting space, and display the contents of their smart-device screens on walls and tables. Participants' operations conducted on the window system are transmitted to the appropriate devices to enable participants to share files between smart devices with intuitive operations on walls and tables.

Using the co-creation support application, which implements this new UI technology, materials that people bring into a space can be displayed and shared in a large format so that information can be easily organized when people are brainstorming or collaborating.

Fujitsu Laboratories, Fujitsu Design, and Fujitsu Social Science Laboratory are deploying the technology and application in workshops held at the HAB-YU platform in field trials to validate their effectiveness in supporting the creation of new ideas. The three companies seek to transform workstyles using ICT for the creation of knowledge.

Background

With the spread of smart devices and advances in cloud and communications technologies, it has become easy to take materials created at one's desk, bring them to a meeting, and show them on a projector, or even operate a large-format touchscreen to edit them.

Issues

Existing technologies that provide large-format interfaces, such as big-screen tablets and electronic whiteboards, however, lack linking features. This has made it difficult to connect multiple display units to establish a unified display system that utilizes the entire space of a room. As such, it has been problematic to apply these technologies to situations of co-creation, such as in workshops, where multiple users together take advantage of large screens to share and exchange information while generating ideas.

Overview of the Field Trial

1. Purpose

   To alleviate problems related to displaying information in large format, develop technology that dynamically connects smart devices and the display hardware installed in the space, and validate whether the technology leads to efficient sharing of ideas between people and lively discussions when used in brainstorming workshops with multiple participants.

2. Period and location

   Period: August 2015 to March 2016 (planned)
   Location: HAB-YU platform (Ark Hills South Tower, 3F, Roppongi, Minato-ku, Tokyo)

3. Description of trial

   Validate the effectiveness of support for the creation of new ideas in brainstorming workshops held in a completely digital environment, with consideration to the following:

   • Distribution of digital prompt cards to individual workspaces
   • Individual work support (creation of digital sticky notes and transfer to shared space)
   • Group work support (organization and sharing of digital sticky notes)
   • Bringing home results of workshop brainstorming
   • Storing and reproducing workshop results

About the Technology

The space-digitalizing technology automatically connects hand-held smart devices and the display hardware installed in the space, and expands the UI throughout the entire space. Multiple devices, each equipped with a projector and camera, are set, and collectively they can cover the walls and tabletops in the space as a single window system. As a result, the smartphone screen content from multiple people can be displayed within the same space.

Features of the technology are as follows:

1. Connects multiple projectors and easily links smart devices

   Merely by configuring simplified layout data to the server of the place-service platform⁽²⁾, multiple display devices in the space can be connected and used as a single large-format display. Not only does the technology enable the use of contiguous spaces, it easily displays and supports the operations of screen contents of smart devices brought into the space. Technology was also developed that converts information on walls and tabletops that is pointed at into input data for participants' smart devices.

2. Sensing technology recognizes the positions of personal smart devices and identifies IDs

   Fujitsu Laboratories developed sensing technology that uses connections between environmental sensors in the space, including cameras, and the sensors built into smart devices to simultaneously track the location of people and the IDs of the smart devices they are carrying. Connecting environmental sensors, which can accurately track location but not device IDs, and information from inertial sensors in smart devices, which can determine device IDs but not location, makes it possible to use people's movements, including walking and stopping, to sense location and ID. In turn, as each smart device and its owner are identified, it becomes possible to share the screen of a specific user's smart device on the big display.

3. Intuitive UI technology connects people's movements to the big display

   Based on the technology described in point 2 above, Fujitsu Laboratories developed an intuitive UI that links to people's movements. Using simple, pre-programmed gestures, a participant can transfer the screen contents of their smart device to the wall in front or launch the application. The Fujitsu Laboratories-developed UI utilizes handwritten text to promote the generation of ideas on large screen displays, such as operations for hand writing notes on content projected on the wall, operations for selecting handwritten notes and having them recognized as text, and operations for displaying the recognized text in the size of a sticky note.

Results

These technologies leap beyond the simple functionality of electronic whiteboards; they open up a new natural interface for using a variety of applications in a large space that responds to the ways people move, and facilitates the sharing and exchanging of information between people to accelerate co-creation. The technologies have potential applications beyond workshops and other co-creation venues. They can also be used for spatial design in showrooms, hotel entrances, and other scenarios.

In addition, because this technology uses the front-end interface and API of Fujitsu Laboratories' previously announced place-service platform, it would be simple to deploy localized services in a variety of situations, such as at offices, schools, and hospitals.

Future Plans

Fujitsu Laboratories will continue refining the technology based on the feedback from the field trials held at the HAB-YU platform, and will work with Fujitsu Social Science Laboratory to apply it to their Web Core Innovation Suite⁽³⁾, a co-creation support and educational support tool already on the market.