CN103197757A

Movatterモバイル変換

Info

Publication number: CN103197757A
Application number: CN2012100063389A
Authority: CN
Inventors: 魏斌
Original assignee: Water Power (beijing) Network Technology Co Ltd
Current assignee: Water Power (beijing) Network Technology Co Ltd
Priority date: 2012-01-09
Filing date: 2012-01-09
Publication date: 2013-07-10

Abstract

The invention provides an immersion type virtual reality system. The system comprises at least one display, at least one motion capture device and at least one data processing device, wherein the display can be worn on the body of a user, and is used for capturing the head turning motion of the user, sending the head turning motion data to the as least one data processing device, receiving data sent by at least one data processing device, and transmitting a video or audio of a virtual environment to the user; the motion capture device captures motion trails of main node parts of the user, and sends the motion trail data to the at least one data processing device; and the data processing device is used for processing data acquired from the at least one display and the at least one motion capture device and transmitted to the data processing device, and sending the processed result to the at least one display. The invention further provides an implementation method of the immersion type virtual reality system.

Description

Immersive virtual reality system and implementation method thereof

Technical Field

The present invention relates to a virtual reality system and method, and more particularly, to an immersive virtual reality system and method.

Background

The Virtual Reality (VR) technology is an advanced and digital man-machine interface technology, and is characterized in that a computer generates an artificial Virtual environment, an artificial environment which is mainly based on visual perception and comprises comprehensive perception of auditory sense and touch sense is generated, people can perceive a Virtual world simulated by the computer through various sensory channels such as visual sense, auditory sense, touch sense, acceleration sense and the like, and can interact with the Virtual world through the most natural modes such as movement, voice, expression, gestures, sight line and the like, so that the experience of being personally on the scene is generated. The virtual reality technology is the comprehensive development of a plurality of technologies such as a computer technology, a sensor technology, a man-machine interaction technology, an artificial intelligence technology and the like, is applied to various aspects such as military affairs, medicine, education, entertainment, manufacturing industry, engineering training and the like at present, and is considered to be one of important technologies influencing the life of people at present and in the future.

Virtual reality systems are classified into immersive systems and non-immersive systems according to the degree of immersion. The immersive virtual reality system is realized by special external devices, a high-performance computer and corresponding software, and the immersive virtual reality system enables people to be completely immersed in a graphic world created by the computer to feel like a real world. Immersive virtual reality systems are advanced virtual reality systems that provide a fully immersive experience that gives the user the feeling of being within a virtual boundary. It uses a head-mounted display or other device to enclose the participant's visual, auditory and other senses, and provides a new, virtual sensory space, and uses position trackers, data gloves, other hand-controlled input devices, sounds, etc. to create the participant with a sense of being personally on, mentally engaged, and immersed.

The immersive virtual reality system comprises a CAVE (CAVE) immersive virtual reality system, a large-screen three-dimensional immersive virtual reality system, a cylindrical annular screen projection immersive virtual reality system and the like, the immersive virtual reality system takes large-format virtual reality stereoscopic projection even with an ultra-large format as a display mode, an input type virtual space environment with high telepresence is provided for participants, and the virtual three-dimensional world to be interacted appears in front of the participants in a highly vivid manner. However, these immersive virtual systems require a large display device to be installed in a fixed place for use, limiting the range of use.

The advent of wearable displays has solved the problem of the bulkiness of display devices, which are small in size, portable and transportable, and most likely become widely popular. However, the tracking of human body parts and the delay in data transmission are technical challenges faced by immersive virtual reality systems that employ wearable displays.

Space tracking systems (Spatial tracking systems) such as CN97198214.7 provide human body part tracking solutions, such as 3d fluid motion capture products and MVN inertial motion capture products by Xsens Technologies, which employ inertial sensors to measure the motion of major skeletal parts of the human body in real time. The advantages of the devices are that the users can be immersed into the virtual world, and the disadvantages are that the system devices are complex, expensive and difficult to popularize.

Disclosure of Invention

As a result of extensive and intensive studies by the present inventors, the present inventors have proposed a technical solution according to the present invention, so that the above-mentioned problems can be solved.

The invention provides an immersive virtual reality system, which at least comprises:

at least one display wearable on a user's body, at least one motion capture device, and at least one data processing device;

wherein,

the display is used for displaying images and sound to a user, capturing head rotation motion of the user, sending head rotation motion data to at least one data processing device, receiving data sent by the at least one data processing device, and transmitting video and/or audio of a virtual environment to the user;

the motion capture device captures the motion trail of the main node part of the user and sends the motion trail data to at least one data processing device;

the data processing device is used for processing data acquired from at least one display and at least one motion capture device and transmitted to the data processing device, and sending a processed result to at least one display.

According to the immersive virtual reality system of the invention, preferably the display comprises at least one angle sensing device. The angle sensing means is capable of at least capturing rotational motion about the z-axis, preferably the angle sensing means is an electronic compass. It is also preferred that the angle sensing means is capable of sensing rotational motion about three axes, x, y and z, and more preferably that the angle sensing means is a three axis gyroscope.

According to the immersive virtual reality system of the invention, preferably the motion capture device comprises at least one infrared emitter and at least one infrared camera. More preferably, the motion capture device is a motion sensing camera, and most preferably the motion capture device is a Kinect motion sensing camera and/or a WAVI rotation motion sensing camera.

According to the immersive virtual reality system of the present invention, it is preferable that the data processing apparatus is a computer, such as a PC, a notebook computer; personal Digital Assistants (PDAs), such as the Dell Axim X51V palm-top computer manufactured by Dell corporation; tablet computers, such as iPad and iTouch from apple, TouchPad from Hewlett packard; mobile phones, such as iPhone manufactured by apple, Nokia N85 manufactured by Nokia; audio processing terminals, such as MP3 and iPod; video processing terminals, such as MP 4; and/or gaming machines such as PS2, PS3, PSP, NGC, NDSL, XBOX, Wii, GBA, 3DS, and NDS.

According to the immersive virtual reality system, at least one display (COM1), at least one motion capture device (COM2) and at least one data processing device (COM3) form a VRS. The virtual reality system according to the present invention may also be made up of a plurality of VRSs.

The immersive virtual reality system according to the present invention further comprises at least one data exchange means for handling data transfer between the plurality of VRSs. Preferably, the data exchange device is a computer, a personal digital assistant, a tablet computer, a mobile phone, an audio processing terminal, a video processing terminal and/or a game console.

According to the immersive virtual reality system of the present invention, the transmission speed between the display and the motion capture device, the display and the data processing device, and/or the motion capture device and the data processing device is preferably not less than 18432000bps, more preferably not less than 55296000bps, more preferably not less than 55296000bps, and still more preferably not less than 147456000 bps. Preferably, the data transmission device adopts a WHDI transmission mode.

The invention also provides a method for realizing immersive virtual reality, which at least comprises the following parts:

capturing the motion trail of the main node part of the user through at least one motion capture device, and sending the motion trail data to at least one data processing device,

capturing head turning movements of a user via at least one display and sending head turning movement data to at least one of the data processing devices,

processing data from the motion capture device and the displays by at least one of the data processing devices and sending the processed data to at least one of the displays;

the processed data received from at least one data processing device is converted into video and/or audio through at least one display, so that a user can feel the virtual reality environment changing along with the movement of the user.

According to the method for realizing immersive virtual reality, the displayer comprises at least one angle sensing device. The angle sensing means is capable of at least capturing rotational motion about the z-axis, preferably the angle sensing means is an electronic compass. It is also preferred that the angle sensing means is capable of sensing rotational motion about three axes, x, y and z, and more preferably that the angle sensing means is a three axis gyroscope.

According to the method for realizing immersive virtual reality of the present invention, it is preferable that the motion capture device includes at least one infrared emitter and at least one infrared camera. More preferably, the motion capture device is a motion sensing camera, most preferably the motion capture device is a Kinect camera and/or a WAVI rotation camera.

According to the method for realizing immersive virtual reality, the data processing device is preferably a computer, such as a PC (personal computer) or a notebook computer; personal Digital Assistants (PDAs), such as Dell Axim X51V palm computers manufactured by Dell; tablet computers, such as iPad and iTouch from apple, TouchPad from Hewlett packard; mobile phones, such as iPhone manufactured by apple, Nokia N85 manufactured by Nokia; audio processing terminals, such as MP3 and iPod; video processing terminals, such as MP 4; and/or gaming machines such as PS2, PS3, PSP, NGC, NDSL, XBOX, Wii, GBA, 3DS, and NDS.

According to the method for realizing the immersive virtual reality, at least one display (COM1), at least one motion capture device (COM2) and at least one data processing device (COM3) form a VRS. The virtual reality system according to the present invention may also be made up of a plurality of VRSs.

The method for realizing the immersive virtual reality further comprises the step of processing data transmission among the plurality of VRSs through at least one data exchange device. Preferably, the data exchange device is a computer, a personal digital assistant, a tablet computer, a mobile phone, an audio processing terminal, a video processing terminal and/or a game console.

According to the method for realizing immersive virtual reality, the transmission speed between the display and the motion capture device, between the display and the data processing device and/or between the motion capture device and the data processing device is preferably not less than 18432000bps, more preferably not less than 55296000bps, more preferably not less than 36864000bps, and still more preferably not less than 147456000 bps. Preferably, the data transmission device adopts a WHDI transmission mode. .

Drawings

FIG. 1 is a block diagram of a display according to the present invention;

FIGS. 2a-2d are schematic diagrams of the x, y, and z axes according to the present invention;

FIG. 3 is a schematic diagram of the operation of a display according to the present invention;

FIG. 4 is a block diagram of a motion capture device according to the present invention;

FIG. 5 is a schematic diagram of the main nodal locations of a human body that can be captured by a motion capture device according to the present invention;

FIG. 6 is a block diagram of a data processing apparatus according to the present invention;

FIG. 7 is a schematic diagram of the operation of a data processing apparatus according to the present invention;

FIG. 8 is a schematic diagram of the operation of a data switching apparatus according to the present invention;

fig. 9 is a topological structure diagram of an immersive virtual reality system according to embodiment 1 of the present invention;

fig. 10 is a topological structure diagram of an immersive virtual reality system according to embodiment 2 of the present invention;

fig. 11 is a topological structure diagram of an immersive virtual reality system according to embodiment 3 of the present invention;

fig. 12 is a topological structure diagram of an immersive virtual reality system according toembodiment 4 of the present invention;

fig. 13 is a topological structure diagram of an immersive virtual reality system according to embodiment 5 of the present invention;

fig. 14 is a topological structure diagram of the immersive virtual reality system according to embodiment 6 of the present invention.

Detailed Description

The invention will be explained in more detail below with reference to the drawings. The examples presented in this specification are intended only to explain, illustrate and describe the invention and are intended to have an inclusive, but non-limiting, meaning and are not intended to limit the content of the invention in any way.

The display (COM1) is used for transmitting the video and/or audio of the virtual environment to the user so as to enable the user to generate the experience of being personally on the scene; and comprises at least one angle sensing means for capturing head turning movements including head rotation movements and small-amplitude deflection movements. The display (COM1) may also be used to enable voice communication between a user and other users.

Referring to fig. 1, the display (COM1) has a display device (101), a microphone (102), an earphone (103), an angle sensing device (104), a signal processing device (106), and a data transmission device (105).

The display device (101) is a part for displaying images, and in order to facilitate users to feel more realistic visual effects, the display device (101) is usually positioned near the eyes of human beings, and the display device can be any display device such as a CRT (cathode ray tube), an LCD (liquid crystal display) and the like. A microphone (102) is a device for capturing sound and converting the sound signal into an analog signal, and is generally placed near the mouth of a person. The earphone (103) is a device that converts an analog signal into a sound signal and broadcasts the sound, and is generally placed near the ear of a person.

The angle sensing device (104) is capable of sensing rotational motion about a z-axis, and more preferably, the angle sensing device (104) is capable of sensing rotational motion about three axes, x, y and z. As shown in fig. 2, the rotation motion around the z-axis refers to a rotation motion of the head of the user about the vertical direction, that is, a motion of generating a change in the angle of rotation (yaw) (α), such as a head side-to-side rotation motion in a user's movement or a substantially stationary state, and also includes a head side-to-side rotation motion in a case where the head and the body of the user have little or no displacement in a standing or sitting state. The rotational motions about the three axes x, y and z refer to motions of axial movement of the head of the user in the vertical direction, i.e. generating a rotation (yaw) angle (α), a yaw (pitch) angle (β) and a pitch (roll) angle (γ), in addition to the rotational motions about the z-axis, such as any head rotation motions of the user, including head rotation motions, nodding motions, reclining motions, head swinging motions, etc., and of course, including head rotation motions, nodding motions, reclining motions, head swinging motions in the case where the head and the body of the user have little or no displacement in the standing or sitting state. In the former case, the angle sensing means (104) may be an electronic compass; in the latter case, the angle sensing device (104) may be a three-axis gyroscope. The angle sensing device (104) enables the rotation action and the small-amplitude deflection action of the head to be recognized by the immersive virtual reality system, overcomes the defect that the rotation action and the small-amplitude deflection action of the head are difficult to recognize by the action capturing device (COM2), and can assist the data processing device (COM3) to render corresponding video graphics, so that the immersive virtual reality system can bring more realistic simulation reality effect for users.

The motion capture device (COM2) is used for capturing the motion track of the main node part of a user and at least comprises at least one infrared emitter (201) and at least one infrared camera (202). As shown in fig. 4, the motion capture device (COM2) may further include a controller (203), an output interface (205), and a focus tracking motor (204). In use, the motion capture device (COM2) faces the user such that the user's range of motion falls within the measurement volume of the motion capture device (COM 2). The motion capture device (COM2) may also include a color camera for capturing color images.

The infrared emitter (201) is used for uniformly projecting infrared light in a measuring space of the motion capture device (COM2), wherein the measuring space is a range covered by the infrared emitter (201). An infrared camera (202) (also known as a monochrome CMOS sensor or an infrared CMOS sensor) is used to record speckle data in the measurement space and transmit the resulting data to a controller (203). The controller (202) is used for processing data from the infrared camera (202), completing motion capture on main node parts of a human body, and transmitting the processed data to other connected devices (such as a data processing device (COM3)) through an output interface (205). The focus tracking motor (204) is used for adjusting the up-down and left-right angles of the camera equipment so as to ensure that the measurement space can better cover the human body object.

When uniform infrared light continuously emitted to the space by the infrared emitter (201) irradiates on a rough object (such as a human body surface or a clothes surface), random reflection spots, namely speckles, are formed. Speckle is highly random and changes pattern with distance, and speckle at any two places in space is different pattern, which is equivalent to marking the whole space, so that the position of any object can be recorded exactly when the object enters the space and moves. The infrared camera (202) can record speckles appearing in the space, so that speckle information can be acquired and transmitted to the controller (203).

The controller (203) forms image data having a 3D depth from the data from the infrared camera (202), and performs processing such as human structure recognition, human skeleton model creation, and the like. The human body structure recognition adopts machine learning technology (machine learning) to establish a large amount of image databases. Each pixel of the 3D depth body image is transmitted into an image library to identify the possibility of which body part the pixel belongs to, respectively, to determine the body part. In order to reduce the amount of calculation, a segmentation mask technology may be adopted, that is, background objects are removed, only the part of the segmentation mask is transmitted in the processing flow, and meanwhile, the identity of each person in the measurement space can be identified by combining basic facial data and skeleton data of the human body. And the skeleton model processing is to determine movable joint points by evaluating the output pixels, then generate a skeleton system according to the movable joint points and accurately evaluate the actual position of the human body. Meanwhile, a plurality of common human body postures are prestored in the controller (203), so that the actions of the player can be guessed and matched when the received image data acquisition information is not complete.

As shown in fig. 5, the motion trajectory captured by the motion capture device (COM2) generally includes a motion trajectory of a main node portion of a human body, and the motion trajectory of the main node portion of the human body generally refers to a displacement trajectory of the main node portion of the human body in space, where the main node portion includes at least a head, a hand, an elbow, a shoulder, a waist, a knee and/or a foot, and may further include a hip, a wrist and/or an ankle. The motion capture device (COM2) may be a motion sensing camera, which is commercially available, such as the Kinect motion sensing camera from microsoft corporation, or the WAVI x motion sensing camera from washingo corporation.

However, although the motion capture device (COM2) can capture the motion trace of the user's main node part, it is difficult or impossible to recognize the head turning motion. The head turning motion comprises a head rotating motion and/or a small-amplitude deflecting motion; the head rotation motion means a rotation motion about the cervical spine with little or no displacement of the head relative to the body, such as a head twisting motion, and of course, includes a head twisting motion and a head shaking motion in the case where the head and the body have little or no displacement in the standing or sitting state of the user; the small-amplitude deflection motion of the head means a swinging motion in which the head is displaced with little respect to the body, such as a small-amplitude head swinging motion, and of course, includes a nodding motion, a head tilting motion, or a head swinging motion in which the head and the body are hardly displaced or are displaced with little displacement in the standing or sitting state of the user. However, these head turning movements can cause the user's viewing angle to change greatly, and therefore the virtual reality system must be able to recognize these movements and cause the virtual environment simulated by the system to react and change to the movements. As a result of extensive and intensive studies by the present inventors, according to the present invention, an angle sensing device (104) is mounted on a display (COM1), the angle sensing device (104) can sense at least a rotational motion around a z-axis, when a user's head performs a rotational motion, such as twisting or shaking, the angle sensing device (104) can sense the user's motion and transmit related data, and a data processing device (COM3) receives and processes the related data, and transmits the processed data back to the display (COM1), so that the user can sense (e.g., see and/or hear) a virtual environment change generated as the twisting motion proceeds from the display (COM1), and the user can have a better experience of being personally on the scene. Preferably, the angle sensing means (104) is capable of sensing rotational movement about three axes x, y and z, i.e. any head turning movement of the user. When the head of the user performs a rotating action and/or a small-amplitude deflection action, for example, even if the head is twisted and/or the head is swung and nod with small amplitude, the angle sensing device (104) can sense the action of the user and send related data, the data processing device (COM3) receives and processes the related data, and sends the processed data back to the display (COM1), so that the user can sense (for example, see and/or hear) the change of the virtual environment generated along with the action from the display (COM1), and the user can have better experience of being personally on the scene.

The data processing device (COM3) is used for processing data acquired from the display (COM1) and the motion capture device (COM2) and sending the processed result to the display (COM 1). As shown in fig. 6, the data processing apparatus (COM3) generally comprises a controller (301), an operator (302), a memory (303), an input device (304), and an output device (305).

The controller (301) is used for controlling program execution, works according to an instruction sequence (program) in the memory, and controls and executes instructions, and all parts of the machine are controlled to operate coordinately according to the instructions. The arithmetic unit (302) is used for performing data processing, such as various arithmetic and logical operations. The memory (303) is used for memorizing programs and data, and storing the programs and data in a binary code form. The input device (304) is used for inputting data sent by the external device into the arithmetic unit (302). The output device (305) is used for outputting the data processed by the arithmetic unit (302) to the external device. Such as a display (COM1), a motion capture device (COM2), a data exchange device (COM4), a mouse, a keyboard and/or a touch screen (e.g. a capacitive screen or a resistive screen).

The input device (304) transmits a desired program and data to the arithmetic unit (302), and the program and data are stored in the memory (303) through the arithmetic unit (302). During the operation process, data is read into the arithmetic unit (302) from the memory (303) for operation, and the operation result is stored in the memory (303) or is output by the arithmetic unit (302) through the output device (305). The instruction is also stored in the memory (303) in a data form, the instruction is sent to the controller (301) from the memory (303) during operation, and the controller (301) generates a control flow to control the flow direction of the data flow and control the work of each part so as to process the data flow.

The data processing device (COM3) may be, for example, a computer, such as a PC, a notebook computer; personal Digital Assistants (PDAs), such as the Dell AximX51V palm-size computer manufactured by Dell corporation; tablet computers, such as iPad and iTouch from apple, TouchPad from Hewlett packard; mobile phones, such as iPhone manufactured by apple, Nokia N85 audio processing terminals manufactured by Nokia, such as MP3 and iPod; video processing terminals, such asMP 4; and/or gaming machines such as PS2, PS3, PSP, NGC, NDSL, XBOX, Wii, GBA, 3DS, and NDS.

The working principle of the data processing device (COM3) is shown in fig. 7. The data processing device (COM3) is responsible for performing data processing (701) on the received data. The received data is local user input data (702) and/or remote user input data (703). The local user input data (702) refers to data acquired from a local user, and includes, for example, audio data captured by a microphone (102) of the local display (COM1), head rotation motion data acquired by the angle sensing device (104) of the local display (COM1), local user body motion trajectory data acquired by the local motion capture device (COM2), and the like. The remote user input data (703) refers to data acquired from a remote user and transmitted to the local, and includes, for example, audio data captured by a microphone (102 ') of a remote display (COM1 '), head turning motion data acquired by an angle sensing device (104 ') of the remote display (COM1 '), remote user body motion trajectory data acquired by a remote motion capture device (COM2 '), and the like.

The data processing (701) generally comprises a data acquisition module (7011), an input data sorting module (7012), a data processing module (7014), a 3D rendering module (7016), an audio rendering module (7015), and an output data sorting module (7017), and further comprises a local data output module (7018) and/or a remote data output module (7013).

The data acquisition module (7011) acquires local user input data (702) and/or remote user input data (703) and transmits the data to the input data sorting module (7012). The input data sorting module (7012) sorts and sorts the received data into data formats usable by the data processing module, such as image data BITMAP, audio data MP3, video data MPEG4, and the like, and transmits the sorted data to the data processing module (7014) and/or the remote data output module (7013). If a remote user is present, the remote data output module (7013) transmits all local user input data to the data processing device (COM 3') of the immersive virtual reality system in which the respective remote user is present. The remote user output data (705) is data transmitted by the local virtual reality system to other virtual reality system data processing devices (COM 3', namely data processing devices of the immersive virtual reality system where the remote user is located), and comprises audio data captured by a microphone (102) of a display (COM1) of the local virtual reality system, head turning motion data acquired by an angle sensing device (104) of the display (COM1), local user human body motion track data acquired by a local motion acquisition device (COM2) and the like. A data processing module (7014) processes the received collated data, arranges the virtual three-dimensional environment, places the remote user and the local user at respective locations in the virtual reality environment, the method comprises the steps of setting the body postures and the head angles of a remote user and a local user, and setting the position of a 3D camera (the 3D camera is a composition concept in a virtual three-dimensional space constructed by computer system operation and is used for defining a virtual camera concept of information such as a specified position direction, a visual field and the like so as to render a picture in a three-dimensional space according to the virtual camera, wherein the picture in the 3D scene observed by the user is a picture shot by the 3D camera in the 3D scene, and the 3D camera is not responsible for rendering the picture and only determines the content to be shot) on the eye of the local user. The data processing module (7014) transmits the processed data to the 3D rendering module (7016) and/or the audio rendering module (7015). An audio rendering module (7015) renders the environmental audio heard by each local user and the audio emitted by other users according to the position of the user in the virtual reality environment; preferably, the audio volume is adjusted according to the distance between the sound source and the user in the virtual reality environment, for example, the audio volume is larger when the distance is closer, and the audio volume is smaller when the distance is farther. And the 3D rendering module (7016) respectively renders video pictures seen by each local user according to the set cameras. And the 3D rendering module (7016) and the audio rendering module (7015) send the rendered data to the output data sorting module (7017). The output data sorting module (7017) sorts and sorts the rendered pictures and audio data to be sent to a display (COM1) worn by a local user; in the case of a plurality of local users, the output data sorting module (7017) sorts the rendered screens and audio data for transmission to the display (COM1) worn by each corresponding local user. The output data sorting module (7017) sends the sorted data to the local data output module (7018). The local data output module (7018) transmits the collated data, i.e., the local user output data (704), to the corresponding local user worn display (COM 1). The local user output data (704) is data processed by the data processor (701) for transmission to the local user, including processed audio data captured by the remote user microphone (102'), other audio data in the virtual environment, video data in the virtual environment, etc.

At least one display (COM1), at least one motion capture device (COM2) and at least one data processing device (COM3) form a VRS. The virtual reality system according to the invention with one display (COM1), one motion capture device (COM2) and one data processing device (COM3) is called VRS-SM, and the virtual reality system according to the invention with more than two displays (COM1), one motion capture device (COM2) and one data processing device (COM3) is called VRS-MM. The virtual reality system according to the present invention may also be made up of a plurality of VRSs.

As shown in fig. 8, the data exchange device (COM4) generally includes a controller (401), an operator (402), a memory (403), an input device (404), and an output device (405).

The controller (401) is used for controlling program execution, works according to an instruction sequence (program) in the memory, and controls and executes instructions, and all parts of the machine are controlled to operate coordinately according to the instructions. The arithmetic unit (402) is used for performing data processing, such as various arithmetic and logical operations. The memory (403) is used for memorizing programs and data, and storing the programs and data in the form of binary codes, for example. The input device (404) is used for inputting data sent by the external device into the arithmetic unit (402). The output device (405) is used for outputting the data processed by the arithmetic unit (402) to the external device. Such as a data processing device (COM3), a mouse, a keyboard, and/or a touch-sensitive display screen (e.g., a capacitive or resistive screen).

Theinput device 404 transmits a necessary program and data to thearithmetic unit 402, and the program and data are stored in the memory 403 through thearithmetic unit 402. During the operation process, data is read from the memory (403) into the arithmetic unit (402) for operation, and the operation result is stored in the memory (403) or is output by the arithmetic unit (402) through the output device (405). The instruction is also stored in the memory (403) in a data form, the instruction is sent to the controller (401) from the memory (403) during operation, and the controller (401) generates a control flow to control the flow direction of the data flow and control the work of each part so as to process the data flow.

The data exchange device (COM4) may be, for example, a computer, such as a PC, a laptop; personal Digital Assistants (PDAs), such as the Dell AximX51V palm-size computer manufactured by Dell corporation; tablet computers, such as iPad and iTouch from apple, TouchPad from Hewlett packard; mobile phones, such as iPhone manufactured by apple, Nokia N85 manufactured by Nokia; audio processing terminals, such as MP3 and iPod; video processing terminals, such asMP 4; and/or gaming machines such as PS2, PS3, PSP, NGC, NDSL, XBOX, Wii, GBA, 3DS, and NDS.

In a virtual reality system according to the invention, the data processing device (COM3) and the data exchange device (COM4) may be the same device. Preferably, in a virtual reality system according to the invention, the data processing device (COM3) and the data exchange device (COM4) are not one and the same device.

According to the virtual reality system, data transmission among the display (COM1), the motion capture device (COM2), the data processing device (COM3) and the data exchange device (COM4) is carried out through data transmission devices (such as a data transmission device (105), an output interface (205), an input device (304), an output device (305), an input device (404) and/or an output device (405)), and the data transmission mode adopted by the data transmission devices can be wired transmission or wireless transmission. However, when the wired transmission method is adopted, the connection line on the data transmission device may interfere with the activities of the user and affect the experience of the user, so according to the present invention, the data transmission device preferably adopts a wireless transmission method, such as bluetooth, infrared, Wifi, WHDI, etc. When wireless transmission is adopted, the most important problem is that the transmission speed is such that the system captures the movement of the user in three-dimensional space, and then transmits the visual and/or auditory feelings that the user should experience in the virtual reality environment to the user, and if the user experiences the experience several seconds ago due to the delay of the transmission speed, the function of the virtual reality system is greatly impaired. The delay perceived by the user is mainly due to the transmission delay of the local data transmission, i.e. the delay caused by the transmission between the devices (between the display and the motion capture device, between the display and the data processing device and/or between the motion capture device and the data processing device) within each VRS of the virtual reality system. The transmission delay is divided into an input delay (for example, delay caused by information transmission from the display (COM1) and/or the motion capture device (COM2) to the data processing device (COM3)) and an output delay (for example, delay caused by information transmission from the data processing device (COM3) to the motion capture device (COM 1)). The input delay is relatively small because the input content is mainly data and voice data generated by human body action, and the requirement on the bandwidth of a transmission medium is relatively low. The output delay is relatively large due to the continuous graphics sequence and audio being transmitted, and the bandwidth of the transmission medium is relatively high. Thus, according to the present invention, it is also preferable that the transmission speed of the data transmission apparatus is not less than 18432000bps (bit per second), more preferably not less than 55296000bps, still more preferably not less than 55296000bps, and still more preferably not less than 147456000 bps. More preferably, the data transmission device adopts a WHDI (Wireless Home Digital Interface) transmission method. The WHDI can realize the data transmission speed of 3Gbps, the working range can reach 30 meters, the WHDI can penetrate through the wall, and the data transmission delay can be less than 1 millisecond. Data transmission devices using this technology are commercially available, such as MIPS-Based chips or AMN2120/AMN2220 chips from ammon corporation.

The immersive virtual reality system according to the invention can be used in a variety of aspects such as gaming, scientific research, multimedia, entertainment, simulation, teleconferencing, and the like.

The present invention will now be described in more detail with reference to examples, which are given by way of illustration only and are not meant to limit the invention in any way, according to the present invention.

Examples

Example 1

A according toThe immersive virtual reality system (VRS1, as shown in figure 9) of the invention comprises a display (COM1), a motion capture device (COM2) and a data processing device (COM 3). The display (COM1) is a binocular head mounted video display with a display device (Myvu Corp.)

A transmissive micro liquid crystal display screen), a microphone (iron triangle PRO 51Q), an earphone (sonhai seoul PX80), an angle sensing device (electronic compass, japan asahi corporation AK8973 digital compass chip), a signal processing device and a data transmission device (STLC 4420 WiFi chip manufactured by jew semiconductor), wherein the signal processing device comprises an image processing device (Microdisplay Controller (Custom) KCD-a210-BA manufactured by Kopin, integrated in Myvu corporation

On a transmissive micro-lcd), a sound processing device (UDA 1341 decoder chip based on I2S bus), a control device (Samsung S3C2440 chip with I2C bus interface) and a memory device (two parallel connected SDRAM chips of K4S511632B-CL75, 512M bytes). The power supply part of the display (COM1) adopts an MCP1700 low-dropout voltage regulator chip to supply power. According to the electrical characteristics of the chips, the circuit diagram of the display (COM1) of the embodiment is designed, the design and the manufacture of a Printed Circuit Board (PCB) are completed according to the circuit diagram, and the display (COM1) is assembled.

The motion capture device (COM2) is a motion sensing camera (Kinet camera of Microsoft corporation), and the data processing device (COM3) is a computer (a imagination E2565 desktop computer externally connected with a TL-WN321G wireless network card produced by TP-LINK corporation).

After the display (COM1) is powered on, the Samsung S3C2440 chip, the UDA1341 decoder chip, the AK8973 digital compass chip and the STLC4420 chip are initialized, and then a main program loop is executed in the Samsung S3C2440 chip of the control device. In the main program loop, the control device obtains data from an angle sensor and a microphone, and data from a data processing device (COM3) through a data transmission device and stores the data in a storage device, and sends the data from the angle sensor and the microphone to the data processing device (COM3) through the data transmission device, and outputs audio and video data to be output to a display device and an earphone.

The motion capture device (COM2) captures the motion of the user, generates depth map data, converts the depth map data into bone motion data, and transmits the bone motion data to the data processing device (COM 3). The data processing device (COM3) processes data from the display (COM1) and the motion capture device (COM2) according to a simulation program written in the data processing device in advance for realizing virtual reality, obtains virtual scene information, comprises information such as the position, the limb and head motion, the sound and the like of a user in a scene, converts the information into audio and video information and sends the audio and video information to the display (COM 1). The display (COM1) receives the audio and video information generated by the data processing device (COM3), stores the data in the storage device, outputs the audio data to be output to the earphone, and outputs the video data to be output to the display device.

The display (COM1) and the data processing device (COM3) adopt WiFi to realize data transmission, and the motion capture device (COM2) and the data processing device (COM3) are connected in a wired mode according to the specification.

The results of the runs indicate that the immersive virtual reality system (VRS1) according to the present invention is able to execute the virtual reality simulation program normally without significant delay.

Example 2

An immersive virtual reality system (VRS2, fig. 10) according to the present invention comprises three displays (COM1), a motion capture device (COM2), and a data processing device (COM 3). The display (COM1) was the same as the display (COM1) in example 1, except that the angle sensing device used a three-axis gyroscope (hornewell HMC5883L three-axis magnetoresistive sensor) and the signal transmission device used a CAT6023 chip.

The motion capture device (COM2) is a motion sensing camera (Kinet camera available from microsoft corporation), and the data processing device (COM3) is a game machine (sony PlayStation 2 externally connected to SEM-ST1O11 model WHDI STICK available from SEAMON corporation).

After the display (COM1) is powered on, the Samsung S3C2440 chip, the UDA1341 decoder chip, the HMC58 5883L three-axis magnetoresistive sensor, and the CAT6023 chip are initialized, and then a main program loop is executed in the Samsung S3C2440 chip of the control device. In the main program loop, each control device obtains data from the angle sensor and the microphone, and data from the data processing device (COM3) through the data transmission device and stores the data in the storage device, and sends the data from the angle sensor and the microphone to the data processing device (COM3) through the data transmission device, and outputs the video and audio data to be output to the display device and the earphone.

The motion capture device (COM2) captures the motion of the user, generates depth map data, converts the depth map data into bone motion data, and transmits the bone motion data to the data processing device (COM 3). The data processing device (COM3) processes data from the display (COM1) and the motion capture device (COM2) according to a simulation program written in the data processing device in advance for realizing virtual reality, processes the interaction relation of all users, obtains virtual scene information, comprises information such as the position, the limb and head motion, the sound and the interactive scene information of each user in a scene, converts the information into audio and video information and sends the audio and video information to the corresponding display (COM 1). The display (COM1) receives the audio and video information generated by the data processing device (COM3), stores the data in respective storage devices, outputs the audio data required to be output to respective earphones, and outputs the video data required to be output to respective display devices.

Data transmission is realized between the display (COM1) and the data processing device (COM3) by adopting WHDI, and the motion capture device (COM2) and the data processing device (COM3) are connected in a wired mode according to the specification.

The results of the runs indicate that the immersive virtual reality system (VRS2) according to the present invention is able to execute the virtual reality simulation program normally without significant delay.

Example 3

An immersive virtual reality system (as shown in fig. 11) according to the present invention is composed of two identical immersive virtual reality systems (VRS3 and VRS 3'), wherein the VRS3 is the same as the VRS1 of embodiment 1 except that the angle sensing device in the display (COM1) is a three-axis gyroscope (hounwell HMC58 5883L three-axis magnetoresistive sensor), the signal transmission device is a CAT6023 chip, and the data processing device (COM3) is an associative homemade E2565 desktop computer externally connected to SEM-ST1011 model WHDI STICK manufactured by amoson corporation. The two VRSs 3 are connected through a router (TL-R478 + type router manufactured by TP-Link company) via data processing devices (COM3 and COM 3') for data transmission.

After the display (COM1, COM 1') is turned on, the Samsung S3C2440 chip, the UDA1341 decoding chip, the HMC5883L three-axis magnetoresistive sensor, and the CAT6023 chip are initialized, and then a main program loop is executed in the Samsung S3C2440 chip of the control device. In the main program loop, the control device of the display (COM1) obtains the data from the angle sensor and the microphone thereof, the data from the data processing device (COM3) received by the data transmission device thereof is stored in the storage device thereof, the data from the angle sensor and the microphone thereof is transmitted to the data processing device (COM3) through the data transmission device thereof, and the video and audio data required to be output is output to the display device and the earphone thereof. The display (COM 1') operates the same as the display (COM 1).

In the VRS3, the motion capture device (COM2) captures the motion of the user, generates depth map data, converts the depth map data into bone motion data, and transmits the bone motion data to the data processing device (COM 3). The data processing device (COM3) processes data from the display (COM1), the motion capture device (COM2) and the data processing device (COM3 ') according to a simulation program written in the data processing device in advance for realizing virtual reality, processes the interaction relation of all users to obtain virtual scene information, wherein the virtual scene information comprises information such as the position, the body and head motions, the voice and the interactive scene information of each user in a scene, sends the virtual scene information of the user corresponding to the VRS3 ' to the data processing device (COM3 '), converts the virtual scene information of the user corresponding to the VRS3 into audio and video information and sends the audio and video information to the display (COM 1).

In the VRS3 ', the motion capture device (COM2 ') captures the motion of the user, generates depth map data, converts the depth map data into bone motion data, and transmits the bone motion data to the data processing device (COM3 '). The data processing device (COM3 ') processes data from the display (COM 1') and the motion capture device (COM2 ') according to a simulation program written in the data processing device in advance for realizing virtual reality, sends the data to the data processing device (COM3), receives virtual scene information from the data processing device (COM3), converts the virtual scene information into audio and video information, and sends the audio and video information to the display (COM 1').

After the displays (COM1 and COM1 ') receive the audio and video information generated by the data processing devices (COM3 and COM 3') from the respective VRSs, the data are stored in the respective storage devices, the audio data required to be output are output to the respective earphones, and the video data required to be output are output to the respective display devices.

The simulation programs are respectively and simultaneously run, and the running results show that the immersive virtual reality system can normally execute the virtual reality simulation programs without obvious delay.

Example 4

An immersive virtual reality system according to the invention (as shown in figure 12) consists of two immersive virtual reality systems (VRS3 and VRS 4). VRS3 is the same as in example 3; the VRS4 is the same as VRS3 except it has three displays (COM 1'). The two VRSs are connected through a data exchange device (COM4) via data processing devices (COM3 and COM3 '), and the data processing devices (COM3 and COM 3') and the data exchange device (COM4) are connected through a router (TL-R478 + type router manufactured by TP-Link company). The data exchange device (COM4) is a yangtian a8800t desktop computer manufactured by association.

As in example 3, the displays (COM1 and COM 1') were power-on initialized and executed the main program loop. In the VRS3, the motion capture device (COM2) captures the motion of the user, generates depth map data, converts the depth map data into bone motion data, and transmits the bone motion data to the data processing device (COM 3). The data processing device (COM3) processes data from the display (COM1) and the motion capture device (COM2) according to a simulation program written in advance for realizing virtual reality, and transmits the data to the data exchange device (COM 4).

The data exchange device (COM4) receives data from the data processing devices (COM3 and COM3 ') of each VRS, processes the interaction relationships of all users to obtain virtual scene information including, for example, the position, the body and head movements, the voice, and the interactive scene information of each user in the scene, transmits the virtual scene information of the user corresponding to the VRS3 to the data processing device (COM3), and transmits the virtual scene information of the user corresponding to the VRS4 to the data processing device (COM 3').

The data processing device (COM3) receives the virtual scene information from the data exchange device (COM4), converts the virtual scene information into audio and video information and sends the audio and video information to the display (COM 1). The display (COM1) receives the audio and video information generated by the data processing device (COM3), stores the data in respective storage devices, outputs the audio data required to be output to respective earphones, and outputs the video data required to be output to respective display devices.

The display (COM1 '), motion capture device (COM2 '), and data processing device (COM3 ') of the VRS4 perform substantially the same operations as the display (COM1), motion capture device (COM2), and data processing device (COM3) of the VRS 3.

Example 5

An immersive virtual reality system (as shown in fig. 13) according to the present invention is composed of five immersive virtual reality systems (VRS1, VRS1 ', VRS1 ", VRS 1'" and VRS1 "") which are the same as the VRS3 of example 3, the five VRSs are connected via data processing devices (COM3, COM3 ', COM3 ", COM 3'" and COM3 "") through a data exchange device (COM4) for data transmission, and the data processing devices (COM3, COM3 ', COM3 ", COM 3'" and COM3 "") are connected with the data exchange device (COM4) through a router (TL-R478 + type router manufactured by TP-Link corporation). The data exchange device (COM4) is a yangtian a8800t desktop computer manufactured by association.

As in example 3, the display (COM1, COM 1', etc.) is initialized at power-on and executes a main program loop. In the VRS3, the motion capture device (COM2) captures the motion of the user, generates depth map data, converts the depth map data into bone motion data, and transmits the bone motion data to the data processing device (COM 3). The data processing device (COM3) processes data from the display (COM1) and the motion capture device (COM2) according to a simulation program written in advance for realizing virtual reality, and transmits the data to the data exchange device (COM 4).

After receiving data from the data processing devices (COM3, COM3 ', etc.) of each VRS, the data exchange device (COM4) processes the interaction relationships of all users to obtain virtual scene information including, for example, information such as the position, body and head movements, voice, and interactive scene information of each user in the scene, transmits the virtual scene information corresponding to each VRS user to the corresponding data processing device, transmits the virtual scene information corresponding to the VRS3 user to the data processing device (COM3), and transmits the virtual scene information corresponding to the VRS3 ' user to the data processing device (COM3 ').

The display (COM1 ', COM1 ", etc.), the motion capture device (COM2 ', COM 2", etc.), and the data processing device (COM3 ', COM3 ", etc.) of other VRSs of the present embodiment execute substantially the same operations as the display (COM1), the motion capture device (COM2), and the data processing device (COM3) of the VRS 3.

Example 6

An immersive virtual reality system according to the invention (as shown in fig. 14) is composed of three immersive virtual reality systems (VRS3, VRS3 ' and VRS3 ') identical to the VRS3 of example 3 and two immersive virtual reality systems (VRS4 and VRS4 ') identical to the VRS4 of example 4, the five VRSs are connected via data processing devices (COM3, COM3 ', COM3 ", COM 3" ' and COM3 "") through a data exchange device (COM4) for data transmission, and the data processing devices (COM3, COM3 ', COM3 ", COM 3" ' and COM3 "") and the data exchange device (COM4) are connected through a router (TL-R478 + type router manufactured by TP-Link corporation). The data exchange device (COM4) is a yangtian a8800t desktop computer manufactured by association.

The display (COM1, COM 1', etc.) is initialized at power-on and executes a main program loop. In the VRS3, the motion capture device (COM2) captures the motion of the user, generates depth map data, converts the depth map data into bone motion data, and transmits the bone motion data to the data processing device (COM 3). The data processing device (COM3) processes data from the display (COM1) and the motion capture device (COM2) according to a simulation program written in advance for realizing virtual reality, and transmits the data to the data exchange device (COM 4).

After receiving data from the data processing devices (COM3, COM3 ', etc.) of each VRS, the data exchange device (COM4) processes the interaction relationship of all users to obtain virtual scene information including information such as the position, body and head movements, voice, and interactive scene information of each user in the scene, and transmits the virtual scene information of each user corresponding to each VRS to the corresponding data processing device, for example, transmits the virtual scene information of the user corresponding to VRS3 to the data processing device (COM3), and transmits the virtual scene information of the user corresponding to VRS4 to the data processing device (COM 3').

Claims

1. An immersive virtual reality system, the system comprising at least:

at least one display wearable on the user's body, at least one motion capture device, and

at least one data processing device;

wherein,

the display is used for displaying images to a user, capturing head rotation motion of the user, sending head rotation motion data to at least one data processing device, receiving data sent by the at least one data processing device, and transmitting video and/or audio of a virtual environment to the user;

2. The virtual reality system of claim 1, wherein the display comprises at least one angle sensing device that senses at least rotational motion about a z-axis.

3. The virtual reality system of claim 2, wherein the angle sensing device is an electronic compass.

4. The virtual reality system according to claim 3, wherein the angle sensing means is capable of sensing rotational motion about three axes, x, y and z.

5. The virtual reality system of claim 4, wherein the angle sensing device is a three-axis gyroscope.

6. The virtual reality system of any one of claims 1 to 5, wherein the motion capture device comprises at least one infrared emitter and at least one infrared camera.

7. The virtual reality system of claim 6, wherein the motion capture device is a motion sensing camera.

8. The virtual reality system of any one of claims 1-7, wherein the data processing device is a computer, a personal digital assistant, a tablet computer, a mobile phone, an audio processing terminal, a video processing terminal, and/or a game console.

9. The virtual reality system of any one of claims 1-8, further comprising at least one data exchange device for handling data transfer between multiple VRSs.

10. The virtual reality system of claim 9, wherein the data exchange device is a computer, a personal digital assistant, a tablet computer, a mobile phone, an audio processing terminal, a video processing terminal, and/or a game console.

11. The virtual reality system of any one of claims 1-10, wherein a transmission speed between the display and the motion capture device, the display and the data processing device, and/or the motion capture device and the data processing device is no less than 18432000 bps.

12. The virtual reality system of claim 11, wherein the transmissions between the display and the motion capture device, the display and the data processing device, and/or the motion capture device and the data processing device are WHDI transmissions.

13. A method of implementing immersive virtual reality, the method comprising at least the following:

capturing the head rotation motion of the user through at least one display and sending the head rotation motion data to at least one data processing device,

14. A method of realising immersive virtual reality as claimed in claim 13, wherein the display comprises at least one angle sensing device which is at least capable of sensing rotational movement about the z-axis.

15. The method of claim 14, wherein the angle sensing device is an electronic compass.

16. The method of claim 14, wherein the angle sensing device is capable of sensing rotational motion about three axes, x, y and z.

17. The method of claim 16, wherein the angle sensing device is a three-axis gyroscope.

18. A method of realising immersive virtual reality according to any of claims 13-17, wherein the motion capture devices comprise at least one infrared emitter and at least one infrared camera.

19. The method of claim 18, wherein the motion capture device is a motion sensing camera.

20. The method of enabling immersive virtual reality of any one of claims 13-19, wherein the data processing device is a computer, a personal digital assistant, a tablet computer, a mobile phone, an audio processing terminal, a video processing terminal, and/or a game console.

21. The virtual reality system of any one of claims 13-20, further comprising processing data transfer between the plurality of VRSs by at least one data exchange device.

22. The virtual reality system of claim 21, wherein the data exchange device is a computer, a personal digital assistant, a tablet computer, a mobile phone, an audio processing terminal, a video processing terminal, and/or a game console.

23. The virtual reality system of any one of claims 13-22, wherein a transmission speed between the display and the motion capture device, the display and the data processing device, and/or the motion capture device and the data processing device is no less than 18432000 bps.

24. The virtual reality system of claim 23, wherein the transmissions between the display and the motion capture device, the display and the data processing device, and/or the motion capture device and the data processing device are WHDI transmissions.