US20070247515A1 - Handheld video transmission and display - Google Patents

Abstract

A handheld, wireless video receiver that receives compressed video, decompresses the compressed video, displays the decompressed video. The receiver is part of a system comprising methods, medium, and handheld, wireless devices which compress, enhance, encode, transmit, decompress and display digital video images in real time. Real time wireless videoconferences connect multiple handheld video devices. Real time compression is achieved by sub-sampling each frame of a video signal, filtering the pixel values, and encoding. Real time transmission is achieved due to high levels of effective compression. Real time decompression is achieved by decoding and decompressing the encoded data to display high quality images. The receiver can alter various setting including but not limited to the format for the compression, image size, frame rate, brightness and contrast. A zoom control can be used select a portion of interest of video being transmitted or being played back. The receiver may also have a touch sensitive display screen providing controls for video display and mobile telephone operation.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
• This application is a continuation of co-pending U.S. patent application Ser. No. 11/262,106, filed on Oct. 27, 2005, published Jun. 1, 2006, as U.S. patent application publication 2006/0114987, entitled “HANDHELD VIDEO TRANSMISSION AND DISPLAY,” which hereby is incorporated by reference.
• U.S. patent application Ser. No. 11/262,106 is a continuation in part of co-pending U.S. patent application Ser. No. 09/467,721, filed on Dec. 20, 1999, and entitled “VARIABLE GENERAL PURPOSE COMPRESSION FOR VIDEO IMAGES (ZLN)”, now U.S. Pat. No. 7,233,619, which hereby is incorporated by reference.
• This application and application Ser. No. 09/467,721 claim priority under 35 U.S.C. § 119(e) of U.S. provisional application Ser. No. 60/113,051, filed on Dec. 21, 1998, and entitled “METHODS OF ZERO LOSS (ZL) COMPRESSION AND ENCODING OF GRAYSCALE IMAGES”, which hereby is incorporated by reference.
• My U.S. patent application Ser. No. 09/312,922, filed on May 17, 1999, and entitled “SYSTEM FOR TRANSMITTING VIDEO IMAGES OVER A COMPUTER NETWORK TO A REMOTE RECEIVER,” now U.S. patent Ser. No ______, is also hereby incorporated by reference.
• My U.S. patent application Ser. No. 09/433,978, now U.S. Pat. No. 6,803,931, filed on Nov. 4, 1999, and entitled GRAPHICAL USER INTERFACE INCLUDING ZOOM CONTROL REPRESENTING IMAGE AND MAGNIFICATION OF DISPLAYED IMAGE”, is also hereby incorporated by reference. A co-pending divisional application of U.S. Pat. No. 6,803,931, is U.S. patent application Ser. No. 10/890,079, filed on Jul. 13, 2004, published on Dec. 9, 2004 as publication No. 2004/0250216, and entitled GRAPHICAL USER INTERFACE INCLUDING ZOOM CONTROL REPRESENTING IMAGE AND MAGNIFICATION OF DISPLAYED IMAGE”, and is also hereby incorporated by reference.
• My U.S. patent application Ser. No. 09/470,566, now U.S. Pat. No. 7,016,417, filed on Dec. 22, 1999, and entitled GENERAL PURPOSE COMPRESSION FOR VIDEO IMAGES (RHN)”, describes a compression method known as the “RHN” method, and is also hereby incorporated by reference.
• My co-pending U.S. patent application Ser. No. 09/473,190, filed on Dec. 20, 1999, and entitled “ADDING DOPPLER ENHANCEMENT TO GRAYSCALE COMPRESSION (ZLD)” is also hereby incorporated by reference.
• My co-pending U.S. patent application Ser. No. 10/154,775, filed on May 24, 2002, published as US 2003/0005428, and entitled “GLOBAL MEDIA EXCHANGE” is also hereby incorporated by reference.
• U.S. patent application Ser. No. 09/436,432, filed on Nov. 8, 1999, and entitled “SYSTEM FOR TRANSMITTING VIDEO IMAGES OVER A COMPUTER NETWORK TO A REMOTE RECEIVER,” now U.S. Pat. No. 7,191,462, is wholly owned by the inventor of the present invention.
BACKGROUND
• 1. Field of the Invention
• This invention relates to handheld devices for video transmission, including video capture, wired and wireless file transfer and live streaming, and display. Embodiments of the invention relate to data compression, specifically to the compression and decompression of video and still images, and relate to graphical user interfaces for controlling video transmission and display.
• 2. Description of Prior Art
• In the last few years, there have been tremendous advances in the speed of computer processors and in the availability of bandwidth of worldwide computer networks such as the Internet. These advances have led to a point where businesses and households now commonly have both the computing power and network connectivity necessary to have point-to-point digital communications of audio, rich graphical images, and video. However the transmission of video signals with the full resolution and quality of television is still out of reach. In order to achieve an acceptable level of video quality, the video signal must be compressed significantly without losing either spatial or temporal quality.
• A number of different approaches have been taken but each has resulted in less than acceptable results. These approaches and their disadvantages are disclosed by Mark Nelson in a book entitledThe Data Compression Book, Second Edition, published by M&T Book in 1996. Mark Morrision also discusses the state of the art in a book entitledThe Magic of Image Processing, published by Sams Publishing in 1993.
• Video Signals
• Standard video signals are analog in nature. In the United States, television signals contain 525 scan lines of which 480 lines are visible on most televisions. The video signal represents a continuous stream of still images, also known as frames, that are fully scanned, transmitted and displayed at a rate of 30 frames per second. This frame rate is considered full motion.
• A television screen has a 4:3 aspect ratio.
• When an analog video signal is digitized each of the 480 lines is sampled 640 times, and each sample is represented by a number. Each sample point is called a picture element, or pixel. A two dimensional array is created that is 640 pixels wide and 480 pixels high. This 640×480 pixel array is a still graphical image that is considered to be full frame. The human eye can perceive 16.7 thousand colors. A pixel value comprised of 24 bits can represent each perceivable color. A graphical image made up of 24-bit pixels is considered to be full color. A single, second-long, full frame, full color video requires over 220 millions bits of data.
• The transmission of 640×480 pixels×24 bits perpixel times 30 frames requires the transmission of221, 184,000 million bits per second. A T1 Internet connection can transfer up to 1.54 million bits per second. A high-speed (56 Kb) modem can transfer data at a maximum rate of 56 thousand bits per second. The transfer of full motion, full frame, full color digital video over a T1 Internet connection, or 56 Kb modem, will require an effective data compression of over 144:1, or 3949:1, respectively.
• A video signal typically will contain some signal noise. In the case where the image is generated based on sampled data, such as an ultrasound machine, there is often noise and artificial spikes in the signal. A video signal recorded on magnetic tape may have fluctuations due the irregularities in the recording media. Florescent or improper lighting may cause a solid background to flicker or appear grainy. Such noise exists in the real world but may reduce the quality of the perceived image and lower the compression ratio that could be achieved by conventional methods.
• Basic Run-length Encoding
• An early technique for data compression is run-length encoding where a repeated series of items are replaced with one sample item and a count for the number of times the sample repeats. Prior art shows run-length encoding of both individual bits and bytes. These simple approaches by themselves have failed to achieve the necessary compression ratios.
• Variable Length Encoding
• In the late 1940s, Claude Shannon at Bell Labs and R.M. Fano at MIT pioneered the field of data compression. Their work resulted in a technique of using variable length codes where codes with low probabilities have more bits, and codes with higher probabilities have fewer bits. This approach requires multiple passes through the data to determine code probability and then to encode the data. This approach also has failed to achieve the necessary compression ratios.
• D. A. Huffman disclosed a more efficient approach of variable length encoding known as Huffman coding in a paper entitled “A Method for Construction of Minimum Redundancy Codes,” published in 1952. This approach also has failed to achieve the necessary compression ratios.
• Arithmetic, Finite Context, and Adaptive Coding
• In the 1980s, arithmetic, finite coding, and adaptive coding have provided a slight improvement over the earlier methods. These approaches require extensive computer processing and have failed to achieve the necessary compression ratios.
• Dictionary-Based Compression
• Dictionary-based compression uses a completely different method to compress data. Variable length strings of symbols are encoded as single tokens. The tokens form an index to a dictionary. In 1977, Abraham Lempel and Jacob Ziv published a paper entitled, “A Universal Algorithm for Sequential Data Compression” in IEEE Transactions on Information Theory, which disclosed a compression technique commonly known as LZ77. The same authors published a 1978 sequel entitled, “Compression of Individual Sequences via Variable-Rate Coding,” which disclosed a compression technique commonly known as LZ78 (see U.S. Pat. No. 4,464,650). Terry Welch published an article entitled, “A Technique for High-Performance Data Compression,” in the June 1984 issue of IEEE Computer, which disclosed an algorithm commonly known as LZW, which is the basis for the GIF algorithm (see U.S. Pat. Nos. 4,558,302, 4,814,746, and 4,876,541). In 1989, Stack Electronics implemented a LZ77 based method called QIC-122 (see U.S. Pat. No. 5,532,694, U.S. Pat. No. 5,506,580, and U.S. Pat. No. 5,463,390).
• These lossless (method where no data is lost) compression methods can achieve up to 10:1 compression ratios on graphic images typical of a video image. While these dictionary-based algorithms are popular, these approaches require extensive computer processing and have failed to achieve the necessary compression ratios.
• JPEG and MPEG
• Graphical images have an advantage over conventional computer data files: they can be slightly modified during the compression/decompression cycle without affecting the perceived quality on the part of the viewer. By allowing some loss of data, compression ratios of 25:1 have been achieved without major degradation of the perceived image. The Joint Photographic Experts Group (JPEG) has developed a standard for graphical image compression. The JPEG lossy (method where some data is lost) compression algorithm first divides the color image into three color planes and divides each plane into 8 by 8 blocks, and then the algorithm operates in three successive stages:
• • (a) A mathematical transformation known as Discrete Cosine Transform (DCT) takes a set of points from the spatial domain and transforms them into an identical representation in the frequency domain.
  • (b) A lossy quantization is performed using a quantization matrix to reduce the precision of the coefficients.
  • (c) The zero values are encoded in a zig-zag sequence (see Nelson, pp. 341-342).
• JPEG can be scaled to perform higher compression ratio by allowing more loss in the quantization stage of the compression. However this loss results in certain blocks of the image being compressed such that areas of the image have a blocky appearance and the edges of the 8 by 8 blocks become apparent because they no longer match the colors of their adjacent blocks. Another disadvantage of JPEG is smearing. The true edges in an image get blurred due to the lossy compression method.
• The Moving Pictures Expert Group (MPEG) uses a combination of JPEG based techniques combined with forward and reverse temporal differencing. MPEG compares adjacent frames and, for those blocks that are identical to those in a previous or subsequent frame, only a description of the previous or subsequent identical block is encoded. MPEG suffers from the same blocking and smearing problems as JPEG.
• These approaches require extensive computer processing and have failed to achieve the necessary compression ratios without unacceptable loss of image quality and artificially induced distortion.
• QuickTime: CinePak, Sorensen, H.263
• Apple Computer, Inc. released a component architecture for digital video compression and decompression, named QuickTime. Any number of methods can be encoded into a QuickTime compressor/decompressor (codec). Some popular codec are CinePak, Sorensen, and H.263. CinePak and Sorensen both require extensive computer processing to prepare a digital video sequence for playback in real time; neither can be used for live compression. H.263 compresses in real time but does so by sacrificing image quality resulting in severe blocking and smearing.
• Fractal and Wavelet Compression
• Extremely high compression ratios are achievable with fractal and wavelet compression algorithms. These approaches require extensive computer processing and generally cannot be completed in real time.
• Sub-Sampling
• Sub-sampling is the selection of a subset of data from a larger set of data. For example, when every other pixel of every other row of a video image is selected, the resulting image has half the width and half the height. This is image sub-sampling. Other types of sub-sampling include frame sub-sampling, area sub-sampling, and bit-wise sub-sampling.
• Image Stretching
• If an image is to be enlarged but maintain the same number of pixels per inch, data must be filled in for the new pixels that are added. Various methods of stretching an image and filling in the new pixels to maintain image consistency are known in the art. Some methods known in the art are dithering (using adjacent colors that appear to be blended color), and error diffusion, “nearest neighbor”, bilinear and bicubic.
• Portable Hand Held Devices: Pen-based Computers and PDAs
• In the early 1990s, a number of pen based computers were developed. These portable computers were characterized by a display screen that could be also used as an input device when touched or stroked with a pen or finger. For example in 1991, NCR developed a “notepad” computer, the NCR3125. Early pen-based computers ran three operating systems: DOS, Microsoft's Windows for Pen Computing and Go Corp.'s PenPoint. In 1993, Apple developed the Newton MessagePad, an early personal digital assistant (PDA). Palm developed the Palm Pilot in 1996. Later, in 2002, Handspring released the Treo which runs the Palm OS and features a Qwerty keyboard. In 2000, the Sony Clie, used the Palm OS and could play audio files. Later versions included a built-in camera and could capture and play Apple QuickTime™ video. Compaq (now Hewlett Packard) developed the iPAQ in 2000. The iPAQ and other PocketPCs run a version of Windows CE. Some PocketPC and PDA have wireless communication capabilities.
• In 2001, Apple released a music player, called the iPod, featuring a small, internal hard disk drive that could hold over 1000 songs and fit in your pocket. The original iPod has a display, a set of controls, and ports for connecting to a computer, such as a Macintosh or PC, via Firewire, and for connecting to headphones. However, the original iPod did not have a color display, a built-in camera, built-in speakers, built-in microphone or wireless communications.
• Portable Hand Held Devices: Cell Phone and Picture Phones
• The first cellular telephones had simple LCD displays suitable for displaying only a limited amount of text. More recently, cell phones have been developed which have larger, higher resolution displays that are both grayscale and color. Some cell phones have been equipped with built-in cameras with the ability to save JPEG still photos to internal memory. In April 2002, Samsung introduced a cell phone with a built-in still photo camera and a color display. The Samsung SCH-X590 can store up to 100 photos in its memory and can transfer still photos wirelessly.
• Cell phones can be used as wireless modems. Initially they had limited data bandwidth. Next, digital cell phones were developed. By early 2002, bandwidth was typically 60-70 Kbps. Higher bandwidth wireless networks are being developed.
• Hand Held Devices are Limited is Size and Weight
• Hand held devices are limited in size and weight. Many users are only willing to use a handheld device that weights a few ounces and can fit inside a typical shirt pocket, or even worn on their waist or arm. These size and weight limitation prevent handheld devices from having the electronic circuitry, processors, and batteries found in laptops and other larger computers. These limitations have made it impossible to provide full frame, full motion video display or live transmission on handheld devices.
• PDAs, PocketPCs, and Picture Phones are Limited by Battery Life, Processor Speed, and Network Bandwidth
• The existing, commercially available hand held devices have not been able to support live or streaming video for a number of reasons. Uncompressed full-motion, full frame video requires extremely high bandwidth that is not available to handheld portable devices. In order to reduce the bandwidth, lossy compression such as MPEG has been used to reduce the size of the video stream. While MPEG is effective in desktop computers with broadband connections to the Internet, decoding and displaying MPEG encoded video is very processor intensive. The processors of existing handheld devices are slower or less powerful than those used in desktop computers. If MPEG were used in a handheld device, the processor would quickly drains the battery of most handheld devices. Further, the higher bandwidth wireless communications interfaces would also place a large strain on the already strained batteries. Live video transmission and reception would be even more challenging. For this reason, handheld device have not been able to transmit or receive streaming, or especially, live video.
• What is needed is an enhanced handheld device that is capable of receiving streaming and live video. Further a handheld device that could capture and transmit live video would provide live coverage of events that would otherwise not be able to be seen. With handheld video devices that both transmit and receive live video, handheld wireless videoconferencing could become a reality. Also a video compression method that requires significantly reduced processing power and would be less draining on the battery of a handheld device is needed. Additionally since, handheld video display screens which are smaller than typical computer screens, a user of a handheld video receiver needs to be able control the portion of a video be transmitted to allow a smaller, higher quality video to be received and viewed on the handheld screen with dimensions smaller than the original video.
SUMMARY OF THE INVENTION
• In accordance with the present invention a handheld device comprises a black and white or color video display screen, speakers or headphones for hearing audio associated with the video display, controls for user input, a memory for storing compressed video data, and a processor for running computer programs which decompress the compressed video data and play the video on the display screen, and the video's audio on speakers and/or headphones. Further, some embodiments of the present invention include a microphone and video camera for inputting audio and video. A plurality of handheld video devices are connected to a network for exchanging video file, streaming video from a pre-recorded video file or live transmission from one device to one or more devices in remote locations. The network connections can be wired or wireless.
• One embodiment of the present invention comprises a video camera that can be removably mounted on an iPod-type device to add the video capture capability. Further the separate camera unit could include a microphone or speakers. Further, wireless communications could be added to the separate camera unit or as yet another removable unit.
• Further, the present invention includes a method of compression of a video stream comprising steps of sub-sampling a video frame, and run-length encoding the sub-sampled pixel values, whereby the method can be executed in real time, and whereby the compressed representation of pixels saves substantial space on a storage medium and requires substantially less time and bandwidth to be transported over a communications link. The present invention includes a corresponding method for decompressing the encoded data.
• Further, the present invention includes a zoom control that is graphically displayed on the display screen and receives input from either the touch screen or the controls of the handheld device. A user may use the zoom control to send remote control commands to a transmitting device to dynamically specify an area to be transmitted. Alternatively, the user may use the zoom control to magnify video that is being played from a file.
OBJECTS AND ADVANTAGES
• Accordingly, beside the objects and advantages of the method described above, some additional objects and advantages of the present invention are:
• • (a) to provide a handheld device for capturing audio and video which can be transmitted to another video display device.
  • (b) to provide a handheld device for displaying video that has been received from a video capture and transmission device.
  • (c) to provide a handheld wireless video conferencing system comprising handheld devices which act as both transmitters and receivers connected over a data network.
  • (d) to provide an add-on module that will allow an iPod-type device to capture, transmit, or receive video.
  • (e) to provide a graphical zoom control on a hand held video display device whereby the user can remotely control the area of the video that is being transmitted in high resolution.
  • (f) to provide a graphical zoom control on a hand held video display device whereby the user can magnify a video being displayed.
  • (g) to provide a method of compressing and decompressing video signals so that the video information can be transported across a digital communications channel in real time.
  • (h) to provide a method of compressing and decompressing video signals such that compression can be accomplished with software on commercially available computers without the need for additional hardware for either compression or decompression.
  • (i) to provide a high quality video image without the blocking and smearing defects associated with prior art lossy methods.
  • (j) to provide a high quality video image that suitable for use in medical applications.
  • (k) to enhance images by filtering noise or recording artifacts.
  • (l) to provide a method of compression of video signals such that the compressed representation of the video signals is substantially reduced in size for storage on a storage medium.
  • (m) to provide a level of encryption so that images are not directly viewable from the data as contained in the transmission.
DRAWING FIGURES
• In the drawings, closely related figures have the same number but different alphabetic suffixes.
• FIG. 1 shows the high level steps of compression and decompression of an image.
• FIGS. 2A to2H show alternatives for selecting a pixel value for encoding.
• FIG. 3A shows the variable encoding format.
• FIG. 3B shows an example of a code where N is 5 bits wide and U is 3 bits wide.
• FIG. 4A shows the flowchart for the compression method.
• FIG. 4B shows an image and a corresponding stream of pixels.
• FIGS. 5A to5C shows the formats for the run-length encoding of the RHN method.
• FIG. 6 shows a series of codes and the resulting encoded stream.
• FIG. 7 shows a series of codes and the resulting encoded stream of the RHN method.
• FIG. 8A shows examples of variable formats.
• FIG. 8B shows a format that preserves 9 bits of color.
• FIG. 9 shows the flow chart for the decompression method.
• FIG. 10 shows image stretching by interpolation.
• FIGS. 11A and 11B show an encryption table and a decryption table.
• FIGS. 12A and 12B show machines for compressing and decompressing, respectively.
• FIG. 12C shows a compressor and decompressor connected to a storage medium.
• FIG. 12D shows a compressor and decompressor connected to a communications channel.
• FIG. 13A shows elements of a compressor.
• FIG. 13B shows an embodiment of an encoding circuit.
• FIG. 13C shows a generic pixel sub-sampler.
• FIGS. 13D through 13J show embodiments of pixel sub-samplers.
• FIGS. 14A through 14C shows embodiments of a machine element for variably altering the number of bits.
• FIG. 15 shows elements of a decompressor.
• FIG. 16A shows elements for setting width, height, frame rate, brightness, and contrast which are variably altered by a receiver.
• FIG. 16B shows elements for setting the number of pixel bits that are variably altered by a receiver.
• FIG. 17 shows a lossless compression step for further compression of an encoded data buffer.
• FIG. 18 shows images being enlarged by stretching.
• FIGS. 19A through 19C show various network configuration comprising handheld video devices.
• FIGS. 20A through 20D show various embodiments of handheld video devices.
• FIGS. 21A through 21C show handheld video devices comprising graphical zoom controls.
REFERENCE NUMERALS IN DRAWINGS
• 100 compression steps
• 110 sub-sampling step
• 130 encoding step
• 140 encoded data
• 150 decompression steps
• 160 decoding step
• 180 image reconstitution step
• 200 32 bit pixel value
• 202 blue channel
• 204 green channel
• 206 red channel
• 208 alpha channel
• 210 24 bit pixel value
• 212 blue component
• 214 green component
• 216 red component
• 220 RGB averaging diagram
• 222 blue value
• 224 green value
• 226 red value
• 228 averaged value
• 230 blue selection diagram
• 232 blue instance
• 234 green instance
• 236 red instance
• 240 selected blue value
• 250 green selection diagram
• 260 selected green value
• 270 red selection diagram
• 280 selected red value
• 290 grayscale pixel
• 292 grayscale blue
• 294 grayscale green
• 296 grayscale red
• 298 selected grayscale value
• 299 filtered pixel value
• 300 N
• 301 U
• 302 W
• 310pixel bit7
• 312pixel bit6
• 314pixel bit5
• 316pixel bit4
• 318pixel bit3
• 320pixel bit2
• 322pixel bit1
• 324pixel bit0
• 325 8 bit pixel
• 330 5 bit sample
• 332sample bit4
• 334sample bit3
• 336sample bit2
• 338sample bit1
• 340sample bit0
• 350 3 low order bits
• 360 formatted code
• 362 encodedbit4
• 364 encodedbit3
• 366 encodedbit2
• 368 encodedbit1
• 370 encodedbit0
• 380 3 bit count value
• 400 encode flowchart
• 402 encode entry
• 403 encode initialization step
• 404 get pixel step
• 405 get value step
• 406 lookup encoded value step
• 408 compare previous
• 410 increment counter step
• 412 check count overflow
• 414 new code step
• 416 check end of data
• 418 set done
• 420 counter overflow step
• 422 check done
• 428 encode exit
• 430 image
• 440 image width
• 450 image height
• 460 pixel stream
• 500 code byte
• 510 flag bit
• 520 repeat code
• 530 count
• 550 data code
• 560 wasted bits
• 565data bit6
• 570data bit5
• 575data bit4
• 580data bit3
• 585data bit2
• 590data bit1
• 595data bit0
• 610 decimal values
• 620 first value
• 622 second value
• 624 third value
• 626 fourth value
• 628 fifth value
• 630 sixth value
• 632 seventh value
• 640 binary code
• 650 first byte
• 651 first data
• 652 first count
• 653 second byte
• 654 second data
• 655 second count
• 656 third byte
• 657 third data
• 658 third count
• 740 RHN binary code
• 803 ZL3 format
• 804 ZL4 format
• 805 ZL5 format
• 808 ZL8 format
• 809 ZL9 format
• 812 ZL12 format
• 820 ZL9C format
• 900 decode entry
• 901 decode initialize step
• 902 get code step
• 908 decode lookup step
• 909 check zero count
• 910 place pixel step
• 914 reset counter step
• 916 check length
• 918 decode exit
• 920 decode flowchart
• 1010 first adjacent pixel
• 1012 second adjacent pixel
• 1014 first subsequent adjacent pixel
• 1016 second subsequent adjacent pixel
• 1052,1054,1056,1058,1060 interpolated pixels
• 1100 encryption table
• 1110 decryption table
• 1200 video frames
• 1205afirst video frame
• 1205bsecond video frame
• 1205nnth video frame
• 1210 compressor
• 1215 video signal
• 1220 series of encoded data
• 1225 encoded data buffer
• 1225afirst encoded data
• 1225bsecond encoded data
• 1225nnth encoded data
• 1230 received encoded data
• 1230afirst received encoded data
• 1230bsecond received encoded data
• 1230nnth received encoded data
• 1235 encoded data stream
• 1238 received encoded data
• 1240 I/O device
• 1245 input encoded data stream
• 1250 decompressor
• 1260 decoded video frame
• 1260afirst decoded video frame
• 1260bsecond decoded video frame
• 1260nnth decoded video frame
• 1268 decoded video frames
• 1270 video sequence
• 1280 storage medium
• 1290 communications channel
• 1310 video digitizer
• 1320path1320
• 1330 video memory
• 1331 scan
• 1332 pixel index
• 1340path1340
• 1350 encoding circuit
• 1360path1360
• 1370 encoded data
• 1380 pixel sub-sampler
• 1380a24 to 5 bit sub-sampler
• 1380b24-bit RGB to S bit sub-sampler
• 1380c32-bit RGB to 5 bit sub-sampler
• 1380dcolor 9-bit sub-sampler
• 1380eYUV sub-sampler
• 1380f36-bit RGB to 24 bit sub-sampler
• 1380g15-bit sub-sampler output
• 1382 pixel extractor
• 1383 value path
• 1384 coder
• 1385path1385
• 1390 data/count
• 1392 code index
• 1395path1395
• 1400 24-bit to variable bit sub-sampler
• 1401 generic 3-bit sub-sampler
• 1402 generic 4-bit sub-sampler
• 1403 generic 8-bit sub-sampler
• 1404 generic 10-bit sub-sampler
• 1410 number of bits selector
• 1420 number of bits indicator
• 1430 36-bit to variable bit sub-sampler
• 1440 24/36 bit variable bit sub-sampler
• 1450 second selector
• 1460 selection logic
• 1470 selection signal
• 1510 decoding circuit
• 1520 decoded pixel values
• 1530 decoder pixel index
• 1540 image memory
• 1600 transmitter
• 1610 receiver
• 1615 setting control path
• 1620 frame sub-sampler
• 1621path1621
• 1630 selected frame
• 1632 pixel from frame
• 1640 transmitter pixel sub-sampler
• 1642path1642
• 1650 run length encoder
• 1660 settings
• 1661 brightness
• 1662 contrast
• 1663 height
• 1664 width
• 1665 frame rate
• 1670 frame selector
• 1675 frame select indicator
• 1680 number of pixel bits setting
• 1690 alternate transmitter
• 1700 run-length encoding step
• 1710 run-length encoded output
• 1720 further lossless compression step
• 1730 further lossless compression
• 1800 unstretched frame
• 1810 enlarged image
• 1820 stretching step
• 1901afirst video
• 1901bfirst reflected video
• 1902asecond video
• 1902bsecond reflected video
• 1910 network
• 1910awired network
• 1910bwireless network
• 1910ccombined network
• 1920afirst node
• 1920bsecond node
• 1920cthird node
• 1920dfourth node
• 1920efifth node
• 1930 reflector
• 1940 point-to-point transmission
• 1942 first indirect path
• 1944 second indirect path
• 2010 first handheld device
• 2012 display
• 2012bsecond display
• 2012dphone display
• 2014 controls
• 2014dphone controls
• 2016 wireless port
• 2016bsecond wireless port
• 2016cintegrated wireless port
• 2016dcellular port
• 2020 headphone
• 2021 right speaker
• 2021bright built-in speaker
• 2021cright integrated speaker
• 2021dright phone speaker
• 2021ephone earphone
• 2022 microphone
• 2022bbuilt-in microphone
• 2022cintegrated microphone
• 2022dphone microphone
• 2023 left speaker
• 2023bleft built-in speaker
• 2023cleft integrated speaker
• 2023dleft phone speaker
• 2024 speaker/microphone cable
• 2030 camera
• 2030bbuilt-in camera
• 2030cintegrated camera
• 2030dphone camera
• 2032 lens
• 2034 camera cable
• 2040 second handheld device
• 2050 wireless connection
• 2051 wired connection
• 2051afirst wired connection
• 2051bsecond wired connection
• 2052 video source
• 2054 video transmitter
• 2056 video storage
• 2060 integrated handheld device
• 2062 A/V module
• 2064 wireless module
• 2070 cellular integrated device
• 2100 zoom control
• 2102ainner region
• 2102bsecond inner region
• 2102cthird inner region
• 2104amagnification factor
• 2104bsecond magnification factor
• 2104cthird magnification factor
• 2106aouter region
• 2106bsecond outer region
• 2106csecond outer region
• 2110avideo display window
• 2110balternate video display window
DESCRIPTION OF THE INVENTION
• FIG. 1-Compression and Decompression Steps
• FIG. 1 illustrates a sequence ofcompression steps100 and a sequence of decompression steps150 of the present invention. The compression steps100 comprise asub-sampling step110 and anencoding step130. After completion of the compression steps100, a stream of encodeddata140 is output to either a storage medium or a transmission channel. The decompression steps150 comprise adecoding step160 wherein the stream of encodeddata140 is processed and animage reconstitution step180.
• FIGS. 2A to2H Selecting Pixel Values for Encoding
• FIGS. 2A to2G illustrate alternatives for selecting a pixel value for encoding. The sub-sampling step110 (FIG. 1) includes sub-sampling of a pixel value to obtain a variable selected number of bits.
• Video digitizing hardware typical has the options of storing the pixel values as a 32bit pixel value200 or a 24bit pixel value210, shown inFIG. 2A andFIG. 2B, respectively. The 32bit pixel value200 is composed of ablue channel202, agreen channel204, ared channel206, and analpha channel208. Each channel contain 8 bits and can represent 256 saturation levels for the particular color channel. For each channel the saturation intensity value of zero represents the fully off state, and the saturation intensity value of “255” represents the fully on state. A common alternative not shown is a sixteen-bit format where the three color channels contain 5 bits each and the alpha channel is a single bit. The present invention anticipates the use of the color channels of 16 bit pixel value is a manner substantially the same as the 32-bit pixel value200 except the number of bits per channel is 5 instead of 8.
• The 24-bit pixel value210 is composed of ablue component212, agreen component214, and ared component216. There is no component for the alpha channel in the 24bit pixel value210. Regardless of the structure, theblue channel202 is equivalent to theblue component212, thegreen channel204 is equivalent to thegreen component214, and thered channel206 is equivalent to thered component216.
• In the present invention, the 32bit pixel value200 alternative is preferred due to the consistent alignment of 32 bit values in most computer memories; however for simplicity of illustration thealpha channel208 will be omitted inFIGS. 2C to2G.
• If the video signal is digitized in color, the three color components may have different values. For example inFIG. 2C, a RGB averaging diagram220 illustrates ablue value222 of 35 decimal, agreen value224 of 15, and ared value226 of 10. One alternative is to sub sample from 24 bits to 8 bits by averaging the three color values to obtain an averagedvalue228 that, in this example, has the value of 20. (10+15+35)/3=20. This will produce a grayscale image. Alternatively, a color image can be preserved by sampling bits from each color component (seeFIG. 8B).
• FIG. 2D illustrates another alternative for selecting an 8 bit value in a blue selection diagram230. In this example, ablue instance232 has the value of 35, agreen instance234 has the value of 15, and ared instance236 has the value of 10. In this alternative theblue instance232 is always selected as a selectedblue value240.
• FIG. 2E illustrates another alternative for selecting an 8 bit value in a green selection diagram250. In this alternative thegreen instance234 is always selected as a selectedgreen value260.
• FIG. 2F illustrates another alternative for selecting an 8 bit value in a red selection diagram270. In this alternative thered instance236 is always selected as a selectedred value280.
• If the video signal being digitized is grayscale, the three color components will have the same values. For example inFIG. 2G, agrayscale pixel290 comprises a grayscale blue292 with a value of decimal 40, a grayscale green294 with a value of 40, and a grayscale red with a value of 40. Because the values are all the same, it makes no difference which grayscale color component is selected, a selectedgrayscale value298 will have the value of 40 in this example.
• The preferred embodiment of this invention uses the low order byte of the pixel value, which is typically the blue component as shown inFIG. 2D.
• FIG. 2H illustrates a filteredpixel value299 of 8 bits that may be selected by one of the alternatives described above. In these examples, the filteredpixel value299 is equivalent to items referenced bynumerals228,240,260,280, or298. This reduction of the 32bit pixel value200 or the 24bit pixel value210 contributes a reduction in data size of 4:1 or 3:1, respectively. This reduction recognizes that for some images, such as medical images or grayscale images, no relevant information is lost.
• For additional compression, the filteredpixel value299 can variably select any number of bits. For example, selection of the most significant four bits instead of all eight bits filters noise that may show up in the low order bits may be very suitable for an image such as one produced by an ultrasound medical device. An example of this is shown byZL4804 inFIG. 8A.
• FIGS.3A and3—Encoding Formats
• Speed of compression and decompression may be enhanced if the algorithms fit into computer memory native storage elements such as 8 bit bytes, 16 bit words, or 32 bit double words, or some other size for which the computer architecture is optimized.
• A grayscale image may be stored at a higher bit level than the actual values require. This may occur when an image is generated by an imaging technology such as radar, ultrasound, x-ray, magnetic resonance, or similar electronic technology. For example an ultrasound machine may only produce 16 levels of grayscale, requiring 4 bits of data per pixel, but the image digitizing may be performed at 8 to 12 bits per pixel. In this example, the low order bits (4 to 8) respectively provide no significant image data.
• In the present invention, a fast and efficient compression and encoding method is implemented by using unused bits to store a repeat count for repeated values.
• The most significant N bits of the pixel value are selected where N is the number of significant bits (determined by data analysis or by user selection). If N is less than W, where W is a native machine data type such as 8 bit byte, 16 bit word, or 32 bit double word or some other size for which the computer architecture is optimized, then W-N equals the number of unneeded bits, U. A repeat count, C, can contain a value from 1 to CMAX where CMA is 2 to the power of U. For example, if U equals 4, C can be a number from 1 to 16. In practice the maximum value will be encoded as a zero because the high order bit is truncated. In the example, decimal 16 has a binary value “10000” will be stored as “0000”.
• For example, when W is 8, value pairs for N and U could include without limitation (2,6), (3,5), (4,4), (5,3), and (6,2). When W is 16, value pairs for N and U could include without limitation (2,14), (3,13), (4,12), (5,11), (6,10), (7, 9), (8, 8), (9, 7), (10, 6), (11, 5), (12,4), (13, 3), and (14, 2). When W is 32, value pairs for N and U could include without limitation all combinations of values pairs for N and U where N+U equals 32 and N>1 and U>1. When W is not a multiple of 8, value pairs for N and U could include without limitation all combinations of values pairs for N and U where N+U equals W and N>1 and U>1.
• FIG. 3A shows the encoded format whereN300 represent the N most significant bits of thepixel value299,U301 represents the bits that are not used for the data and are used for the repeat count, andW302 where W is the width of the encoded data and equal to sum of N and U
• FIG. 3B illustrates bit sub-sampling where N's300 bit width is 5, U's301 bit width is 3, andW302 is 8. Thehigh order 5 bits310-318 of an 8bit pixel325 are extracted to form a fivebit sample330. The lower 3 bits of330 are ignoredbits350. In the formattedcode360, the ignoredbits350 are replaced with therepeat count value380.
• Encoding
• The most significant N bits of each pixel are selected from the image to obtain value V.
• In the encryption embodiment of this invention V may be used to select an encoded value, E, from the encoding table. E is also a N-bit value. The number of elements in the encode table1100 (FIG. 11) is 2 to the Nth power.
• In the other embodiments of this invention V is used as E.
• E is saved as the prior value, P. For each subsequent pixel, the encoded value, E, is obtained and compared to the prior value, P. If the prior value, P, is the same as E, then a repeat counter, C, is incremented; otherwise the accumulated repeat count, C, for the prior value, P, is merged with P and placed in an array A that implements the encoded data140 (FIG. 1) buffer. For example, if W is 8 and N is 4 and C is 10, U is 4, CMAX is 16, and ((P<<U)|C) is the merged value. If the repeat count, C, is greater CMAX, then CMAX is merged with P ((P<<U)|CMAX) and placed in the encoded data140 (FIG. 1) buffer, A. CMAX is subtracted from C and merged values are placed in A until C is less than CMAX. All pixels are processed in this manner until the final value is compressed and encoded. The length, L, of the encoded data140 (FIG. 1) is also placed in the encodeddata140 buffer.
• FIG. 4A—Encode Flowchart
• FIG. 4A illustrates the encodeflowchart400 which represents the details of the encryption embodiment of the encoding step130 (FIG. 1) for the present invention.
• The encoding begins at an encodeentry402. In an encodeinitialization step403, a prior value P is set to a known value, preferably decimal “255” or hexadecimal 0xFF, a repeat counter C is set to zero, an encoded length L is set to 0, and a completion flag “Done” is set to a logical value of false. Next, aget pixel step404 obtains a pixel from the image being encoded. At aget value step405, a value V is set to the N bitfiltered pixel value299 as derived from the pixel using one of the methods shown inFIGS. 2C to2G, preferably the fastest as explained above, and extracting the N most significant bits. At a lookup encodedvalue step406, an encoded value E is set to the value of one of the codes1105 (FIG. 11A) of the encode table1100 as indexed by V. (In the non-encrypted embodiment of this invention,step406 is bypassed because V is used as E) Next, a compare previous408 decision is made by comparing the values of E and P. If the values are the same, anincrement counter step410 is executed and flow continues to theget pixel step404 that obtains the next pixel from the image.
• If the encode value E does not match the prior value P, then acheck count overflow412 decision is made. If the counter C is less than or equal to CMAX, then anew code step414 is executed, otherwise acounter overflow step420 is executed.
• Atstep414, the counter C is masked and bit-wise OR-ed with P shifted left by U bit positions and is placed in the A at the next available location as indexed by the encoded length L. Then, continuing insideflowchart step414, L is incremented, the repeat count C is set to 1 and the prior value P is set to E. Afterstep414, a “check end of data” decision is made by checking to see if there are any more pixels in the image, and, if not, if the last value has been processed. Because this method utilizes a readahead technique step414 must be executed one more time after the end of data is reached to process the last run-length. If there is more data in the image, flow continues to a check of the completion flag “Done” atstep422. If the check indicates that the process is not completed, flow continues to step404.
• If the end of data is reached but the completion flag “Done” is still false, flow continues to a set donestep418. Atstep418, the completion flag “Done” is set to logical true, and flow continues todecision412 where the last run-length will be output and flow will eventually exit throughstep414,decision416,decision422, and then terminate at encodeexit428.
• It is possible for the repeat count C to become larger than CMAX requiring more bits than allocated by this method. This situation is handled by making thecheck count overflow412 decision and executing thecounter overflow step420. Atstep420, the counter C is masked and bit-wise OR-ed with P shifted left by U bit positions and is placed in the A at the next available location as indexed by the encoded length L. Then, continuing insideflowchart step414, L is incremented, and the repeat count C is decrement by CMAX. Afterstep420, flow continues to thecheck count overflow412 decision. Thus when the encode value E repeats more than CMAX times, multiple sets of repeat counts and encoded values are output to the encodeddata140 buffer.
• This entire process is repeated for each image or video frame selected during optional image sub-sampling (see110 inFIG. 1) and the encoded length L is transmitted with the encoded data associated with each frame. The encoded length varies from frame to frame depending on the content of the image being encoded.
• FIG. 4—Image and Pixel Stream
• FIG. 4B illustrates an image and its corresponding stream of pixels. Arectangular image430 is composed of rows and columns of pixels. Theimage430 has awidth440 and aheight450, both measured in pixels. In this illustrative embodiment, pixels in a row are accessed from left to right. Rows are accessed from top to bottom. Some pixels in the image are labeled from A to Z. Pixel A is the first pixel and pixel Z is the last pixel. Scanning left to right and top to bottom will produce apixel stream460. In thepixel stream460, pixels A and B are adjacent. Also pixels N and0 are adjacent even though they appear on different rows in the image. If adjacent pixels have the same code the process inFIG. 4A will consider them in the same run.
• Because the video signal being digitized is analog there will be some loss of information in the analog to digital conversion. The video digitizing hardware can be configured to sample the analog data into theimage430 with almost anywidth440 and anyheight450. The present invention achieves most of its effective compression by sub-sampling the data image with thewidth440 value less than the conventional640 and theheight450 value less than the convention480. In a preferred embodiment of the invention, for use in a medical application with T1 Internet transmission bandwidth, image dimensions are sub-sampled at 320 by 240. However an image dimension sub-sampling resolution of 80 by 60 may be suitable for some video application.
• FIGS. 5A to5C—Run-length Encoding Formats of the RHN Method
• FIGS. 5A to5C show use of a different structure than the present invention.FIGS. 5A to5C show the formats for the run-length encoding of RHN. InFIG. 5A, acode byte500, with its high order bit designated as aflag bit510.
• FIG. 5B shows arepeat code520 comprising a Boolean value one in itsflag bit510 and a 7 bit count530 in the remaining 7 low order bits. The seven bit count530 can represent 128 values with a zero representing “128” and 1 through 127 being their own value.
• FIG. 5C shows adata code550 comprising:
• • 1. a Boolean value zero in itsflag bit510
  • 2. two unused data bits: data bit6 reference by565 and data bit5 reference by570, and
  • 3. five bits,data bits4 to0, reference by575,580,585,590, and595, respectively.
• FIG. 5C shows that in every byte of theRHN data code550 two bits are unused and one bit is used for the flag bit, so that only five of the eight bits are used for data. The remaining three bits are wastedbits560. The present invention uses a different structure by placing the repeat count in bits that the RHN format would not have used for data (U). The corresponding ZLN format, ZL5 (where N is 5, U is 3, and W is 8), always uses five bits for data and the remaining 3 bits for the repeat count. In practice, repeat counts are small and often can fit in 3 bits, so this embodiment of the present invention will result in superior compression performance over the RHN method.
• In addition, the present invention provides for a larger count when the bit filtering is larger. For example, the alternate ZLN format where each byte contains 4 data bits, ZL4 (where N is 4 and U is 4), allows for a four bits of repeat count. For example, in practice, ZIA is superior to RHN on a typical ultrasound image containing 16 shades of gray.
• FIG. 6—Encoded Data Stream
• FIG. 6 shows a series of exemplarydecimal values610 comprising afirst value620 equal to decimal 0, asecond value622 equal to 0, athird value624 equal to 0, afourth value626 equal to 0, afifth value628 equal to 0, asixth value630 equal to 2, and aseventh value632 equal to 10. The value of zero is merely exemplary and could be any binary value. After the encoding step130 (FIG. 1), the corresponding encoded data140 (FIG. 1) would be compressed down to three bytes ofbinary code640 comprising afirst byte650, asecond byte653, and athird byte656 each containing a merged value and count, (651,652), (654,655), and (657,658), respectively. Thefirst data651 has a binary value of “00000” which equals the exemplary repeated decimal value. Thefirst count652 has a binary value “101” which equals decimal five representing the run-length of the repeating value in the first five of the decimal values610. Thesecond data654 has a binary value of “00010” which equals the non-repeated decimal value two. Thesecond count655 has a value of 1. Thethird data657 has a binary value of “01010” which equals the non-repeated decimal value ten. Thethird count658 has a value of 1.
• FIG. 7—RHN Codes and Encoded Stream
• FIG. 7 shows the same series of decimal values610 (FIG. 6) comprising thefirst value620 equal to decimal 0, thesecond value622 equal to 0, thethird value624 equal to 0, thefourth value626 equal to 0, the fifth value728 equal to 0, the sixth value730 equal to 2, and the seventh value732 equal to 10. After encoding by RHN, the corresponding encoded data140 (FIG. 1) would be compressed down to four bytes of RHNbinary code740.
• The embodiment of the present invention shown inFIG. 6 only requires three bytes to encode the same data. In this example, the present invention is 25% better than the RHN format.
• FIGS.8A and8—ZLN Formats
• The ZLN method of the present invention provides for variable formats. The values ofN300,U301, andW302 can be dynamically changed between frames. For ease of communication a format is named with the prefix “ZL” and a digit representing the value of N. For example, “ZL5” refers to a format where bit width of N is equal to 5. There are multiple values of U depending of the W. To also specify the bit width of U a hyphen and a number can be appended. For example, “ZL5-13” represents a format where N=5 and U=
• 13. “ZL5-3” is a common format and may be imprecisely referred to as “ZL5.”
• FIG. 8A shows a number of formats with adjacent labels:ZL3803,ZM4804,ZL5805,ZL8808,ZL9809, andZL12812. Data bits are represented by “D,” and count bits are represented by “C”.
• FIG. 8B shows how the most significant 3 bits of each color component (216,214, and212 ofFIG. 2B) are extracted and formatted in ZL9-7C format (the “C” append indicates that the color is preserved). With three red bits represented by “R”, three green bits represented “G” and three blue bits represented by “B”.
• Decoding
• To decode the compressed array, the decoder has a decode table that corresponds with the encode table. For W*4 bit color pixels, the decode table contains the appropriate alpha, red, green, and blue values. For W*3 bit color pixels, the alpha value is not used. The compressed array is processed W bits at a time as X. The repeat count, C, is extracted from X by masking off the data value (C=X & (((2**N)−1)<<U)). The encoded value, E, is extracted from X by masking off the count (E=X & ((2**U)−1)). The encoded value, E may be used to index into the decryption. The decoded pixels are placed in a reconstructed image and repeated C times. Each element of the compressed array, A, is processed until its entire length, L, has been processed.
• FIG. 9—Decode Flowchart
• FIG. 9 illustrates thedecode flowchart920 which presents the details of the decryption embodiment of the decode step160 (FIG. 1) and the image reconstitution step180 (FIG. 1).
• The decoding begins at adecode entry900. In a “decode initialization”step901, a repeat counter C is set to one, an encoded length L is set to the value obtained with the encoded data140 (FIG. 1), and an index I is set to 0. Next, a “get code”step902 obtains a signed byte X from the encoded data140 (FIG. 1) array A. The index I is incremented. The count (for example the 3-bit count380 as shown inFIG. 3B) is extracted from X by masking off the data bits and placed in the repeat counter C (C=X & ((2**N)−1<<U). The value of E is extracted from X by masking off the count bits (E=X & (2**U)−1). In practice, the count mask and value mask can be pre-computed with the following two lines of code in the C programming language:
• valueMask=−1<<U;
• countMask=˜valueMask;
• In this illustrative decryption embodiment of the present invention, flow goes to a “decode lookup”step908 where the value of E is used to index into the decode table1110 (FIG. 11) to obtain a pixel value V. In the other embodiments where E is not encrypted, E is used as V and step908 is bypassed. Flow continues to a “check zero count”909 decision.
• The909 decision always fails the first time ensuring that aplace pixel step910 is executed. Theplace pixel step910 places the pixel value V in the next location of the decompressed image and decrements the repeat counter C and returns to the909 decision. The pixel value V is placed repeatedly until C decrements to zero. Then the909 decision branches flow to a “reset counter”step914. Atstep914 the repeat counter is reset to 1.
• Flow continues to the “check length”916 decision where the index I is compared to the encoded length L to determine if there are more codes to be processed. If I is less than L flow returns to step902, otherwise the decode process terminates at a “decode exit”918.
• The entire decode process is repeated for each encoded frame image.
• FIG. 10—Interpolation
• FIG. 10 illustrates interpolation when twoadjacent pixels1010 and1012 and two subsequent rowadjacent pixels1014 and1016 are stretched to insert a new row and column of pixels.
• Pixels1052,1054,1056,1058 and1060 are inserted due to the enlargement of the image. Their values are calculated by averaging the values of the two pixels above and below or to the left or the right of the new pixel. A preferred sequence is calculation of:
• 1.1052 between1010 and1012
• 2.1054 between1010 and1014
• 3.1058 between1012 and1016
• 4.1056 between1054 and1058
• Pixel1060 can be calculated on the interpolation for the subsequent row.
• FIG. 11—Encryption
• By using corresponding encoding and decoding tables the data can be encrypted and decrypted without using actual values. Encryption provides a level of security for the encodeddata140 while in storage or transit.
• FIG. 11 shows an example of an encryption table1100, where N is 3 and W is 8, and a decryption table1110, where N is 3 and U is 5.
• The encode table1100 is 2 the power of N in length. If the target color image format is W*4 bit color, then the decode table1110 has W bits for alpha, red, green, and blue each, respectively. If the target color image format is W*3 bit color, then the alpha value is not used. If the image is W bit grayscale then only the grayscale value is used to create the decompressed and decoded image.
• The corresponding table elements are mapped to each other. For example, 0 could encode to 22 as long as the 22^ndelement of the decode table returns (θxff<<24|θ<<16 |θ<<8 |θ).
• When these versions of the tables are used, the encode and decode processes and their speed of execution are substantially the same but the encoded data140 (FIG. 1) becomes a cipher and has a higher level of security. It should be recognized by one with ordinarily skill in the art that there are other embodiments of the present invention with different encryption/decryption table rearrangements.
• FIGS.12A through12D—Compression and Decompression Devices
• FIGS. 12A and 12B show devices for compressing and decompressing, respectively, a stream video frames.
• FIG. 12A shows avideo signal1215 being compressed and encoded by acompressor1210 to form an encodeddata stream1235, which is sent to an I/O device1240. Thevideo signal1215 comprises a series ofvideo frames1200, shown asfirst video frame1205a,second video frame1205b, . . . throughnth video frame1205n. The encodeddata stream1235 comprises a series of encodeddata1220, shown as first encoded data1225a, second encoded data1225b, . . . , through nth encoded data1225n.
• FIG. 12B shows an input encodeddata stream1245 being received from an I/O device1240, and then, decoded and decompressed by adecompressor1250 to form avideo sequence1270. The input encodeddata stream1245 comprises received encodeddata1238, shown as first received encodeddata1230a, second received encodeddata1230b, . . . , through nth received encodeddata1230n. Thevideo sequence1270 comprises a series of decodedvideo frames1268, shown as first decodedvideo frame1260a, second decodedvideo frame1260b, . . . , through nth decodedvideo frame1260n.
• FIG. 12C shows an embodiment where the I/O device1240 ofFIGS. 12A and 12B is astorage medium1280. The encodeddata stream1235 from thecompressor1210 is stored in thestorage medium1280. Thestorage medium1280 provides the input encodeddata stream1245 as input to thedecompressor1250.
• FIG. 12D shows an embodiment where the I/O device1240 ofFIGS. 12A and 12B is acommunications channel1290. The encodeddata stream1235 from thecompressor1210 is transmitted over thecommunications channel1290. Thecommunications channel1290 provides the input encodeddata stream1245 as input to thedecompressor1250.
• FIGS.13A through13J—Compressor Details, Encoding Circuit, and Bitwise Pixel Sub-Samplers
• FIG. 13A shows details of an embodiment of thecompressor1210, which comprises avideo digitizer1310, avideo memory1330, anencoding circuit1350, and encodeddata1370. Eachvideo frame1205 in the series ofvideo frames1200 is digitized by thevideo digitizer1310 and stored alongpath1320 in thevideo memory1330. Theencoding circuit1350 access the digitized video frame viapath1340 and outputs the encodeddata1370 alongpath1360. The encodeddata1225 corresponding to eachvideo frame1205 is then output from thecompressor1210.
• FIG. 13B shows further details of an embodiment of theencoding circuit1350. Apixel sub-sampler1380 scans each pixel from the digitized video frame in thevideo memory1330. Apixel index1332 is used to drive ascan1331 signal to select each pixel from the video memory, in a predetermined sequence. A novel aspect of the present invention is that the compression method can be accomplished with a single scan of the video memory for each frame. Thepixel sub-sampler1380 selects a predetermined number of bits from each pixel and outputs the data value alongpath1385. Alternatively, thepixel sub-sampler1380 encodes the sub-sampled data by using a lookup table similar toFIG. 11A.Different pixel sub-samplers1380 will be discussed in reference toFIGS. 13C through 13J. The data/count1390 unit increments the count each time the output of thepixel sub-sampler1380 is the same; otherwise, when the output of thepixel sub-sampler1380 is different (or when the counter reaches the maximum count value, the data and count are combined as a code and output alongpath1395 to the encodeddata1225 for the frame currently in thevideo memory1330. The location of the code in the encodeddata1225 is selected by thecode index1392 signal.
• FIG. 13C shows further details of ageneric pixel sub-sampler1380. When a pixel is scanned from video memory alongpath1340, it has an original pixel bit width, P.A pixel extractor1382 extracts a subset of bits from each pixel with a value bit width, V, alongvalue path1383. The value bit width V is less than the pixel bit widthP. A coder1384 takes the V bits from thepixel path1383 and outputs a code with an encoded bit width, E, as the data value alongpath1385. One embodiment of the coder is a null coder, or pass-through coder. Another embodiment of the coder uses an encryption table to encrypt the data value as an encrypted data value.
• FIGS. 13D through 13J show embodiments of pixel sub-samplers.
• FIG. 13D illustrates a 24 to 5 bit sub-sampler1380a, where the pixel bit width, P, is 24; the value bit width, V, output from thepixel extractor1382 is 8 (seeFIG. 2H); and the encoded bit width, E, output from thecoder1384 is 5. In this embodiment, the extracted 8 bits could be any component of the grayscale (e.g.FIG. 2G) or thehigh order 8 bits of the 24-bit value.
• FIG. 13E illustrates a 24-bit RGB to 5 bit sub-sampler1380b, where the pixel bit width, P, is 24 divided into 8 bits of red, green, and blue (RGB, seeFIG. 2B); the value bit width, V, output from thepixel extractor1382 is 8; and the encoded bit width, E, output from thecoder1384 is 5. In this embodiment, the extracted 8 bits could be an average (e.g.FIG. 2C) or one of the colors (e.g.FIGS. 2D, 2E, or2F).
• FIG. 13F illustrates a 32-bit RGB to 5 bit sub-sampler1380c, where the pixel bit width, P, is 32 divided into 8 bits of red, green, blue, and alpha (seeFIG. 2A); the value bit width, V, output from thepixel extractor1382 is 8; and the encoded bit width, E, output from thecoder1384 is 5. In this embodiment, the extracted 8 bits could be an average (e.g.FIG. 2C) or one of the colors (e.g.FIGS. 2D, 2E, or2F).
• FIG. 13G illustrates a color 9-bit sub-sampler1380d, where the pixel bit width, P, is 24 divided into 8 bits each of red, green, and blue; the value bit width, V, output from thepixel extractors1382 is 9; and the encoded bit width, E, output from thecoder1384 is 9. In this embodiment, thehigh order 3 bits of each color component are selected (e.g. ZL9C shownFIG. 8B).
• FIG. 13H illustrates aYUV sub-sampler1380e, where the pixel bit width, P, is 24 divided into 8 bits for each of YUV; the value bit width, V, output from thepixel extractors1382 is 8; and the encoded bit width, E, output from thecoder1384 is 5. In this embodiment, four bits of the Y value is extracted and 2 bits of each of the U and V values are extracted. This 8 bit value is further coded as a 5 bit value.
• FIG. 13I illustrates a 36-bit RGB to 24 bit sub-sampler1380f, where the pixel bit width, P, is 36 divided into 12 bits each of red, green, and blue; the value bit width, V, output from thepixel extractors1382 is 24; and the encoded bit width, E, output from thecoder1384 is also24. In this embodiment, thehigh order 8 bits of each 12-bit color component are selected.
• FIG. 13J illustrates a 15-bit sub-sampler1380g, where the pixel bit width, P, is 24 divided into 8 bits from each color component; the value bit width, V, output from thepixel extractor1382 is 15; and the encoded bit width, E, output from thecoder1384 is 15. In this embodiment, thehigh order 5 bits of each 8-bit color component are selected.
• FIGS.14A through14C—Variable Selection of Bit-wise Sub-sampling
• FIGS. 14A through 14C shows embodiments of a device for variably altering the number of bits.
• FIG. 14A illustrates 24-bit to variable bit sub-sampler1400. When a pixel is scanned from video memory alongpath1340, it has an original pixel bit width, P, equal to 24 bits. These 24 bits are passed as input to a number of sub-samplers. The variable number of bits is selected by a number ofbits selector1410 as indicated by a number ofbits indicator1420 and outputs a code with an variable encoded bit width, E, as the data value alongpath1385. A user atremote receiver1610 sets the number of bits indicator1420 (see discussion regardingFIGS. 16A and 16B). The variable bit sub-sampler comprises a generic 3-bit sub-sampler1401, a generic 4-bit sub-sampler1402, generic 8-bit sub-sampler1403, and generic 10-bit sub-sampler1404 which are embodiments of the generic sub-sampler shown inFIG. 13C with specific values for E. The variable bit sub-sampler further comprises nested sub-samplers: the 24 to 5 bit sub-sampler1380aofFIG. 13D, the1380dofFIG. 13G, and the 15-bit sub-sampler1380gofFIG. 13J. This is illustrative of the types of bit sub-samplers that can be variably selected.
• Likewise,FIG. 14B illustrates a 36-bit to variable bit sub-sampler1430, where P is 36 and the number of bit that can be selected are 12, 15, or 24, respectively.
• FIG. 14C shows that the 24 bit to variable bit sub-sampler1400 ofFIG. 14A and the 36-bit to variable bit sub-sampler1430 ofFIG. 14B can be further combined to form at 24/36 bit variable bit sub-sampler1440 where asecond selector1450 is used to selected either the 24 bit inputs or the 36 bit inputs usingselection logic1460 that also receives the number ofbits indicator1420. Aselection signal1470 enables either the output of 24-bit to variable bit sub-sampler1400 or the output of 36-bit to variable bit sub-sampler1430.Sub-samplers1400 and1430 both receive the number ofbits indicator1420 as shown inFIG. 14A andFIG. 14B. In this way any number of bits may reasonably be selected from either a 36 or 24-bit pixel bit width.
• FIG. 15—Decompressor Elements
• FIG. 15 shows details of an embodiment of thedecompressor1250, which comprises adecoding circuit1510 which inputs received encodeddata1230 and outputs decodedpixel values1520 to animage memory1540. Adecoder pixel index1530 selects the location in theimage memory1540 to store the decoded pixels values1520. Theimage memory1540 delivers each decodedvideo frame1260 to the video display.
• FIGS.16A and16B—Parameters Altered by a Remote Receiver
• FIG. 16A shows a system for setting width, height, frame rate, brightness, and contrast in atransmitter1600 which are variably altered by areceiver1610. The receiver sends commands to thetransmitter1600 via settingcontrol path1615. The commands alter thetransmitter settings1660.
• Thesettings1660 includebrightness1661,contrast1662,height1663,width1664, andframe rate1665. Thebrightness1661,contrast1662,height1663, andwidth1664 setting alter the attributes of each frame as it is digitized in aframe sub-sampler1620. Thebrightness1661 andcontrast1662 settings alter the video digitizer1310 (FIG. 13A) as it senses the video frame. Theheight1663 and1664 allow for optionally selecting a subset area of each frame; this is area sub-sampling. Alternatively,height1663 and1664 allow for optionally selecting a subset of pixels from an array of pixels that make up a single frame, by skipping pixels in a row or by skipping rows; this is image sub-sampling. Theframe rate1665 setting alters theframe selector1670 which drives the frameselect indicator1675 to optionally sub-sample frames from a sequence of video frames; this is frame sub-sampling.
• Theframe sub-sampler1620 outputs a selectedframe1630 alongpath1621. Thetransmitter pixel sub-sampler1640 scans the selectedframe1630 getting each pixel fromframe1632 and outputs data values alongpath1642 to arun length encoder1650. The encodeddata stream1235 is then transmitted to theremote receiver1610.
• FIG. 16B shows additional elements of a system for setting the number of pixel bits in analternate transmitter1690 which is variably altered by areceiver1610. The receiver sends commands to thetransmitter1600 via settingcontrol path1615. The commands alter thetransmitter settings1660. The settings include a number of pixel bits setting1680 which affect the number of bits selected by thetransmitter pixel sub-sampler1640. Thepixel sub-sampler1640 could be any pixel sub-sampler, for example, seeFIGS. 13C through 13J and14A through14C. Thetransmitter pixel sub-sampler1640 scans the selected frame1630 (as inFIG. 16A) getting each pixel fromframe1632 and outputs data values alongpath1642 to arun length encoder1650. The encodeddata stream1235 is then transmitted to theremote receiver1610.
• These embodiments illustrate the novel feature of the present invention of allowing a user at aremote receiver1610 to control aspects of thetransmitter1600 or1690 from a remote location, including brightness, contrast, frame dimensions, frame rate, image area, and the type of compression used.
• FIG. 17—Further Lossless Compression Step
• FIG. 17 shows a lossless compression step for further compressing an encoded data buffer. After a run-length encoding step1700 in the transmitter, a run-length encodedoutput1710 can be further processed with a furtherlossless compression step1720 resulting in furtherlossless compression output1730. The furtherlossless compression step1720 could be implemented as a variable length coding, arithmetic coding, or other compression step known in the art.
• FIG. 18—Image Stretching
• FIG. 18 shows images being enlarged by stretching. Anunstretched frame1800 is stretched during stretchingstep1820 resulting in anenlarged image1810. When a frame is image sub-sampled or area sub-sampled, the remaining data can be stretched to fill the full display area on thereceiver1610. This results in an interpolated image or magnified image, respectively.
• FIGS.19A through19C—Handheld Video Transmission Networks
• FIGS. 19A through 19C show various network configuration comprising handheld video devices.
• FIG. 19A illustrates an exemplary network1910 comprising afirst node1920a, asecond node1920b, and anoptional reflector1930. The network1910 is shown as awired network1910a. Thefirst node1920ais displaying afirst video1901aof a man. Thesecond node1920bis displaying asecond video1902aof a woman. This illustrates a videoconference between the man at thesecond node1920band the woman at thefirst node1920a. In the first mode of operation, the respective videos are transmitted over a point-to-point transmission1940 path between the two nodes over the network1910. In another mode of operation each of the videos is transmitted to the reflector where both videos are displayed as first reflectedvideo1901band second reflectedvideo1902b. Thesecond video1902aoriginates at thefirst node1920ais transmitted to the reflector over firstindirect path1942. Thesecond video1901aoriginates at thesecond node1920bis transmitted to the reflector over secondindirect path1944. The reflector then retransmits the two videos to the respective display nodes,1920aand1920b, over the indirect paths. In other configurations, the reflector would also transmit the combined video to other nodes participating in the videoconference.
• FIG. 19B shows an example of three nodes,third node1920c,fourth node1920d, andfifth node1920ein a wireless network. The wireless connections are shown as waves. The three nodes operate in the same manner as the three nodes inFIG. 19A.
• FIG. 19C shows an example of a combinednetwork1910cwhere five nodes are connect in a network comprised of both awired network1910aand awireless network1910b. Any of the five nodes could transmit video to any of the other nodes in the combined network. Any node, for examplethird node1920cas shown, could act as areflector1930.
• In another embodiment of the present invention, any node could act as a video server and transmit pre-recorded video to one or more other nodes.
• These illustrations are exemplary. In practice, combined networks could consist of any number of nodes. Any of the nodes in the network could be a handheld video device.
• FIGS.20A through20D—Handheld Video Devices
• FIGS. 20A through 20D show various embodiments of handheld video devices.
• FIG. 20A shows a handheld video transmitter comprising avideo source2052, avideo transmitter2054, andvideo storage2056.
• FIG. 20B shows two handheld video devices in communication over either awireless connection2050 or awired connection2051.
• Afirst handheld device2010 comprises adisplay2012,manual controls2014, awireless port2016, and a firstwired connection2051a. While either thewireless port2016 or thewired connection2051acould be present, only one of the two would be necessary to receive video from or transmit video to other nodes in the network1910. In this example, the first handheld device is shown as an iPod-type device with an internal hard disk drive. Thefirst handheld device2010 further comprises aheadphone2020, connected via a speaker/microphone cable2024, and acamera2030, connected via acamera cable2034. Theheadphone2020 comprises aright speaker2021, amicrophone2022, and aleft speaker2023. Thecamera2030 has alens2032 and internal circuitry that converts the light that passes through thelens2032 into digital video data.
• In the best mode for this embodiment, the iPod-type device is implemented using a standard Apple iPod (enhanced with an audio input for the microphone and, optionally, with a wireless port, and appropriate software), and thecamera2030 is implemented using an iBot Firewire camera manufactured by Orange Micro, a lower performing Connectix USB camera, or similar camera. Alternatively, if the iPod-type device were only used of viewing video, the Apple iPod could be used without hardware modification. In another variation, the microphone could be build into the camera (not shown) instead of the headphones.
• Asecond handheld device2040 comprises asecond display2012b, asecond wireless port2016b, and a secondwired connection2051b. While either thewireless port2016bor thewired connection2051bcould be present, only one of the two would be necessary to receive video from or transmit video to other nodes in the network1910. In this example, the second handheld device is shown as a device with a touch screen. Thesecond handheld device2040 further comprises a right built-inspeaker2021b, a built-inmicrophone2022b, a left built-inspeaker2023b, and a built-incamera2030bwithlens2032.
• The configuration of thesecond handheld device2040 has the advantage of eliminating the cables for the external headphone and camera of thefirst handheld device2010 by having all elements built-in.
• These two devices are exemplary. A two-device handheld videoconferencing network could have two identical handheld devices, such as thefirst handheld device2010. Further, a single device with a camera (as shown) could transmit video for display on any number of hand held devices that do not have cameras or microphones.
• FIG. 20C illustrates anintegrated handheld device2060 comprising aniPod type device2010, an A/V module2062 and anoptional wireless module2064. TheiPod type device2010 comprisesdisplay2012, controls2014, and awired connection2051. The A/V module2062 comprises a rightintegrated speaker2021c, anintegrated microphone2022c, a leftintegrated speaker2023c, and aintegrated camera2030cwithlens2032. The A/V module2062 could be manufactured and marketed separately (as shown) as an add-on module for standard iPods, or could be incorporated into the iPod packaging as an enhanced iPod-type device. Thewireless module2064 comprises anintegrated wireless port2016c. Thewireless module2064 also could be manufactured and marketed separately (as shown) as an add-on module for standard iPods, or could be incorporated into the iPod packaging as an enhanced iPod-type device.
• The configuration of theintegrated handheld device2060 has the advantage of eliminating the cables for the external headphone and camera of thefirst handheld device2010 by having all elements integrated into removably attached modules that form a single unit when attached. The user can configure the standard iPod based on the user's intended use. If only a wireless connection is needed, only thewireless module2064 can be attached to the iPod; in this configuration video can be received and displayed but not transmitted. If only video transmission is necessary and a wired connection is convenient, thewireless module2064 can be omitted. Either configuration provides a single integrated unit that can be carried in the user's pocket and can store and display videos.
• FIG. 20D illustrates an cellularintegrated device2070 comprisingphone display2012d, phone controls2014d(including a number keypad), acellular port2016d, aright phone speaker2021d, aphone earphone2021e,phone microphone2022d, leftphone speaker2023d, and aphone camera2030dwithlens2032.
• Any of the handheld devices shown inFIGS. 20A through 20D could be nodes in video transmission networks, such as those shown inFIGS. 12D and 19A through19C. Each transmitting device preferably would include acompressor1210 as shown inFIGS. 12A and 12D. Each receiving device preferably would include adecompressor1250 as shown inFIGS. 12B and 12D. Thecompressor1210 anddecompressor1250 preferably would implement one or more embodiments of the compression methods discussed above.
• FIGS.21A through21C—Handheld Video Devices with Graphical Zoom Control
• FIGS. 21A through 21C show exemplary handheld video devices comprising graphical zoom controls.
• A graphical user interface (GUI) graphically corresponds to a video display window2110 through which a single image or a stream of video frames is displayed. The GUI and the video display window2110 are displayed on a display2012 (or2012bor2012d). The GUI includes azoom control2100 having an inner region2102 positioned within an outer region2106. Thezoom control2100 is a graphical way for the user of a remote receiver1610 (seeFIGS. 16A and 16B) to send remote control commands to set the parameters of a video transmitter (1600 or1690) for control the area of the video to be compressed and transmitted.
• FIG. 21A shows an embodiment of the iPod-type handheld device2010 ofFIG. 20C displaying azoom control2100 having aninner region2102apositioned within anouter region2106a. The zoomed video image is show invideo display window2110a. In this embodiment thezoom control2100 is displayed on top of thevideo display window2110a. The size and position of theinner region2102arelative to the outer region1206ashows the user which portion of the original video is being received and magnified. Only the selected portion of the original video (in this example, the hair and top of the face) needs to be transmitted in full resolution or high quality. A low resolution, or thumbnail version of the original video frame is optionally displayed in theouter region2106a. The thumbnail can be updated at a rate slower than the frame rate of the magnified video, such as once or twice a second. Themagnification factor2104ashows thetext 2× showing that the portion being displayed in thevideo display window2110ais being displayed at twice the size.
• FIG. 21B shows an embodiment of the cellularintegrated device2070 ofFIG. 20D displaying azoom control2100 having an secondinner region2102bpositioned within an secondouter region2106b. The zoomed video image is shown in alternatevideo display window2110b. In this embodiment, thezoom control2100 is displayed outside and below the alternatevideo display window2110b. The size and position of the secondinner region2102brelative to the second outer region1206bshows the user which portion of the original video is being received and magnified. Only the selected portion of the original video (in this example, the lower face and tie) needs to be transmitted in full resolution or high quality. A low resolution, or thumbnail version of the original video frame is optionally displayed in the secondouter region2106b. Thesecond magnification factor2104bshows thetext 2× showing that the portion being displayed in the alternatevideo display window2110bis being displayed at twice the size.
• FIG. 21C shows an embodiment of thesecond handheld device2040 ofFIG. 20B displaying azoom control2100 having an thirdinner region2102cpositioned within an thirdouter region2106c. The zoomed video image is shown in avideo display window2110ashown filling the second display2112b. In this embodiment, thezoom control2100 is displayed over thevideo display window2110a. The size and position of the thirdinner region2102crelative to the third outer region1206cshows the user which portion of the original video is being received and magnified. Only the selected portion of the original video (in this example, the right shoulder of the woman) needs to be transmitted in full resolution or high quality. A low resolution, or thumbnail version of the original video frame is optionally displayed in the thirdouter region2106c. Thethird magnification factor2104cshows thetext 3× showing that the portion being displayed in thevideo display window2110ais being displayed at three times the size. In this embodiment the controls (similar in function to controls2014) are incorporated into a touch screen of thesecond display2012b. The user enters zoom in, zoom out, and pan commands by tapping the thirdinner region2102cor the third outer region2106, or by selecting and dragging the outline of the thirdinner region2102c.
• Operation of Graphical Zoom Controls
• A user controls aspects and changes parameters of the image displayed within the video display window2110 using thecontrols2014 to enter input commands within thezoom control2100 by selecting appropriate parts of the controls2104 (or regions of thezoom control2100 on a touch screen or with a pointing device). Thecontrols2014 can be a touch screen, touch pad, iPod-like scroll pad, remote control or other device, depending on the configuration of the handheld device.
• The size of the inner region2102 relative to the outer region2106 represents the magnification of the portion of the image being displayed within the video display window2110. A magnification factor104 representing the current magnification of the image being displayed within the video display window2110 from the original image is displayed within the inner region2102. The magnification of the image being displayed is increased by tapping within the inner region2102, or while in zoom control mode, pressing the “zoom in” button on a iPod-type control2104 orcell phone control2014d. As the magnification is thus increased, the size of the inner region2102 is decreased appropriately relative to the outer region2106 and the magnification factor104 is appropriately incremented. The magnification of the image being displayed is decreased by tapping outside of the inner region but inside of the outer region, or while in zoom control mode clicking the “zoom out” button on a iPod-type control2104 orcell phone control2014d. As the magnification is thus decreased, the size of the inner region102 is increased appropriately relative to the outer region2106 and the magnification factor104 is appropriately decremented.
• The position of the inner region2102 within the outer region2106 represents the portion of the entire original image being displayed within the video display window2110. The portion of the image being displayed within the video display window2110 is changed by moving the inner region2102 to the desired position within the outer region2106 using the touch screen, a pointing device, or thecontrols2014 or2014d. As the position of the inner region2102 changes within the outer region2106, the portion of the image displayed within the video display window2110 changes appropriately.
• Thedisplay2012 including the video display window2110 and a graphical user interface including thezoom control2100, according to the present invention. Thezoom control2100 of the present invention preferably includes two regions2102 and2106. The outer region2106 forms the outer edge of thezoom control2100 and represents the entire available original image. The inner region2102, is included and positioned within the outer region2106 and represents a region of interest of the original image currently being displayed within the video display window2110. Within the inner region2102, a magnification factor104 is optionally displayed, representing the current magnification being applied to the image displayed within the video display window2110.
• The magnification factor104 is changed by using the touch screen or controls2014 (or2014d) to zoom in or zoom out. By zooming in a number of times, the inner region102 becomes continually smaller in size and the magnification factor104 is incremented a number of times equal to the number of times that the control zoomed in.
• A user zooms out on a specific portion of the image to decrease the magnification factor104; the inner region102 becomes appropriately larger in size and the magnification factor104 is decremented. By zooming out a number of times, the inner region102 becomes increasingly larger with each zoom out and the magnification factor104 is decremented a number of times equal to the number of times the user zooms out, until the magnification factor is equal to 1.
• The inner region2102 also has a pan or positional feature within the outer region2106, such that the position of the inner region2102 within the outer region2106 represents the portion of the entire original image that is being displayed within the video display window2110. The position of the inner region2102 is changed within the outer region2106 by using the touch screen, a pointing device, or controls2014 to move the inner region2102 to the desired position within the outer region2106. Accordingly, the inner region2102 graphically represents what portion of the entire image is currently being displayed within the video display window2110 and what magnification factor104 is currently being used to make this selected portion of the original image fit within the video display window2110.
• Advantages
• Video Coverage of Remote Events
• The present invention will allow low cost, portable, video transmission of events of interest whenever and wherever they happen. These handheld wireless video transmitters will be able to provide news coverage of wars, natural disasters, terrorist attacks, traffic and criminal activities in a way that has never before been possible.
• Improved Continuous Communication
• The present invention will enabled enhanced personal communication between friends, family, and co-workers in ways never before possible.
• Improved Entertainment and Education
• The present invention will enabled the transmission of video-based entertainment and education in ways never before possible. User will be able to use pocket-sized, handheld device to watch video that are downloaded from a global media exchange, streamed from a video server, or transmitted live from a performance, classroom, laboratory, or field experience.
• Improved Healthcare
• The present invention would enable a physician or medical specialist to receive medical quality video any time in any location. For example, a critical emergency room ultrasound study could be monitored while it is being performed by less skilled emergency room personnel ensuring that the best medical image is acquired. A rapid diagnosis can be made and the results of a study can be verbally dictated for immediate transcription and use within the hospital.
• Further, the present invention could be used to transmit medical quality video from a remote, rural location, including a battle ground. It could also be used to transmit guidance and advice from an expert physician into a remote, rural location.
• Thus, the present invention can improve medical care, reduce the turnaround for analysis of medical studies, reduce the turnaround for surgery, and provide medical professionals with continuous access to medical quality imaging.
• Noise Filtering and Image Enhancement
• The removal of the least significant bits of pixel values results in high quality decompressed images when the original image is generated by an electronic sensing device, such as an ultrasound machine, which is generating only a certain number of bits of grayscale resolution. By variably altering the number of most significant bits, various filters can be implemented to enhance the image quality. Such a noise filter can be beneficial when the image is generated by an imaging technology such as radar, ultrasound, x-ray, magnetic resonance, or similar technology. Variations can be made to enhance the perceived quality of the decompressed image. Therefore, altering the number of data bits selected and altering the width of the repeat count is anticipated by this invention and specific values in the examples should not be construed as limiting the scope of this invention.
• Dynamic Variable Formats
• While a video stream is being viewed a viewer on the decoding end of the transmission can vary the settings for the compressor. Different tradeoffs between image spatial and temporal quality can be made. As the contents of the video signal change an appropriate format can be selected. Control signals can be sent back to the compressor via a communications link.
• Execution Speed
• The preferred embodiment of this invention uses a number of techniques to reduce the time required to compress and decompress the data.
• The methods require only a single sequential pass through the data. Both the compression steps100 and the decompression steps150 access a pixel once and perform all calculations.
• When selecting the filteredpixel value299, the preferred embodiment selects the low order byte from the 32bit pixel value200 or the 24bit pixel value210 so that an additional shift operation or addressing operation is avoided.
• The shift operation is a fast and efficient way to convert a byte or word to the filteredpixel value299.
• General Purpose
• The lossless compression of the sampled data achieved by the preferred embodiment of the present invention results in high quality video streams that have general purpose application in a number of areas including, without limitation, video conferencing, surveillance, manufacturing, rich media advertising, and other forms of video transmission, storage, and processing.
• Lossless Nature/No Artifacts
• Once the analog signal is sub-sampled and filtered to select a filtered pixel value that eliminates some of the real world defects, the methods of the present invention compress and decompress the data with no irreversible data loss. Unlike JPEG and MPEG, the decompressed image never suffers from artificially induced blocking or smearing or other artifacts that are result of the lossy compression algorithm itself. As a result even a small sub-sample of the image remains clear and true to the perceived quality of the original image.
• Superior Features over RHN Format
• When compared against the RHN format, the format and methods of the present invention provide a number of advantages, including, but not limited to, faster speed and smaller size of encoded data, better performance for both medical and typical video images, and a typically closer representation of the original video signal.
CONCLUSION, RAMIFICATION, AND SCOPE
• Accordingly, the reader will see that handheld wireless devices are used to receive and display high quality video. The video can be displayed as it is received live and a graphical zoom control can be used to dynamically control the area of the source image that is to be transmitted in full resolution. In other embodiments, a handheld wireless device captures the video with an attached video camera and microphone and the device transmits the video images live as they are captured. A single handheld wireless video transmitter can transmit to multiple handheld wireless receivers. A plurality of handheld wireless video devices which capture, transmit, receive, and display video over a network are used for mobile video conferencing. In other embodiments the video data is transferred as a video file or streamed from a video server contain pre-recorded video files.
• Further the compression and decompression steps of the present invention provides a means of digitally compressing a video signal in real time, communicating the encoded data stream over a transmission channel, and decoding each frame and displaying the decompressed video frames in real time.
• Furthermore, the present invention has additional advantages in that:
• 1. it enables live video transmission and display on pocket-sized handheld devices;
• 2. it enables wireless videoconferencing with portable, handheld video devices;
• 3. it provides an iPod-type device which is able to display high quality color video;
• 4. it provides an iPod-type device which is able to be used as a wireless video transmitter or receiver.
• 5. it enables video coverage of remote events or catastrophic events;
• 6. it improves interpersonal communication, productivity, and effectiveness;
• 7. it improves education;
• 8. it improves entertainment;
• 9. it improves and expands healthcare at lower costs;
• 10. it provides a means of filtering real world defects from the video image and enhancing the image quality;
• 11. it allows for execution of both the compression and decompression steps using software running on commonly available computers without special compression or decompression hardware;
• 12. it provides decompressed images that have high spatial quality that are not distorted by artifacts of the compression algorithms being used;
• 13. it provides a variably scalable means of video compression; and
• 14. it provides a means for reducing the space required in a storage medium.
• Although the descriptions above contain many specifics, these should not be construed as limiting the scope of the invention but as merely providing illustrations of some of the preferred embodiments of this invention. For example, the physical layout, cable type, connectors, packaging, and location of the video display or video camera can all be altered without affecting the basic elements of the claimed embodiments. Further, bit ordering can be altered and the same relative operation, relative performance, and relative perceived image quality will result. Also, these processes can each be implemented as a hardware apparatus that will improve the performance significantly.
• Thus the scope of the invention should be determined by the appended claims and their legal equivalents, and not solely by the examples given.

Claims (26)

1-25. (canceled)

26. A handheld video receiver, in a system for transmission and display of video and other media, the system comprising a digital network, a global media exchange and a plurality of devices, each connected to the digital network, wherein at least one device is capable of transmitting a plurality of compressed video frames which make up a video stream, said handheld video receiver comprising:

a) a color display having a predetermined display width and display height and having a touch screen for receiving input from a user, wherein the display width is greater than or equal to 320 pixels and the display height is greater than or equal to 480 pixels,

b) an audio output,

c) a program memory, for storing computer programs,

d) a processor, for executing computer programs from the program memory,

e) a data memory, for storing data,

f) a user interface on the touch screen comprising controls for interfacing with the executing computer programs, the controls comprising:

i) a zoom interface for issuing zoom in, zoom out, and pan commands, wherein the magnification and position of the portion of the displayed media are changed in response to one or more of the zoom in, zoom out, and pan commands, and

ii) mobile telephone controls for entering numeric telephone numbers and for controlling mobile telephone calls,

g) a wireless network interface for connecting to the digital network via wireless a communications channel, wherein the handheld video receiver maintains connection to the digital network when being moved from one location to another location or while being freely carried by the user and wherein mobile telephone calls are transmitted via the wireless network interface,

h) a wired network interface for connecting to the digital network via a wired communications channel,

i) a decompressor, cooperating with the processor,

j) a video camera for capturing images in real-time,

k) a microphone for capturing an audio portion in real-time, and

l) a compressor configured to compress captured images,

wherein the handheld video receiver has a weight of less than about six ounces,

wherein the handheld device has physical dimension allowing it to be held substantially in one hand of the user,

wherein the user interface is operated by one or more fingers of another hand of the user,

wherein each of the communications channels from the digital network to the handheld video receiever has a predetermined bandwidth such that the maximum sustainable bandwidth measured in millions of bits per second,

wherein the compressed video frames are received over one of the communications channel,

wherein the decompressor decompresses the compressed video frames in real time,

wherein the video stream is stored in a video file,

wherein the data memory is configured to store a plurality of video files,

wherein at least a portion of the decompressed video frames is displayed on the color display in real time,

wherein the audio portion plays on the audio output,

wherein the compressed captured images are transmitted over the network from the handheld video receiver to one of the other devices connected to the network,

wherein the computer programs further comprise programs for receiving and displaying still graphic images and text downloaded from the network, and

wherein the handheld video receiver receives one or more video files from the global media exchange after the user agrees to pay a license fee determined by interaction of the user with the global media exchange over the digital network,

whereby a rights holder of the one or more video files is compensated for the use of the downloaded video files,

whereby user enters commands via the touch screen to play back any of the stored video files, and

whereby the video is displayed while the user carries the handheld video receiver.

27. A handheld video receiver, in a system for transmission and display of video and other media, the system comprising a digital network, a plurality of devices, each connected to the digital network, wherein at least one device is capable of transmitting a plurality of compressed video frames which make up a video stream, said handheld video receiver comprising:

a) a color display having a predetermined display width and display height and having a touch screen for receiving input from a user,

b) an audio output,

c) a program memory, for storing computer programs,

d) a processor, for executing computer programs from the program memory,

e) a data memory, for storing data,

f) a user interface on the touch screen comprising controls for interfacing with the executing computer programs, the controls comprising:

i) a zoom interface for issuing zoom in, zoom out, and pan commands, and

ii) mobile telephone controls for entering numeric telephone numbers and for controlling mobile telephone calls,

g) a wireless network interface for connecting to the digital network via a communications channel, and

h) a decompressor, cooperating with the processor,

wherein the handheld device has physical dimension allowing it to be held substantially in one hand of the user,

wherein the user interface is operated by one or more fingers of another hand of the user,

wherein the communications channel from the digital network to the handheld video receiever has a predetermined bandwidth such that there is a maximum sustainable bandwidth,

wherein the compressed video frames are received over the communications channel,

wherein the decompressor decompresses the compressed video frames in real time,

wherein at least a portion of the decompressed video frames is displayed on the color display in real time, and

wherein the audio portion plays on the audio output, whereby the video is displayed while the user carries the handheld video receiver.

28. The handheld video receiver ofclaim 27,

wherein the display height is greater than or equal to 480 pixels.

29. The handheld video receiver ofclaim 27,

wherein the display width is greater than or equal to 320 pixels.

30. The handheld video receiver ofclaim 27,

wherein the display width is less than or equal to 640 pixels.

31. The handheld video receiver ofclaim 27,

wherein the handheld video receiver has a weight of less than about five ounces.

32. The handheld video receiver ofclaim 27,

wherein the maximum sustainable bandwidth measured in millions of bits per second.

33. The handheld video receiver ofclaim 27,

wherein the maximum sustainable bandwidth is less than 10 million bits per second.

34. The handheld video receiver ofclaim 27,

wherein the magnification and position of the portion of the displayed media are changed in response to one or more of the zoom in, zoom out, and pan commands.

35. The handheld video receiver ofclaim 27,

wherein the video stream is pre-recorded in a video file and the transmission is a file transfer,

wherein the data memory is configured to store a plurality of pre-recorded video files,

whereby user enters commands via the touch screen to play back any of the stored video files.

36. The handheld video receiver ofclaim 27,

wherein the video stream is pre-recorded in a video file and the transmission is real-time streaming, and

wherein the video stream is decompressed in real time as the video stream is received,

whereby the video is displayed in real time while the user carries the handheld video receiver.

37. The handheld video receiver ofclaim 27,

wherein the video stream is live and the transmission is live streaming,

wherein the video stream is decompressed in real time as the video stream is received, and,

wherein the decompressed video is displayed live while the user carries the handheld video receiver.

38. The handheld video receiver ofclaim 27,

wherein the controls further comprises commands relating to video quality and transmission,

wherein the handheld video receiver dynamically interprets control commands entered by the user via the touch screen,

whereby the user of the receiver views the optimum quality live video.

39. The handheld video receiver ofclaim 27,

wherein the data memory comprises a hard disk drive,

wherein the video stream is stored on the hard disk drive as a video file, and

wherein the video file stored on the hard disk drive is played back.

40. The handheld video receiver ofclaim 27, further comprising:

a) a video camera for capturing the video frames in real-time,

b) a microphone for capturing an audio portion in real-time, and

c) a compressor configured to compress the video in real-time as the video is captured by the video camera,

wherein the captured video and audio portion are compressed in real-time.

41. The handheld video receiver ofclaim 40,

wherein the data memory comprises a hard disk drive, and

wherein the compressed video is stored on the hard disk drive.

42. The handheld video receiver ofclaim 40,

wherein the compressed video is transmitted over the network from the handheld video receiver to one of the other devices connected to the network.

43. The handheld video receiver ofclaim 27, further comprising a wired network interface for connecting to the digital network via a wired communications channel,

wherein video stream is transferred over the wired communication channel.

44. The handheld video receiver ofclaim 27,

wherein video stream is transferred via the wireless network interface,

wherein the handheld video receiver maintains connection to the digital network when being moved from one location to another location or while being freely carried by the user,

whereby the processing of the video continues when the handheld video receiver is being moved from one location to another location or while being carried by the user.

45. The handheld video receiver ofclaim 27,

wherein the wireless network interface is a wireless mobile telephone network interface, and

wherein mobile telephone calls and the video stream are transmitted via the wireless mobile telephone network interface.

46. The handheld video receiver ofclaim 27,

wherein the system further comprises a global media exchange, said global media exchange connected to the digital network,

wherein the video stream is pre-recorded in a video file,

wherein the handheld video receiver receives one or more pre-recorded video files from the global media exchange after the user agrees to pay a license fee determined by interaction of the user with the global media exchange over the digital network,

whereby a rights holder of the downloaded video files is compensated for the use of the downloaded video files.

47. The handheld video receiver ofclaim 27 wherein the video stream is compresssed using a color lossless compression method.

48. The handheld video receiver ofclaim 27, further comprising a battery with a limited battery life,

wherein the processor is less powerful than the processors typically found in non-handheld devices,

wherein the processor and decompressor cooperate to decompress each of the video frames in a single pass,

whereby the processor speed requirements are reduced and the battery life is extended while maintaining high quality video with optimal effective compression ratios.

49. The handheld video receiver ofclaim 27, wherein the digtial network of the system is the Internet, and

wherein the computer programs further comprise programs for receiving and displaying still graphic images and text downloaded from the network.

50. A method of exchanging digital video files between a global media exchange and a handheld video receiver, comprising the steps of:

a) capturing real-time, full-frame, full-color video from a video input device,

b) digitizing the captured video,

c) compressing the digital video with a compressor configured to preserve the level of spatial quality and motion quality of the compressed video that the handheld video receiver is capable of displaying,

d) storing the compressed video in a data memory in a device in communication with a digital network,

e) transferring the compressed video to the global media exchange via the network,

f) downloading the compressed video from the global media exchange via the network to the handheld video receiver after a user of the handheld video receiver agrees to pay a fee for use of the downloaded video,

g) storing the downloaded video in a data memory of handheld video receiver,

h) decompressing the downloaded video to a format that is optimal for display on a touch screen display on the handheld video receiver,

i) displaying at least a portion of the decompressed video on the display,

j) playing an audio portion of the decompressed video on an audio output of the handheld video receiver,

whereby the user downloads and plays back video from the global media exchange.

Images