US20180043829A1 - Method and Apparatus for Providing Automatic Mirror Setting Via Inward Facing Cameras - Google Patents

Abstract

A method or system is able to adjust an exterior mirror of a vehicle via an automatic mirror-setting (“AM”) model managed by a virtuous cycle containing a cloud based network (“CBN”). The system includes a set of mirrors, a set of inward facing cameras, a vehicle onboard computer (“VOC”), and AM module. In one embodiment, the mirrors, attached to a vehicle, are configured to capture at least a portion of external environment in which the vehicle operates. The inward facing cameras, mounted in the vehicle, are configured to collect internal images including operator facial features showing operator visual characteristics. VOC, which is coupled to CBN, is configured to determine operator vision metadata based on the internal images, operator visual characteristics, and historical stored data. The AM module is able to adaptively set a mirror to an optimal orientation so that area of external blind spot is minimized.

Description

PRIORITY
• This application claims the benefit of priority based upon U.S. Provisional Patent Application having an application Ser. No. 62/372,999, filed on Aug. 10, 2016, and having a title of “Method and System for Providing Information Using Collected and Stored Metadata,” which is hereby incorporated by reference in its entirety.
RELATED APPLICATION
• This application is related to the following co-pending application assigned to the Assignee of the present invention.
• a. Application Ser. No. 15/672,747, filed Aug. 9, 2017, entitled “Method and Apparatus for Providing Information via Collected and Stored Metadata Using Inferred Attentional Model,” invented by the same inventors with an Attorney's docket No. 1152.P0002US;
• b. Application Ser. No. 15/672,832, filed Aug. 9, 2017, entitled “Method and Apparatus for Providing Driver Information Via Audio and Video Metadata Extraction,” invented by the same inventors with an Attorney's docket No. 1152.P0006US; and
• c. Application No. ______, filed Aug. 10, 2017, entitled “Method and Apparatus for Providing Goal Oriented Navigational Directions,” invented by the same inventors with an Attorney's docket No. 1152.P0008US.
FIELD
• The exemplary embodiment(s) of the present invention relates to the field of communication networks. More specifically, the exemplary embodiment(s) of the present invention relates to operating an intelligent machine using a virtuous cycle between cloud, machine learning, and containerized sensors.
BACKGROUND
• With increasing popularity of automation and intelligent electronic devices, such as computerized machines, IoT (the Internet of Things), smart vehicles, smart phones, drones, mobile devices, airplanes, artificial intelligence (“AI”), the demand of intelligent machines and faster real-time response are increasing. To properly provide machine learning, a significant number of pieces, such as data management, model training, and data collection, needs to be improved.
• A conventional type of machine learning is, in itself, an exploratory process which may involve trying different kinds of models, such as convolutional, RNN (recurrent neural network), et cetera. Machine learning or training typically concerns a wide variety of hyper-parameters that change the shape of the model and training characteristics. Model training generally requires intensive computation. As such, real-time response via machine learning model can be challenging.
• A drawback associated with traditional automobile or vehicle is that most of mirrors especially external mirrors on both side of vehicle are typically set incorrectly so that driver often time has a large or big blind spot to both side of car. Typically, driver head position is different between the time setting the mirror and the time driving the vehicle.
SUMMARY
• One embodiment of the presently claimed invention discloses a method and/or system capable of adaptively adjusting one or more mirrors mounted on a vehicle via an automatic mirror-setting (“AM”) model managed by a virtuous cycle containing a machine learning center (“MLC”) and cloud based network (“CBN”). The system or AM system includes a set of mirrors, a set of inward facing cameras, a vehicle onboard computer (“VOC”), and AM module. In one embodiment, the mirrors, attached to the vehicle, are configured to capture at least a portion of external environment in which the vehicle operates. In one example, the mirrors include a left exterior side mirror, a right exterior side mirror, and an interior center mirror. The external environment includes road, nearby structures, pedestrians, traffic condition, nearby cars, and traffic lights.
• The inward facing cameras, mounted inside of the vehicle, are configured to collect internal images including operator facial features showing operator visual characteristics. VOC, which is coupled to CBN, is configured to determine operator vision metadata based on the internal images, operator visual characteristics, and historical stored data. In one example, the inward facing cameras include multiple exteriorly mounted image sensors capable of capturing internal images relating to position of driver relative to driver seat and interior of the vehicle. The operator visual characteristics include number of eyes on operator facial feature. Also, the operator visual characteristics include peripheral vision, vision boundary, and height of visual center.
• The AM module is able to adaptively set a mirror to an optimal orientation so that the area of external blind spot is minimized. The AM module includes at least a portion of an AM model which is able to dynamically adjust orientation of at least one of the plurality of mirrors to show an event associated with the external environment based on the external images and historical data from the CBN. In one embodiment, the AM model includes an abnormal tracking function which is able to realign orientation of at least one of the plurality of mirror to continuously track an abnormal event in response to the external images and real-time cloud data submitted by other nearby vehicles. It should be noted that the AM model is trained by MCL which is coupled to the VOC. A function of MCL is to train and improve the AM model based on the labeled data from the CBN.
• In one aspect, the AM system further includes outward facing cameras mounted on the vehicle for collecting external images representing the surrounding environment in which the vehicle operates. The CBN is wirelessly coupled to the VOC and configured to correlate and generate labeled data associated with AM data based on historical cloud data, internal images, and external images. The outward facing cameras are configured to capture real-time images as the vehicle moves across a geographical area.
• In an alternative embodiment, the presently claimed invention discloses a method or process for interactively setting a mirror mounted on a vehicle via metadata extraction utilizing a virtuous cycle including sensors, MLC and CBN. The process is capable of receiving a mirror resetting signal indicating at least one mirror mounted on a vehicle requiring an adjustment. Upon activating at least a portion of inward facing cameras mounted in the vehicle for capturing internal images including driver eye level respect to interior of the vehicle, historical cloud data associated with the vehicle and driver is obtained from a virtuous cycle. The process subsequently adjusts at least one mirror to an orientation with minimal blind spot in accordance with driver head position shown in the internal image and historical cloud data. In one aspect, the internal images are continuously obtained for a predefined wait period until the driver settling down so that the accurate calculation of driver head position can be computed. It should be noted that the set of outward facing cameras mounted on a vehicle can be activated for recording external surrounding images representing a geographic environment in which the vehicle operates. In one aspect, AM model is capable of tracking surrounding environmental event in accordance with the external surrounding images and historical data supplied by the virtuous cycle.
• In an alternative embodiment, AM system is able to perform a process configured to utilizing one of external mirror mounted on a vehicle to dynamically track an abnormal event facilitated via the virtuous cycle. After receiving a message of detecting an abnormal event nearby surrounding area in which the vehicle operates from cloud based data pushed by the MLC, images showing driver head position captured by a set of interior cameras is obtained while the driver operates moving vehicle. The process is capable of adaptively adjusting orientation of at least one mirror to track the abnormal event based on projected location according to the message so that the driver is able to see the abnormal event. In one example, AM model is able to issue a notice telling the driver to watch the abnormal event at the newly reoriented mirror. It should be noted that the labeled data representing driver reaction responding to the abnormal event is uploaded back to the CBN for facilitating AM model training at the MLC.
• Additional features and benefits of the exemplary embodiment(s) of the present invention will become apparent from the detailed description, figures and claims set forth below.
BRIEF DESCRIPTION OF THE DRAWINGS
• The exemplary embodiment(s) of the present invention will be understood more fully from the detailed description given below and from the accompanying drawings of various embodiments of the invention, which, however, should not be taken to limit the invention to the specific embodiments, but are for explanation and understanding only.
• FIGS. 1A-1B are block diagrams illustrating a virtuous cycle facilitating an automatic mirror-setting (“AM”) system capable of adaptively adjusting mirror(s) via a virtuous cycle in accordance with one embodiment of the present invention;
• FIGS. 1C-1E are diagrams illustrating AM model providing mirror adjustment using inward and/or outward facing cameras via a virtuous cycle in accordance with one embodiment of the present invention;
• FIGS. 1F-1H is a block diagram illustrating a pipeline process of outward facing camera capable of identifying and classifying detected object(s) using a virtuous cycle in accordance with one embodiment of the present invention;
• FIGS. 2A-2B are block diagrams illustrating a virtuous cycle capable of facilitating AM model detection in accordance with one embodiment of the present invention;
• FIG. 3 is a block diagram illustrating a cloud based network using crowdsourcing approach to improve AM model(s) in accordance with one embodiment of the present invention;
• FIG. 4 is a block diagram illustrating an AM model or system using the virtuous cycle in accordance with one embodiment of the present invention;
• FIG. 5 is a block diagram illustrating an exemplary process of correlating AM data in accordance with one embodiment of the present invention;
• FIG. 6 is a block diagram illustrating an exemplary process of real-time data management for AM model in accordance with one embodiment of the present invention;
• FIG. 7 is a block diagram illustrating a crowd sourced application model for
• AM model in accordance with one embodiment of the present invention;
• FIG. 8 is a block diagram illustrating a method of storing AM related data using a geo-spatial objective storage in accordance with one embodiment of the present invention;
• FIG. 9 is a block diagram illustrating an exemplary approach of analysis engine analyzing AM data in accordance with one embodiment of the present invention;
• FIG. 10 is a block diagram illustrating an exemplary containerized sensor network used for sensing AM related information in accordance with one embodiment of the present invention;
• FIG. 11 is a block diagram illustrating a processing device or computer system which can be installed in a vehicle for facilitating the virtuous cycle in accordance with one embodiment of the present invention; and
• FIG. 12 is a flowchart illustrating a process of AM model or system capable of providing driver rating in accordance with one embodiment of the present invention.
DETAILED DESCRIPTION
• Embodiments of the present invention are described herein with context of a method and/or apparatus for facilitating automatic mirror adjustment based on images captured by inward facing cameras via an AM model continuously trained by a virtuous cycle containing cloud based network, containerized sensing device, and machine learning center (“MLC”).
• The purpose of the following detailed description is to provide an understanding of one or more embodiments of the present invention. Those of ordinary skills in the art will realize that the following detailed description is illustrative only and is not intended to be in any way limiting. Other embodiments will readily suggest themselves to such skilled persons having the benefit of this disclosure and/or description.
• In the interest of clarity, not all of the routine features of the implementations described herein are shown and described. It will, of course, be understood that in the development of any such actual implementation, numerous implementation-specific decisions may be made in order to achieve the developer's specific goals, such as compliance with application- and business-related constraints, and that these specific goals will vary from one implementation to another and from one developer to another. Moreover, it will be understood that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skills in the art having the benefit of embodiment(s) of this disclosure.
• Various embodiments of the present invention illustrated in the drawings may not be drawn to scale. Rather, the dimensions of the various features may be expanded or reduced for clarity. In addition, some of the drawings may be simplified for clarity. Thus, the drawings may not depict all of the components of a given apparatus (e.g., device) or method. The same reference indicators will be used throughout the drawings and the following detailed description to refer to the same or like parts.
• In accordance with the embodiment(s) of present invention, the components, process steps, and/or data structures described herein may be implemented using various types of operating systems, computing platforms, computer programs, and/or general purpose machines. In addition, those of ordinary skills in the art will recognize that devices of a less general purpose nature, such as hardware devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or the like, may also be used without departing from the scope and spirit of the inventive concepts disclosed herein. Where a method comprising a series of process steps is implemented by a computer or a machine and those process steps can be stored as a series of instructions readable by the machine, they may be stored on a tangible medium such as a computer memory device (e.g., ROM (Read Only Memory), PROM (Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), FLASH Memory, Jump Drive, and the like), magnetic storage medium (e.g., tape, magnetic disk drive, and the like), optical storage medium (e.g., CD-ROM, DVD-ROM, paper card and paper tape, and the like) and other known types of program memory.
• The term “system” or “device” is used generically herein to describe any number of components, elements, sub-systems, devices, packet switch elements, packet switches, access switches, routers, networks, computer and/or communication devices or mechanisms, or combinations of components thereof. The term “computer” includes a processor, memory, and buses capable of executing instruction wherein the computer refers to one or a cluster of computers, personal computers, workstations, mainframes, or combinations of computers thereof.
• One embodiment of the presently claimed invention discloses a method or system capable of adjusting an exterior mirror of a vehicle via an automatic mirror-setting (“AM”) model managed by a virtuous cycle containing machine learning center (“MLC”) and cloud based network (“CBN”). The system or AM system includes a set of mirrors, a set of inward facing cameras, a vehicle onboard computer (“VOC”), and AM module. In one embodiment, the mirrors, attached to a vehicle, are configured to capture at least a portion of external environment in which the vehicle operates. The inward facing cameras, mounted in the vehicle, are configured to collect internal images including operator facial features showing operator visual characteristics. VOC, which is coupled to CBN, is configured to determine operator vision metadata based on the internal images, operator visual characteristics, and historical stored data. The AM module is able to adaptively set a mirror to an optimal orientation so that area of external blind spot is minimized.
• In an alternative embodiment, AM system is able to perform a process configured to utilizing one of external mirror mounted on a vehicle to dynamically track an abnormal event facilitated via the virtuous cycle. After receiving a message of detecting an abnormal event nearby surrounding area in which the vehicle operates from cloud based data pushed by the MLC, images showing driver head position captured by a set of interior cameras is obtained while the driver operates moving vehicle. The process is capable of adaptively adjusting orientation of at least one mirror to track the abnormal event based on projected location according to the message so that the driver is able to see the abnormal event. In one example, AM model is able to issue a notice telling the driver to watch the abnormal event at the newly reoriented mirror. It should be noted that the labeled data representing driver reaction responding to the abnormal event is uploaded back to the CBN for facilitating AM model training at the MLC.
• FIG. 1A is a block diagram100 illustrating a virtuous cycle facilitating an automatic mirror-setting (“AM”) system capable of adaptively adjusting mirror(s) via a virtuous cycle in accordance with one embodiment of the present invention. Diagram100 illustrates a virtuous cycle containing avehicle102,CBN104, andMLC106. In one aspect,MCL106 can be located remotely or in the cloud. Alternatively,MCL106 can be a part ofCBN104. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (circuit or elements) were added to or removed from diagram100.
• Vehicle102, in one example, can be a car, automobile, bus, train, drone, airplane, truck, and the like, and is capable of moving geographically from point A to point B. To simplify forgoing discussing, the term “vehicle” or “car” is used.Vehicle102 includes wheels with ABS (anti-lock braking system), body,steering wheel108, exterior or outward facingcameras125, interior (or360° (degree)) or inward facingcamera126,antenna124, onboard controller orVOC123, and operator (or driver)109. It should be noted that outward facing cameras and/or inward facing cameras125-126 can be installed at front, side-facing, stereo, and inside ofvehicle102. In one example,vehicle102 also includes various sensors which senses information related to vehicle state, vehicle status, driver actions, For example, the sensors, not shown in FIG. IA, are able to collect information, such as audio, ABS, steering, braking, acceleration, traction control, windshield wipers, GPS (global positioning system), radar, ultrasound, lidar (Light Detection and Ranging), and the like.
• VOC oronboard controller123 includes CPU (central processing unit), GPU (graphic processing unit), memory, and disk responsible for gathering data from outward facing orexterior cameras125, inward facing orinterior cameras126, audio sensor, ABS, traction control, steering wheel, CAN-bus sensors, and the like. In one aspect,VOC123 executes AM model received fromMLC106, and interfaces withantenna124 to communicate withCBN104 via awireless communication network110. Note that wireless communication network includes, but not limited to, WIFI, cellular network, Bluetooth network, satellite network, or the like. A function ofVOC123 is to gather or capture real-time surrounding information as well as exterior information whenvehicle102 is moving.
• CBN104 includes various digital computing systems, such as, but not limited to,server farm120, routers/switches121,cloud administrators119, connected computing devices116-117, andnetwork elements118. A function ofCBN104 is to provide cloud computing which can be viewed as on-demand Internet based computing service with enormous computing power and resources. Another function ofCBN104 is to improve or refine AM labeled data via correlating captured real-time data with relevant cloud data. The refined AM labeled data is subsequently passed toMLC106 for model training via aconnection112.
• MLC106, in one embodiment, provides, refines, trains, and/or distributesmodels115 such as AM model based on information or data such as AM labeled data provided fromCBN104. It should be noted that the machine learning makes AM model based on models generated and maintained by various computational algorithms using historical data as well as current data. A function ofMLC106 is that it is capable of pushing information such as revised AM model tovehicle102 via awireless communications network114 in real-time.
• To identify or collect current operator driving style viavehicle102, an onboard AM model which could reside inside ofVOC123 receives a triggering event or events from built-in sensors such as driver body language, external surrounding condition, internal detected images, ABS, wheel slippery, turning status, engine status, and the like. The triggering event or events may include, but not limited to, activation of ABS, texting, drinking, smoking, arguing, playing, fighting, rapid steering, rapid breaking, excessive wheel slip, activation of emergency stop, and on. Upon receiving triggering events via vehicular status signals, the recording or recorded images captured by inward facing camera or 360 camera are rewound from an earlier time stamp leading to the receipt of triggering event(s) for identifying, for example, AM labeled data which contains images of driver head position or abnormal events. After correlation of labeled data with historical sampling data at CBN, the AM model is retrained and refined atMLC106. The retrained AM model is subsequently pushed back ontovehicle102.
• It should be noted that by detecting the position of the driver's head, the system can automatically set the mirrors to the safest possible position. For example, by keeping track of the full range of positions the driver's head has been in, the system can determine the “center point” of their normal driving position. Note that the head position is usually different slightly from where they might place their head when performing a mirror adjustment. In one aspect, the AM model has a delay element that, during operation, upon pressing the button for “auto mirror set,” a delay and a tone are issued to allow the driver to position themselves as they will be when driving.
• During an operation, inward facingcamera126 captures facial images of driver oroperator109 including the driver head position and eye level. Upon verifying withCBN104, afocal direction107 ofoperator109 is identified. After obtaining and processing external images relating tofocal direction107, apossible trajectory105 in which the location is looked at is obtained.Trajectory105 andfocal direction107 are subsequently processed and combined in accordance with stored data in the cloud. The object, which is being looked at byoperator109, is identified. In this example, the object is ahouse103 nearby the road. After identifying driver vision scope and trajectory, the eye level is determined wherein the eye level or head position will be used to adjust the mirrors to optimal orientations with minimal blind spots.
• An advantage of using AM system is to reduce blind spots whereby traffic accidents should be reduced.
• FIG. 1B illustrates a block diagram140 showing an operator or driver monitored by AM system for adaptively adjusting mirrors via a virtuous cycle in accordance with one embodiment of the present invention. Diagram140 illustrates adriver148, inward facing camera(s)142, rightexternal side mirror143, andexterior camera144. In one aspect,camera142, also known as interior camera or 360 camera, monitors or captures driver'sfacial expression146 and/or driver (or operator) body language such as head position. Upon readingstatus149 which indicates stable with accelerometer, ahead with gaze, hands on steering wheel (no texting), AM model can conclude that driver is behaving normally or abnormally.
• During an operation, the interior images captured by inward facing camera(s)142 can show a location in whichoperator148 is focusing based on relative eye positions ofoperator148. Once the direction of location such asdirection145 is identified, AM system obtains external images captured by outward facing camera(s)144. After identifyingimage145 is where operator pays attention based ondirection145, theimage145 is recorded and process. Based on detectedtrajectory145, AM model is able to identify the driver vision associated to side mirrors. For example, the AM model can identify the optimal orientation formirror143 in view ofdriver vision141 with minimum blind spots.
• FIG. 1C illustrates diagrams180 and198 showing AM model containing inward facing cameras to automatically setting mirrors using a virtuous cycle in accordance with one embodiment of the present invention. Diagram180, in one embodiment, includesinterior car189,exterior car187,steering wheel181,dashboard182,driver head position184, inward facingcamera190, and leftmirror188. With assistance of virtuous cycle, onboard vehicle computer can calculatedriver head position184 based on the observed images captured by inward facingcamera190. After calculation of driver head position and his peripheral vision, leftmirror188 is automatically adjusted to viewrear view186 with the coverage with minimum blind spot as indicated bynumeral183. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (components or elements) were added to or removed from diagram180.
• Diagram180 illustrates a heuristic surrounding using a stream of head position data to choose center point of eyes. In one embodiment, the AM system employs one or more internal cameras to extract metadata regarding the head position of a driver, and use that known position in space in order to automatically set the rearview mirrors to the optimal position for that driver. The AM model is, in one aspect, capable of calculating the optimal setting for rear mirrors based on obtained driver head position. For example, when drivers are sitting in their normal driving positions, their head may not stay in one place; instead they sweep through a range of space. The system is able to use extracted metadata about head position, and more importantly, the exact position of the driver's eyes. By combining information about the exact distance from the eyes to the rear mirrors, a vertical and horizontal angle can be calculated that will allow the driver to see what is happening behind them, while minimizing the size of the “blind spot” that can occur.
• To operate functions of AM model, vehicle geometry metrics may be used. For example,left mirror188 has a metric of horizontal distance to center of steering wheel and vertical distance to center of steering wheel. The metrics also define mirror width, length, and height. An advantage of employing AM model is that it is able to dynamically adjust external and/or internal mirrors to minimize blind spot(s).
• Diagram198 shows a heuristic illustration used for filtered and weighted average of eye position. After capturing inward facing camera output atblock191, face and eye position is detected atblock192. Upon generation of time series of extracted position of eyes atblock193, filter outlier data points are generated and/or extracted atblock194. Atblock195, weighted average of eye position over a predefined time interval (t1, t2) is calculated and/or obtained.
• FIG. 1D shows diagrams1600-1602 illustrating real-time coverage of rear view mirror using AM model via a virtuous cycle in accordance with one embodiment of the present invention. Diagram1600 includesvehicle1606,blind cars1610, andvisible cars1612 wherein the rear view mirrors ofvehicle1606 are set improperly whereby leftside viewing coverage1614 and rightside viewing coverage1618 missblind cars1610. Although coverage1614-1618 coversvisible cars1612,blind cards1610 are in blind spots. Diagram1602 illustrates a scenario in which rear view mirror ofvehicle1606 are set properly wherebyblind cars1610 are visible by new coverage1624-1628. In one example,vehicle1606 displays real-time coverage of rear view mirror that all cars1610-1612 are observed leaving minimum blind spots.
• FIG. 1E is a block diagram1700 illustrating a dynamic tracking function of AM model containing inward and outward facing cameras via a virtuous cycle in accordance with one embodiment of the present invention. Diagram1700 includes three lanes1702-1706,vehicle1706, cars1708-1712, wireless transmission towers1711-1712, andvirtuous cycle1708. In one example,car1712 is acting or driving recklessly which constitutes an abnormal event. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more cars (components or elements) were added to or removed from diagram1700.
• In one embodiment, AM model is able to notify the driver an abnormal event nearby and direct the driver to monitor the situation via one or more mirrors which is dynamic tracking an abnormal event facilitated by thevirtuous cycle1708. In one aspect, the dynamic tracking function is able to move or turn mirror(s) to track the movement caused by the abnormal event. For example, whencar1712 is detected bycar1710 for speeding and changing multiple lanes at once as indicated by numeral1716,car1710 reports the reckless driving behavior as abnormal event viawireless signal1718 tovirtuous cycle1708 viawireless tower1711. After determining the abnormal event based on cloud data,virtuous cycle1708 pushes abnormal event tovehicle1706 via wireless signals and connection1722-1724. The left external side mirror is automatically adjusted fromoriginal cover1728 tosituational coverage1730 which will track the movement ofcar1712. The tracking will allow the driver to monitor the abnormal situation more effectively.
• An advantage of using a dynamic tracking function of AM model is that it provides an additional vision to the driver via mirror operation or adjustment.
• FIG. IF is a logic block diagram illustrating apipeline process150 of outward facing camera capable of identifying and classifying detected object(s) using a virtuous cycle in accordance with one embodiment of the present invention. Outward facingcamera151 collects images and the images are stored in aqueue152. After scaling the images byimage scaling component153, the scaled image is forwarded to objectdetection154.Object detection154 generates a collection of objection information which is forwarded to queue155. The object information which includes bounding-box, object category, object orientation, and object distance is forwarded tocomponent156 androuter157. Upon categorizing the object information atblock156, the categorized data is forwarded to map158. After recognizing the object based onmap158, the recognizer is forwarded torouter157. After routing information atrouter157, the output images are forwarded to block159 which uses classifier130-131 to classify the images and/or objects.
• Pipeline process150 illustrates a logic processing flow which is instantiated for the purpose of processing incoming data, extracting metadata on a frame by frame or data packet basis, and forwarding both frames and metadata packets forward through the pipeline. Each stage of the pipeline can contain software elements that perform operations upon the current audio or video or sensor data frame. The elements in the pipeline can be inserted or removed while the pipeline is running, which allows for an adaptive pipeline that can perform different operations depending on the applications. The pipeline process is configured to adapt various system constraints that can be situationally present. Additionally, elements in the pipeline can have their internal settings updated in real-time, providing the ability to “turn off,” “turn on” elements, or to adjust their configuration settings on the fly.
• Pipeline process150 includes a metadata packet schema which includes name/value pairs with arbitrary nesting and basic primitive data types such as arrays and structures that is used to create a self-describing and both machine and human readable form of the extracted real-time metadata flowing through the system. Such a generalized schema allows multiple software components to agree on how to describe the high level events that are being captured and analyzed and acted upon by the system. For example, a schema is constructed to describe the individual locations within a video frame of a person's eyes, nose, mouth, chin line, etc. Such a data structure allows a downstream software component to infer even higher level events, such as “this person is looking up at34 degrees above the horizon” or “this person is looking left18 degrees left of center.” The process can subsequently construct additional metadata packets and insert them into the stream, resulting in higher level semantic metadata that the system is able to act upon.
• FIG. 1G is a logic block diagram illustrating apipeline process160 capable of identifying and classifying face detection, head and gaze orientation, and mouth features using a virtuous cycle in accordance with one embodiment of the present invention. Inward facingcamera161 collects images and the images are stored in aqueue162. After scaling the images byimage scaling component163, the scaled image is forwarded to face andhead detection164. The output ofdetection164 is forwarded to image transform (“IT”) components165-166. After transformation, the transformed image is forwarded to blocks169-170. After facial feature extraction inblock169, the feature map is forwarded to block167 for pose normalization.Block168 receives face images fromIT component165 and transformed images fromblock167; the normalized face image is forwarded to block172. Upon processing normalized face with embedding network atblock172, a face ID is identified.
• Block170 extracts mouth feature and generates mouth feature(s) of driver.Block171 processes head and gaze based on output ofIT component166 which receives information with both scaled and unscaled images. In one example, block171 is capable of generating various features, such as gaze, head, number of eyes, glasses, and the like.
• FIG. 1H is a logic block diagram175 illustrating a process of classifying detected object(s) using a virtuous cycle in accordance with one embodiment of the present invention.Block176 is a software element used to classify a pedestrian based on collected external images captured by outward facing cameras. Based on collected data and historical data, pedestrian may be identified.Block177 is a software element used to classify a vehicle based on collected external images captured by outward facing cameras. Based on collected data and historical data, vehicle information can be identified. The exemplary classification information includes model of the vehicle, license plate, state of vehicle registration, and the like. In addition, formation such as turn-signals, brake lights, and headlights can also be classified via facilitation of virtuous cycle.Block178 is a software element used to classify traffic signals or conditions according to collected external images captured by outward facing cameras. For example, according to collected data as well as historical data, the traffic signal can be classified. The exemplary classification includes sign, speed limit, stop sign, and the like.
• FIG. 2A is a block diagram200 illustrating a virtuous cycle capable of detecting or monitoring AM system in accordance with one embodiment of the present invention. Diagram200, which is similar to diagram100 shown inFIG. 1A, includes a containerizedsensor network206, real-world scale data202, andcontinuous machine learning204. In one embodiment,continuous machine learning204 pushes real-time models to containerizedsensor network206 as indicated bynumeral210.Containerized sensor network206 continuously feeds captured data or images to real-world scale data202 with uploading in real-time or in a batched format. Real-world scale data202 provides labeled data tocontinuous machine learning204 for constant model training as indicated bynumeral212. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (or elements) were added to or removed fromFIG. 2A.
• The virtuous cycle illustrated in diagram200, in one embodiment, is configured to implement AM system wherein containerizedsensor network206 is similar tovehicle102 as shown inFIG. 1A and real-world scale data202 is similar toCBN104 shown in FIG. IA. Also,continuous machine learning204 is similar toMCL106 shown in FIG. IA. In one aspect, containerizedsensor network206 such as an automobile or car contains a containerized sensing device capable of collecting surrounding information or images using onboard sensors or sensor network when the car is in motion. Based on the AM model, selective recording the collected surrounding information is selectively recorded to a local storage or memory.
• Real-world scale data202, such as cloud or CBN, which is wirelessly coupled to the containerized sensing device, is able to correlate with cloud data and recently obtained AM data for producing labeled data. For example, real-world scale data202 generates AM labeled data based on historical AM cloud data and the surrounding information sent from the containerized sensing device.
• Continuous machine learning204, such as MLC or cloud, is configured to train and improve AM model based on the labeled data from real-world scale data202. With continuous gathering data and training AM model(s), the AM system will be able to learn, obtain, and/or collect all available data for the population samples.
• In one embodiment, a virtuous cycle includes partition-able Machine Learning networks, training partitioned networks, partitioning a network using sub-modules, and composing partitioned networks. For example, a virtuous cycle involves data gathering from a device, creating intelligent behaviors from the data, and deploying the intelligence. In one example, partition idea includes knowing the age of a driver which could place or partition “dangerous driving” into multiple models and selectively deployed by an “age detector.” An advantage of using such partitioned models is that models should be able to perform a better job of recognition with the same resources because the domain of discourse is now smaller. Note that, even if some behaviors overlap by age, the partitioned models can have common recognition components.
• It should be noted that more context information collected, a better job of recognition can be generated. For example, “dangerous driving” can be further partitioned by weather condition, time of day, traffic conditions, et cetera. In the “dangerous driving” scenario, categories of dangerous driving can be partitioned into “inattention”, “aggressive driving”, “following too closely”, “swerving”, “driving too slowly”, “frequent breaking”, deceleration, ABS event, et cetera.
• For example, by resisting a steering behavior that is erratic, the car gives the driver direct feedback on their behavior—if the resistance is modest enough then if the steering behavior is intentional (such as trying to avoid running over a small animal) then the driver is still able to perform their irregular action. However, if the driver is texting or inebriated then the correction may alert them to their behavior and get their attention. Similarly, someone engaged in “road rage” who is driving too close to another car may feel resistance on the gas pedal. A benefit of using AM system is to identify driver head position and adjust mirror(s) based on driver head position.
• In one aspect, a model such as AM model includes some individual blocks that are trained in isolation to the larger problem (e.g. weather detection, traffic detection, road type, etc.). Combining the blocks can produce a larger model. Note that the sample data may include behaviors that are clearly bad (ABS event, rapid deceleration, midline crossing, being too close to the car in front, etc.). In one embodiment, one or more sub-modules are built. The models include weather condition detection and traffic detection for additional modules intelligence, such as “correction vectors” for “dangerous driving.”
• An advantage of using a virtuous cycle is that it can learn and detect object such as AM in the real world.
• FIG. 2B is a block diagram230 illustrating an alternative exemplary virtuous cycle capable of detecting AM in accordance with one embodiment of the present invention. Diagram230 includesexternal data source234,sensors238,crowdsourcing233, andintelligent model239. In one aspect, components/activities above dottedline231 are operated incloud232, also known as in-cloud component. Components/activities below dottedline231 are operated incar236, also known as in-device or in-car component. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (or elements) were added to or removed fromFIG. 2B.
• In one aspect, in-cloud components and in-device components coordinate to perform desirable user specific tasks. While in-cloud component leverages massive scale to process incoming device information, cloud applications leverage crowd sourced data to produce applications. External data sources can be used to contextualize the applications to facilitate intellectual crowdsourcing. For example, in-car (or in-phone or in-device) portion of the virtuous cycle pushes intelligent data gathering to the edge application. In one example, edge applications can perform intelligent data gathering as well as intelligent in-car processing. It should be noted that the amount of data gathering may rely on sensor data as well as intelligent models which can be loaded to the edge.
• FIG. 3 is a block diagram300 illustrating a cloud based network using crowdsourcing approach to improve AM model(s) in accordance with one embodiment of the present invention. Diagram300 includes population ofvehicles302,sample population304,models deployment306,correlation component308, andcloud application312. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (or samples) were added to or removed fromFIG. 3.
• Crowdsourcing is a process of using various sourcing or specific models generated or contributed from other cloud or Internet users for achieving needed services. For example, crowdsourcing relies on the availability of a large population of vehicles, phones, or other devices to sourcedata302. For example, a subset of available devices such assample304 is chosen by some criterion such as location to perform data gathering tasks. To gather data more efficiently, intelligent models are deployed to a limited number ofvehicles306 for reducing the need of large uploading and processing a great deal of data in the cloud. It should be noted that the chosen devices such ascars306 monitor the environment with the intelligent model and create succinct data about what has been observed. The data generated by the intelligent models is uploaded to the correlated data store as indicated bynumeral308. It should be noted that the uploading can be performed in real-time for certain information or at a later time for other types of information depending on the need as well as condition of network traffic.
• Correlatedcomponent308 includes correlated data storage capable of providing a mechanism for storing and querying uploaded data.Cloud applications312, in one embodiment, leverage the correlated data to produce new intelligent models, create crowd sourced applications, and other types of analysis.
• FIG. 4 is a block diagram400 illustrating an AM system using the virtuous cycle in accordance with one embodiment of the present invention. Diagram400 includes a correlateddata store402,machine learning framework404, andsensor network406. Correlateddata store402,machine learning framework404, andsensor network406 are coupled by connections410-416 to form a virtuous cycle as indicated bynumeral420. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (circuit or elements) were added to or removed fromFIG. 4.
• In one embodiment, correlateddata store402 manages real-time streams of data in such a way that correlations between the data are preserved.Sensor network406 represents the collection of vehicles, phones, stationary sensors, and other devices, and is capable of uploading real-time events into correlateddata store402 via awireless communication network412 in real-time or in a batched format. In one aspect, stationary sensors includes, but not limited to, municipal cameras, webcams in offices and buildings, parking lot cameras, security cameras, and traffic cams capable of collecting real-time images.
• The stationary cameras such as municipal cameras and webcams in offices are usually configured to point to streets, buildings, parking lots wherein the images captured by such stationary cameras can be used for accurate labeling. To fuse between motion images captured by vehicles and still images captured by stationary cameras can track object(s) such as car(s) more accurately. Combining or fusing stationary sensors and vehicle sensors can provide both labeling data and historical stationary sampling data also known as stationary “fabric”. It should be noted that during the crowdsourcing applications, fusing stationary data (e.g. stationary cameras can collect vehicle speed and position) with real-time moving images can improve ML process.
• Machine Learning (“ML”)framework404 managessensor network406 and provides mechanisms for analysis and training of ML models.ML framework404 draws data from correlateddata store402 via acommunication network410 for the purpose of training modes and/or labeled data analysis.ML framework404 can deploy data gathering modules to gather specific data as well as deploy ML models based on the previously gathered data. The data upload, training, and model deployment cycle can be continuous to enable continuous improvement of models.
• FIG. 5 is a block diagram500 illustrating an exemplary process of correlating AM data in accordance with one embodiment of the present invention. Diagram500 includessource input504, real-time data management508,history store510, and crowd sourced applications512-516. In one example,source input504 includes cars, phones, tablets, watches, computers, and the like capable of collecting massive amount of data or images which will be passed onto real-time data management508 as indicated bynumeral506. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (or elements) were added to or removed fromFIG. 5.
• In one aspect, a correlated system includes a real-time portion and a batch/historical portion. The real-time part aims to leverage new data in near or approximately real-time. Real-time component ormanagement508 is configured to manage a massive amount ofinflux data506 coming from cars, phones, andother devices504. In one aspect, after ingesting data in real-time, real-time data management508 transmits processed data in bulk to the batch/historical store510 as well as routes the data to crowd sourced applications512-516 in real-time.
• Crowd sourced applications512-516, in one embodiment, leverage real-time events to track, analyze, and store information that can be offered to user, clients, and/or subscribers. Batch-Historical side of correlateddata store510 maintains a historical record of potentially all events consumed by the real-time framework. In one example, historical data can be gathered from the real-time stream and it can be stored in ahistory store510 that provides high performance, low cost, and durable storage. In one aspect, real-time data management508 andhistory store510 coupled by aconnection502 are configured to perform AM data correlation as indicated by dotted line.
• FIG. 6 is a block diagram600 illustrating an exemplary process of real-time data for AM system in accordance with one embodiment of the present invention. Diagram600 includesdata input602,gateway606,normalizer608,queue610,dispatcher616,storage conversion620, andhistorical data storage624. The process of real-time data management further includes acomponent614 for publish and subscribe. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (circuit or elements) were added to or removed fromFIG. 6.
• The real-time data management, in one embodiment, is able to handle a large numbers (i.e.,10′s of millions) of report events to the cloud as indicated bynumeral604. API (application program interface)gateway606 can handle multiple functions such as client authentication and load balancing of events pushed into the cloud. The real-time data management can leverage standard HTTP protocols. The events are routed to stateless servers for performing data scrubbing and normalization as indicated bynumeral608. The events frommultiple sources602 are aggregated together into a scalable/durable/consistent queue as indicated bynumeral610. Anevent dispatcher616 provides a publish/subscribe model forcrowd source applications618 which enables each application to look at a small subset of the event types. The heterogeneous event stream, for example, is captured and converted to files for long-term storage as indicated bynumeral620. Long-term storage624 provides a scalable and durable repository for historical data.
• FIG. 7 is a block diagram700 illustrating a crowd sourced application model for AM model in accordance with one embodiment of the present invention. Diagram700 includes agateway702,event handler704,state cache706,state store708,client request handler710,gateway712, andsource input714. In one example,gateway702 receives an event stream from an event dispatcher andAPI gateway712 receives information/data frominput source714. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (or elements) were added to or removed fromFIG. 7.
• The crowd sourced application model, in one embodiment, facilitates events to be routed to a crowd source application from a real-time data manager. In one example, the events entergateway702 using a simple push call. Note that multiple events are handled by one or more servers. The events, in one aspect, are converted into inserts or modifications to a common state store.State store708 is able to hold data from multiple applications and is scalable and durable. For example,State store708, besides historical data, is configured to store present data, information about “future data”, and/or data that can be shared across applications such as predictive AI (artificial intelligence).
• State cache706, in one example, is used to provide fast access to commonly requested data stored instate store708. Note that application can be used by clients.API gateway712 provides authentication and load balancing.Client request handler710 leveragesstate store708 for providing client data.
• In an exemplary embodiment, an onboard AM model is able to handle real-time AM detection based on triggering events. For example, after ML models or AM models for AM detection have been deployed to all or most of the vehicles, the deployed ML models will report to collected data indicating AM system for facilitating issuance of real-time warning for dangerous event(s). The information or data relating to the real-time dangerous event(s) or AM system is stored instate store708.Vehicles714 looking for AM detection can, for example, access the AMsystem using gateway712.
• FIG. 8 is a block diagram800 illustrating a method of storing AM related data using a geo-spatial objective storage in accordance with one embodiment of the present invention. Diagram800 includesgateway802,initial object804, putcall806, findcall808, getcall810, SQL (Structured Query Language)812,non-SQL814, and geo-spatial object storage820. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (circuit or elements) were added to or removed fromFIG. 8.
• Geo-spatial object storage820, in one aspect, stores or holds objects which may include time period, spatial extent, ancillary information, and optional linked file. In one embodiment, geo-spatial object storage820 includes UUID (universally unique identifier)822,version824, start and endtime826, bounding828,properties830,data832, and file-path834. For example, whileUUID822 identifies an object, all objects have version(s)824 that allow schema to change in the future. Start and endtime826 indicates an optional time period with a start time and an end time. Anoptional bounding geometry828 is used to specify spatial extent of an object. An optional set ofproperties830 is used to specify name-value pairs.Data832 can be binary data. Anoptional file path834 may be used to associate with the object of a file containing relevant information such as MPEG (Moving Picture Experts Group) stream.
• In one embodiment,API gateway802 is used to provide access to the service. Before an object can be added to the store, the object is assigned an UUID which is provided by the initial object call. Once UUID is established for a new object, the put call804 stores the object state. The state is stored durably inNon-SQL store814 along with UUID. A portion of UUID is used as hash partition for scale-out. The indexible properties includes version, time duration, bounding, and properties which are inserted in ascalable SQL store812 for indexing. TheNon-SQL store814 is used to contain the full object state.Non-SQL store814 is scaled-out using UUID as, for example, a partition key.
• SQL store812 is used to create index tables that can be used to perform queries.SQL store812 may include three tables816 containing information, bounding, and properties. For example, information holds a primary key, objects void, creation timestamp, state of object and object properties “version” and “time duration.” Bounding holds the bounding geometry from the object and the id of the associated information table entry. Properties hold property name/value pairs from the object stored as one name/value pair per row along with ID of associated info table entry.
• Findcall808, in one embodiment, accepts a query and returns a result set, and issues a SQL query toSQL store812 and returns a result set containing UUID that matches the query.
• FIG. 9 is a block diagram900 illustrating an exemplary approach of analysis engine analyzing AM data in accordance with one embodiment of the present invention. Diagram900 includeshistory store902,analysis engine904, and geo-spatial object store906. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (circuit or elements) were added to or removed fromFIG. 9.
• In one aspect, diagram900 illustratesanalysis engine904 containing ML training component capable of analyzing labeled data based on real-time captured AM data and historical data. The data transformation engine, in one example, interacts with Geo-spatial object store906 to locate relevant data and with history store to process the data. Optimally, the transformed data may be stored.
• It should be noted that virtuous cycle employing ML training component to provide continuous model training using real-time data as well as historical samples, and deliver AM detection model for one or more subscribers. A feature of virtuous cycle is able to continuous training a model and able to provide a real-time or near real-time result. It should be noted that the virtuous cycle is applicable to various other fields, such as, but not limited to, business intelligence, law enforcement, medical services, military applications, and the like.
• FIG. 10 is a block diagram1000 illustrating an exemplary containerized sensor network used for sensing AM system related information in accordance with one embodiment of the present invention. Diagram1000 includes asensor bus1002,streaming pipeline1004, andapplication layer1006 whereinsensor bus1002 is able to receive low-bandwidth sources and high-bandwidth sources.Streaming pipeline1004, in one embodiment, includes ML capable of generating unique model such asmodel1008. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (circuit or elements) were added to or removed fromFIG. 10.
• FIG. 11 is a block diagram1100 illustrating a processing device or computer system which can be installed in a vehicle to support onboard cameras, CAN (Controller Area Network) bus, Inertial Measurement Units, Lidar, et cetera for facilitating virtuous cycle in accordance with one embodiment of the present invention. Computer system orAM system1100 can include aprocessing unit1101, aninterface bus1112, and an input/output (“IO”)unit1120.Processing unit1101 includes aprocessor1102, amain memory1104, asystem bus1111, astatic memory device1106, abus control unit1105, I/O element1130, andAM element1185. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (circuit or elements) were added to or removed fromFIG. 11.
• Bus1111 is used to transmit information between various components andprocessor1102 for data processing.Processor1102 may be any of a wide variety of general-purpose processors, embedded processors, or microprocessors such as ARM® embedded processors, Intel® Core™ Duo, Core™ Quad, Xeon®, Pentium™ microprocessor, Motorola™ 68040, AMD® family processors, or Power PC™ microprocessor.
• Main memory1104, which may include multiple levels of cache memories, stores frequently used data and instructions.Main memory1104 may be RAM (random access memory), MRAM (magnetic RAM), or flash memory.Static memory1106 may be a ROM (read-only memory), which is coupled tobus1111, for storing static information and/or instructions.Bus control unit1105 is coupled to buses1111-1112 and controls which component, such asmain memory1104 orprocessor1102, can use the bus.Bus control unit1105 manages the communications betweenbus1111 andbus1112.
• I/O unit1120, in one embodiment, includes adisplay1121,keyboard1122,cursor control device1123, andcommunication device1125.Display device1121 may be a liquid crystal device, cathode ray tube (“CRT”), touch-screen display, or other suitable display device.Display1121 projects or displays images of a graphical planning board.Keyboard1122 may be a conventional alphanumeric input device for communicating information betweencomputer system1100 and computer operator(s). Another type of user input device iscursor control device1123, such as a conventional mouse, touch mouse, trackball, or other type of cursor for communicating information betweensystem1100 and user(s).
• AM element1185, in one embodiment, is coupled tobus1111, and configured to interface with the virtuous cycle for facilitating AM performance. For example, ifAM system1100 is installed in a car,AM element1185 is used to operate the AM model as well as interface with the cloud based network. IfAM system1100 is placed at the cloud based network,AM element1185 can be configured to handle the correlating process for generating labeled data for AM data.
• Communication device1125 is coupled tobus1111 for accessing information from remote computers or servers, such asserver104 or other computers, through wide-area network102.Communication device1125 may include a modem or a network interface device, or other similar devices that facilitate communication betweencomputer1100 and the network.Computer system1100 may be coupled to a number of servers via a network infrastructure such as the Internet.
• The exemplary embodiment of the present invention includes various processing steps, which will be described below. The steps of the embodiment may be embodied in machine or computer executable instructions. The instructions can be used to cause a general purpose or special purpose system, which is programmed with the instructions, to perform the steps of the exemplary embodiment of the present invention. Alternatively, the steps of the exemplary embodiment of the present invention may be performed by specific hardware components that contain hard-wired logic for performing the steps, or by any combination of programmed computer components and custom hardware components.
• FIG. 12 is aflowchart1200 illustrating a process of AM system capable of automatically setting mirrors with minimum blind spots in accordance with one embodiment of the present invention. The process, atblock1202, is capable of receiving a mirror resetting signal indicating at least one mirror mounted on a vehicle requiring an adjustment. Upon activating at least a portion of inward facing cameras mounted in the vehicle for capturing internal images including driver eye level respect to interior of the vehicle atblock1204, the historical cloud data associated with the vehicle and driver atblock1206 is obtained from a virtuous cycle. Atblock1208, the process subsequently adjusts at least one mirror to an orientation with minimal blind spot in accordance with driver head position shown in the internal image and historical cloud data. In one aspect, the internal images are continuously obtained for a predefined wait period until the driver settling down so that the accurate calculation of driver head position can be computed. It should be noted that the set of outward facing cameras mounted on a vehicle can be activated for recording external surrounding images representing a geographic environment in which the vehicle operates. In one aspect, AM model is capable of tracking surrounding environmental event in accordance with the external surrounding images and historical data supplied by the virtuous cycle.
• While particular embodiments of the present invention have been shown and described, it will be obvious to those of ordinary skills in the art that based upon the teachings herein, changes and modifications may be made without departing from this exemplary embodiment(s) of the present invention and its broader aspects. Therefore, the appended claims are intended to encompass within their scope all such changes and modifications as are within the true spirit and scope of this exemplary embodiment(s) of the present invention.

Claims (20)

What is claimed is:

1. A system for interactively adjusting a mirror mounted on a vehicle, comprising:

a plurality of mirrors attached to a vehicle and configured to capture at least a portion of external environment in which the vehicle operates;

a plurality of inward facing cameras mounted in the vehicle configured to collect internal images including operator facial features showing operator visual characteristics;

a vehicle onboard computer (“VOC”), coupled to a cloud based network (“CBN”) and the plurality of inward facing cameras, configured to determine operator vision metadata in accordance with the internal images, operator visual characteristics, and historical stored data; and

an automatic mirror-setting (“AM”) module coupled to the VOC and configured to adaptively set at least one of the plurality of mirrors to an optimal orientation so that area of external blind spot is reduced.

2. The system ofclaim 1, wherein the plurality of mirrors includes a left exterior side mirror, a right exterior side mirror, and an interior center mirror.

3. The system ofclaim 1, wherein the external environment includes road, nearby structures, pedestrians, traffic condition, nearby cars, and traffic lights.

4. The system ofclaim 1, wherein the plurality of inward facing cameras includes multiple exteriorly mounted image sensors capable of capturing internal images relating to position of driver relative to driver seat and interior of the vehicle.

5. The system ofclaim 1, wherein the operator visual characteristics includes number of eyes on operator facial feature.

6. The system ofclaim 1, wherein the operator visual characteristics includes peripheral vision, vision boundary, and height of visual center.

7. The system ofclaim 1, further comprising a plurality of outward facing cameras mounted on the vehicle collecting external images representing the surrounding environment in which the vehicle operates.

8. The system ofclaim 7, wherein the AM module includes at least a portion of an AM model which is able to dynamically adjust orientation of at least one of the plurality of mirrors to show an event associated with the external environment based on the external images and historical data from the CBN.

9. The system ofclaim 8, wherein the AM model includes an abnormal tracking function which is able to realign orientation of at least one of the plurality of mirror to continuously track an abnormal event in response to the external images and real-time cloud data submitted by other nearby vehicles.

10. The system ofclaim 8, wherein the AM model is trained by a machine learning center (“MLC”) which is coupled to the VOC and configured to train and improve the AM model based on the labeled data from the CBN.

11. The system ofclaim 10, wherein the CBN is wireles sly coupled to the VOC and configured to correlate and generate labeled data associated with AM data based on historical cloud data, internal images, and external images.

12. The system ofclaim 8, wherein the plurality of outward facing cameras is configured to capture real-time images as the vehicle moves across a geographical area.

13. The system ofclaim 1, wherein the plurality of inward facing cameras is configured to extract metadata associated with operator head pose, gaze direction, and looking at a mobile device.

14. A method for interactively setting a mirror mounted on a vehicle via metadata extraction utilizing a virtuous cycle including sensors, machine learning center (“MLC”), and cloud based network (“CBN”), comprising:

receiving a mirror resetting signal indicating at least one of a plurality of mirrors mounted on a vehicle requiring an adjustment;

activating at least a portion of inward facing cameras mounted in the vehicle for capturing internal images including driver eye level with respect to interior of the vehicle;

obtaining historical cloud data associated with the vehicle and driver from a virtuous cycle; and

adjusting at least one of the plurality of mirrors to an orientation with minimal blind spot in accordance with driver head position shown in the internal image and historical cloud data.

15. The method ofclaim 14, further comprising obtaining internal images continuously for a predefined wait period until the driver settling down before calculating driver head position.

16. The method ofclaim 14, further comprising activating a set of outward facing cameras mounted on a vehicle for recording external surrounding images representing a geographic environment in which the vehicle operates.

17. The method ofclaim 14, further comprising tracking surrounding environmental event in accordance with the external surrounding images and historical data supplied by the virtuous cycle.

18. A method configured to utilizing one of external mirror mounted on a vehicle to dynamically track an abnormal event facilitated by an automatic mirror-setting (“AM”) model via a virtuous cycle containing sensors, machine learning center (“MLC”), and cloud based network (“CBN”), comprising:

receiving a message of detecting an abnormal event nearby surrounding area in which the vehicle operates from cloud based data pushed by the MLC;

obtaining images showing driver head position captured by a set of interior cameras while the driver operates moving vehicle;

adaptively adjusting orientation of at least one mirror to track the abnormal event based on projected location according to the message so that the driver is able to see the abnormal event.

19. The method ofclaim 18, further comprising issuing a notice of watching the abnormal event at reoriented mirror to the driver.

20. The method ofclaim 19, further comprising uploading the labeled data representing driver reaction responding to the abnormal event back to the CBN for facilitating AM model training at the MLC.

Images