US20180348742A1 - Planning and Adapting Projects Based on a Buildability Analysis - Google Patents

Abstract

Disclosed herein is a worksite automation process that involves: generating a first sequence of tasks to build the product according to a model. The process further involves causing one or more robotic devices to build the product by beginning to execute the first sequence of tasks. Further, during the execution of the first sequence of tasks, performing a buildability analysis to determine a feasibility of completing the product by executing the first sequence of tasks. Based on the analysis, determining that it is not feasible to complete the product by executing the first sequence of tasks, and in response, generating a second sequence of tasks to complete the product according to the model. Then, causing the one or more robotic devices to continue building the product by beginning to execute the second sequence of tasks.

Description

BACKGROUND
• Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
• Automated manufacturing processes may involve the use of one or more robotic devices that may be used to construct an output product, such as a car, a wall, a piece of furniture, or any number of other physical fabrications. The robotic devices may be equipped with end-effector-mounted tools, such as a gripper or a drill, which may be used during a construction process. The robotic devices may be programmed with sequences of specific motion commands and commands for other operations in order to cause the robotic devices to complete a process.
SUMMARY
• The present disclosure provides systems and processes that relate to robotic automation of tasks in worksites in order to build or assemble products. As the automated tasks are being performed in the worksite, the systems and processes disclosed herein can adjust to deviations from a model of the product that is being built (e.g., a structure). For instance, a worksite automation system could coordinate a sequence of tasks in order to build an end product. As the system is building the product, the system could perform a buildability analysis to detect any issues that could prevent the system from building the product according to the model. If the buildability analysis indicates that the product is not buildable using the current sequence of tasks, the system could generate a new sequence of tasks that allows the system to continue building the product without necessarily changing the product's model. For example, if the system detects an issue while building the product, the system could generate a new sequence of tasks that allows the system to build off of the portion of the product that has been built thus far (i.e., “as-built”) and to complete the product such that the product meets the requirements of the model. Thus, the system could preserve any work that has been accomplished before the issue was detected.
• In one aspect, a computer-implemented method is provided. The method involves during a pre-build phase of building a product, generating a first sequence of tasks to build the product according to a model of the product. The method also involves during a build phase, causing one or more robotic devices to build the product by beginning to execute the first sequence of tasks. Further, the method involves, during the execution of the first sequence of tasks, performing a buildability analysis to determine a feasibility of completing the product by executing the first sequence of tasks. The method further involves determining, based on the analysis, that it is not feasible to complete the product by executing the first sequence of tasks. And in response to determining that it is not feasible to complete the product by performing the first sequence of tasks, the method involves generating a second sequence of tasks to complete the product according to the model, where the second sequence of tasks is different from the first sequence of tasks. Yet further the method involves causing the one or more robotic devices to continue building the product by beginning to execute the second sequence of tasks.
• In another aspect, a worksite automation system is provided. The system includes one or more robotic devices, and a control system including one or more processors and one or more data storage devices. The control system is configured to: receive an instruction to build a product, where the instruction comprises one or more constraints on the product. The control system is also configured to during a pre-build phase: (i) determine a model for the product, and (ii) generate a first sequence of tasks to build the product according to the model. Further, the control system is configured to: during a build phase, cause the one or more robotic devices to build the product by beginning to execute the tasks of the first sequence of tasks, and execution of the first sequence of tasks, perform a buildability analysis of the first sequence of tasks to determine a feasibility of completing the product by executing the first sequence of tasks. Based on the analysis, if it is not feasible to complete the product, the control system is configured to generate a second sequence of tasks to complete the product according to the model, where the second sequence of tasks is different from the first sequence of tasks, and cause the one or more robotic devices to continue building the product by beginning to execute the second sequence of tasks.
• In yet another aspect, a method is provided. The method involves during a design phase, generating a first sequence of tasks for a first model of a product. The method also involves during a build phase, causing one or more robotic devices to build the product by beginning to execute the first sequence of tasks. The method further involves during the performance of the first sequence of tasks, performing a first buildability analysis of the first model to determine a feasibility of building the product according to the first model, and based on the analysis, determining a second model. Yet further, the method involves causing the one or more robotic devices to continue building the product according to the second model.
• These as well as other aspects, advantages, and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description with reference where appropriate to the accompanying drawings. Further, it should be understood that the description provided in this summary section and elsewhere in this document is intended to illustrate the claimed subject matter by way of example and not by way of limitation.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1 depicts a worksite, according to an example embodiment.
• FIG. 2 is a simplified block diagram depicting components of a robotic device control system, according to an example embodiment.
• FIG. 3 is a flowchart depicting phases of building a product, according to an example embodiment.
• FIG. 4 is a flowchart depicting processes of the phases of building a product, according to an example embodiment.
• FIG. 5 is a flowchart depicting processes that are performed during a design phase, according to an example embodiment.
• FIG. 6 illustrates a template of a chair, according to an example embodiment.
• FIG. 7A depicts constraints on a table, according to an example embodiment.
• FIGS. 7B, 7C, 7D, 7E, and 7F each depict a design of a table, according to an example embodiment.
• FIG. 8A illustrates a tree structure, according to an example embodiment.
• FIG. 8B depicts a robotic device performing a task, according to an example embodiment.
• FIG. 9 is a flowchart illustrating a method, according to an example embodiment.
• FIG. 10 is a flowchart illustrating another method, according to an example embodiment.
• FIG. 11 shows a view of a robot, according to an example embodiment.
• FIG. 12 is a simplified block diagram depicting components of a robotic device, according to an example embodiment.
• FIG. 13 is a simplified block diagram depicting components of a computing device, according to an example embodiment.
DETAILED DESCRIPTION
• Example methods and systems are described herein. Any example embodiment or feature described herein is not necessarily to be construed as preferred or advantageous over other embodiments or features. The example embodiments described herein are not meant to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated herein.
• Furthermore, the particular arrangements shown in the Figures should not be viewed as limiting. It should be understood that other embodiments might include more or less of each element shown in a given Figure. Further, some of the illustrated elements may be combined or omitted. Yet further, an example embodiment may include elements that are not illustrated in the Figures.
I. OVERVIEW
• Disclosed herein is a system that could automate a sequence of tasks for building a product. The product that is being built by the system could be a discrete product or could be a product that is part of a larger product or structure. The product could be designed by a user, who could also specify constraints on the product. The specified constraints could be constraints on functional and/or non-functional features of the product. The constraints could also be performance constraints on a process of building the product. In particular, to build the product, the system could determine a model of the product that satisfies the constraints. Then, the system could determine a sequence of tasks that could be performed to build the product according to the model.
• In particular, building a product could involve two phases. The first phase is a pre-build phase that involves two sub-phases: a design phase and a simulation phase. In the design phase, the system could determine a model for the product, and could also determine a sequence of tasks to build the product according to the model. And in the simulation phase, the system could simulate building the product, perhaps by simulating the performance of the sequence of tasks to build the product, the behavior of the product as the product is being built (e.g., stability of the product), and/or intermediate stages of the product. The second phase of building a product is a build phase. In the build phase, the system could perform the sequence of tasks in order to build the physical product.
• However, in any of the phases, errors that could impede the system from building the product could occur. In particular, the errors could cause the system to build a product that deviates from the desired specifications. For example, during the build phase, deviations of the as-built product from the model could occur. Such deviations could accumulate to create an undesirable effect (e.g., a defect) in the final product. In general, the system could encounter issues in the pre-build phase and/or the build phase that could create undesirable effects in the final product.
• Accordingly, to avoid errors that could impede the system from building the product or could cause the system to build a product that does not meet the desired specifications, the system could perform a buildability analysis that determines the feasibility of building a product, or a certain portion thereof. Generally, the system could perform pre-build buildability analyses during the pre-build phase, and then could perform build buildability analyses during the build phase (e.g., at the beginning of the build and/or in the midst of the build). Pre-build buildability analyses could allow the system to accurately and efficiently determine a model for the product. The pre-build buildability analyses could also allow the system to validate that a particular sequence of tasks could result in a successful completion of the product. And the build buildability analyses could allow the system to detect and efficiently adapt to any issues or changes that may occur while the system is building the product. By detecting and adapting to issues, the system could improve the probability that the product is built according to the desired specifications.
• In an example, the system could perform a first buildability analysis as the system is determining a model of the product in the design phase. In particular, the first buildability analysis could determine whether the model satisfies any constraints on the product. As explained herein, product constraints define features of a class of products that encompasses the product that is being built. For example, a product constraint on a table could be that the table has one or more legs or supporting members that support a surface such that the surface will be level or horizontal when the legs are placed on a level floor. After determining the model, the first buildability analysis could determine whether the table's model describes a table that satisfies the constraints on the minimum number of legs and the orientation of the top surface, among other constraints. If the analysis determines that model does not satisfy the constraints, then the system could determine that the model is not buildable. But if the model satisfies the constraints, then the analysis could determine that the model is buildable.
• The buildability analysis that is performed during the design phase could also determine whether the model satisfies other desired constraints on the table, which could be specified by a user, for example. For example, the constraints could define desired features or properties of the table such as the table's load limit, center of gravity, stability, stresses (e.g., bending stress), strengths (e.g., buckling strength), among other features or properties. The buildability analysis during the design phase could also determine whether the model satisfies non-functional features defined by the constraints. For example, the system could determine whether the model satisfies desired aesthetic features (e.g., engravings, fixtures, etc.) that are defined by the constraints.
• If the buildability analysis determines that a model is not buildable, the system could then determine a new model, and could also perform a buildability analysis on the new model. This process is cyclical, and therefore, could be repeated until the buildability analysis determines a model that is buildable by the system. Once the system determines a buildable model, the system could generate a sequence of tasks that could be performed to build a product that satisfies the buildable model. In an embodiment, the system could use a tree structure to generate possible sequences of tasks to build the product. Then, the system could select a possible sequence of tasks to build the product.
• Once the sequence of tasks is selected, the system could proceed to the simulation phase during which the system could simulate the performance of the sequence of tasks. Specifically, the system could simulate the performance of the tasks using resources available to the system. The resources could include available workers to perform the tasks, available tools that the workers can use to perform the tasks, and available materials to build the product. For example, the system could simulate robotic devices performing the sequence of tasks using resources (e.g., parts, materials, etc.) available to the system. By simulating the sequence of tasks using the resources available to the system, the system could perform a buildability analysis to determine whether it has the resources to build the product according to the determined model. For example, during the simulation, the analysis could determine whether the system has access to robotic devices that could be configured to perform each task of the sequence of tasks. By way of example, one of the tasks of building a table could be a sanding task, and the buildability analysis could determine whether the system includes a robotic device that could be configured to perform the sanding task.
• During the simulation phase, the system could also determine an order in which the tasks of the sequence are to be performed (also referred to as an “order of operations”). In such a system, the buildability analysis (during the simulation phase) could also involve determining whether the tasks could be performed according to the determined order of operations. In an example, the analysis could determine whether a product is stable while being built if the tasks are performed in the determined order. In another example, the analysis could also determine whether any conflicts occur when building the product. For instance, the analysis could determine whether any timing or spatial conflicts occur between the robotic devices assigned to perform tasks. In yet another example, the analysis could determine whether the performance constraints are satisfied if the product is built using the determined sequence of tasks. For instance, based on the simulation, the analysis could determine the estimated time to build the product, and could then determine whether the performance time meets the performance constraints.
• By performing the buildability analysis during the simulation phase, the system could determine whether the product is buildable using the resources available to the system. If the product is not buildable, the system could determine a new sequence of tasks to build the product according to the model. In some examples, the system could test a predetermined threshold number of sequences, each of which the analysis determines is not buildable. In these examples, the system could determine that the model is not buildable, and could return to the design phase to determine a new model for the product. Conversely, if the analysis determines that the product is buildable using the system's resources, the system could move to the build phase.
• In the build phase, the system could build the product in the worksite by sending instructions to robotic devices located in the worksite. The instructions cause the robotic devices to perform the tasks according to the determined order of operations. In this phase, the system could perform a buildability analysis to determine whether any issues or changes have occurred that could require the system to adapt. Specifically, the system could receive data (e.g., from sensors and/or feedback from the robotic devices) indicative of the worksite, and the analysis could use the data to determine whether any deviations from the model have occurred. The analysis could also determine whether any deviations could occur in the future. The analysis could be based on conditions of the worksite (e.g., environmental conditions, obstacles), the current status of the product (e.g., status of completed structures and/or completed tasks), available resources, changes to the constraints (e.g., by a user), and deviations from the design (e.g., delay in time or deviations from structural design). If the analysis detects a deviation (either current or anticipated), the analysis could indicate the product is not buildable.
• In an embodiment, in response to determining that the product is not buildable, the system could return to the design phase. In the design stage, the system could use the tree structure to generate a new sequence of tasks to build the product that is to be performed instead of the original sequence of tasks. In particular, the system could use the tree structure to generate a sequence of tasks that builds on the portion of the product that has been built thus far. As such, the new sequence of tasks preserves the work that has been performed. Preserving the work that has been performed could save time and costs. Further, generating a new sequence of tasks could be advantageous since generating a new model could require intensive computing power and a significant amount of time, which will incur high costs and cause delays in completing the product.
II. EXAMPLE SYSTEMS AND METHODS
• A. Example Worksite Coordinate Frame
• FIG. 1 depicts a worksite coordinateframe100, according to an example embodiment. The worksite coordinateframe100 may define a portion of a physical environment in which objects, machines, and perhaps humans may be located. The worksite coordinateframe100 may take on a three-dimensional form and may be used for various purposes. For instance, the worksite coordinateframe100 may be defined for a construction site where the construction of a building or another project is being or is about to be carried out. As such, the worksite coordinateframe100 may include a physical stage or stages on which a physical building process is planned or is occurring within the physical world.
• However, while various aspects of the disclosure are discussed below in the context of a construction site, example implementations are not limited to construction sites and may extend to a variety of other worksite coordinate frames, such as retail spaces, manufacturing facilities, distribution facilities, office spaces, shopping centers, festival grounds, and/or airports, among other examples. Additionally, while one worksite coordinateframe100 is shown inFIG. 1, example implementations may be carried out in the context of a plurality of worksite coordinate frames.
• As depicted inFIG. 1, the worksite coordinateframe100 includes a plurality of resources. The resources may include one or more actors, devices, hardware components, and/or physical materials. The physical materials may beconstruction materials140, which may be any materials or tools located in a construction site. For example, inFIG. 1, theconstruction materials140 are depicted as a palette of bricks. Within examples, the actors in the worksite coordinateframe100 may include robotic actors and human actors. The human actors may have one or more skills. For example, a human actor may be a carpenter. Further, the human actors may interface with the robotic actors and devices using computing devices.
• Aworker robot130 may be a robotic device configured to perform a task within the worksite coordinateframe100. In the illustrated scenario, theworker robot130 includes an end effector tool mounted to a robotic arm. The end effector tool may be configured to perform a task on a work surface, such as drilling, milling, cutting, welding, nailing, riveting, sanding, spraying, gripping, extruding, etching, carving, or any other task typically performed during construction of a building. Further, the robotic arm of theworker robot130 may include a mount to which different types of end effectors can be attached. As such, different end effectors may be swapped out such that theworker robot130 can perform different types of tasks. Further, theworker robot130 may be capable of moving throughout the worksite. For example, as depicted inFIG. 1, theworker robot130 may include wheels. However, other configurations for providing mobility are possible as well (e.g., biped devices, quadruped devices, treads, tracks, etc.).
• As illustrated inFIG. 1, the worksite coordinateframe100 includes a number ofpylon markers104. Thepylon markers104 may be located at the boundaries of the worksite coordinateframe100 and/or throughout the worksite coordinateframe100 and may be used to establish a three-dimensional coordinatesystem102 within the worksite coordinateframe100. For example, the three-dimensional coordinatesystem102 may be a Cartesian coordinate system with an x-axis, a y-axis, and a z-axis. One of thepylon markers104 may be designated as the origin of the coordinatesystem102, and the remainingpylon markers104 may be spaced a known distance from the origin. As a result, each of thepylon markers104 may be associated with a known (x, y, z) coordinate within the coordinatesystem102, where the x-, y-, and z-coordinates correspond to a distance from the origin pylon marker along the x-, y-, and z-axes respectively.
• Thepylon markers104 do not necessarily need to be located at the boundaries of the worksite coordinateframe100, but may alternatively or additionally be arranged at various known locations throughout the worksite coordinateframe100. For example, in some embodiments, thepylon markers104 may be arranged in a two-dimensional or three-dimensional grid throughout the worksite. However, other configurations are possible as well, and thepylon markers104 may be arranged in any manner of known locations in the worksite coordinateframe100.
• Thepylon markers104 may be retroreflective such that the laser tracker of arobotic device120 could measure the location of thepylon markers104 with respect to therobotic device120. By determining the location of a pylon marker with known coordinates from therobotic device120, the coordinates of therobotic device120 may be derived. As therobotic device120 moves about the worksite coordinateframe100, it may occasionally provide a line of sight between its laser tracker and apylon marker104. This provides updated coordinates for the location of therobotic device120 as it moves about the worksite coordinateframe100.
• In addition to thepylon markers104, the worksite coordinateframe100 may include a number ofadditional markers112. Themarkers112 may be attached to various target objects throughout the worksite coordinateframe100. For example, as depicted inFIG. 1,respective markers112 may be attached to therobotic device120, theworker robot130, and/or theconstruction materials140. The location of themarkers112 may be measured to provide coordinates within the coordinatesystem102 associated with therobotic device120,worker robot130, and/orconstruction materials140.
• In some embodiments, determining the location of a target object in the worksite coordinateframe100 may involve more than simply determining the location of a single point within a three-dimensional coordinate system. For instance, in some embodiments, the locations of a set of points may be determined to define a volume of the target object. For example, referring toFIG. 1, a three-dimensional space representing the volume of theconstruction materials140 may be determined. A robotic device may determine the locations of a set ofmarkers112 attached to theconstruction materials140. Themarkers112 may be attached to the boundaries or edges of theconstruction materials140, or in some embodiments, themarkers112 may be arranged on theconstruction materials140 in any known manner. By determining the location of the set ofmarkers112 arranged on theconstruction materials140, the location of a three-dimensional volume may be determined indicating a shape of theconstruction materials140 within the worksite coordinateframe100. Similarly, by determining the locations of sets ofmarkers112 arranged on various target objects, three-dimensional volumes and their positions within the worksite coordinateframe100 may be determined for themover robot120, theworker robot130, or any other object in the worksite coordinateframe100.
• In some embodiments, determining the location of a target object in the worksite coordinateframe100 may include determining a pose of the target object relative to the worksite coordinateframe100. The pose of the target object may include a combination of the position and orientation of the object. Various processes may be used to determine the pose of a target object, including analytic or geometric methods, genetic algorithm methods, and/or learning-based methods, among others.
• In some embodiments, where the target object is a robotic device, the pose of the robot may be determined based on its operational state. The robotic device may have various operational states that result in different poses. A control system may determine the operational state of the robotic device. Given that the volume and/or shape of the robotic device is already known or has otherwise been determined, the control system may determine the pose of the robotic device based on the determined operational state.
• For example, referring toFIG. 1, theworker robot130 includes a robotic arm that may be positioned in any number of poses based on its operational state. One operational state may include configuring the robotic arm to perform a task on a work surface. However, an operational state may include any configuration of a robotic device in the worksite coordinateframe100 that results in an associated pose. The volume and shape of the various components of the robotic arm are known or have otherwise been determined. Thus, by determining the pose (e.g., the position and orientation) of the robotic arm based on the operational state of theworker robot130, a three-dimensional volume representing the space occupied by the robotic arm within the worksite coordinateframe100 may be determined.
• B. Example Robotic Control Systems
• FIG. 2 illustrates an example configuration of arobotic control system200 that could be used in connection with the embodiments described herein. Therobotic control system200 may be configured to operate autonomously, semi-autonomously, and/or using directions provided by user(s). Therobotic control system200 could be responsible for managing a worksite, such as a construction site, a production site, a manufacturing site, an inspection or quality control site, etc. For example, in a construction or manufacturing worksite, therobotic control system200 could be responsible for coordinating the construction or manufacturing of a product (also referred to interchangeably as “output product”). In particular, such a robotic control system could control one or more robotic devices to construct the product, and could also monitor the environment using one or more sensors.
• As illustrated inFIG. 2, the roboticdevice control system200 includes arobotic device210 configured to control an end-effector220. The end-effector220 could be a tool end-effector that is configured to perform a task on a work surface (e.g., a surface of the output product) and could be mounted to a moveable component, such as a robotic arm, of therobotic device210. Therobotic device210 could be located within a worksite (e.g.,site100 depicted inFIG. 1).
• According to one example, the worksite could be a factory floor where robotic devices install parts in an assembly line to assemble a product (e.g., a table, airplane wing, etc.). According to an additional example, rather than an assembly line, the worksite could be a worksite where robotic devices combine a variety of parts to construct a physical structure. In these examples, the worksite could be a temporary location from which the final physical structure may be delivered (e.g., as a product) to another location (e.g., a distributor or customer location) when completely built.
• According to yet another example, the worksite could be a municipal site where robotic devices work with heavy construction materials to construct a bridge or a road. According to a further example, the worksite could be a construction site where robotic devices work with construction materials to construct a house or a building. The worksite could also be the interior of a house where robotic devices install housing materials to construct a section of the house. In these examples, the final physical structure is installed in the worksite.
• Therobotic control system200 could further include local sensor(s)230 and global sensor(s)240 configured to provide environment data representative of the worksite. For example, the local sensor(s)230 and global sensor(s)240 could determine the location of various objects in the worksite, such as a product that is being built by the system, for example. As another example, the local sensor(s)230 and the global sensor(s)240 could provide therobotic control system200 with data that therobotic control system200 can use to generate a “world map” of the worksite. The world map could be indicative of a real-time or near real-time representation of the worksite. Thus, the local sensor(s)230 and global sensor(s)240 could provide therobotic control system200 with data such that therobotic control system200 could update the world map continuously or periodically. Therobotic control system200 could also use the sensor data and/or the world map to track the performance of a task in the worksite. In particular, therobotic control system200 could track any robotic devices participating in the task, any materials used in performing the task, and any changes to the as-built product as a result of performing the task.
• Additionally, the local sensor(s)230 could be arranged on or within therobotic device210 and could be configured to measure the location of the end-effector220 with respect to a work surface (e.g., a surface of the product being built). The local sensor(s)230 could also be configured to scan or capture features of the work surface. The global sensor(s)240, on the other hand, could be arranged within the worksite and could be configured to measure the location of the output product with respect to a coordinate system in the worksite. The global sensor(s)240 could also be configured to measure the location of the end-effector220 with respect to the coordinate system or with respect to another object (e.g., location of the base of the robotic device). Further, the global sensor(s) could also be configured to measure the location of therobotic device210.
• In an embodiment, the global sensor(s)240 could include a laser tracker system with very high resolution (e.g., hundredths of a millimeter). The laser tracker system could be used to determine locations of objects in the worksite. However, the global sensor(s)240 are not limited to laser tracker systems, but could include any sensor capable of capturing features of objects located in the worksite, such as motion capture sensors, scanners, light detection and ranging (LIDAR) sensors, point cloud sensors, ultrasonic range sensors, Global Positioning System (GPS) receivers, sonar, optical sensors, Radio Frequency identification (RFID) systems, Near Field Communication (NFC) chips, wireless sensors, radio sensors, radars, cameras (e.g., color cameras, grayscale cameras, and/or infrared cameras), and/or range sensors (e.g., ultrasonic and/or infrared), among others.
• And the local sensor(s)230 could include a high speed camera for providing optical flow data or an inertial measurement unit (IMU). However, the local sensor(s)230 are not limited to high speed cameras or IMUs, but could include any sensor capable of measuring the location of the end-effector220 with respect to a work surface or capable of capturing features of the work surface. Such sensors include force sensors, proximity sensors, motion sensors (e.g., gyroscopes, and/or accelerometers), load sensors, position sensors, thermal imaging sensors, depth sensors (e.g., RGB-D, laser, structured-light, and/or a time-of-flight camera), ultrasonic range sensors, infrared sensors, optical sensors, Radio Frequency identification (RFID) systems, Near Field Communication (NFC) chips, wireless sensors, light sensors, touch sensors (e.g., capacitive sensors), scanners, cameras (e.g., color cameras, grayscale cameras, and/or infrared cameras), and/or range sensors (e.g., ultrasonic and/or infrared), among others. In some embodiments, the location of the end-effector220 with respect to a work surface could be determined using wheel odometry and/or robot forward kinematics.
• Additionally, the local sensor(s)230 and global sensor(s)240 could be positioned within or in the vicinity of the worksite, among other possible locations. For example, the local sensor(s)230 could be attached to therobotic device210. In some embodiments, the global sensor(s)240 could be arranged in fixed locations throughout the worksite, for example, as a dedicated sensing installation. Further, an example implementation may also use sensors incorporated within existing devices, such as mobile phones, laptops, and/or tablets. These devices may be in possession of workers located in the production site, such as construction workers in a construction site.
• FIG. 2 also depicts acontroller250 that could receive data from the local sensor(s)230 and global sensor(s)240. In particular, the local sensor(s)230 and global sensor(s)240 could provide sensor data to thecontroller250 through acommunication unit260. Thecommunication unit260 could include wired links and/or wireless links (e.g., using various wireless transmitters and receivers). A wired link may include, for example, a parallel bus or a serial bus such as a Universal Serial Bus (USB). A wireless link may include, for example, Bluetooth, IEEE 802.11 (IEEE 802.11 may refer to IEEE 802.11-2007, IEEE 802.11n-2009, or any other IEEE 802.11 revision), Cellular (such as GSM, GPRS, CDMA, UMTS, EV-DO, WiMAX, HSPDA, or LTE), or Zigbee, among other possibilities. Furthermore, multiple wired and/or wireless protocols may be used, such as “3G” or “4G” data connectivity using a cellular communication protocol (e.g., CDMA, GSM, or WiMAX, as well as for “WiFi” connectivity using 802.11).
• In other examples, therobotic control system200 could include access points through which the local sensor(s)230 and global sensor(s)240 and/orcontroller250 could communicate with a cloud server. Access points may take various forms such as the form of a wireless access point (WAP) or wireless router. Further, if a connection is made using a cellular air-interface protocol, such as a CDMA or GSM protocol, an access point may be a base station in a cellular network that provides Internet connectivity via the cellular network. Other examples are also possible.
• Thecontroller250 is shown to include one or more processor(s)252,data storage254,program instructions256, an input/output unit258, and apower source262. Note that thecontroller250 is shown for illustration purposes only, as thecontroller250 could include additional components and/or have one or more components removed without departing from the scope of the disclosure. Further, note that the various components of thecontroller250 could be arranged and connected in any manner. Thecontroller250 could be incorporated in whole or in part into therobotic device210 or could take the form of a desktop computer, a laptop, a tablet, a wearable computing device, and/or a mobile phone, among other possibilities.
• Each processor, from the one or more processor(s)252, could be a general-purpose processor or a special purpose processor (e.g., digital signal processors, application specific integrated circuits, etc.). The processor(s)252 could be configured to execute computer-readable program instructions256 that are stored in thedata storage254 and are executable to provide the functionality of thecontroller250 described herein. For instance, theprogram instructions256 could be executable to provide for processing of sensor data received from the local sensor(s)230 and global sensor(s)240.
• Thedata storage254 could include or take the form of one or more computer-readable storage media that can be read or accessed by the processor(s)252. The one or more computer-readable storage media could include volatile and/or non-volatile storage components, such as optical, magnetic, organic or other memory or disc storage, which could be integrated in whole or in part with the processor(s)252. In some embodiments, thedata storage254 could be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc storage unit), while in other embodiments, thedata storage254 could be implemented using two or more physical devices. Further, in addition to the computer-readable program instructions256, thedata storage254 could include additional data such as diagnostic data, among other possibilities. Further, thecontroller250 could also include apower source262 configured to supply power to various components of thecontroller250. Any type of power source could be used, such as direct current from a battery or alternating current from mains electricity.
• FIG. 1 further depicts thecontroller250 including an input/output unit258. The input/output unit258 could output information to a user through a display. The display could take on any form and may be arranged to project images and/or graphics to a user of thecontroller250. In an example arrangement, a projector within the input/output unit258 could be configured to project various projections of images and/or graphics onto a surface of the display. The display could include: an opaque or a transparent (or semi-transparent) matrix display, such as an electroluminescent display or a liquid crystal display, one or more waveguides for delivering an image to the user's eyes, or other optical elements capable of delivering an image to the user. A corresponding display driver could be disposed within thecontroller250 for driving such a matrix display. Other arrangements could also be possible for the display. As such, the display could show a graphical interface that could provide an application through which the user could interact with the systems disclosed herein.
• Further, therobotic control system200 could display the world map on the display of the input/output unit258. Therefore, the input/output unit258 could display a real-time or near real-time representation of the worksite, including the as-built product. Accordingly, a user could monitor the progress of building or assembling the output product. Based on the real-time feedback data (e.g., data from local sensor(s)230 and global sensor(s)240) indicative of the worksite, the displayed world map could be updated to reflect the real-time changes in the worksite.
• The input/output unit258 could also include task controls. The task controls could provide a user with real-time control of task execution. For instance, the user could be able to provide an input that could start, stop, skip, or modify a task. For instance, a graphical interface displayed on display could include a task queue of the tasks that therobotic control system200 will perform. The graphical interface could allow the user to start, stop, skip, or modify a task. In some implementations, the graphical interface could allow the user to enter parameters relating to the output product. The graphical interface could allow the user to enter parameters that could relate to aspects of the output product, including dimensions, density, curvature properties, other geometric properties, materials to be used, and/or other numeric inputs.
• In further examples, the graphical interface could contain a timeline of the building the output product. The timeline could have a cursor representing a current timestamp, which could represent a particular point in time of the process of building the output product. In addition, the timeline could contain buttons to play through the process at a particular speed, or fast-forward or rewind through the process. The timeline could be used to control the point in time at which the geometry and/or other aspects of the worksite are displayed within the display. Further, the timeline could be used to indicate a particular point in time either for purposes of simulating the output product or for visualizing within software an actual physical building process taking place within the worksite. Further, a user could modify the design of the output product via the graphical interface.
• In some examples, the display could provide users with multiple 3D views of the worksite, and could allow a user to change the orientation and/or zoom of a particular view. In other examples, the display could present other types of representations of the worksite, such as numerical representations, as well or instead. In further examples, users could be provided with a three-dimensional (3D) modeling graphical interface that allows the user to alter one or more variables describing a worksite and/or the desired output product that affect a building process in the worksite.
• In further examples, the graphical interface could include parameters describing aspects of the process during runtime. In particular, robot parameters could be displayed that describe characteristics of therobotic device210, such as the position of therobotic device210, physical tools currently being used by therobotic device210, and/or axes along which therobotic device210 is currently operating within the worksite. Additionally, tool parameters could be displayed describing operating characteristics of the end-effector220. For instance, an amount of power being supplied to a spindle or an amount of force being used with a gripper could be displayed within an example graphical interface. Additionally, the graphical interface could display sensor data. The graphical interface could also contain controls related to ordering and/or speed of execution of tasks. Further, the graphical interface could contain controls relating to the robot actors, such as robot positions and diagnostics. Additionally, the graphical interface could allow for control of different attributes of the output product. Within the graphical interface, controls could be provided for manipulating one or more tasks being executed during runtime. For example, a user could be able to interact with graphical using touch input in order to modify a building process by altering planned tasks in real time or almost real time.
• In some examples, a graphical interface could include a device control in order to select a particular device within a worksite. For example, the graphical interface could display the robot actors within worksite and could allow for a selection of a particular robotic device. Additionally, the graphical interface could include robot parameters, such as position information describing the current position of robotic devices. In some examples, the position could be displayed as Cartesian coordinates, as robot axes values, or both. In further examples, the position information could reflect the position of an end-effector of a robot actor or of a physical tool mounted on the robot's end-effector.
• Additionally, the input/output unit258 could receive user-input (e.g., from the user of the controller250). In particular, the input/output unit258 could allow for interaction with the graphical interface such as for scrolling, providing text, and/or selecting various features of the application, among other possible interactions. The input/output unit258 could take on various forms. In one example, the input/output unit258 could include a pointing device such as a computing mouse used for control of the graphical interface. However, if the input/output unit258 includes a touch screen display, touch-input could be received (e.g., such as using a finger or a stylus) that allows for control of the graphical interface. In another example, the input/output unit258 could include a keyboard that provides for selection of numbers, characters and/or symbols to be displayed via the graphical interface. For instance, in the arrangement where the input/output unit258 includes a touch screen display, portions the display could show the keyboard. Thus, touch-input on the portion of the display including the keyboard could result in user-input such as selection of specific numbers, characters, and/or symbols to be shown on the graphical interface through the display. In yet another example, the input/output unit258 could include a voice input device that receives audio input, such as from a user through a microphone, that is then interpretable using one of various speech recognition techniques into one or more characters that may be shown through the display. Other examples may also be possible.
• C. Worksite Automation System
• Example embodiments may provide for a system and processes for worksite automation in manufacturing, fabrication, and construction worksites, among other types of worksites. Worksite automation could involve automating the process of designing and/or building a product in a worksite. In particular, the worksite automation system could determine a model for a product, and could then generate a sequence of tasks that could be performed to build the product according to the model. And the sequence of tasks could be executed by available actors that are located in the worksite. When executing the tasks, the actors could utilize available resources disposed in or near the worksite. In an example, the tasks could be executed by robotic devices, and therefore, the product could be built, partially or entirely, by robotic devices.
• FIG. 3 illustratesphases300 of building a product, according to an exemplary embodiment. As illustrated inFIG. 3, building a product could involve three phases: adesign phase302, asimulation phase304, and abuild phase306. In particular, thedesign phase302 and thesimulation phase304 could involve processes that are performed prior to building the product, and therefore, thedesign phase302 and thesimulation phase304 could collectively be referred to as a pre-build phase. However, as explained below, the system could return to the pre-build phase after thebuild phase306 has commenced.
• Furthermore, each of thephases300 could be performed with little to no input from a user. As such, the system could semi-autonomously or autonomously build the product. Further, thedesign phase302 and thesimulation phase304 could be performed using a computing device. In some examples, thedesign phase302 and thesimulation phase304 could be performed using the same computing device that provides instructions to actors during thebuild phase306.
• In an embodiment, and as illustrated inFIG. 3, the worksite automation system could receive aninput308. Theinput308 could be received from a user (e.g., via an input to a computing device) or could be received from another computing device. Within examples, theinput308 could be indicative of an instruction to build or assemble a product in the worksite. As such, responsive to receiving theinput308, the system could build or assemble the product. In particular, the system could build the product by performing processes associated with each of thephases300. As illustrated inFIG. 3, responsive to receiving theinput308, the system could determine a design of the product by performing processes associated with thedesign phase302.
• a. Design Phase
• FIG. 4 illustrates processes that are associated with each of thephases300 of building a product, according to an exemplary embodiment. As illustrated inFIG. 4, in afirst process402 of thedesign phase302, the system could be configured to determine a model for the product. The model of the output product could be a two-dimensional (2D) or three-dimensional (3D) representation of the product. Generally, a model could indicate how the physical structure of the product will be configured and how the product will function once built by the system. The model could also be indicative of other features of the product, such as characteristics of the materials and parts that are used to build the product.
• In one implementation, the system could determine the model for the product by receiving the model from a user of the system. For instance, the user could design the model, and could then include the model as part of theinput308. Typically, the user could use design software to develop the model. In particular, the designed model could be a 2D or 3D computer-aided design (CAD), among other types of models. For instance, in a construction worksite, building information modeling (BIM) could be used to design and model a construction project. The type of model could depend on factors such as the conventions of the product type and/or the user's preferences.
• In another implementation, the system could determine the model for the product by receiving the model from another system or device. For instance, responsive to receiving theinput308, the system could retrieve the model from a database. The database could include models of a variety of products, such as products that are commonly constructed or assembled. Additionally and/or alternatively, the database could be a commercial database that stores models that could be purchased by the system.
• In yet another implementation, the system could determine the model for the product by generating the model for the product. In this implementation, theinput308 could be indicative of a type of the product and one or more constraints on the product. The product type and the one or more constraints could be defined by the user, for instance. Once the system receives theinput308, the system could then generate based on the product type and the constraints the model for the product. More specifically, the system, based on the product type, could determine characteristics that define the product type. The system could then use the determined characteristics and the one or more constraints to generate the model such that the model satisfies the product characteristics and the desired constraints.
• In an embodiment, the constraints that are included in theinput308 could define desired high-level specifications of the product, such as the genus or species of the product. The high-level specifications could also include materials to be used in the product, functional features of the product, aesthetic features of the product, etc. For instance, the constraints could define maximum and/or minimum values for parameters (e.g., dimensions) of the product. Additionally and/or alternatively, the constraints could be indicative of spatial considerations in the worksite (e.g., where to build the product, location of obstacles, etc.). Additionally and/or alternatively, the constraints could define low-level specifications, which could be indicative of constraints on relationships (e.g., relative positioning) between components of the product. Additionally and/or alternatively, the constraints could be indicative of performance constraints on a building process of the product. Examples of performance constraints include, but are not limited to, building time, speed, manufacturing methods, efficiency, cost, material waste, resources used, etc.
• Consider, for example, that theinput308 indicates that the product type is a chair. Theinput308 could include high-level constraints that indicate a type of chair (e.g., armchair), desired materials to be used in the armchair, load support constraints (e.g., minimum/maximum weight support, load bearing surface locations, etc.), where to install the armchair in the worksite, etc. Then, based on theinput308, the system could generate a model for the armchair.
• FIG. 5 illustrates steps of theprocess402 of determining a model, according to an exemplary embodiment. As illustrated inFIG. 5, the first step502 could be for the system to determine characteristics that define the type (e.g., class) of product. The characteristics could be functional characteristics that define what constitutes a structurally sound structure of the product type. The characteristics could also be non-functional characteristics that define aesthetic features of the product type. The characteristics could be shared amongst all or a portion of products that fall with the product type. Within examples, the characteristics could be stored in a database that could be accessed by the system.
• As illustrated inFIG. 5, thenext step504 could be for the system to determine components of the product. In particular, the system could use the product type to determine the possible components of the product. The system could, for example, determine from a database the components that are used in the product type. For instance, the database could include templates of various product types. A template is indicative of components that could be included in the product type, such as components that are common amongst products of the product type. In some examples, a template could also be indicative of a sample arrangement of components in a product of the product type.
• FIG. 6 illustrates an example template of anarmchair600, according to an exemplary embodiment. As illustrated inFIG. 6, the template could include components of thearmchair600, such as an acceptable number of legs and arms. For instance, thearmchair600 includes fourlegs606. The template could also specify that thearmchair600 could include a back-support610, aseat604, and twoarms602. The template could also specify a sample arrangement of the components of the armchair. For example, as illustrated inFIG. 6, the template could indicate each of thelegs606 is coupled one end to theseat604. Further, the template could be indicative of possible relationships between components of the product (e.g., relative positioning of the components).
• Additionally, the template could indicate possible characteristics of thearmchair600. As illustrated inFIG. 6, the template could indicate a range of acceptable values of dimensions of the various components of thearmchair600. In this example, the template of thearmchair600 illustrates a height H1 indicative of a height of theseat604, an armrest height H2, an arrest width W2, a seat width W1, a backrest height H3, a seat length L2, an armrest length L1, among other dimensions. Each value could be indicative of a range of acceptable values for the respective feature. The template could also be indicative of non-functional characteristics, such as acceptable materials from which the components of thearmchair600 could be manufactured. Examples of acceptable materials that could be used in thearmchair600 include solid wood, wood slats, padded leather, stuffed fabric, metal, molded plastic, among other materials.
• Returning toFIG. 5, after the system performs theprocess504 of determining the product's components, the system could then perform theprocess506 of determining a model for each of the components. In one implementation, the template of the product type could specify sample models of each component, and therefore, the system could determine from the template the model for each component. If the system uses the sample component models, the system could, if necessary, adjust the model of each component such that each model satisfies any constraints on the product.
• In another implementation, the system could generate a model for each component. In this implementation, the system could use characteristics of a component and the constraints to iteratively generate a geometry for the component. Iteratively generating a component's geometry could involve the system generating based on the component's characteristics a model for the component, and then iteratively adjusting the model until the model satisfies the constraints on the component. Alternatively, the system could generate based on a component's characteristics a plurality of models for the component, and then could select a model that satisfies the constraints.
• As illustrated inFIG. 5, after performingprocess506, the system could then performprocess508 of generating a model for the product. In an example, the system could iteratively generate one or more models by incorporating the one or more models of each component of the product (that were determined during process506). Iteratively generating one or more models could involve generating one or more models for the product such that each model includes a different permutation of the various models of the components. Further, in some examples, the models of the components could be discrete such that the model may not include a design of a connector that connects one component to another component. In such examples, generating a product model may include generating designs of connectors between the different components. Additionally and/or alternatively, generating a product model could include designing aesthetic features of the product.
• In one example, the system could generate a model in a bottom up process. The system could first determine how to incorporate two components together. This step could include determining functional features of the components, the designs of connectors between the two components, aesthetic features, etc. The system could then connect the two components, and could iteratively adjust the model of the two components until the model satisfies any constraints on the two components. Next, the system could determine how to incorporate another component with the two components. This process could be repeated until all of the components are incorporated into the model.
• Once the system has generated a model of the product, the system could then iteratively adjust the model until the model satisfies the constraints. In an embodiment, iteratively adjusting a model so that the model satisfies the constraints may include adjusting one or more parameters of the model. The parameters that could be adjusted include the geometry of the components, dimensions of the components, materials of the components, etc. The system could adjust these parameters as long as the values of the parameters fall within the range of values defined by the product's characteristics. In addition the system could adjust the parameters such that the model satisfies any performance constraints. If the system cannot adjust a model to satisfy the constraints, the system could determine that the model is not buildable, and may therefore discard the model. Other methods of generating a model for a product are also possible. For instance, the system could generate a model by randomly and iteratively adjusting parameters of the different components of the product until the model satisfies the constraints on the model.
• Once the system has generated one or models of the product, the system could then select a model from the one or more models. In an example, the system could select a model that best satisfies a performance constraint. For example, the performance constraint could indicate that the system should optimize the model for cost. Accordingly, the system could select the model that has the least estimated cost. In some examples, a user of the system may define a default performance constraint for which to optimize. Alternatively, if there isn't a performance constraint on the project, the system may select any of the models that satisfy the constraints. Examples of performance constraints include optimizing for an aesthetic or functional property, optimizing for time, optimizing for accuracy, among other examples.
• FIGS. 7A-7F illustrate an example of the system performing theprocess402 of determining a model of a product. In this example, theinput308 into the system could indicate that the product is a table. And theinput308 could also indicate desired constraints on the table. In an embodiment, responsive to receiving theinput308, the system could commence thedesign phase302 by performing theprocess402 of determining a model of the table. In particular, the first step of theprocess402 is determining the product characteristics of the table (step502 inFIG. 5). Within examples, the system could use a template of a table to determine the characteristics of the table.
• FIG. 7A illustrates anexample table template700, according to an exemplary embodiment. The template could be indicative of characteristics of a table, such as the different components of the table, possible arrangements of the components, possible dimensions of the table, materials from which the table could be constructed, among other characteristics. As illustrated inFIG. 7A, thetable template700 could include asurface702 that is supported by a support structure. For instance, thetable template700 could indicate that the support structure of the table is one or more legs.
• Furthermore,FIG. 7A also indicates the desired constraints on the table. For instance, the constraints could be indicative of desired dimensions of the table, such as a maximum perimeter of the table, which is indicated inFIG. 7A by a maximum width w₁and maximum length l₁. The constraints could also be indicative of a desired height h₁of the table. As explained above, the constraints could also be indicative of dimensions of the table, materials from which to construct the table, performance constraints, load constraints, etc.
• Once the system has determined the characteristics of the table, the system could then generate a model for each of the components of the table. In particular, the system could generate, using the processes described above, one or more models of components of the table. In this example, the components of the table are asurface702 and a structure that supports thesurface702 such that thesurface702 is level or horizontal when the table is placed on a level surface. And once the system has determined one or more models for each of the components, the system could, using the process above, generate one or more models for the table.
• FIGS. 7B-7F each illustrate a model for the table generated by the system, according to exemplary embodiments. In particular,FIGS. 7B, 7C, 7D, 7E, and 7F, illustratetable models704,708,710,712, and714, respectively. As illustrated in these figures, each table model satisfies the characteristics of a table since each table includes a surface and a support structure. Furthermore, as illustrated inFIGS. 7B-7F, each table satisfies the constraints on the table. For instance, each table is within the maximum dimensions specified by the constraints.
• Returning toFIG. 4, afterstep402 of determining the model, the system could be configured to perform abuildability analysis404. In particular, thebuildability analysis404 could determine whether the model satisfies the product constraints. For example, a product constraint on table700 inFIG. 7A could be indicative of a maximum perimeter of the table as indicated by a maximum width w₁and maximum length l₁of the table. In this example, thebuildability analysis404 could determine whether a generated model (e.g., one of the models illustrated inFIGS. 7B-7F) describes a table that satisfies the constraint. If the analysis determines that model does not satisfy that constraint, then the system could determine that the model is not buildable. On the other hand, if the model satisfies the constraints, then the analysis could determine that the model is buildable.
• Thebuildability analysis404 could also determine whether the model satisfies other functional features of the product. For example, the functional features could define the product's properties such as load limits, center of gravity, stability, stresses (e.g., bending stress), strengths (e.g., buckling strength), etc. Thebuildability analysis404 could also determine whether the model satisfies the non-functional features defined by the constraints. For example, the system could determine whether the model satisfies aesthetic features (e.g., engravings, fixtures, etc.) defined by the constraints. Based on the analysis, the system could determine whether to proceed at thedecision element406. If the model is not buildable, the system could return to step402 to determine a new model. If the model is buildable, the system could proceed to step408.
• Instep408, the system could generate a sequence of tasks to build the product according to the model. A task could be any type of task that could be performed in connection with building a product. Example types of tasks include construction tasks, manufacturing tasks, assembly tasks, processing tasks, etc. Additionally, a task could involve interacting with objects located in the worksite, such as parts, tools, obstacles, etc. Furthermore, a task could be performed using one or more tools available in the worksite.
• In an embodiment, the system could generate permutations of the sequence of tasks that can be performed to build a product according to the selected model. In an example, the system could generate the permutations of the sequence of tasks using a tree structure. A tree structure could include a plurality of nodes between a root node and one or more final nodes. The nodes are arranged such that the root node branches into one or more nodes, each of which also branches into one or more nodes. The nodes continually branch into one or more nodes until a final node is reached. Furthermore, the root node represents the current status of the product, and each node could represent a task. And a final node could be indicative of a completion or final task of building a product.
• FIG. 8A illustrates atree structure820, according to an exemplary embodiment. As illustrated inFIG. 8A, thetree structure822 includes aroot node822, which could be indicative of the current statute of the build and/or the worksite. As further illustrated inFIG. 8A, one or more first steps could branch from the root node. In this example, two possiblefirst steps824 and826 branch from theroot node822. As further illustrated, threesecond steps828,830, and832 branch from thefirst step824, and threesecond steps834,836, and838. Each node could branch into one or more nodes until a final node is reached, where a final node represents a final step. As illustrated inFIG. 8A, there are one or more nodes between each of the second nodes (represented by the ellipses) andfinal nodes840,842,844,846,848, and850.
• A permutation of the sequence of tasks could be determined by selecting a continuous sequence of nodes from the root node to a final node. The sequence of selected nodes represents steps of a permutation of the building process. For example, if no portion of the product has yet been built, the root node could be indicative of the current status of the worksite in which the product will be built. And the first node that is selected from the nodes that branch from the root node is the first step of a sequence of tasks to build the product in the worksite. However, if there is a portion of the product that has been built, then the root node is indicative of the built portion of the product. And the first node that is selected from the nodes that branch from the root node is the first step of a sequence of tasks that build off of the built portion of the product.
• The system could then select a permutation of the sequences of tasks that satisfies the constraints. In an example, the system could select a permutation based on any specified performance constraints. For example, a performance constraint could indicate that the system should optimize the model for cost. Accordingly, the system could select the permutation of the sequence of tasks that has the least estimated cost. In some examples, a user of the system could define a default performance constraint to optimize for. Alternatively, if there isn't a performance constraint on the product, the system could select any of the permutations of the sequence of tasks that satisfy the constraints.
• Once the system has selected the permutation of the sequence of tasks, the system could proceed to thesimulation phase304. In thesimulation phase304, the system could perform a simulation atstep410. Specifically, in order to perform the simulation, the system could determine an order of operations in which to perform the selected sequence of tasks. This could include determining resources for each task, assigning the necessary resources to that respective task, and determining an order in which tasks are to be performed. In some examples, the system may not be able to determine an order of operations for the sequence of tasks, and therefore, the system could determine that it is not possible to execute the selected sequence of tasks. In this scenario, the system could determine the permutation of the sequence of tasks is not feasible and may select a different permutation of the sequence of tasks. This process could be repeated until the system selects a permutation of that satisfies the constraints on the model.
• Once the system has determined an order of operations, the system could simulate the performance of the sequence of tasks. During the simulation, the system could perform thebuildability analysis412 to determine whether the sequence of tasks can be performed using the system's resources. In one aspect, the system could determine whether the selected sequence of tasks could be performed by robotic devices available to the system. In an example, the system could determine whether each task of the selected permutation could be performed by a robotic device. In another aspect of the buildability analysis, the system could determine whether the system has the materials and parts to build the product according to the selected model and the selected sequence of tasks. If the system detects that buildability could fail for any reason (e.g., lack of resources, etc.), then the system could return to thestep408 of generating a sequence of tasks in order to select a different permutation of the sequence of tasks.
• Furthermore, during thesimulation step410, the system could assign tasks of the selected sequence of tasks to one or more actors that are available in the worksite. In an example, the system could maintain a worker schedule, which indicates a schedule of the workers (both robotic and human) located in the worksite. The system could then assign the tasks of the building process to the available actors in the worksite. In an example, the system could assign all of the tasks to available robotic devices such that the product is built exclusively by robotic devices. The task that a robotic device is assigned could depend on a type of end-effector of the robotic device. The system could assign a task to a robotic device that includes an end-effector that is configured to perform that particular task. For instance, a task that involves moving an object could be assigned to a robotic device that includes a gripper end-effector.
• b. Simulation Phase
• Once the tasks have been assigned to actors, the system could simulate the sequence of tasks. In this step, the system could simulate the actors utilizing resources to execute the one or more tasks of the sequence of tasks in the worksite. The simulation could be indicative of how the robotic devices in the worksite would execute the tasks in the order specified by the order of operations. In some examples, the simulation could be performed at the same speed at which the robotic devices would perform the tasks in the worksite. In other examples, the simulation could be performed at a faster speed. Further, in some examples, a representation of the simulation could be displayed on a display of a computing device. Additionally and/or alternatively, the simulation could be recorded for future analysis.
• Furthermore, thebuildability analysis412 could determine whether the simulated robotic devices can perform their assigned tasks using the specified resources and in the order indicated by the order of operations. In one aspect of thebuildability analysis412, the system could determine whether each simulated robotic device can perform its assigned task. In an example, the system could determine whether each simulated robotic device (that is assigned a task) will encounter a problem when performing its assigned task. For instance, the buildability analysis could detect failures or problems due to kinematic limitations of the robotic devices (e.g., joint limits and/or reachability limits). In another example, the system could determine velocity or acceleration constraints on our workers, and could identify whether certain kinds of tasks (e.g. glue deposition toolpaths) exceed the capabilities of the available robotic devices.
• Additionally, in another aspect, the buildability analysis could detect any collisions that could occur between any of the objects that are located in the worksite, such as potential collisions between robotic devices performing their respective tasks, collisions between a robotic device and an object in the worksite, among other examples. The system could also determine whether each simulated robotic device could reach the areas where it needs to be located in order to perform its assigned task.
• In yet another aspect of thebuildability analysis412, the system could determine whether the simulated structure is stable throughout the construction of the product. In yet another aspect, the system could also determine whether the resources available to the system are adequate to complete the project. During thebuildability analysis412, if the system detects that the design is not buildable using the selected building process (i.e., selected model and/or selected sequence of tasks), the system could make a decision atdecision element414 to return to thestep408 of generating a sequence of tasks in order to select a different permutation of the sequence of tasks. Conversely, if the system completes the simulation and does not detect any issues with the sequence of tasks, the system could make a decision atdecision element414 to proceed to thebuild phase306.
• c. Build Phase
• In thebuild step416, the system could implement the selected model in the worksite by causing actors in the worksite to perform the selected sequence of tasks. To perform a step of the sequence of tasks, the system could send instructions to the actors that cause the actors to perform their assigned tasks. For instance, the instructions for a task could be sent to a robotic device when the robotic device is scheduled to perform the task. In another example, the system could send instructions to human workers via computing devices, which the workers could be using to interface with the system.
• Additionally, in thebuild phase306, the system could generate and maintain a world map that includes data indicative of the worksite. The system could use data received from devices (e.g., global andlocal sensors230 and240 ofFIG. 2) located in the worksite to generate the world map. The world map could include an overview of the all of the actors in the worksite, the tasks that are carried out by the actors, and the relevant locations of resources and objects in the worksite. The world map could also include a description, location, and amount of each of the resources in the worksite. For instance, the world map could include an inventory of each type of material available to the system. Furthermore, the system could dynamically update the world map in real-time using at least the data received from devices (e.g., robotic devices, sensors, etc.) located in the worksite. As such, the world map not only could define spatial features of the worksite, but could also include physical details of the worksite, such as changes that could be occurring to the worksite in real-time. Accordingly, a live link between the world map and the physical world could be established.
• FIG. 8B illustrates aworld map800 of arobotic device802 constructing abridge804. As illustrated inFIG. 8B, theworld map800 depicts an “as-built”portion808A of thebridge804. As also illustrated inFIG. 8B, the word map also depicts an outline of a design808 of a portion of thebridge804 that is to be built. In this example, the robotic device has commenced the build phase of building the bridge. Therefore, therobotic device802 could be performing tasks associated with a sequence of tasks to build thebridge804 usingresources806.
• In an embodiment, the system could, periodically or continuously, perform abuildability analysis418 during thebuild phase306. Thebuildability analysis418 analyzes the feasibility of building the product according to the sequence of tasks and/or the model. Within examples, thebuildability analysis418 could be performed before building the product. Additionally and/or alternatively, thebuildability analysis418 could be performed while the product is being built. Note that althoughprocess416 andprocess418 are shown as two separate processes inFIG. 4, the system could perform the process simultaneously. As such, the system could perform thebuildability analysis418 as the workers are performing thebuild416.
• In an embodiment, to perform thebuildability analysis418, the system could analyze data from the world map to detect any issues that could affect the feasibility of building the product. In one aspect, the system could compare the as-built portion of the product to the model of the product. If the system detects a deviation of the as-built portion from the model, the system could then determine whether the product would still satisfy the constraints. Additionally and/or alternatively, the system could determine whether the tasks of the sequence of tasks that have not been performed could still be performed. Additionally and/or alternatively, the system could determine whether performing the tasks that have not been performed would further compound on the deviation such that the final product would not satisfy the constraints.
• For example, the system could analyze theworld map800 of the bridge project. For instance, system could determine whether the as-builtportion808A is built according to the model, perhaps by comparing the dimensions of the as-builtportion808A to the dimensions in the corresponding portion of the model. The system could also determine characteristics of the as-builtportion808A, such as the weight bearing capacity of the bridge, and could compare the determined characteristics to the desired characteristics. The system could also determine whether therobotic device802 could build the unbuilt portion of thebridge804, perhaps by determining whether theresources806 include the resources necessary to build thebridge804. The system could also determine whether tasks associated with building theunbuilt portion808B could be performed. For instance, the system could determine whether there was any delay in building thebridge804, and if there was a delay, whether the system could still complete the unperformed tasks within a time-limit that is set by the constraints.
• The system could analyze other data from the world map that could affect the buildability of a project. In one example, the system could determine whether the resources in the worksite are sufficient to complete the construction of the product. In another example, the system could detect any changes or events in the worksite that could disrupt the performance of the sequence of tasks. For instance, the system could detect if the conditions in the worksite change in a way that can affect the building process (e.g., significant temperature change). The system could also detect any new obstacles that are introduced into the worksite, which may impact the performance of the sequence of tasks.
• In another aspect of thebuildability analysis418, the system could receive an input that indicates a change to the design and/or constraints. And in response, the system could determine whether the product is still buildable using the selected sequence of tasks in light of the changes to the design and/or constraints. For example, the system could receive an input that indicates a change to a performance constraint. In response, the system could determine whether the current sequence of tasks could satisfy the new performance constraint.
• If the system determines that the product is not buildable using the current sequence of tasks, the system could determine to select a new sequence of tasks. In an embodiment, in response to determining that the product is not buildable, the system could detect an error atdecision element420. Then, the system could return to generate sequence of tasks step408. Atstep408 of generating a sequence of tasks, the system could generate a new sequence of tasks. In an example, the new sequence of tasks could achieve the same design as the previous sequence of tasks, but possibly using different steps.
• In an embodiment, the system could use the tree structure to generate a new sequence of tasks. As explained above, the root node of the tree structure represents the current state of the project. In this case, the as-built portion of the product is current state of the project, and therefore, is represented as the root node. As such, the first nodes that branch from the root node represent first steps that build off of the built portion of the product. Then, the system could select a sequence of tasks that starts from the root node and ends at a final node of tree structure. The final node of the new sequence of tasks could be the same final node as the original sequence of tasks, or could be a different final node. By generating the new sequence of tasks in this manner, the system could build on the as-built portion of the output product, and thus the system could preserve work that has already been performed.
• To facilitate generating the new sequence of tasks, the system could determine the last task that was performed before the system determined to generate a new sequence of tasks. The system could then determine the node in the building tree with which that task is associated, and could designate that node as the root node. The system may then select a continuous sequence of nodes between the root node and a final node. The new sequence of nodes is different than the originally selected sequence of nodes. Once the new sequence is generated, the system could then perform the processes as indicated inFIG. 4.
• In some examples, the system could determine that a sequence of tasks for the original design is not achievable. The system could then return to step402 of determining a model for the product. Atstep402, the system could select one of the models that were previously generated and that satisfy the constraints. Alternatively, the system could generate a new model that builds off of the as-built portion of the product. Once the new model is generated the system could perform the processes of each phase as described above. In some examples, after the system after completing the processes of thedesign phase302, may skip thesimulation phase304 and move directly to thebuild phase306.
• Nonetheless, once the new sequence of tasks and/or model is generated, the system could resume building the product. In some examples, if the system detects a potential issue before the issue actually occurs, the system could adjust to the issue without stopping thebuild416. However, if the issue that arises affects the task that is currently being performed, the system could pause thebuild416 until the system determines an adjustment. Once the adjustment is determined, the system could resume thebuild416.
• Operations relating to the robotic control system described above may be implemented as a method by one or more processors. As explained above, the robotic control system may operate one or more robotic devices. Therefore, there may be exchange of signals between the robotic device and the robotic control system.Example methods900 and1000 that describe the operations of a robotic control system are illustrated in the form of flowcharts inFIGS. 9 and 10 respectively.
• FIG. 9 illustrates anexample method900 of the worksite automation system building an output product, according to an example embodiment. As illustrated inblock902,method900 involves during a pre-build phase of building a product, generating a first sequence of tasks to build the product according to a model of the product. As illustrated byblock904, themethod900 also involves during a build phase, causing one or more robotic devices to build the product by beginning to execute the first sequence of tasks. As shown inblock906, themethod900 further involves during the execution of the first sequence of tasks, performing a buildability analysis to determine a feasibility of completing the product by executing the first sequence of tasks.
• As illustrated byblock908, themethod900 further involves determining, based on the analysis, that it is not feasible to complete the product by executing the first sequence of tasks. As shown byblock910, themethod900 additionally involves in response to determining that it is not feasible to complete the product by performing the first sequence of tasks, generating a second sequence of tasks to complete the product according to the model, where the second sequence of tasks is different from the first sequence of tasks. As shown byblock912, themethod900 yet further involves causing the one or more robotic devices to continue building the product by beginning to execute the second sequence of tasks. In particular, the system begins to execute the second sequence of tasks instead of executing the first sequence of tasks. Furthermore, this process can be cyclical. For instance, during the execution of the second sequence of tasks, the system could determine, based on a buildability analysis, that it is not feasible to complete the product by performing the second sequence of tasks. Accordingly, the system could generate a third sequence of tasks. Then, the system could begin to execute the third sequence of tasks instead of executing the second sequence of tasks, and so forth.
• FIG. 10 illustrates anotherexample method1000 of the worksite automation system building an output product, according to an example embodiment. As shown byblock1002, themethod1000 involves during a pre-build phase, generating a first sequence of tasks for a first model of a product. Further, as shown byblock1004, themethod1000 involves, during a build phase, causing one or more robotic devices to build the product by beginning to execute the first sequence of tasks. As shown byblock1006, themethod1000 also involves during the execution of the first sequence of tasks, performing a first buildability analysis of the first model to determine the feasibility of building the product according to the first model. Furthermore, as shown byblock1008, themethod1000 involves based on the analysis, determining a second model. Additionally, as shown byblock1010, themethod1000 involves causing the one or more robotic devices to continue building the product according to the second model.
• D. Example Robotic Device
• FIG. 11 illustrates a robotic device, according to an example embodiment. In particular,robotic device1100 may include arobotic arm1102 with anend effector1104 capable of being equipped with one or more different tools, grippers, or guides. Therobotic arm1102 may be capable of motion along six degrees of freedom, depicted inFIG. 11A as A1-A6. In certain examples,robotic device1100 may be further capable of motion along one or more axes A0, such as along a rail which is not shown that allows side to side movement. In certain embodiments, instructions may be given to positionend effector1104 at a specific location, and the positions of therobotic arm1104 along A1-A6 and/or ofrobotic device actor1100 along one or more axes A0 may be calculated by a process of the related controller. In alternative embodiments, position control ofrobotic device1100 and/orrobotic arm1102 may require separate, individual settings and control commands. Robotic devices operating with fewer degrees of freedom may be used in some examples as well or instead.
• Therobotic device1100 may have a fixed end effector or may be able to interchange end effectors. In order to interchange end effectors, therobotic device1100 may have access to a plurality of end effectors that may be stored on or near therobotic device1100. The plurality of end effectors may include end effectors of different types, such as tool end effectors, gripper end effectors, and guide end effectors. As such, therobotic device1100, which has the ability to interchange end effectors, may be assigned different tasks that require different types of end effectors. As explained herein, arobotic device1100 may select an end effector based on a task that is assigned to therobotic device1100.
• FIG. 12 shows an example configuration of arobotic device1200. Generally, arobotic device1200 may be any device that has a computing ability and interacts with its surroundings with an actuation capability and/or with ability to emit/generate physical phenomena such as light and/or sound, among others. For instance, therobotic device1200 may be a humanoid robot, a robotic arm, or a quadruped robot, among others. A robotic device may also be any device that is generally understood to those of ordinary skill in the art as being a “robotic.” Therobotic device1200 may also be referred to as a robotic device, a robotic manipulator, a robot client, or a robot, among others.
• Therobotic device1200 is shown to include processor(s)1202,data storage1204,program instructions1206,controller1208, sensor(s)1210, power source(s)1212, actuator(s)1214, and movable component(s)1216. Note that therobotic device1200 is shown for illustration purposes only androbotic device1200 may include additional components and/or have one or more components removed without departing from the scope of the disclosure. Further, note that the various components ofrobotic device1200 may be arranged and connected in any manner.
• Moreover, the above description of processor(s)252,data storage254,program instructions256, sensors (e.g., local sensor(s)230 and global sensor(s)240), and/orpower source262, may apply to any discussion below relating to the respective component being used in another system or arrangements. For instance, as noted,FIG. 12 (among other possible figures) illustrates processors, data storage, program instructions, sensors, and/or power as being incorporated in another arrangement. These components at issue may thus take on the same or similar characteristics (and/or form) as the respective components discussed above in association withFIG. 2. However, the components at issue could also take on other characteristics (and/or form) without departing from the scope of the disclosure.
• As noted, therobotic device1200 may include a controller1208 (e.g., taking the form of a microcontroller). Thecontroller1208 may include processing unit and data storage, and may be arranged to manage or carry out various operations (e.g., individually or in collaboration with processor(s)1202). Thus, thiscontroller1208 could take on the same or similar characteristics (and/or form) as the above-mentionedcontroller250, but could take on other characteristics (and/or form) as well. So in some implementations, thecontroller250 may be incorporated as part therobotic device1200 and thuscontroller250 may itself becontroller1208. In other implementation,controller1208 may be included as part of therobotic device1200 andcontroller250 may be separate from therobotic device1200. Regardless of the implementations, these controllers may take various forms. For instance, a controller may take the form of a chip set, a server system, a digital signal processor, a programmable logic controller, and/or a sampled-data system, among other possibilities. Moreover, a controller could also be referred to herein as a control system, among other.
• Additionally, therobotic device1200 may also include one or more actuator(s)1214. An actuator is a mechanism that may be used to introduce mechanical motion. In particular, an actuator may be configured to convert stored energy into movement of one or more components. Various mechanisms may be used to power an actuator. For instance, actuators may be powered by chemicals, compressed air, hydraulics, or electricity, among other possibilities. With this arrangement, actuator(s)1214 may cause movement of various movable component(s)1216 of therobotic device1200. The moveable component(s)1216 may include appendages/members such as robotic arms, legs, and/or hands, among others. The moveable component(s)1216 may also include a movable base, wheels, and/or end effectors, among others. Further, when arobotic device1200 includes at least one end effector, such an end effector may be a tool (e.g., a screwdriver, drill, welding iron, or some combination thereof) and/or a gripper, among others as discussed above.
• E. Example Computing Device
• FIG. 13 is a block diagram showing components of anexample computing device1300 that includes one ormore processors1302,data storage1304,program instructions1306, power source(s)1308,sensors1310,display1312, and Input Method Editor (IME)1314. Note that thecomputing device1300 is shown for illustration purposes only andcomputing device1300 may include additional components and/or have one or more components removed without departing from the scope of the disclosure. Further, note that the various components ofcomputing device1300 may be arranged and connected in any manner
• Display1312 may take on any form (e.g., LED, LCD, OLED, etc.). Further,display1312 may be a touchscreen display (e.g., a touchscreen display on a tablet).Display1312 may show a graphical user interface (GUI) that may provide an application through which the user may interact with the systems disclosed herein.
• Further, thecomputing device1300 may receive user input (e.g., from the user of the computing device1300) viaIME1314. In particular, theIME1314 may allow for interaction with the GUI such as for scrolling, providing text, and/or selecting various features of the application, among other possible interactions. TheIME1314 may take on various forms. In one example, theIME1314 may be a pointing device such as a computing mouse used for control of the GUI. However, ifdisplay1312 is a touch screen display, user touch input can be received (e.g., such as using a finger or a stylus) that allows for control of the GUI. In another example,IME1314 may be a text IME such as a keyboard that provides for selection of numbers, characters and/or symbols to be displayed via the GUI.
• For instance, in the arrangement wheredisplay1312 is a touch screen display, portions of thedisplay1312 may show theIME1314. Thus, touch-input on the portion of thedisplay1312 including theIME1314 may result in user-input such as selection of specific numbers, characters, and/or symbols to be shown on the GUI viadisplay1312. In yet another example, theIME1314 may be a voice IME that may be used that receives audio input, such as from a user via a microphone of thecomputing device1300, that is then interpretable using one of various speech recognition techniques into one or more characters than may be shown viadisplay1312. Other examples may also be possible.
• Thecomputing device1300 may also include a communication unit1316. The communication unit1316 may include wired links and/or wireless links (e.g., using various wireless transmitters and receivers). A wired link may include, for example, a parallel bus or a serial bus such as a Universal Serial Bus (USB). A wireless link may include, for example, Bluetooth, IEEE 802.11 (IEEE 802.11 may refer to IEEE 802.11-2007, IEEE 802.11n-2009, or any other IEEE 802.11 revision), Cellular (such as GSM, GPRS, CDMA, UMTS, EV-DO, WiMAX, HSPDA, or LTE), or Zigbee, among other possibilities. Furthermore, multiple wired and/or wireless protocols may be used, such as “3G” or “4G” data connectivity using a cellular communication protocol (e.g., CDMA, GSM, or WiMAX, as well as for “WiFi” connectivity using 802.11).
• Thecomputing device1300 may be coupled with one or more sensors such as optical flow sensors, force sensors, proximity sensors, motion sensors (e.g., gyroscopes, and/or accelerometers), load sensors, position sensors, thermal imaging sensors, depth sensors (e.g., RGB-D, laser, structured-light, and/or a time-of-flight camera), ultrasonic range sensors, infrared sensors, optical sensors, Radio Frequency identification (RFID) systems, Near Field Communication (NFC) chip, wireless sensors, light sensors, touch sensors (e.g., capacitive sensors), cameras (e.g., color cameras, grayscale cameras, and/or infrared cameras), and/or range sensors (e.g., ultrasonic and/or infrared), among others.
III. ADDITIONAL FEATURES
• A. Serialization
• Within examples, the various data that is generated by the system could be serialized and stored in a data storage of the system (e.g.,data storage1304 inFIG. 13). For instance, the system could serialize a product model, a tree used to generate possible task sequences, a world map of the worksite, among other data generated by the system. In particular, serializing the data could allow the system to pause building a product and resume at a later time. For example, the building process could be paused once working hours are over, and could be resumed at the start of working hours. Once the building process is resumed, the system could retrieve the last instance of the data (e.g., the last model, tree structure, world map, etc.). Then, the system could update the data to account for any changes that may have occurred while the system was paused. Once the data is updated, the system could perform a buildability analysis to determine whether the last instance of the model and/or sequence of tasks could still be used.
• B. Machine Learning
• In addition to the examples described above, the system could use machine learning, statistical analysis, and/or other analysis techniques to learn how a particular product is defined. In particular, the system could learn what a particular product is by ingesting 3D models of chairs, e.g., from online catalogs or other sources. Then the system could learn the functional properties of the chair and/or of components thereof. For instance, the system could analyze data such as a set of properties of the product. For example, the set of properties could be indicative of aesthetic properties, such as color, markings, visual patterns, and surface finish/texture, among other properties. Additionally or alternatively, the set of properties could be indicative of mechanical properties, such as bending strength, brittleness, bulk modulus, coefficient of friction, compressive strength, creep, elasticity, fatigue strength, flexibility, fracture toughness, hardness, plasticity, resilience, shear strength, stiffness, stress/strain properties, surface roughness, tensile strength, toughness, viscosity, yield strength, and weight, among other properties. Additionally and/or alternatively, the set of properties could be indicative of geometric properties such as shape, size, orientation, angles, etc.
• Additionally or alternatively, the set of properties could be indicative of electrical and/or magnetic properties, such as capacitance, conductivity, density, dielectric strength, field properties, inductance, permittivity, and resistance, among other properties. Additionally or alternatively, the set of properties could be indicative of chemical properties, such as corrosion resistance, flammability, pH, reactivity, stability, surface energy/tension, and toxicity, among other properties. Additionally or alternatively, the set of properties could be indicative of manufacturing properties for coating, cutting, drilling, forming and shaping processes, heat treating, joining, machining, rolling, sanding, and welding, among other techniques. Additionally or alternatively, the set of properties could be indicative of optical properties, such as absorbance, fluorescence, photosensitivity, reflectivity, refractive index, scattering, and transmittance, among other properties. Additionally or alternatively, the set of properties could be indicative of thermal properties, such as boiling point, critical point, emissivity, melting point, specific heat, thermal conductivity, thermal diffusivity, and thermal expansion, among other properties.
• For example, the system could learn what a chair is by learning possible load support constraints (e.g., weight of a person, load bearing surface locations), among other features, that define a chair. Such a system could then generate a model of a chair per its definition of a chair. Furthermore, the system could receive feedback that it could use to refine its definition or understanding of a particular product. Such feedback could include feedback from a user of the product. Additionally, the feedback could be input into the machine learning model so that the system could continuously refine its definition of a product.
IV. CONCLUSION
• The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its spirit and scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those enumerated herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.
• The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example embodiments described herein and in the figures are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.
• A block that represents a processing of information, such as a block of a method described above, may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and/or related data may be stored on any type of computer readable medium such as a storage device including a disk or hard drive or other storage medium.
• The computer readable medium may also include non-transitory computer readable media such as computer-readable media that stores data for short periods of time like register memory, processor cache, and random access memory (RAM). The computer readable media may also include non-transitory computer readable media that stores program code and/or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.
• Moreover, a block that represents one or more information transmissions may correspond to information transmissions between software and/or hardware modules in the same physical device. However, other information transmissions may be between software modules and/or hardware modules in different physical devices.
• The particular arrangements shown in the figures should not be viewed as limiting. It should be understood that other embodiments can include more or less of each element shown in a given figure. Further, some of the illustrated elements can be combined or omitted. Yet further, an example embodiment can include elements that are not illustrated in the figures.
• While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims (20)

We claim:

1. A computer-implemented method comprising:

during a pre-build phase of building a product, generating a first sequence of tasks to build the product according to a model of the product;

during a build phase, causing one or more robotic devices to build the product by beginning to execute the first sequence of tasks;

during the execution of the first sequence of tasks, performing a buildability analysis to determine a feasibility of completing the product by executing the first sequence of tasks;

determining, based on the analysis, that it is not feasible to complete the product by executing the first sequence of tasks;

in response to determining that it is not feasible to complete the product by performing the first sequence of tasks, generating a second sequence of tasks to complete the product according to the model, wherein the second sequence of tasks is different from the first sequence of tasks; and

causing the one or more robotic devices to continue building the product by beginning to execute the second sequence of tasks.

2. The computer-implemented method ofclaim 1 further comprising:

determining, based on the analysis, that it is feasible to complete the product by executing the first sequence of tasks; and

responsively causing the one or more robotic devices to continue building the product by continuing the execution of the first sequence of tasks.

3. The computer-implemented method ofclaim 1, wherein generating a first sequence of tasks comprises:

selecting a sequence of nodes between a root node and a final node of a tree structure, wherein each node is associated with a respective one of the tasks.

4. The computer-implemented method ofclaim 3, wherein generating a second sequence of tasks comprises:

determining (i) a last-performed task of the first sequence of tasks, and (ii) a last-performed node associated with the last-performed task; and

selecting one or more nodes between the last-performed node and a second final node such that the selected nodes comprise a sequence of nodes between the last-performed node and the second final node.

5. The computer-implemented method ofclaim 1, wherein performing a buildability analysis of the first sequence of tasks comprises:

determining that the first building process does not satisfy a specified performance constraint; and

responsively determining that it is not feasible to complete the product by performing the first sequence of tasks.

6. The computer-implemented method ofclaim 5, wherein the specified performance constraint comprises at least one of: time, cost, and efficiency.

7. The computer-implemented method ofclaim 1, wherein the product is being built in a worksite, and wherein the method further comprises:

generating a map of the worksite, wherein the map of the worksite comprises a description of one or more resources available in the worksite and a description of a built portion of the product.

8. The computer-implemented method ofclaim 7, wherein performing a buildability analysis of the first sequence of tasks comprises:

based on the description of the built portion of the product, determining a deviation between the built portion and the model of the product;

determining whether the deviation affects the feasibility of completing the product; and

if the deviation affects the feasibility of completing the product, determining that it is not feasible to complete the product by executing the tasks of the first sequence of tasks.

9. The computer-implemented method ofclaim 7, wherein performing a buildability analysis of the first sequence of tasks comprises:

analyzing the map of the worksite;

based on the analysis, determining that it is not feasible to execute tasks of the first sequence of tasks that have not yet been executed.

10. The computer-implemented method ofclaim 9, wherein analyzing the map of the worksite comprises at least one of:

detecting obstacles in the worksite that impede a robotic device from executing a task of the first sequence of tasks, and detecting a geometry of the robotic device impedes the robotic device from executing a task of the first sequence of tasks.

11. The computer-implemented method ofclaim 1, wherein the method further comprises:

during the pre-build phase, simulating the one or more robotic devices executing the first sequence of tasks;

during the simulation, performing a simulation buildability analysis of the first sequence of tasks to determine the feasibility of executing the first sequence of tasks using the resources available in the worksite; and

based on the analysis, determining to generate a new sequence of tasks.

12. The computer-implemented method ofclaim 1 further comprising:

during the execution of the second sequence of tasks, performing a buildability analysis to determine a feasibility of completing the product by executing the second sequence of tasks;

determining, based on the analysis, that it is not feasible to complete the product by executing the second sequence of tasks;

responsively generating a third sequence of tasks to complete the product according to the model, wherein the third sequence of tasks is different from the second sequence of tasks; and

causing the one or more robotic devices to continue building the product by beginning to execute the third sequence of tasks.

13. A worksite automation system comprising:

one or more robotic devices;

a control system including one or more processors and one or more data storage devices, the control system configured to:

receive an instruction to build a product, wherein the instruction comprises one or more constraints on the product;

during a pre-build phase: (i) determine a model for the product, and (ii) generate a first sequence of tasks to build the product according to the model;

during a build phase, cause the one or more robotic devices to build the product by beginning to execute the tasks of the first sequence of tasks;

during the execution of the first sequence of tasks, perform a buildability analysis of the first sequence of tasks to determine a feasibility of completing the product by executing the first sequence of tasks;

based on the analysis, if it is not feasible to complete the product:

generate a second sequence of tasks to complete the product according to the model, wherein the second sequence of tasks is different from the first sequence of tasks; and

cause the one or more robotic devices to continue building the product by beginning to execute the second sequence of tasks.

14. The worksite automation system ofclaim 13, wherein the control system is further configured to:

if it is feasible to complete the product by performing the first sequence of tasks:

cause the one or more robotic devices to continue building the product by executing the first sequence of tasks.

15. The worksite automation system ofclaim 13, wherein the constraints comprise design constraints and performance constraints on the model of the product.

16. The worksite automation system ofclaim 13, wherein the control system configured to generate a first sequence of task comprises the control system configured to:

select a sequence of nodes between a root node and first final node of a tree structure, wherein each node is associated with a respective one of the tasks.

17. The worksite automation system ofclaim 16, wherein the control system configured to generate a second sequence of tasks for the model comprises the control system configured to:

determine (i) a last-performed task of the first sequence of tasks, and (ii) a last-performed node associated with the last-performed task; and

select one or more nodes between the last-performed node and a second final node such that the selected nodes comprise a sequence of nodes between the last-performed node and the second final node.

18. A computer-implemented method comprising:

during a pre-build phase, generating a first sequence of tasks for a first model of a product;

during a build phase, causing one or more robotic devices to build the product by beginning to execute the first sequence of tasks;

during the execution of the first sequence of tasks, performing a first buildability analysis of the first model to determine a feasibility of building the product according to the first model;

based on the analysis, determining a second model; and

causing the one or more robotic devices to continue building the product according to the second model.

19. The computer-implemented method ofclaim 18, wherein generating the second model comprises generating a second sequence of tasks for the second model, and wherein causing the one or more robotic devices to continue building the product according to the second model comprises causing the one or more robotic devices to execute the second sequence of tasks.

20. The computer-implemented method ofclaim 18, wherein performing a buildability analysis of the first model comprises:

determining a deviation between the built portion and the first model of the product;

determining whether the deviation affects the feasibility of completing the product; and

if the deviation affects the feasibility of completing the product according to the first model, determining that it is not feasible to complete the product according to the first model.

Images