WO2025133222A1

Movatterモバイル変換

Info

Publication number: WO2025133222A1
Application number: PCT/EP2024/088059
Authority: WO
Inventors: Hadil ABUKWAIK; Norman VOLKERT
Original assignee: BASF SE
Current assignee: BASF SE
Priority date: 2023-12-22
Filing date: 2024-12-20
Publication date: 2025-06-26
Anticipated expiration: 2026-06-22

Abstract

The disclosure relates to the field of generative artificial intelligence in industrial use, in particular the monitoring, control, assessment, evaluation and/or implementation of industrial processes. Disclosed are methods, apparatuses, process representations, models, uses, computer elements and systems for monitoring, controlling, operating, assessing, evaluating, and/or implementing industrial processes using generative modelling of industrial processes.

Description

PERFORMANCE OPTIMIZATION IN PROCESS ENGINEERING

TECHNICAL FIELD

The disclosure relates to the field of generative artificial intelligence in industrial use, in particular the monitoring, controlling, assessing, evaluating, planning and/or implementing of industrial processes. Disclosed are methods, apparatuses, process representations, models, uses, computer elements and systems for monitoring, controlling, operating, assessing, evaluating, and/or implementing industrial processes using generative modelling of industrial processes.

TECHNICAL BACKGROUND

The present disclosure relates in general terms to the operation of industrial processes.

SUMMARY

In another aspect a method for operating an industrial process, wherein the industrial process includes multiple process components for operating the industrial process, the method comprising the steps: providing input data including at least one instruction related to at least one performance objective for a specified industrial process, generating one or more numeric representation (s) of the input data, providing the one or more numeric representation(s) to a process model , e.g. to generate at least one operation representation representing multiple process components specific to the at least one industrial process according to the performance objective and optionally receiving the generated at least one operation representation, wherein the process model is trained to generate from the one or more numeric representation(s) of the input data at least one operation representation specific to the performance objective for the specified industrial process, wherein the operation representation specifies process components for operating the industrial process according to the performance objective, providing the at least one operation representation for operating the specified industrial process according to the performance objective.

In another aspect an apparatus for operating an industrial process, wherein the industrial process includes multiple process components for operating the industrial process, the device comprising: an input interface configured to provide input data including at least one instruction related to at least one performance objective for a specified industrial process, a representation generator configured to generate one or more numeric representation (s) of the input data, an operation representation generator configured to provide the one or more numeric representation (s) to a process model, e.g. to generate at least one operation representation representing multiple process components specific to the at least one industrial process according to the performance objective and optionally receiving the generated at least one operation representation, wherein the process model is trained to generate from the one or more numeric representation (s) of the input data at least one operation representation specific to the performance objective for the specified industrial process, wherein the operation representation specifies process components for operating the industrial process according to the performance objective, an output interface configured to provide the at least one operation representation for operating the specified industrial process according to the performance objective.

In another aspect disclosed is a use of any of the at least one operation representation specific to the at least one industrial process generated according to any of the methods disclosed herein or by the apparatuses disclosed herein for operating the at least one industrial process.

In another aspect disclosed is a use of any of the at least one two-dimensional operation representation specific to the at least one industrial process according to any of the methods disclosed herein or by the apparatuses disclosed herein for enriching at least one three-dimensional operation representation specific to the at least one industrial process.

In yet another aspect disclosed is a computer element, such as a computer program product or a machine-readable medium, with instructions, which when executed on one or more computing node(s) or processor(s) is/are configured to carry out the steps of the method(s) disclosed herein or configured to be carried out by the apparatus(es) disclosed herein.

Any disclosure, embodiments and examples described herein relate to the methods, the systems, apparatuses, uses, and computer elements lined out above and below. Advantageously, the benefits provided by any of the embodiments and examples equally apply to all other embodiments and examples.

EMBODIMENTS

In the following, embodiments of the present disclosure will be outlined by ways of embodiments and/or examples. It is to be understood that the present disclosure is not limited to said embodiments and/or examples.

The disclosure relates to the field of generative artificial intelligence in industrial use, in particular the monitoring, controlling, assessing, evaluating, planning and/or implementing industrial processes. Industrial processes are operated according to safety and performance standards, such as operational performance, quality performance, environmental performance. In particular the safety and performance of chemical processes including chemical reactions are sensitive to operation conditions of equipment and instrumentation used in such processes. Depending on the equipment, instrumentation and process flow safe and reliable performance needs to be assured in operation of the process. To do so any physical setup of industrial processes or parts thereof undergo stringent technical validation before any new physical setup is installed or any changes are made to existing physical set up of the process. Similarly, any control or monitoring setup of industrial processes or parts thereof undergo stringent technical validation before any new control or monitoring setup is installed or any changes are made to existing control or monitoring set up of the process. Validation is hence executed by way of digital representations or digital twins. Different forms of digital twins or representations exist ranging from process and instrumentation diagrams (P&IDs) via process flow diagrams (PFDs) to fully enriched 3D representations including architectural limitations or fully enriched operation representations including real-time sensor data for real-time control and/or monitoring. To ensure reliable and safe operation of the industrial process based on digital representations efficient and human centric human machine interaction is required.

By generating one or more numeric representation (s) of the input data related to the industrial process and providing the one or more numeric representation(s) to a data driven process model to generate at least one operation representation specific to the industrial process, monitoring, controlling, operating, assessing, evaluating, planning and/or implementing industrial processes can be enhanced. In particular, the input data provided to the data driven model can be structured and/or unstructured in the sense of human-type input, e.g. unstructured by way of natural language or human language- or structured by way of machine-readable process outputs from monitoring and/or control systems, and can be mapped to a machine-processable representation. Further, the model can be trained on respective machine-processable representation to generate suitable operation representation (s) of industrial process(es). Owing to the mapping into a multi-dimensional numeric representation space relating different data types, the input data or the processing by the data-driven model may handle mappings between multiple data types. This allows for interchangeable handling regarding the provided data types of the input data, while still reliably generating operation representation (s) specific to the industrial process(es), since correlations between input data of different data types can be captured in a multi-dimensional probability space depending on the numeric representation(s) for different data types. In view of the need for technical validation of industrial processes or changes in industrial processes to be implemented, the generation of such operation representation (s) specific to the industrial process(es) allows for effective and reliable technical validation followed by implementation.

The use of generative artificial intelligence enables novel areas of application. Generative neural networks are a subgroup of artificial intelligence models, that fall into the category of deep learning. Generative neural networks are machine learning models that employ one or more layers of nonlinear units to generate an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. Such complex modelling structures are adapted herein for use in industrial use cases that are safety and process performance relevant.

The method disclosed herein may for instance be performed, carried-out, executed and/or controlled by an/the computing apparatus, for instance a server, a server cloud, a computer-system, or part thereof. The disclosed method or any step of the method may be computer-implemented. For instance, the method or any step of the method may be performed and/or controlled by using at least one processor e.g. of an/the apparatus.

In one embodiment any of the method may include any of the following steps in the following embodiment: providing input data related to the at least one industrial process including at least one indication specific to the at least one industrial process, wherein the input data may be based on one or more data type(s); and/or depending on the data type(s) provided by the input data generating one or more numeric representation (s) of the input data, and/or providing the one or more numeric representation(s) to a process model, e.g. to generate at least one operation representation representing multiple process components specific to the at least one industrial process and optionally receiving the generated at least one operation representation, wherein the process model is trained to generate from the one or more numeric representation (s) of the input data at least one operation representation representing multiple process components specific to the at least one industrial process, wherein the data type of the input data and the data type of the at least one operation representation specific to the at least one industrial process differ at least in part, and/or providing the at least one operation representation specific to the at least one industrial process for operating the at least one industrial process.

In one embodiment input data relates to at least one data type that is different to the data type of the at least one operation representation generated by the process model. Input data may relate to one or more data type(s) that at least in part is/are different to the data type of the at least one operation representation generated by the process model.

In one embodiment input data relates to or includes one or more instruction(s). At least one instruction may relate to at least one data type that is different to the data type of the at least one operation representation generated by the process model. Instruction elements may include task instructions relating to the objective of the task to be executed with respect to operating the industrial process, context instructions relating to at least one specification of the industrial process, performance instructions relating to at least one performance indicator of the industrial process and/or constraint instructions relating to at least one constraint to be fulfilled by the industrial process according to the operation representation. The input data may include one or more instructions providing at least one indication specific to the industrial process. The one or more instructions or instruction element(s) may include at least one indication specific to the industrial process.

In one embodiment the input data includes a sequence of one or more text elements relating to at least one instruction or instruction element including at least one indication specific to at least one industrial process. The input data may include text data representing natural language and relating to at least one instruction or instruction element. The input data may include a sequence of one or more text elements relating to at least one instruction. The input data may include a sequence of one or more elements such as text elements relating to at least one of: the task instruction relating to the objective of the task to be executed with respect to operating the industrial process, the context instruction relating to at least one specification of the industrial process, the performance instruction relating to at least one performance indicator of the industrial process to be fulfilled according to the operation representation, and/or the constraint instruction relating to at least one constraint to be fulfilled by the operation of the industrial process according to the operation representation.

The input data may include a sequence of one or more text elements relating to at least one task instruction including an indication of the task and/or the objective and to at least one context instruction including an indication specifying the industrial process. The task instruction may relate to a generation task of the operation representation, such as generate operation representation from existing, e.g. based on existing industrial process, and/or from scratch, e.g. based on industrial process to be implemented or to be built. The task instruction may relate to the generation task to amend from existing, e.g. based on existing industrial process, and/or from scratch, e.g. based on industrial process to be implemented or to be built. The task instruction may relate to the generation task to generate alternative from existing, e.g. based on existing industrial process, and/or from scratch, e.g. based on industrial process to be implemented or to be built. The task instruction may relate to the generation task to evaluate, assess, convert, optimize and/or visualize from existing, e.g. based on existing industrial process, or from scratch, e.g. based on industrial process to be implemented or to be built. The context instruction may relate to the specific industrial process the operation representation is to be generated for. The input data may optionally include a sequence of one or more text elements relating to the performance instruction and/or the constraint instruction. In this context optionally refers to data input on use of the methods disclosed herein, that is not required to perform the methods disclosed herein. In other words, the methods may be performed on use without the specific data input.

In one embodiment the input data includes image, text and/or numerical data, related to one or more industrial processes) to generate at least one operation representation including structured representation, such as json, tables, graphs, etc., representing at least one diagram with graphical symbols specific to the industrial process according to the at least one instruction with at least one indication specific to at least one industrial process. This way multiple data types may be used to increase reliability in generating operation representations representing diagrams with graphical symbols specific to the industrial process. Since these operation representations representing diagrams are used for operating the industrial process, reliability can be increase by using different data sources as input in this safety and industrial process performance relevant field.

The image data may relate to at least one diagram with graphical symbols associated with one or more industrial processes), e.g. existing or to be implemented. The input data may include image data related to representations or diagrams with graphical elements or symbols associated with the industrial operation. The image data may include one or more graphical representations or diagrams with graphical elements or symbols associated with components of the industrial process. The image data may include one or more graphical representations or diagrams with graphical elements or symbols associated with process components including e.g. piping components, equipment components, instrumentation components for measurement and/or control, connections and material flows between process components, process conditions and/or operating conditions. The image data may include one or more graphical representations including pre-defined graphical elements associated with the components of the industrial process. The image data may include process and instrumentation diagrams (P&IDs), process flow diagrams (PFDs), three dimensional (3D) representations of the physical layout of the industrial process, operational measurement representations based on or from measurement and/or control components, measurement representations based on sensor layouts signifying sensor positions in relation to piping and/or equipment of the industrial process. The context instruction may include at least one image-based and/or text-based specification of the industrial process.

A Process Flow Diagram (PFD) may describe the relationships between major components in an industrial process, it may however dispense with describing minor components, piping systems, or instrumentation. A PFD may focus on the flow of chemical fluids and the equipment involved in the process, highlighting some properties such as temperature, pressure, fluid density, and flow rate. A PFD may give a broad overview of the production process and may be updated e.g. if equipment or material flows are changed or meant to be changed. A Piping and Instrumentation Diagram (P&ID) may include more detailed information than a PFD. It may describe major equipment, piping details (such as service, size, specification, and rating), and instrumentation details (such as pressure, temperature, and flow instruments). The P&ID may e.g. include control valves, safety valves, and other minor components. A P&ID may provide detailed information allowing operation of the industrial process, e.g. to produce a chemical product. PFDs and P&IDs may be examples of graphical representations of at least a part of the industrial process.

The input data may include numerical data related to measurements taken in association with operating the industrial process for example by control and/or monitoring elements. The task, context, performance and/or constraint instruction may include numerical data associated with operating the industrial process. The context and/or constraint instruction may include numerical data associated with measurements from operating the existing industrial process. The context and/or constraint instruction may include numerical data associated with measurements from operating existing industrial process(es) that are comparable to the industrial process to be implemented or built.

The input data may include at least text data and the image and/or numerical data may be provided together with the text data as part of the input data. The input data may optionally include the image and/or numerical data. In this context optionally refers to data input on use of the methods disclosed herein, that is not required to perform the methods disclosed herein. In other words, the methods may be performed on use without the specific data input. The image and/or numerical data may be provided as part of the input data. The image and/or numerical data may be retrieved from a data base and added to the input data to provide enriched input data.

In one embodiment the input data includes data representations, in particular structured representations, such as json, tables, graphs, etc., representing at least one diagram with graphical symbols, specific to the industrial processes) and corresponding text data for guiding the process model and, optionally at least one instruction with at least one indication specific to at least one industrial process, to generate at least one operation representation including structured representation representing at least one diagram with graphical symbols specific to the industrial process according to the at least one instruction with at least one indication specific to at least one industrial process. In such case the input data includes training instructions including structured representations of one or more dia- gram(s) with graphical symbols, related to one or more industrial process(es) and corresponding text data. This way few shot learning may be enabled, which is particularly useful when the process model performance needs to be improved without re-training the model. With few-shot learning the process model can be guided without training reducing resource usage for training while ensuring adequate outcomes in this safety and industrial process performance relevant field.

The input data may hence be based on types of data designed to be easily understood and interpreted by humans, and usable to communicate complex information in a simple and concise manner. This simplifies the use of the methods, apparatuses and computer elements though improved human-machine interaction and increases reliability in generating operation representation (s) used for operating the industrial process.

The input data may be provided on request by a user and/or a machine. The input data may be pre-processed depending on the data type(s) provided by the input data. The data type(s) may be discerned by the format the data is provided in. The data type(s) may be discerned by the file format the unstructured data is provided in. The data type(s) may be separated per data type such as text, numerical and/or image data. The separated input data may be separately provided for further pre-processing.

For example, the pre-processing may include the generation of one or more numeric representation(s) of the input data. In one embodiment generating one or more numeric representation(s) of the input data includes providing input data separated by data type to one or more embedding model(s). The embedding model may be configured to map the input data per data type to the numeric representation. The embedding model may be configured to map one data type to a lower dimensional representation. The embedding model per data type may be configured to generate numerical data from non-numerical data by mapping non-numerical data into a multidimensional vector space. The embedding model per data type may be configured to vectorize non-numerical data, such as text and/or image data. The embedding model per data type may be configured to vectorize non-numerical data, such as text and/or image data. The embedding model may be configured to map one or more data type(s) to a lower level or dimensional representation. The embedding model may be configured to generate joint or shared representations of one or more data type(s), such as text, numerical and/or image data. The embedding model may be configured to generate one or more numeric representation(s) by including mappings between elements of the data it transforms. For text, such correlations may include semantics embedded in a trained probability distribution of the embedding model.

The pre-processing may include enriching the input data with regard to the at least one instruction with at least one indication specific to at least one industrial process. Enriching may be based on enrichment data of the industrial processes, e.g. existing industrial processes and/or industrial processes to be implemented or built or voice commands. In particular enrichment data may include historical image and/or numerical data associated with existing industrial processes and/or industrial processes to be implemented or built. The enrichment data may relate to image and/or numerical data associated with existing industrial processes and/or industrial processes to be implemented and/or built. The enrichment data may relate to meta data signifying one or more numeric representation(s) of the enrichment data, in particular image and/or numerical data, and/or of annotations of the enrichment data, in particular text data.

By translating human natural language data such as text into machine processable data such as numeric values that take the meaning of the text into account and by generating lower dimensional representations, enrichment can be computed efficiently and more focused instructions according to the meaning of the human natural language data can be generated.

The process model may be trained to generate from the one or more numeric representation (s) of the input data, including or excluding enrichment data, at least one operation representation specific to the industrial process as indicated by the input data. The data type(s) of the input data and of the at least one operation representation specific to the industrial process may differ at least in part. The input data may include at least one data type different to the data type of the at least one operation representation specific to the industrial process. The input data may include at least one data type different for specifying the instructions to or for the process model different to the data type of the at least one operation representation specific to the industrial process.

In another embodiment the input data includes at least one text data type for specifying the instructions to the process model, wherein the at least one operation representation specific to the industrial process and generated by the process model represents at least one data type corresponding to at least one diagram with graphical symbols specific to the industrial process according to the at least one instruction with at least one indication specific to at least one industrial process. The input data may include at least one text data type for specifying the instructions to or for the process model. The at least one operation representation specific to the industrial process and generated by the process model may include at least one image data type, in particular diagram types with predefined symbols associated with the industrial process or components thereof. The process model may be configured to map the input data including at least one text data type for specifying the instructions to or for the process model to at least one operation representation specific to the industrial process including at least one image data type, in particular diagram types with predefined symbols associated with the industrial process or components thereof. The process model may be configured to map the input data including at least one text data type for specifying the instructions to or for the process model to at least one operation representation specific to the industrial process of image data type, in particular diagram types with predefined symbols associated with the industrial process or components thereof.

In another embodiment the process model maps between at least two different data type(s) of the input data and the at least one operation representation specific to the industrial process. The process model may be configured to map one or more data types of the input data to one or more different data types of the at least one operation representation specific to the industrial process. The process model may be configured to map between data types of the input data and the at least one operation representation specific to the industrial process. The process model may be configured to map input data of one or more data type(s) into a multi-dimensional numeric representation space correlating different data types. The process model may be configured to generate from input data of one or more data type(s) at least one operation representation (s) specific to the industrial process(es) including at least one different data type.

In another embodiment the process model is a data-driven model or a numeric transformation model trained or parametrized on one or more numeric representation(s) of input data relating to respective industrial process(es) and corresponding or expected output data relating to operation representation(s) specific to the respective industrial processes). For instance, a data-driven model or a numeric transformation model may be or comprise a generative adversarial network (GAN) and/or a variational autoencoder (VAE). The process model may hence be trained or parameterized on pre-defined input-output pairs for training. The process model may be trained to map from text-based input data to image-based output operation representation(s) specific to the industrial process. The process model or numeric transformation model may map between at least two different data type(s) of the input data and the at least one operation representation specific to the industrial process;

The process model may be trained based on a training process using at least on network component for generation and at least one network component for discrimination. The process network may include a Generative Adversarial Network (GAN) architecture and/or a Variational Autoencoder (VAE) network architecture. The process model may be trained to map the input data, in particular text-based data, to the numerical representation or a multidimensional vector in a multidimensional vector space or latent space and to map the numerical representation, e.g. a multidimensional vector in the multidimensional vector space or latent space, to operation representation (s) specific to the industrial process, in particular image-based data. The training of the mapping is performed based on numerical representation generation from text-based input data and generation of image-based operation representation (s) specific to the industrial process. The process model may be trained to map from text-based input data to image-based operation representation (s) specific to the industrial process based on a shared or joint representation of numerical representations of image and text in the multidimensional vector space or latent space. On training the discrimination is performed based on providing text-based input and image-based output pairs and adjusting the weights of the network such that a loss function discriminating the generated image-based output and the provided image-based output from the provided pair is minimized or maximized. The process model may be trained to map from text-based input data to image-based operation representation(s) specific to the industrial process based on a generator model trained in conjunction with a discriminator model. On training the discrimination is performed based on providing textbased input and image-based output pairs and adjusting the weights of the generator network such that a loss function based on the generated image of the generator network and the generated image of the discriminator network is minimized or maximized.

In another embodiment the process model may be a pre-trained model. The pre-trained model may be a general purpose model trained or parametrized based on general data sets including input, such as text-based, and output, such as, image-based, data pairs not specific to a task, in particular not specific to generating operation specifications for operating industrial processes, further in particular not specific to input data including at least one instruction with at least one indication specific to at least one industrial process and the operation representation (s) specific to the industrial process.

The process model may be fine-tuned based on training a pre-trained model. The pre-trained model may be trained or parametrized on numeric representations of input data relating to industrial process(es) and corresponding output data relating to operation representation (s) specific to the industrial process. The weights of the pre-trained model may be adjusted according to the task-specific data, in particular specific to generating operation specifications for operating industrial processes, further in particular specific to input data including at least one instruction with at least one indication specific to at least one industrial process and the operation representation (s) specific to the industrial process. The thus trained process model may be trained to map from text-based input data to image-based operation representation (s) specific to the industrial process.

The operation representation specific to the industrial process as generated by the process model based on the input data may include processed image data related to representations or diagrams with graphical elements or symbols associated with the industrial operation. Processed image data may relate to the image data generated by the process model. Processed image data may include one or more graphical representations of the industrial operation. Processed image data may include one or more graphical representations or diagrams with graphical elements or symbols associated with components of the industrial process. Processed image data may include one or more graphical representations or diagrams with graphical elements or symbols associated with process components including e.g. piping components, equipment components, instrumentation components for measurement and/or control, connections and material flows between process components, process conditions and/or operating conditions. Processed image data may include one or more graphical representations including pre-defined graphical elements associated with the components of the industrial process. Processed image data may include process and instrumentation diagrams (P&IDs), process flow diagrams (PFDs), three dimensional (3D) representations of the physical layout of the industrial process, operational measurement representations based on or from measurement and/or control components, measurement representations based on sensor layouts signifying sensor positions in relation to piping and/or equipment of the industrial process. In one embodiment the performance objective relates to an environmental performance associated with the environmental impact of the specified industrial process and/or a production performance associated with the one or more output product(s) producible by the specified industrial process.

The production performance associated with the one or more output product(s) producible by the specified industrial process may relate to process efficiency properties, such as yields, energy usage, resource usage or the like, produced product quality properties, maintenance properties, such as maintenance intervals in relation to operation conditions, safety properties, such as number of warnings in relation to operation of the industrial process.

One or more environmental property contributions related to the production of one or more output material(s) may relate to contributions of production components used by the chemical production network. Production components may include input materials, chemical processes and/or transports. One or more environmental property contributions related to the production of one or more output material(s) may relate to input materials, chemical processes and/or transports used to produce the output material. The contributions related to chemical process may include contributions directly or indirectly related to the chemical process. For example, external energy inputs to chemical processes such as electricity may be indirectly related to the chemical processes. Further for example, internal energy inputs to chemical processes such as steam or heat may be directly related to the chemical processes.

In another embodiment the input data representing the performance objectives to be reached by the industrial processes) specifies environmental performance including at least chemical production routes, emission characteristics, energy characteristics, waste characteristics, reuse characteristics and/or recycling characteristics. In another embodiment the input data representing the performance objectives to be reached by the industrial processes) specifies production performance including at least yield characteristics, total volume characteristics, output product quality characteristics, input material characteristics and/or auxiliary supplies characteristics. The input data may include structured or unstructured representations of production performance including at least yield characteristics, total volume characteristics, output product quality characteristics, input material characteristics and/or auxiliary supplies characteristics. For example, text data in natural language or text data with sequence of text elements may relate to production performance including at least yield characteristics, total volume characteristics, output product quality characteristics, input material characteristics and/or auxiliary supplies characteristics.

In another embodiment the process model is provided with input-output pairs including operation representations including structured representation representing industrial process(es) by graphical symbols and corresponding input data representing at least the at least one performance objective reached by the industrial process(es). The inputoutput pairs include performance objectives that relate to an environmental performance associated with the environmental impact of the specified industrial process and/or a production performance associated with the one or more output product(s) producible by the specified industrial process. For example, the input-output pairs may include the performance objective and the operation representation specifying process components for operating the industrial process according to the performance objective and. The input-output pairs may unstructured representation, such as text representations specifying the performance objective and structured representations, such as structured representation representing industrial process(es) by graphical symbols, relating to the operation representation specifying process components. The input-output pairs may be retrieved from a data base on training and/or on enrichment as described above to enrich the input data.

In another embodiment the input-output pairs are provided as part of the input data, wherein the process model includes a pre-trained model, wherein the pre-trained model is a general-purpose model trained on general data sets including input-output data pairs not specific to generating operation representations for operating industrial processes.

In another embodiment the input data includes structured representation representing industrial process(es) by one or more diagram(s) with graphical symbols related to one or more industrial process(es) and corresponding text data representing at least one instruction related to at least one performance objective for a specified industrial process to guide the process model to generate operation instructions representing at least one diagram with graphical symbols for the specified industrial process according to the at least one instruction related to at least one performance objective for the specified industrial process.

In another embodiment the input-output pairs are provided on training of the process model, wherein the process model includes a pre-trained model trained based on the provided input-output pairs, wherein the pre-trained model is a general-purpose model trained on general data sets including input-output data pairs not specific to generating operation representations for operating industrial processes. In another embodiment the input data relates to one or more instruction(s) including at least one of task instruction related to the task to be executed by the process model with respect to operating the industrial process, performance instruction relating to at least one performance objective to be fulfilled by the industrial process according to the operation representation, context instruction related to at least one specification of the industrial process and/or constraint instruction relating to at least one constraint to be fulfilled by the industrial process according to the operation representation, wherein the one or more instruction (s) are provided using one or more data type(s), wherein depending on the data type, the one or more numeric representation (s) are generated by an embedding model trained to map the one or more instruction(s) to the one or more numeric representation (s) to be provided to the process model.

In another embodiment the process model is trained to map input data of at least one first data type to at least one operation representation of at least one second data type, wherein the first and the second data type are different data types.

In another embodiment the input data includes at least one instruction specifying the performance objective in text with words, wherein one or more numeric representation (s) are generated by an embedding model trained to map the text to the one or more numeric representation(s) based on a sequence of words.

In another embodiment the operation representation includes at least one structured representation of at least one diagram with graphical symbols specific to the industrial process, wherein the graphical symbols are pre-defined and represent process components of the specified industrial process fulfilling at least one performance objective.

In another embodiment providing the at least one operation representation for operating the specified industrial process according to the performance objective includes generating a diagram with pre-defined graphical symbols representing process components of the industrial process fulfilling at least one performance objective.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following, the present disclosure is further described with reference to the enclosed figures. The same reference numbers in the drawings and this disclosure are intended to refer to the same or like elements, components, and/or parts.

Fig. 1 illustrates the validation for a chemical process of a chemical plant based on a piping and instrumentation diagram (P&ID) representing a two-dimensional graphic of the chemical process operation.

Fig. 2 illustrates an example of an industrial process in the form of a piping and instrumentation diagram (P&ID) representing a chemical plant operation.

Fig. 3 illustrates a method for operation of an industrial process capable of mapping between different data types for validation. Fig. 4 illustrates a method for evaluating or assessing an industrial process by an operator of the chemical process based on the P&ID generated by the process model engine.

Figs. 5a and 5b illustrate examples of the input-output structure of the process model engine.

Fig. 6 illustrates examples of data classes of input and output data pairs that may be used for training, fining tuning or few shot learning of the process model engine.

Figs. 7a, 7b, 8 and 9 illustrate examples of the data structures per example data class.

Figs. 10a, b and c illustrate the generation of numeric representations based on unstructured input data such as text and image data.

Fig. 11a, b illustrates a GAN architecture including training setup e.g. based on the training data sets shown in Figs. 6-9 and inference.

Figs. 12a, b illustrate an autoencoder architecture including a training setup e.g. based on the training data sets shown in Figs. 6-9 and inference.

Fig. 12c illustrates a trained multimodal model.

Fig. 13 illustrates a sub-network based architecture including a training setup e.g. based on the training data sets shown in Figs. 6-9 and inference.

Fig. 14 illustrates the use of a pre-trained network including potential fine-tuning by training e.g. based on the training data sets shown in Figs. 6-9 and inference.

DETAILED DESCRIPTION

The validation process may include a human in the loop setup. In such processes the chemical engineer may want to validate a change to be made to the chemical process. For example, if the change of a faulty equipment part is required, the implementation of the new equipment part may be executed, if the technical validation is successful.

For validation different aspects of the chemical process operation may be validated depending on the change to be made. For example, if the equipment is changed the transition to the new equipment may be validated regarding operation conditions of the chemical process and in interaction with the unchanged equipment. The validation may result in this example in the following outcomes: 1) Equipment change may be implemented with unchanged operation conditions and set points.

2) Equipment change may be implemented with change in operation conditions and/or set points.

3) Equipment changes lead to adverse effects on operation conditions and set points in combination with unchanged equipment.

Other changes may relate to the instrumentation such as measurement, controlling and/or monitoring equipment, equipment operation conditions and/or set points, physical process flows between equipment components, digital processing engines for monitoring and/or controlling equipment or the like.

To digitally support the chemical engineer in validation a validation engine may be provided. The validation engine may ingest the validation request provided by the chemical engineer. The validation request may include input data to be validated and a validation objective. The validation engine may process such request based on the validation objective by applying a validation rule and optionally validation data from a validation repository. Such rule-based setups may include multiple validation engines depending on the validation objectives and associated rule base. The validation engine may output the validation results, e.g. as in the example above for equipment change. Depending on the outcome of the validation engine the chemical engineer may adapt the input data to be validated and rerun the process.

Like the change of an existing chemical process, the validation process may be executed for new chemical processes to be implemented. In such case the new chemical process may be provided by way of a P&ID and the validation objective. The validation engine may generate the validation result based on the applicable validation rule and provide it to the chemical process engineer for further assessment. Validation is only one example task for assessment of chemical processes. Other examples include generation of P&IDs for new chemical process setups, generation of P&IDs in relation to performance objectives, correction of P&IDs, enhancement of P&IDs, completion of P&IDs, concatenation of P&IDs for chemical processes to be connected, or the like. Hence multiple types of tasks exist.

P&IDs are only one example of graphical data types suitable for assessment of chemical processes. Other data types include PFDs, 3D representations or real-time measurement data representations. For each of these data types, different engines processing different rules acting on such data types are required. Hence, processing such information is cumbersome and requires correlating different data types to avoid errors and corrections required to be made at a very late stage, e.g. on implementation of the industrial process or the change to the industrial process.

The example of the P&ID shows a plant section of an Ammonia production plant. The P&ID in this illustrative example specifies piping and equipment components. The equipment components may include coolers, absorbers, compressors, storage, or the like. The piping components may specify the connection between equipment components. In other examples instrumentation, measurement and control components may be further specified.

Fig. 3 illustrates a method for operation of an industrial process capable of mapping between different data types for validation.

Unstructured input data related to the industrial process including at least one representation specific to the industrial process is provided. The representation may be based on one or more data type(s). Data types may refer to different formats of data such as graphical and/or text data. The unstructured input data may comprise at least one data type relating to text data such as string data. The text data may include a sequence of elements such as words, numbers or other elements representing natural language such as human readable language. The text data may represent at least one task instruction relating to the task to be executed such as generation of P&ID, evaluation of P&ID, optimization of P&ID, visualization of P&ID, assessment of P&ID, or the like. The text data may represent at least one context instruction relating to the industrial process, such as industrial process specification, industrial process operations, industrial process performance, industrial process constraints or the like. The unstructured input data may comprise at least one data type relating to graphical data such as image data. For example, text data may be provided to generate a new P&ID or text and graphical data may be provided to correct a provided P&ID.

Providing one or more input data structure(s) associated with one or more industrial processes. The input data structure may include one or more data modalities. The one or more data modalities may relate to, include or be one or more data type(s). The input data structure(s) may include unstructured data. The input data structure(s) may include at least text data and/or image data as data modality or data type. The input data structure(s) may include at least text data associated with one or more industrial processes and/or P&ID image data associated with one or more industrial processes. Text data may represent the one or more industrial processes, one or more component(s) of the one or more industrial processes, one or more properties of the one or more industrial processes or combinations thereof. P&ID image data may represent one or more component(s) of the one or more industrial processes or per industrial process. P&ID image data may represent one or more graphical symbols of one or more component(s) of the one or more industrial processes or per industrial process. The data type(s) mentioned here shall not be considered limiting. Other data type(s) like audio, video, measurement data of the industrial process, control data of the industrial process may also be provided in the context of the input data structure(s).

The input data structure(s) may include instructions specific to one or more industrial process(es). The input data structure(s) may include any instructions related to one or more industrial process(es). The input data structure(s) may be provided by an interface associated with an operating system or an operator of the industrial process. The input data structure(s) may relate to or include unstructured data specifying the instructions specific to one or more industrial process(es). The input data structure(s) may include one or more generation instruction(s) related to a task the model engine may be requested to execute. The generation instructions may relate to a generation of a digital representation of the one or more industrial process(es). The generation instructions may relate to the generation of P&ID image data based on text data. The generation instructions may relate to the generation of text data based on P&ID image data. The generation instruction(s) may relate to a classification of a digital representation of the one or more industrial processes). The classification may relate to P&ID image data to be classified into one or more class(es) of text data. The classification may relate to text data to be classified into one or more class(es) of P&ID image data. The data type(s) and tasks mentioned here shall not be considered limiting.

The input data structure(s) associated with one or more industrial processes may include text data describing at least part of the industrial process(es) and/or P&ID image data representing at least part of the industrial process(es). The image data may represent at least part of the industrial process(es) by way of standardized symbols. Such image data may include P&IDs specifying at least part of industrial process(es) by way of graphical and/or textual symbols. This may for example include symbols as specified in technical standards for documentation of production systems. Specifically for chemical production the following technical standards exist:

Diagrams for the chemical and petrochemical industry - Part 1 : Specification of diagrams (ISO 10628-1 :2014); German version EN ISO 10628-1 :2015,

Diagrams for the chemical and petrochemical industry - Part 2: Graphical symbols (ISO 10628-2:2012); German version EN ISO 10628-2:2012,

PIP - PIC001, Piping and Instrumentation Diagram Documentation Criteria, March 1, 2018, ISA - 5.1, Instrumentation Symbols and Identification, 1 January 2022 and/or any related standards cited therein.

The above references are not limiting and many other technical standards for documentation of production system exist such as for example:

Technical product documentation - Simplified representation of pipelines - Part 1 : General rules and orthogonal representation (ISO 6412-1 :2017); German version EN ISO 6412-1:2018, Technical product documentation - Simplified representation of pipelines - Part 2: Isometric projection (ISO 6412-2:2017); German version EN ISO 6412-2:2018, Technical product documentation - Simplified representation of pipelines - Part 3: Terminal features of ventilation and drainage systems (ISO 6412-3:2017); German version EN ISO 6412-3:2018,

Graphical symbols for diagrams - Part 1 : General information and indexes (ISO 14617-1 :2005); English version ISO 14617-1:2005(en)

Graphical symbols for diagrams - Part 1 : General information and indexes (ISO/DIS 14617-1: under development) and/or related parts such as

— Part 2: Symbols having general application

— Part 3: Connections and related devices

— Part 4: Actuators and related devices

— Part 5: Measurement and control devices

— Part 6: Measurement and control functions

— Part 7: Basic mechanical components

— Part 8: Valves and dampers

— Part 9: Pumps, compressors and fans

— Part 10: Fluid power converters

— Part 11 : Devices for heat transfer and heat engines

— Part 12: Devices for separating, purification and mixing

— Part 13: Devices for material processing

— Part 14: Devices for transport and handling of material

— Part 15: Installation diagrams and network maps and/or any related standards cited therein.

The input data structure(s) may relate to the generation of the digital process representation of the one or more industrial processes under one or more technical constraints for the one or more industrial processes. The technical constraints may relate to the industrial process as running, the industrial process to be run or the industrial process to be retro-fitted. The technical constraints may relate to technical parameters of equipment, measurement and/or control system, and/or material flows. The technical constraints may relate to performance criteria of the industrial process. The performance criteria may relate to the industrial process as running, the industrial process to be run or the industrial process to be retro-fitted. The performance criteria may relate to production capacity, production yields and/or environmental impact.

Depending on the data type(s) provided by the unstructured input data, one or more numeric representation (s) of the input data may be generated. The numeric representations may be generated based on one or more embedding model(s). Different embedding models mapping text and/or graphic data to numeric representations are described in more detail in the context of Figs. 10a to 14. In general, these techniques map the text and/or graphical data to an embedding space based on numerical representations of the sequence of text or graphical elements. This may also be referred to as vocabulary. Based on the mapping the unstructured input data including text and/or graphical data are transformed to vectors or vectorized. The unstructured input data may hence be transformed from human processable instructions such as text or graphics to machine processable instructions such as numeric vector representations. The embedding model may be specific to one data type and may map the respective data type. Multiple embedding model(s) per data type may be provided and selected based on the data type. For example, a text embedding model and a graphics embedding model may be provided and selected based on the data type. The input data may include the piping and instrumentation diagram (P&ID) representing a two-dimensional graphic of the chemical process operation in association with text data.

The one or more numeric representation (s) may be provided to a data driven process model. The data driven process model may be trained to generate from the one or more numeric representation (s) of the input data at least one operation representation specific to the industrial process. The data type of the unstructured input data and the at least one operation representation specific to the industrial process may differ at least in parts. The data driven process model may be parametrized based on numeric representations of input data relating to industrial process(es) and corresponding output data relating to operation representation (s) specific to the industrial process(es). For example, the process model may be parametrized to map text data to graphical data or vice versa. Further for example, the process model may be parametrized to map text and graphical data to graphical data or vice versa. The process model may generate operation representations in the form of piping and instrumentation diagrams (P&ID) representing a two-dimensional graphic of the chemical process operation. Different process model architectures, training procedures and inference setups exist, examples of which will be described in more detail in the context of Figs. 10a to 14.

The one or more input data structure(s) may be provided to a process model engine mapping the one or more input data structure(s) to one or more digital process representations. If the input data structure(s) include generation instruction , the one or more input data structure(s) may be mapped to one or more digital process representations according to the one or more generation instruction(s). The process model engine may be trained to map one or more input data structure(s) to one or more digital process representations. The process model engine may be configured to map multimodal input data structure(s) including one or more data modalities or data type(s) to one or more digital process representations. The process model engine may be configured to map text data associated with one or more industrial processes and/or P&ID image data associated with one or more industrial processes to one or more digital process representations.

The process model engine may include one or more processing layers. The processing layers may be configured depending on the model architecture. For example, one processing layer may be configured to map the input data structure(s) to a vector representation depending on the data modality provided. Another processing layer may be configured to map the vector representation to a fused or joint vector representation. Another processing layer may be configured to map the fused or joint vector representation to the one or more digital process representation (s) of the one or more industrial process(es). The at least one operation representation specific to the industrial process may be provided for validating operation of the industrial process. Validation may include further data analysis based on existing representations of the existing chemical process, representations generated from measurement data of the existing chemical process, or representations generated for comparable chemical processes. Validation may include further assessment by a human operator of the chemical process. Validation may be based the piping and instrumentation diagram (P&ID) representing a two-dimensional graphic of the chemical process operation.

The generated one or more digital process representation(s) may be provided for monitoring, controlling, assessing, evaluating, quantifying and/or implementing one or more industrial process(es). For example, a running industrial process may be evaluated in view of one or more retro-fits to be made to the industrial process to reach one or more performance criteria. Further for example, an industrial process to be run may be evaluated in view of one or more technical constraints of the equipment layout.

Fig. 4 illustrates a method for evaluating or assessing an industrial process by an operator of the chemical process based on the P&ID generated by the process model engine.

The process model engine described in the context of Fig. 3 and as will be described in the context of Figs. 10a to 14 may be used by the operator of the chemical plant. The operator may send a request to the process model engine. The request may include unstructured input data as described in the context of Fig. 3. As an example a text prompt "Generate P&ID design with needed instrumentation to enable a predictive maintenance model for oven X in plant Y producing chemical Z via chemical process U”. In addition an image file in image file format such as an .pdf, .png, .svg or other suitable file formats may be provided. Such file may be uploaded by the user or retrieved from a data base based on the text prompt. In the latter case the text prompt may be vectorized and the vector representation may be compared to vector representations of a specification of the file content stored in the data base. The file related to the specification with highest similarity score as determined based on the vector representation may be provided.

In the example of Fig. 3 text data relating to instructions for generating of P&ID image data may be provided as input data. The text data may include instructions in the form of natural language. The text data may be unstructured data that may be mapped to a vector or latent space based on vocabulary tokens to make the unstructured data processable by the model. Such processing of unstructured text data will be described in more detail below.

The process model engine may ingest the vectorized text data. The process model engine may be configured to generate P&ID image data based on the vectorized text data. Possible model architectures may differ depending on the concrete task, training datasets, the model performance with respect to the output generation and/or the processing performance. The process model engine may be based on at least one variational autoencoder, at least one generative adversarial network, at least one pre-trained transformer-based model, at least one image generation model or combinations thereof.

The process model engine may be adapted or trained to operate across at least two different data type(s). For example in case of a generation task, the process model engine may be adapted to ingest text data and to output P&ID image data The process model engine may be adapted or trained to process at least two different data type(s) on the input. For example in case of an modification task, the process model engine may be adapted to ingest text data and P&ID image data and to output P&ID image data. This way the model may be fed with task-specific context provided by more than one data type.

The process model engine may generate one or more digital process representation (s) as output for monitoring, controlling, assessing, evaluating, quantifying and/or implementing one or more industrial process(es). For example, prior to or during the lifecycle phase of a chemical plant configured to perform chemical processes the physical setup of the plant may be provided by a topological representation of equipment, instruments, and connections for the material flows and/or logical flows. The topological representation may be provided by the P&ID image data. The topological representation may represent the chemical plant or specific part, such as section or process of the chemical plant. The chemical plant may be a running plant or a plant to be build. In both instances, providing the P&ID image data or topology representation is a cumbersome and error prone requiring many iterations of design, validation, and correction. This can result in less efficient and reliable monitoring, controlling, assessing, evaluating and/or quantifying performance of the chemical processes or any modifications thereto. Similarly the implementation of new chemical processes or modifications of chemical process setups can be hampered in view of the resulting performance of the chemical processes or any modifications thereto.

In this context performance may relate different properties of the chemical processes. For example performance may relate to production purposes, e.g., producing specific output material at specific yields, technical restrictions, e.g., with respect to time or equipment properties, or qualities, e.g. composition of output material or environmental impact properties. Further for example, performance may relate to safety measures such as pressure, temperature, or volume thresholds. In particular, for implementation or modification of the physical layout of the chemical processes errors in planning can lead to costly rework and delays in the construction process.

By using the process model engine for generating one or more digital process representation (s) as output the monitoring, controlling, assessing, evaluating, quantifying and/or implementing of one or more industrial process(es) can be improved with respect to reliability and robustness from initial design to the running of the industrial process(es). Moreover, the enablement though artificial intelligence allows for more rapid generation and evaluation of multiple layout options. This can reduce the time and cost for monitoring, controlling, assessing, evaluating, quantifying and/or implementing of one or more industrial process(es). Improved design for specific quality: Generative Al can generate designs that are optimized for specific criteria, such as energy efficiency, structural integrity, or aesthetics. This can result in designs that are more efficient and sustainable. The methods disclosed herein have the further advantages:

Reduced design errors: Generative Al can detect and correct design errors early in the process, reducing the risk of costly errors and delays later on. This can improve the P&ID quality and safety of the final design. Cost savings: Automating the design process can reduce the need for manual labor and increase efficiency, resulting in cost savings for businesses.

Enable inexperienced engineers to get expert-like guidance on P&ID quality and sustainability enabling materials, costs, and design possibilities. It helps them make confident and educated P&l D/Topology design choices with the help of a virtual assistant.

Generate full (e.g., Acytelene Plant) or specific sections or processes of a plant P&ID, given: Plant type information on the targeted product (e.g., Acytelene) and input/output material. Context information on the desired plant (e.g., land area, size, etc.). Optimization requirements of the plant resources (e.g., cost/time/material). Industrial constraints related to the production (e.g., yield per hour, safety, etc.). Digitalization enablement through installed instrumentation (e.g., to enable process digital solutions like predictive maintenance model install specific sensors and controllers. Sustainability requirements to follow green infrastructure design (e.g., by incorporating rainwater harvesting systems, greywater recycling, or integration with renewable energy generation). Generate multiple P&ID design alternatives that meet the specified criteria above to explore a wide range of possibilities and identify innovative solutions that may not have been manually considered otherwise.

Evaluate P&ID design: The solution would use generative Al algorithms to evaluate P&ID designs based on the specified requirements and constraints.

Optimize P&ID design: The solution could include optimization feature that enable the generative Al algorithms to optimize for specific criteria, such as production capacity, efficiency, safety, and sustainability. Visualize created P&ID: The solution could include tools for visualizing the generated P&ID in its original format or as a graph and in 2D or 3D plant models.

Interact with human P&ID designer: The solution could include tools for interaction between automatically generated P&IDs and human engineers, enabling them to provide feedback and iterate on the generated designs.

Integration with existing P&ID engineering tools: The solution could integrate with typical P&ID engineering software enabling designers to refine and modify the generated designs using industry-standard tools.

Different embodiments of model architectures, input data structures, handling of different data modalities and training methods will be described in more detail in the following.

Fig. 6 illustrates examples of data classes of input and output data pairs that may be used for training, fining tuning or few shot learning of the process model engine. Figs. 7a-9 illustrate examples of the data structures per example data class. For training, fine tuning or few-shot-learning, as case may be, in relation to any of the model architecture suitable for generating one or more digital process representation(s) associated with one or more industrial process(es), input data sets are required. Figs. 7a-9 illustrate examples of input-output pairs that include text data in relation to P&ID image data. In particular, equipment, instrumentation and chemical process related data pairs may be provided for training, finetuning or few shot learning of the process model engine, as case may be. In other embodiments further data type(s) such as audio, video, measurement data from sensors or control data from instrumentation, may be included in the data type(s).

In the example, equipment related data pairs including the standardized symbol of the equipment and corresponding text data signifying the natural language name of the equipment may form one class of the input-output data. Such data may be retrieved from standard documentation as for example cited in the context of Fig. 3. Another source for equipment related data pairs may include annotated P&IDs. Examples of standardized documentation and annotated equipment P&IDs are shown in Fig. 7a, b.

Instrumentation related data pairs including the standardized symbol of the instrumentation and corresponding text data signifying the natural language name of the instrumentation may form one class of the input-output data. Such data may be retrieved from standard documentation as for example cited in the context of Fig. 3. Another source for instrumentation related data pairs may include annotated P&IDs. Examples of standardized documentation and annotated instrumentation P&IDs are shown in Fig. 8.

Process or section related data pairs including the standardized symbol of the process or section and corresponding text data signifying the natural language specification of the instrumentation may form one class of the input-output data. Such data may be retrieved from standard documentation as for example cited in the context of Fig. 3. Another source for instrumentation related data pairs may include annotated section or process P&IDs. Examples of annotated section or process P&IDs are shown in Fig. 9.

To generate numeric representations different embedding models for different data type(s) exist. The examples of Figs. 10a, b and c show illustrative embedding models for words, images and symbols. Other embedding models exist also for other data types and the examples depicted here shall not be considered limiting.

Fig. 10a illustrates a word embedding. The generation of word embeddings may for example use similar methods to the ones described in US9037464B1. The text embedding may be generated by a text embedding mechanism such as Word2Vec or GloVe generating 300-dimensional word vectors. As illustrated in Fig. 10a, the text data may be composed of elements representing words of a vocabulary. The vocabulary may include pre-defined mappings of each word to a feature vector. Via the vocabulary or a so-called on hot encoding unstructured input data based on natural language may be transformed to numeric vectors. W_vn and W_nv may be trained matrices. Weight matrix W_vn may map the input x to the hidden layer (V*N dimensional matrix). Weight matrix W_nv may map the hidden layer outputs to the final output layer (N*V dimensional matrix). This way C contexts may be considered for determining the word embedding. The numeric representations may be continuous representations represented using floating-point numbers. Positions of representations in the high-dimensional space may reflect semantic similarities, syntactic similarities, or both, between words represented by the numeric representations. Another example called skip-gram is shown on the left side of Fig. 10a and illustrates the reverse logic, where for each word C probability distributions of V probabilities, one for each word are generated and selected based on the softmax function.

Fig. 10b illustrate the generation of numeric representations for a symbolic language such as P&IDs. The symbols may be translated to text and the numeric representation may be generated based on such text e.g. as described in the context of Fig. 10a. The encoding method illustrated in Fig. 10b for generating the numeric representation includes graph encoding, where the nodes encode the information regarding individual elements such as equipment and the edges encode the relation between such symbols. Fig. 10c illustrates the generation of numeric representations for images such as P&ID diagrams. The image may be patched and the patches may be transformed to numeric representation including positional encoding.

Based on the numeric representations generated by any of the encoding methods described above or any other encoding method known in the art, models may be trained.

Figs. 11a, 11b illustrate a GAN architecture including training setup e.g. based on the training data sets shown in Figs. 6-9 and inference.

The process model may include a Generative Adversarial Network (GAN). The GAN architecture is a type of deep learning model that can be used for generating a wide variety of data types such as new text, diagrams or images. The training process for a GAN involves two main components: a generator and a discriminator. The generator is a neural network that takes a random noise vector as input and produces a synthetic text, diagram or image output. The goal of the generator is to produce outputs that are indistinguishable from real-world text, symbols or images. The discriminator is also a neural network that takes a text, diagram or image input (either real or synthetic) and outputs a probability that the input is real. The goal of the discriminator is to correctly distinguish between real and synthetic inputs. During training, the generator and discriminator are trained simultaneously. The generator tries to produce outputs that can fool the discriminator into thinking they are real, while the discriminator tries to correctly classify the inputs it receives.

The training of the GAN may include a generator and a discriminator network as for example illustrated in Fig. 11 b. The training may be performed on a training data set including input-output pairs. In the example of Fig. 11a, 11b text data may be provided as input data and corresponding image or diagram data may be provided as output. This should only serve as one example of data types or multi modal input-output pairs. Other choices of datatypes and related input-output pairs are feasible and the example should not be considered limiting. The generator and discriminator may be trained simultaneously. In the example of Figs. 11 a, 11 b, the generator network may ingest text data and may embed such text data into a numeric representation such as described in the context of Figs. 10a, b, c. Noise may be added to the embedding vector to produce a synthetic output via an up-sampling network that may include a convolutional neural network architecture configured to generate images and/or diagrams. The synthetically generated image and/or diagram may be concatenated with the text embedding and down sampled for sigmoid operation.

The image and/or diagram data related to the respective text data may be provided to discriminator. The image and/or diagram data may be real or synthetic data of the input-output pair. The discriminator may generate the image and/or diagram embedding as described for example in the context of Figs. 10a, b. The generated text embedding of the generator may be concatenated to the image and/or diagram embedding. The embedding may be upsampled to generate at least one diagram and/or image e.g. by using a convolutional neural network architecture. The generated image and/or diagram may be concatenated with the text embedding and down sampled for sigmoid operation. One or more loss function(s) may be defined that relates to the probabilities that generated data is real or fake. One loss function may be defined for discriminator and generator network, while the generator and the discriminator have opposing goals such as one maximizes the loss function while the other minimizes the loss function. Based on this opposing setup the generator may learn to generate real outputs and may be used for image and/or diagram generation from text input. This way the generator may learn to map between different data types or modalities such as input data of one data type to output data of another data type.

For P&ID generation, the input data may include at least one text instruction with at least one indication specific to at least one industrial process. Further data type(s) such as existing P&IDs or sensor data may be provided. The generator may be trained as described above in relation to the discriminator based on the training data including input data including at least one instruction with at least one indication specific to at least one industrial process and respective output data including at least operation representations.

Depending on the data type(s) provided by the input data, one or more numeric representation (s) of the input data may be generated by the generator of the GAN. If more than one data type is present in the input data, the data types may be provided to respective embedding layers configured to generate the respective embeddings or numeric representations such as for example described in the context of Figs. 10a, b, c. The numeric representations per data type may be fused, e.g. by concatenation or transformation operation(s) such as dot products, addition or the like of the numeric representations. In other embodiments numeric representations may be generated based on multiple data types.

The one or more numeric representation (s) may be provided to the trained generator configured to generate at least one operation representation representing multiple process components specific to the industrial process. This may include representations of P&IDs including at least one diagram with graphical symbols specific to the industrial process. The generated at least one operation representation specific to the industrial process may be used for operating the industrial process.

The process model may include autoencoder architecture, in particular a variational autoencoder architecture. Fig. 12a illustrates the training process for the variational autoencoder architecture. The architecture in this example is configured to generate representations of P&ID diagrams based on text input data. Other examples od input-output data and data type(s) are possible, and this example should not be considered limiting.

The variational autoencoder architecture includes encoders configured to map the input data to the numeric representation per data type, in this example image and text data. Other examples may exist, where one or more encoders may be configured to map multiple input data type(s). The variational autoencoder architecture includes decoders configured to map one or more numeric representations per data type, in this example image and text. Other examples may exist, where one or more decoders may be configured to map multiple input data type(s).

On training, e.g. based on the training data sets illustrated in Figs. 6-9, the one or more encoder(s) may map their respective input data to a joint latent space. A multimodal representation may be generated as illustrated in Fig. 12a by fusing the one or more representations generated per encoder based on the input data provided per encoder. In the example, fusion may include a scalar or dot product of the numeric representations generated based on the text data by the text encoder and the image data by the image encoder. Other fusion techniques may be suitable, and the given example should be considered non-limiting.

The joint latent space distribution may be sampled based on the fused multimodal representation. Based on the sampled representation the text sampled latent vector and image sampled latent vector may be provided to the text encoder and image encoder respectively. The text and image may be reconstructed. The outputs of the decoders may be compared to the inputs of the encoders. Based on the difference between the input and generated output the weights of the joint latent space representation mapping may be adjusted. This way the joint latent space distribution may be trained to be used for multimodal mapping in the example between text and image or image and text data.

Fig. 12b illustrates the inference process for the trained variational autoencoder architecture mapping between at least two different data type(s). In the example text data may be providing at least one instruction with at least one indication specific to at least one industrial process. The text data may be mapped to one or more numeric representations) and provided to the trained text encoder. The encoded text representation may be used to sample the joint latent space distribution trained as described above. This may allow to generate image sampled latent vector and hence change data type or modality. The image sampled latent vector may be provided to the image decoder configured to generate from the text numeric representation (s) of the input data at least one image of the operation representation representing multiple process components specific to the industrial process. This may include representations of P&IDs including at least one diagram with graphical symbols specific to the industrial process. The generated at least one operation representation specific to the industrial process may be used for operating the industrial process.

Fig. 13 illustrates a sub-network-based architecture including a training setup e.g. based on the training data sets shown in Figs. 6-9 and inference.

The process model may include a sub-network-based architecture. The following illustrates an example training process for a sub-network-based architecture. The architecture in this example is configured to generate representations of P&ID diagrams based on text input data. Other examples of input-output data and data type(s) are possible, and this example should not be considered limiting.

The sub-network-based architecture includes mapping networks per data type configured to map the input data to the numeric representation per data type, in this example image and text data. Other examples may exist, where one or more sub-networks may be configured to map multiple input data type(s). The generated numeric representations per data type may be fused such as concatenated. The sub-network-based architecture may include one generative model such as a normalizing flow or a recurrent neural network configured to map fused numeric representations, in this example a fused image and text representation. Other examples may exist, where one or more generative models may be configured to map multiple input data type(s).

On training, e.g. based on the training data sets illustrated in Figs. 6-9, the one or more sub-network(s) may map their respective input data to their respective numeric representation(s). A multimodal representation may be generated as illustrated in Fig. 13 by fusing the one or more representations generated per sub-network based on the representation provided per sub-network. In the example, fusion may include concatenation of the numeric representations generated based on the text data by the text sub-network and the image data by the image sub-network. Other fusion techniques may be suitable, and the given example should be considered non-limiting.

The fused multimodal representation may be provided to the generative model configured to generate an image representation. The image may be reconstructed. The output of the generative model may be compared to the expected output as provided by the input-output pairs of the training data set. Based on the difference between the generated and expected output the weights of the sub-networks and the generative model may be adjusted. This way the subnetworks and the generative model may be trained to be used for multimodal mapping in the example between text and image or image and text data.

Fig. 13 illustrates the inference process for the sub-network-based architecture mapping between at least two different data type(s). In the example text data may be providing at least one instruction with at least one indication specific to at least one industrial process. Further image data such as existing P&ID diagrams may be provided as input. The text and image data may be mapped to the respective one or more numeric representation (s) and provided to the trained generative model. The respective one or more text or image numeric representation (s) may be used to generate images by the generative model trained as described above. This may allow to generate images and hence at least on part change data type or modality. The respective one or more text or image numeric representation (s) may be provided to the generative model configured to generate from the respective one or more text or image numeric representation (s) of the input data at least one image of the operation representation representing multiple process components specific to the industrial process. This may include representations of P&IDs including at least one diagram with graphical symbols specific to the industrial process. The generated at least one operation representation specific to the industrial process may be used for operating the industrial process.

In this example the process model may include a pre-trained model. The pre-trained model may be or include a general-purpose model parametrized based on general data sets including input-output-data pairs not specific to input data including at least one instruction with at least one indication specific to at least one industrial process and the operation representation (s) specific to the industrial process. The pre-trained model may include a multimodal model that depending on the data type(s) provided by the input data may generate one or more numeric representation(s) of the input data and provide such representations to the pre-trained model. The pre-trained model may be trained to generate from the one or more numeric representation (s) of the input data at least one operation representation representing multiple process components specific to the industrial process as indicated by the input data. The one or more data type(s) of the input data and of the at least one operation representation specific to the industrial process may differ at least in part. In the example of the Fig. 14, text data is provided and image data representing at least one operation representation specific to the industrial process is generated. This may include representations of P&IDs including at least one diagram with graphical symbols specific to the industrial process. The generated at least one operation representation specific to the industrial process may be used for operating the industrial process.

The process model may be or include a pre-trained model. The pre-trained model may be or include a general-purpose model parametrized based on general data sets including input-output-data pairs not specific to input data including at least one instruction with at least one indication specific to at least one industrial process and the operation representation(s) specific to the industrial process. The pre-trained model may be trained or parametrized on numeric representations of input data relating to industrial process(es) and corresponding output data relating to operation representation(s) specific to the industrial process. The pre-trained model weights may hence be adjusted to the specific task based e.g. on training data sets shown in Figs. 6-9. The adjustment of weights may include transfer learning and/or fine tuning techniques. Transfer learning may refer to using an existing pre-trained model such as one of the LLM family like BERT (NLU), GPT (NLG), etc that was trained on a particular task (source task - NLU or NLG) and adapting it for a different but related task (target task). The target task may relate to domain adaptation of source task. The goal may be to leverage the knowledge gained from the source task to achieve better performance on the target task, especially when there is limited labeled data for the target task. Transfer learning may involve freezing at least some layer weights of the pre-trained model and training e.g. an existing layer or a new top layer while keeping other layers fixed. Fine-tuning may involve taking a pre-trained model that has been trained on a large dataset (usually for a related or broader task) and adapting the weights for a specific task by continuing the training process on a smaller, task-specific dataset. Fine tuning may involve adjusting specific layers and weights of the pre-trained model for the new task. This way the pre-trained model may be tailored for a particular task or domain.

By generating one or more numeric representation (s) of the input data related to the industrial process and providing the one or more numeric representation(s) to a data driven process model to generate at least one operation representation specific to the industrial process, monitoring, controlling, operating, assessing, evaluating, planning and/or implementing industrial processes can be enhanced. In particular, the human task of validating safety and/or performance relevant operations can be enhanced. Through the human centric, simplified interaction enabled by the "translation” between different data modalities or types the monitoring, controlling, operating, assessing, evaluating, planning and/or implementing industrial processes can be structured in a simple, efficient and human centric manner, which allows operators ro detect errors or other factors influencing the operation of the industrial process early and to avoid maintenance or downtimes impeding the operation efficiency of the industrial process. The methods, apparatuses, models and uses proposed herein can be viewed as the human centric assistant for industrial process operators.

The present disclosure has been described in conjunction with preferred embodiments and examples as well. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed invention, from the studies of the drawings, this disclosure and the claims.

Any steps presented herein can be performed in any order. The methods disclosed herein are not limited to a specific order of these steps. It is also not required that the different steps are performed at a certain place or in a certain computing node of a distributed system, i.e. each of the steps may be performed at different computing nodes using different equipment/data processing.

As used herein ..determining" also includes ..initiating or causing to determine", "generating" also includes ..initiating and/or causing to generate" and "providing” also includes "initiating or causing to determine, generate, select, send and/or receive”. "Initiating or causing to perform an action” includes any processing signal that triggers a computing node or device to perform the respective action.

In the claims as well as in the description the word "comprising” or "including” or similar wording does not exclude other elements or steps and shall not be construed limiting to the elements or steps lined out. The indefinite article "a” or "an” does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation or further elements may be included.

Providing in the scope of this disclosure may include any interface configured to provide data. This may include an application programming interface, a human-machine interface such as a display and/or a software module interface. Providing may include communication of data or submission of data to the interface, in particular display to a user or use of the data by the receiving entity.

Any disclosure and embodiments described herein relate to methods, systems, apparatuses, devices, chemicals, materials, services, uses, computer program elements lined out above and vice versa. Advantageously, the benefits provided by any of the embodiments and examples equally apply to all other embodiments and examples and vice versa.

All terms and definitions used herein are understood broadly and have their general meaning if not indicated other- wise.

Claims

CLAIMS:

1. A method for operating an industrial process, wherein the industrial process includes multiple process components for operating the industrial process, the method comprising the steps: providing input data including at least one instruction related to at least one performance objective for a specified industrial process, generating one or more numeric representation (s) of the input data, providing the one or more numeric representation(s) to a process model, wherein the process model is trained to generate from the one or more numeric representation (s) of the input data at least one operation representation specific to the performance objective for the specified industrial process, wherein the operation representation specifies process components for operating the industrial process according to the performance objective, providing the at least one operation representation for operating the specified industrial process according to the performance objective.

2. The method of claim 1 , wherein the performance objective relates to an environmental performance associated with the environmental impact of the specified industrial process and/or a production performance associated with the one or more output product(s) producible by the specified industrial process.

3. The method of any of the preceding claims, wherein the input data representing the performance objectives to be reached by the industrial process(es) specifies environmental performance including at least chemical production routes, emission characteristics, energy characteristics, waste characteristics, reuse characteristics and/or recycling characteristics.

4. The method of any of the preceding claims, wherein the input data representing the performance objectives to be reached by the industrial process(es) specifies production performance including at least yield characteristics, total volume characteristics, output product quality characteristics, input material characteristics and/or auxiliary supplies characteristics.

5. The method of any of the preceding claims, wherein the process model is provided with input-output pairs including operation representations including structured representation representing industrial process(es) by graphical symbols and corresponding input data representing at least the at least one performance objective reached by the industrial process(es).

6. The method of any of the preceding claims, wherein the input-output pairs are provided as part of the input data, wherein the process model includes a pre-trained model, wherein the pre-trained model is a general- purpose model trained on general data sets including input-output data pairs not specific to generating operation representations for operating industrial processes.

7. The method of any of the preceding claims, wherein the input data includes structured representation representing industrial process(es) by one or more diagram(s) with graphical symbols related to one or more industrial process(es) and corresponding text data representing at least one instruction related to at least one performance objective for a specified industrial process to guide the process model to generate operation instructions representing at least one diagram with graphical symbols for the specified industrial process according to the at least one instruction related to at least one performance objective for the specified industrial process.

8. The method of any of the preceding claims, wherein the input-output pairs are provided on training of the process model, wherein the process model includes a pre-trained model trained based on the provided input-out- put pairs, wherein the pre-trained model is a general-purpose model trained on general data sets including in- put-output data pairs not specific to generating operation representations for operating industrial processes.

9. The method of any of the preceding claims, wherein the input data relates to one or more instruction(s) including at least one of task instruction related to the task to be executed by the process model with respect to operating the industrial process, performance instruction relating to at least one performance objective to be fulfilled by the industrial process according to the operation representation, context instruction related to at least one specification of the industrial process and/or constraint instruction relating to at least one constraint to be fulfilled by the industrial process according to the operation representation, wherein the one or more instruction(s) are provided using one or more data type(s), wherein depending on the data type, the one or more numeric representation (s) are generated by an embedding model trained to map the one or more instruction(s) to the one or more numeric representation(s) to be provided to the process model.

10. The method of any of the preceding claims, wherein the process model is trained to map input data of at least one first data type to at least one operation representation of at least one second data type, wherein the first and the second data type are different data types.

11 . The method of any of the preceding claims, wherein the input data includes at least one instruction specifying the performance objective in text with words, wherein one or more numeric representation(s) are generated by an embedding model trained to map the text to the one or more numeric representation(s) based on a sequence of words.

12. The method of any of the preceding claims, wherein the operation representation includes at least one structured representation of at least one diagram with graphical symbols specific to the industrial process, wherein the graphical symbols are pre-defined and represent process components of the specified industrial process fulfilling at least one performance objective.

13. The method of any of the preceding claims, wherein providing the at least one operation representation for operating the specified industrial process according to the performance objective includes generating a diagram with pre-defined graphical symbols representing process components of the industrial process fulfilling at least one performance objective.

14. An apparatus for operating an industrial process, wherein the industrial process includes multiple process components for operating the industrial process, the device comprising: an input interface configured to provide input data including at least one instruction related to at least one performance objective for a specified industrial process, a representation generator configured to generate one or more numeric representation (s) of the input data, an operation representation generator configured to provide the one or more numeric representation (s) to a process model, wherein the process model is trained to generate from the one or more numeric representations) of the input data at least one operation representation specific to the performance objective for the specified industrial process, wherein the operation representation specifies process components for operating the industrial process according to the performance objective, an output interface configured to provide the at least one operation representation for operating the specified industrial process according to the performance objective.

15. Use of the at least one operation representation specific to the industrial process as generated according to the methods of any of claims 1 to 13 or by the apparatus according to claim 14 for operating the industrial process or for enriching at least one three-dimensional operation representation specific to the industrial process.