Background
New view synthesis refers to rendering a picture corresponding to a target pose given a source image and the source pose and the target pose, and generally involves three-dimensional understanding of a scene. The new view synthesis has wide application in the fields of 3D reconstruction, AR/VR and the like. In recent years, deep learning, especially convolutional neural networks, neural radiation fields, three-dimensional gaussian sputtering and other models play a key role in new view synthesis, and can learn complex scene representations and illumination models to generate more realistic images. For example, three-dimensional gaussian sputtering (3 DGaussian Splatting) uses a series of gaussian functions (typically 3D gaussian distributions) to represent objects in a scene. The scene is broken down into a number of gaussian ellipsoids, each with its own center position, color, direction and size. Upon rendering, these ellipsoids are rasterized into pixels, and the color value of each pixel is determined by an integration or sampling process, thereby producing a final image. Compared with traditional volume rendering or grid-based rendering, three-dimensional gaussian sputtering can achieve real-time rendering speeds.
With the rapid development of the three-dimensional Gaussian sputtering technology, the new view synthesis technology has a remarkable breakthrough in rendering quality, efficiency and interactivity, and the wide expansion of the application of the vision industry is strongly promoted. The application covers the fields of digital people, automatic driving scene simulation, three-dimensional content generation, unmanned aerial vehicle rapid mapping, large-scale scene reconstruction, three-dimensional language fields and the like. Typically, such commercial applications require that a multi-view scene picture be input and then the scene reconstruction work be performed at the high performance server side. Although the training efficiency of new view synthesis applications has been significantly improved, this limits the quality of the reconstruction effort, as its construction basis relies on sparse point clouds estimated by the Structure-from-Motion technique during the camera calibration phase. Motion restoration structures are methods that automatically restore the three-dimensional structure of a scene and the camera's internal and external parameters from a plurality of images or video sequences, typically estimating the depth of the scene and the trajectory of the camera based on geometric constraints and motion models. The workflow of the motion recovery structure comprises the steps of feature detection, feature matching, relative pose estimation, triangularization, global optimization, dense reconstruction and the like. Major challenges faced by current motion restoration structures include handling occlusion, illumination changes, low texture regions, and image noise, which can lead to feature matching errors and reconstruction inaccuracies. This reliance on the efficiency and accuracy of the motion restoration structure matching often means that the user is required to wait for a long period of time for a service oriented application.
Furthermore, supporting multi-view pictures captured in an arbitrary manner by an arbitrary device is of great importance for new view composition applications, in particular for application scenarios where services are popular on common mobile devices. The camera pose estimation method based on the motion restoration structure faces serious experiments in the face of sparse photo input with insufficient labeling of pose data. In an environment with a sparse view, there may be a lack of sufficient matching texture, which directly affects the accuracy of camera pose estimation.
In order to solve the problems of low efficiency and inaccurate estimation of camera pose in the three-dimensional Gaussian sputtering initialization stage of the motion restoration structure method, two main stream improvement strategies exist at present:
1) A more accurate and robust camera pose estimation method is employed. The core of the direction is to adopt a deep neural network based on a visual transducer architecture to improve the accuracy and the robustness of the camera pose estimation. The transducer model exhibits excellent performance in dealing with complex spatial relationships and long-range dependencies with its strong sequence modeling capabilities and parallel computing advantages. Specifically, by introducing a self-attention mechanism, the transducer can effectively capture the relationship between elements in the input sequence, thereby realizing accurate estimation of the pose of the camera without explicit geometric constraint. In addition, the training process of the deep neural network can automatically learn rich characteristic representations, so that the adaptability to shielding, illumination change and dynamic scenes is further enhanced, and the reliability of an estimation result is improved.
2) The camera pose is used as part of the iterative training parameters. This strategy is to incorporate camera pose parameters into the iterative training process, treating them as learnable variables, rather than a fixed, a priori knowledge. The core idea of the method is to optimize the pose parameters of the camera by using a gradient descent algorithm and minimizing loss functions such as reprojection errors or luminosity errors. In the training process, the model can continuously adjust the pose of the camera until the pose is converged to the optimal estimated value.
The prior art, as analyzed, has mainly the following drawbacks:
1) The motion restoration structure technique is capable of restoring the motion trajectory of a camera and the three-dimensional structure of a scene from a series of images. In the process, the pose of the camera can be deduced, and the generated scene point cloud is a key basis for scene initialization in three-dimensional Gaussian sputtering training, so that the final scene training effect is influenced. Although the accuracy of camera pose prediction under a sparse view can be improved by adopting a deep neural network method, if a three-dimensional Gaussian sputtering motion recovery structure method is directly replaced, the accuracy of scene initialization is inevitably damaged, and a reconstruction result is blurred.
2) The combination of camera pose parameters and iterative training of a three-dimensional Gaussian sputtering scene can provide a more accurate model optimization approach in theory, but the model optimization process is often caused to become abnormally complex in actual operation due to the deep fusion of the camera pose parameters and the three-dimensional Gaussian sputtering scene. The excessive coupling not only makes the model difficult to reach the global optimal solution, but also can cause the problems of unstable training and the like, and limits the further improvement of the performance of the model. More importantly, the camera pose parameters are incorporated into the training link, meaning that each iteration needs to be recalculated and adjusted, the number of iterative training times can be significantly increased, the overall training time is prolonged, and the consumption of computing resources and the time efficiency are severely challenged.
In summary, in the technical field of new view synthesis, camera pose information is an essential part of scene training. The prior art either presets that camera pose information is known or extrapolates the position of the camera by applying motion restoration structure algorithms from dense views. This process is accompanied by high time costs, especially due to the time-consuming overall matching calculation. In addition, in the face of viewing angle occlusion or highly repetitive texture structures in sparse scenes, it is often difficult for motion restoration structure algorithms to accurately restore camera pose, resulting in inaccurate positioning. In addition, the point cloud data preliminarily generated by the motion restoration structure, although serving as a starting point of the neural field training to construct the scene model, is limited in quality by the uncertainty of the above-described pose estimation.
Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless it is specifically stated otherwise.
The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any specific values should be construed as merely illustrative, and not a limitation. Thus, other examples of exemplary embodiments may have different values.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.
The three-dimensional Gaussian sputtering technology is an important innovation in the field of new view synthesis, and has the potential of reshaping the production flow of digital content. The invention solves the problem of scene training under sparse pose-free view, can effectively expand the general capability and hardware scene of new view synthesis technology, thereby promoting the development of visual applications such as digital people, automatic driving scene simulation, three-dimensional content generation, unmanned aerial vehicle rapid mapping, large-scale scene reconstruction, three-dimensional language field and the like, and being beneficial to promoting the popularization of new view synthesis application to common mobile equipment.
In general, referring to fig. 1, the provided three-dimensional gaussian sputtering optimization method for pose-free input mainly comprises four parts of content including camera density ray prediction, three-dimensional gaussian scene initialization (three-dimensional gaussian initial training scene), visual shell redundancy elimination and camera pose regularization.
Camera density ray prediction is improved from camera ray prediction. The core idea of camera ray prediction is that neural network learning generally benefits from an over-parameterized distributed representation, and therefore, rather than using a compact camera feature representation, such as an internal and external matrix, the possible orientation direction of the camera is represented by a set of ray bundles. The light beam is from the pixel point of the image, converges to the center point of the camera, and is represented by the Pluker coordinates. The neural network is to recover the position and direction of the light beam as much as possible from a set of images, so as to solve the intrinsic and extrinsic properties of the camera according to the converging position of the light beam, for example, using a least squares optimization method. The camera density ray prediction of the present invention supports ray information with density to provide prior information for scene initialization, as will be described in detail later.
Three-dimensional gaussian scene initialization, the three-dimensional gaussian points are initialized initially by means of a sparse point cloud of scene structures derived from motion restoration structures. In order to form a more accurate pose estimation method, an end-to-end training process is formed, and priori is provided for scene initialization by means of over-parameterized information of camera density rays, so that camera pose estimation and initialization of a three-dimensional Gaussian scene are integrated. The three-dimensional gaussian points are sampled on the light according to the density relationship, rather than using conventional uniform random sampling.
In order to achieve redundancy elimination of the visual shell, depth estimation information of an image is obtained from an input multi-view picture, the visual shell of a scene is constructed, and then the false initialization point cloud positions outside the shell are eliminated. In the training process of the three-dimensional Gaussian scene, the visual shell provides rough structural information of scene objects and is used for eliminating unreasonable Gaussian point cloud distribution, so that the generation of visual floaters is restrained, and the consistency of the scene is maintained.
In the three-dimensional Gaussian scene training process, training regular terms of camera pose parameters are added, and the camera pose parameters are adjusted to achieve a scene reconstruction effect with higher precision.
In the invention, in the initialization process of a three-dimensional Gaussian scene, a sparse picture (or called an image) without pose mark is input, and the depth information and the pose information of the sparse picture are estimated at first so as to perform joint method optimization. Specifically, referring to fig. 2, the provided three-dimensional gaussian sputtering optimization method for pose-free input comprises the following steps:
Step S210, for an input image, predicting the light beam distribution by using a light prediction model to obtain distribution characteristics of the light beam form, including the momentum of the light, the direction of the light and the volume density of the light.
Camera density ray prediction is a proxy task for camera pose estimation, and a large number of distribution parameters of rays are output. The camera light method only provides the function of recovering the pose of the camera from a plurality of pictures, and has no function of deriving an initialization point cloud like a motion recovery structure algorithm. In the embodiment of the invention, the constructed light distribution type characterization is improved, so that the light characteristic under the Pluker coordinate is improved into the light characteristic with density. Based on the characteristics of three-dimensional Gaussian sputtering volume rendering, rays with volume density information are predicted, and a basis is provided for sampling an initial three-dimensional Gaussian training scene from a ray bundle.
FIG. 3 is a dense ray prediction model constructed based on a visual transducer model in which the pose parameters λ of the camera are encoded as a series of raysPi denotes any point on the ith ray, and di denotes the direction vector of the ith ray. The position information of each ray in space is represented by the pllck coordinates in the predictive model. To achieve initialization of a three-dimensional gaussian scene, in one embodiment, a key parameter of ray density ρ is introduced. Thus, each ray may be specifically represented as an expanded version of li＝[di;mi;ρi, where di and mi are the direction and momentum characteristics of the ray, respectively, calculated from mi＝pi×di. ρi is the spatial volume density information carried by the modified light. Specifically, the input image I is subjected to a block processing to form a sequence S, and the sequence with the position code Φ (·) is input into the visual transducer model V to predict the light direction d, the momentum m, and the density ρ corresponding to the pixel block, which are expressed as:
Wherein,Representing predicted ray bundle distribution parameters, including parameter information for a series of rays. xi denotes a tile position, yi denotes tile input information, phi (xi) denotes position coding information corresponding to a tile, and N denotes the number of tiles. Hereinafter, also adopted areReplacement ofAre used to refer to the distribution of light bundles unless otherwise indicated by context.
In summary, the light prediction model is used to obtain the distribution characteristics of the light beam, including the momentum of the light, the direction of the light, and the bulk density of the light.
Step S220, calculating the pose of the camera based on the distribution characteristics of the light beam form.
Camera pose estimation based on the distribution position of a large number of light rays, camera poses associated with a plurality of view angles are calculated from least squares optimization. For example, for light bundles formed for input picture IThe camera light prediction method solves internal and external parameters lambda of a camera through a least square method, wherein the internal and external parameters lambda comprise specific pose parameters theta.
Because the three-dimensional Gaussian sputtering technology is generally used for inputting a plurality of pictures with multiple visual angles, the joint optimal solution of the light beams of the pictures with the same visual angles is considered in the camera pose calculation, and the light beams can be combined into one light beam for the calculation of least square optimization due to the distributed representation of the light beam forms.
In one embodiment, in order to further improve the accuracy of pose prediction, to cope with prediction uncertainty caused by view angle deletion in a sparse view, the predicted light direction and momentum can be further adjusted. For example, by means of a diffusion model, the light is subjected to noise adding and denoising, so that richer light beam distribution characteristics are obtained, and further, more accurate light prediction under a sparse view is obtained.
In summary, in step S220, based on the camera lighting method, a proxy task for predicting the light is used to recover pose information of the camera. By considering the application characteristics of the volume rendering technology in the three-dimensional Gaussian scene modeling, the prediction task is improved, the key dimension of density is increased, and the density information related to the initial scene is output while the pose of the camera is acquired. Such ray-carried density information essentially represents the inverse operation of the volume rendering process, i.e., the inverse derivation of scene structure from the density information. And initializing Gaussian point clouds of the scene through the density information, and providing a basis for training a high-precision three-dimensional Gaussian scene.
Step S230, for the ray bundle distribution, sampling is performed based on the volume density of the rays, and an initial spatial distribution of the three-dimensional gaussian point cloud focused on the visual center area is obtained.
In this step, under the camera ray prediction framework with density, the initialization of the three-dimensional gaussian scene, that is, the distribution of gaussian in the three-dimensional space based on ray density sampling is initialized.
The process begins with a proxy task, i.e., camera ray prediction acquires a large number of ray parametersEach ray li is defined by a direction vector di and a momentum vector mi in the plgram coordinate system. By resolving these ray parameters, their locations can be reconstructed in three-dimensional space and the volume density ρ of the rays used to reveal the point cloud distribution characteristics of the scene. The strategy utilizes the agent task of regression prediction of the light beam adopted by the pose estimation method based on the camera light, and captures prior information closely related to the camera pose by analyzing the distribution characteristics of the light beam in the scene space.
In particular, starting from an image pixel, along a ray path through the center of the cameraAnd (3) extracting samples (M is the number of ray paths) to serve as a starting point cloud position of three-dimensional Gaussian sputtering scene reconstruction. Each ray li maps from three-dimensional world spaceTo two-dimensional image spaceBy sampling the beam of light, an initial point cloud layout that is focused on the visual center region can be obtained.
FIG. 4 is a schematic diagram of an initial spatial distribution of ray sampling based on density to construct a three-dimensional Gaussian point cloud, specifically comprising the steps of:
Step S41, the light parameters in the Pluker coordinate systemMapping back to the actual position in the three-dimensional space to obtain the distribution of light rays in the three-dimensional space
Step S42, initializing a set of voxel grids in three-dimensional spaceAnd light rays are processed according to a certain ruleDivided into a plurality of segments. Wherein each voxel gridMay contain a plurality of light raysAnd each ray also carries its density distribution along the path { ρi(z)}z∈[0,L], k is the voxel grid index, i is the ray index intersecting the voxel grid, z is the argument, L represents the grid width,Representing a bundle of rays intersecting the voxel grid.
Step 43, initializing a three-dimensional gaussian sphere point in the center of the voxel grid based on the information, wherein the size of the gaussian point is based on the integration of a plurality of ray densities in the voxel grid, thereby forming an initial spatial distribution of the three-dimensional gaussian point cloud. Initializing a three-dimensional Gaussian distribution point gk at the central position of each voxel grid vk according to the light density information, and determining the size s of the three-dimensional Gaussian distribution point according to the integral density of the light in the voxelThereby constructing an initial spatial distribution of the three-dimensional Gaussian point cloudK represents the number of gaussian kernels in the scene.
These gaussian points will be further tuned in subsequent scene density control training and a number of small gaussian points will form larger gaussian points by deletion and gradient updating. In a subsequent scene density control training phase, these gaussian points will undergo further optimization adjustments, and the small gaussian points gk are evolved into larger gaussian points by gradient descent or merging operations, ultimately forming a finer three-dimensional scene representation. The process not only strengthens the geometric structure of the scene, but also improves the accuracy of scene density prediction.
In summary, the method is used for initializing the scene point cloud based on the transform predicted ray density distribution, and sampling the initial three-dimensional Gaussian points in a voxel grid integration mode, so that the initial spatial distribution of the three-dimensional Gaussian point cloud is formed. The method reduces the iteration times of the density control algorithm in the subsequent scene training process, and improves the accuracy of rendering results.
In step S240, a visual shell reflecting structural information of the scene object is obtained through view cone projection and object mask calculation.
In order to further improve the quality and robustness of three-dimensional scene training, in one embodiment, a visual shell is introduced, such as a three-dimensional visual shell is constructed based on two-dimensional depth segmentation information to eliminate redundancy, and the visual shell incorporates prior knowledge of the structure of objects in the input image. For example, the visual shell is calculated by view cone projection and object masking, and the visual shell can better preserve structural consistency required by multi-view reconstruction and has the capability of rejecting invalid point cloud data which may lead to unreasonable gaussian distribution compared with sparse point clouds generated by sampling alone. FIG. 5 is a schematic diagram of a visual shell combined with bundle density information to initialize a three-dimensional scene, forming a scene density grid suitable for initialization.
It is worth mentioning that the calculation cost of the visual shell is relatively low, and the visual shell is generated from a small number of key mask views only by means of the segmentation model, so that the visual shell can be easily integrated into an initialization process, and efficient and accurate three-dimensional scene reconstruction starting conditions are realized.
In one embodiment, in a three-dimensional scene reconstruction process, constructing a three-dimensional mesh in combination with a visual shell includes the steps of:
Step S51, extracting depth information and segmentation mask of the two-dimensional picture from the semantic segmentation modelWhere Ii denotes I input images, Di and Mi correspond to their depth maps and segmentation masks, respectively, and N is the number of images.
In step S52, the camera internal parameter matrix K and the external parameter matrix E of the picture are acquired to determine the conversion relationship between the camera coordinate system and the world coordinate system.
Step S53, initializing the scene voxel gridProjecting the camera parameter matrix K and E to a two-dimensional plane to obtain a corresponding two-dimensional projection grid
Step S54, filtering the projections of the non-reconstructed object content, and using the object mask Mi to apply to the two-dimensional projection gridFiltering to eliminate the content of non-reconstructed target object to form three-dimensional grid with filtered visual shell in three-dimensional spaceThis process ensures the accuracy and effectiveness of three-dimensional reconstruction while reducing the complexity of subsequent processing.
In sum, the redundant point cloud elimination technology based on the visual shell can effectively accelerate the model training speed and obviously inhibit the occurrence of unreasonable visual elements such as visual floaters and the like. The visual shell technology remarkably improves the efficiency and the precision of the three-dimensional scene initialization stage, reduces unnecessary calculation load, lays a solid foundation for the subsequent Gaussian point cloud optimization and scene density control, ensures the definition and the consistency of the reconstruction result,
And S250, carrying out three-dimensional Gaussian sputtering scene training based on the initial spatial distribution of the three-dimensional Gaussian point cloud and the visual shell by taking the set loss function minimization as an optimization target, and obtaining a three-dimensional scene reconstruction model, wherein the loss function comprises training regular terms of camera pose parameters.
In this step, three-dimensional scene training is initiated based on the initial spatial distribution of the three-dimensional gaussian point cloud and the visual shell until a set loss function criterion is met. In the training process, the loss function contains a training regular term of the camera pose parameter, and the scene reconstruction effect with higher precision can be realized by adjusting the pose parameter in the three-dimensional Gaussian scene training process.
Correction of camera parameters becomes particularly important when dealing with scenes from multiple perspectives, because of the dimensional differences that may exist between different perspectives. In addition, the final correction effect is poor due to the inconsistency of illumination between the viewing angles or slight motion blur generated when the handheld device photographs. Therefore, the invention optimizes the external parameters of the camera and the three-dimensional model simultaneously in the scene training process. In this process, the camera parameters are co-participated in the adjustment with the gaussian properties to achieve finer control. In addition, the invention applies a certain regularization condition lambda·IIE-E0 II to the camera parameters. Wherein E0 is a camera extrinsic matrix of an initial pose predicted by a camera ray method, and is also a global camera pose for scene training, E represents a camera extrinsic matrix of a current view angle, and lambda is a coefficient of a regularization term. In order to prevent the position and the posture of the optimized camera from being excessively shifted, the alignment quality in the whole three-dimensional Gaussian sputtering scene training process is optimized.
In one embodiment, the loss function is set to:
The loss function represents an optimization process for three-dimensional gaussian scene and camera pose parameters. Wherein,Representing an optimized three-dimensional Gaussian scene, Ej* representing an optimized camera outlier matrix at the current view j,Is an initial three-dimensional Gaussian scene to be optimized in the training process, and Ej represents a camera external parameter matrix to be optimized under the current view angle j.Representing a three-dimensional Gaussian scene to be optimizedAnd rendering into a 2D image under the corresponding pose parameters of the visual angle j.The real input image of the ith sheet under the visual angle j is represented, and N represents the number of the input images corresponding to the visual angle j. For the canonical term, λ is the coefficient of the canonical term,An initial camera outlier matrix representing the perspective j association, V represents the perspective classification of the input image.
In the training process, a regular term for the pose of the camera is effectively added, and the real pose of the camera is optimized through a neural network. Because the camera light method provides an initial term of camera pose parameters, the estimation of the camera pose can be more accurate and robust only by a small amount of constraint and training cost, so that a more accurate scene reconstruction result is obtained from a sparse view.
After training is completed, a three-dimensional scene reconstruction model can be obtained and further applied to new view synthesis, and the model can synthesize a multi-view new view aiming at an input image without pose.
In summary, compared with the prior art, the invention has the following advantages:
1) Aiming at the emerging three-dimensional Gaussian sputtering method in the technical field of new view synthesis, the invention provides a sparse pose-free input-oriented scene initialization strategy which can efficiently realize the precise estimation of the camera pose and the joint initialization of the scene. The invention combines the advantages of a camera ray method, provides critical initialization scene information for three-dimensional Gaussian sputtering training on the basis of ensuring the estimation efficiency and accuracy of the camera pose in the motion recovery structure, and remarkably improves the quality and detail richness of the final three-dimensional structure.
2) The invention provides an end-to-end implementation scheme for a sparse pose-free input-oriented three-dimensional Gaussian sputtering technology. On the premise of ensuring scene synthesis quality and consistency, the efficiency of initializing the three-dimensional Gaussian sputtering scene is greatly improved. The camera light method is integrated into the three-dimensional Gaussian sputtering training process, so that the unfixed problem of camera pose prediction caused by insufficient information and repeated textures in a sparse view is solved. The estimation precision and efficiency of the camera ray method under the sparse view are better than those of the current motion recovery structure.
3) The invention fully utilizes the agent task of light prediction, increases the dimension of light density, and realizes the prediction of the density information of the three-dimensional space structure. And the camera pose prediction and the three-dimensional Gaussian scene initialization process are combined to initialize the three-dimensional Gaussian point cloud, so that the accuracy and the rendering quality of the training scene are ensured.
4) According to the invention, the visual depth segmentation model is used for acquiring the depth information of the picture, and the two-dimensional scene is re-projected into the three-dimensional visual shell, so that the occurrence of redundant point clouds in the training process is restrained, unreasonable visual floaters are reduced, and the calculation burden of training is lightened.
5) The method and the device are combined with the training mode of camera pose optimization, so that the inconsistency of the camera pose in the multi-view picture is further corrected, and the accuracy of camera pose estimation under the sparse view is improved. Meanwhile, excessive offset of the initial camera pose predicted value is restrained by using a regular term, updating is terminated in advance to avoid introducing excessive calculation cost, and the alignment quality in the whole three-dimensional Gaussian sputtering scene training process can be optimized with less calculation amount.
6) Through multiple simulation tests, the method and the device can synthesize the expected multi-view picture no matter for the input image with or without pose, and improve the quality of the synthesized image while improving the calculation efficiency.
The present invention may be a system, method, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present invention.
The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.
The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as SMALLTALK, C ++, python, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information for computer readable program instructions, which can execute the computer readable program instructions.
Various aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.
The foregoing description of embodiments of the invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the technical improvements in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.