Disclosure of Invention
Because the existing method has the problems, the embodiment of the invention provides a method and a device for processing meteorological data and satellite remote sensing data.
In a first aspect, an embodiment of the present invention provides a method for processing meteorological data and satellite remote sensing data, including:
standardizing the received meteorological data and satellite remote sensing data according to a preset data format to generate standardized data;
dividing the standardized data into image data, vector data and elevation data according to data types, and performing unified storage, management and access on the image data, the vector data and the elevation data by adopting a file system and a relational database;
and receiving an access request sent by a client through a standardized interface, inquiring in the relational database according to the open geographic space information alliance OGC standard and the access request, and automatically issuing the result obtained by inquiry according to an issuing strategy.
Optionally, the dividing the standardized data into image data, vector data, and elevation data according to data types, and performing unified storage, management, and access on the image data, the vector data, and the elevation data by using a file system and a relational database specifically includes:
dividing the standardized data into image data, vector data and elevation data according to data types, abstracting the image data, the vector data and the elevation data into each product object according to the data types, data resource paths, palette resource paths and data timeline resource paths, and uniformly storing, managing and accessing each product object by adopting a file system and a relational database.
Optionally, the automatically publishing the result obtained by querying according to the publishing policy specifically includes:
and constructing an image pyramid for the inquired result according to the image pyramid technology, and automatically publishing the inquired result according to the current resolution, the image pyramid and a publishing strategy.
Optionally, the automatically publishing the result obtained by querying according to the publishing policy specifically includes:
and acquiring a Uniform Resource Locator (URL) corresponding to the inquired result, and automatically publishing the inquired result according to a publishing strategy and the URL.
Optionally, the standardized data includes cloud image data, cloud detection data, cloud phase data, cloud top height data, precipitation estimation data, water body monitoring data, fire monitoring data, surface temperature data, and convection nascent data.
In a second aspect, an embodiment of the present invention further provides a meteorological data and satellite remote sensing data processing apparatus, including:
the data standardization module is used for carrying out standardization processing on the received meteorological data and the satellite remote sensing data according to a preset data format to generate standardized data;
the data storage module is used for dividing the standardized data into image data, vector data and elevation data according to data types, and uniformly storing, managing and accessing the image data, the vector data and the elevation data by adopting a file system and a relational database;
and the data query module is used for receiving an access request sent by a client through a standardized interface, querying in the relational database according to the open geographic space information alliance OGC standard and the access request, and automatically issuing a result obtained by querying according to an issuing strategy.
Optionally, the data storage module is specifically configured to divide the standardized data into image data, vector data, and elevation data according to a data type, abstract the image data, the vector data, and the elevation data into each product object according to a data type, a data resource path, a palette resource path, and a data timeline resource path, and perform unified storage, management, and access on each product object by using a file system and a relational database.
Optionally, the data query module is specifically configured to construct an image pyramid from the query result according to an image pyramid technology, and automatically publish the query result according to the current resolution, the image pyramid, and a publication policy.
In a third aspect, an embodiment of the present invention further provides an electronic device, including:
at least one processor; and
at least one memory communicatively coupled to the processor, wherein:
the memory stores program instructions executable by the processor, which when called by the processor are capable of performing the above-described methods.
In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium storing a computer program, which causes the computer to execute the above method.
According to the technical scheme, the meteorological data and the satellite remote sensing data which are multi-source, multi-scale and multi-form are divided into different types of data after being subjected to standardized processing, are uniformly stored, managed and accessed, and are inquired by adopting the OGC standard, so that the meteorological data and the satellite remote sensing data can be effectively managed and accessed.
Detailed Description
The following further describes embodiments of the present invention with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.
Fig. 1 shows a schematic flow chart of a meteorological data and satellite remote sensing data processing method provided by this embodiment, including:
s101, standardizing the received meteorological data and satellite remote sensing data according to a preset data format to generate standardized data.
The standardized data comprises cloud image data, cloud detection data, cloud phase data, cloud top height data, precipitation estimation data, water body monitoring data, fire point monitoring data, earth surface temperature data and convection nascent data.
The preset data format is a preset data format.
Specifically, in data processing, firstly, in order to ensure the expandability and consistency of the system, standardized processing is performed on input meteorological data and satellite remote sensing data, and a uniform data format defined inside the system is generated in the aspects of file names, file formats, storage modes, data management and the like of the data.
And S102, dividing the standardized data into image data, vector data and elevation data according to data types, and performing unified storage, management and access on the image data, the vector data and the elevation data by adopting a file system and a relational database.
Specifically, after the standardized data is obtained, the standardized meteorological data and the standardized satellite remote sensing data need to be processed into three data types of an image, a vector and an elevation according to different types. At present, the processing of wind and cloud series satellite remote sensing data, carbon satellite remote sensing data and numerical weather forecast data is supported. In the aspect of data management, a unified management mode of a file system and a relational database is adopted, various types of data are uniformly stored, managed and accessed through the relational database and the file system, and the storage, management and access efficiency of the system is ensured through optimized design in multiple aspects of storage, a database server, a database, middleware, an application system and the like.
S103, receiving an access request sent by a client through a standardized interface, inquiring in the relational database according to the open geographic space information alliance OGC standard and the access request, and automatically issuing the inquired result according to an issuing strategy.
Specifically, in product release, vector, raster and image data can be accessed at the client through a standardized interface by adopting a standard and open OGC service (including WMS, WFS and WCS). Meanwhile, the release strategy can be customized according to needs, the real-time release of full-automatic meteorological and satellite remote sensing products without manual intervention is realized, and the automatic generation and release of multi-source, multi-scale and multi-form meteorological data and satellite remote sensing data are supported.
According to the embodiment, the multi-source, multi-scale and multi-form meteorological data and satellite remote sensing data are divided into different types of data after being subjected to standardized processing, are uniformly stored, managed and accessed, are inquired by adopting the OGC standard, and can be effectively managed and accessed.
Further, on the basis of the above method embodiment, S102 specifically includes:
dividing the standardized data into image data, vector data and elevation data according to data types, abstracting the image data, the vector data and the elevation data into each product object according to the data types, data resource paths, palette resource paths and data timeline resource paths, and uniformly storing, managing and accessing each product object by adopting a file system and a relational database.
Specifically, in data management, all meteorological data and satellite remote sensing data are abstracted into a product object with attributes such as a data type, a data resource path, a palette resource path, a data timeline resource path and the like, and each product object is managed after configuration, so that more effective management can be performed.
Further, on the basis of the above method embodiment, the automatically issuing the result obtained by querying according to the issuing policy in S103 specifically includes:
and constructing an image pyramid for the inquired result according to the image pyramid technology, and automatically publishing the inquired result according to the current resolution, the image pyramid and a publishing strategy.
For example, for the generated remote sensing image product, an image pyramid is established according to a certain resolution by an image pyramid technology. The image pyramid construction method includes two methods: one is that the data source of multi-resolution automatically constructs the pyramid, and the other is that the image data of other layers is constructed by sampling and extracting from the bottom layer data except that the bottom layer data of the pyramid is the original image data.
By adopting the image pyramid technology, the display speed of the remote sensing image product during release can be increased.
Further, on the basis of the above method embodiment, the automatically issuing the result obtained by querying according to the issuing policy in S103 specifically includes:
and acquiring a Uniform Resource Locator (URL) corresponding to the inquired result, and automatically publishing the inquired result according to a publishing strategy and the URL.
Specifically, the data access scheme may be in the form of a resource-oriented RESTful architecture, and any data resource (including images, files, etc.) has a corresponding URL identified in the architecture, and a user needs to operate through the URL corresponding to the resource to acquire data, where the data is transmitted in some representation form, such as Html, JSON, JPEG, etc. In addition to the above-mentioned acquisition form of the tile data resources of the image product, the data can be processed through restful api, so as to provide a plurality of data types such as GEOjson, KML and the like for the user.
For example, as shown in fig. 2, the present embodiment mainly includes three parts in combination with an architecture diagram of an automated production and distribution system for meteorological data and satellite remote sensing data: the system comprises multi-source data standardization processing, a universal data access service and an internet-based interactive product display program supporting multiple layers, multiple forms and full resolution.
As shown in fig. 2, the input data of the system includes wind cloud series satellite remote sensing product data (wind cloud 3C, wind cloud 3D, wind cloud 4A, etc.), various numerical forecast product data (T639 data, NCEP data, EC data, etc.), and other satellite remote sensing product data (NPP, NOA, sunflower 8, etc.). The geospatial data related to the product has the characteristics of multilevel, multiple subjects, multiple scales, multiple forms, mass and the like, and comprises various data types such as cloud picture images, cloud detection, cloud phase states, cloud top height, precipitation estimation, water body monitoring, fire point monitoring, surface temperature, convection birth, basic geographic information data and the like. The construction of the system fully considers the diversity, complexity and huge data quantity of the data, and meanwhile, the efficiency of data processing, data storage, data retrieval, data loading and data browsing is improved through various optimization means. The automatic release of multi-level, multi-topic, multi-scale and multi-form mass products is realized.
The specific execution steps are as follows:
firstly, dividing product data into primary data (cloud image products after quality inspection, calibration and positioning), secondary data and tertiary data according to grades in a business processing link, carrying out standardized design on file names of output data, and unifying naming rules. For example:
FY4A_AGRIX_L1_NOM_20171225_2220_2000M_EVB0730.jpg
FY4A_AGRIX_L3_GLL_20171225_POAD_1000M_VWI.jpg
satellite name _ instrument name _ data level _ projection type _ observation start date time _ spatial resolution _ product name jpg.
The generated products are then managed through a unified directory structure, for example:
~/FY4A/L2/IMAGE/LDA/AOD1/20171226234500
v/satellite name/data level/data type/product name/data set name/time
For example, as shown in fig. 3, is a partial catalog of telemetry data products for FY4 satellites.
And establishing an image pyramid for the generated remote sensing image product according to a certain resolution ratio by an image pyramid technology. The image pyramid construction method includes two methods: one is that the data sources of multiple resolutions automatically construct a pyramid; the other method is that image data of other layers are constructed by sampling and extracting from the bottom layer data, except that the bottom layer data of the pyramid is the original image data. At present, 8-layer pyramid images are built according to the resolution of satellite remote sensing data. The relation between the slice coordinates and the longitude and latitude is as follows:
x=(int)((tempLong-TileFullminLong)/(resolution*Tilesize))
y=(int)((templat-TileFullminLat)/(resolution*Tilesize))
where resolution is the resolution (degrees) of each layer, as shown in the table below; TileFullminLong is the starting longitude-180 of the slice image; TileFullminLat is the initial latitude of the slice image-90; tilesize is the pixel width (or height) 256 of the slice; tempLong is the longitude of a point; templat is the latitude of a point. The following table gives a list of resolutions for each level:
the size of the layer data block is 256 × 256 block size currently adopted after considering the network transmission efficiency and the data loading efficiency.
The sliced data are stored in the form of "/Z/Y/x.jpg ("/hierarchy number/row number/column number. jpg). And maps this file path to the DocumentRoot through the Apache server. When data is accessed, the data can be quickly retrieved by analyzing the URL path. For example: the method comprises the steps of mapping/COMM into a root directory, directly retrieving data in a mode of a root path + a relative path when acquiring data through http,// 10.24.10.95/FY4A/L2/IMAGE/PRJ/500M/0064/20171227023834/4/28/10.jpg, adopting a resource-oriented RESTful architecture form by a/COMM/FY 4A/L2/IMAGE/PRJ/500M/0064/20171227023834/4/28/10.jpg data access scheme, identifying any data resource (including IMAGEs, files and the like) in the architecture by a corresponding URL, and operating the user by the URL corresponding to the resource when the user wants to acquire the data, wherein the data in the operations are transmitted in a certain expression form of the user, such as Html, JSON, JPEG and the like. In addition to the above-described acquisition form of the tile data resources of the image product, the data can be processed through restful api, so as to provide a plurality of data types such as GEOjson and KML for the user.
In data management, all meteorological and satellite remote sensing products are abstracted into a product object with the attributes of data type, data resource path, palette resource path, data timeline resource path and the like, and are managed through configuration. After the product display program obtains the configuration information, interactive operation can be provided for the user according to the definition of the product attribute.
By the reasonable comprehensive application of the various means, the college organization and management of multi-level, multi-subject and multi-form mass meteorological and satellite remote sensing data can be well realized. The whole system has the advantages of openness, easy expansion, easy maintenance and the like.
Fig. 4 shows a schematic structural diagram of a meteorological data and satellite remote sensing data processing device provided by this embodiment, the device includes: adata normalization module 401, adata storage module 402, and adata query module 403, wherein:
thedata standardization module 401 is configured to standardize the received meteorological data and satellite remote sensing data according to a preset data format to generate standardized data;
thedata storage module 402 is configured to divide the standardized data into image data, vector data, and elevation data according to data types, and perform unified storage, management, and access on the image data, the vector data, and the elevation data by using a file system and a relational database;
thedata query module 403 is configured to receive an access request sent by a client through a standardized interface, query in the relational database according to the open geospatial information federation OGC standard and the access request, and automatically publish a result obtained by the query according to a publication policy.
Specifically, thedata standardization module 401 standardizes the received meteorological data and satellite remote sensing data according to a preset data format to generate standardized data; thedata storage module 402 divides the standardized data into image data, vector data and elevation data according to data types, and performs unified storage, management and access on the image data, the vector data and the elevation data by adopting a file system and a relational database; thedata query module 403 receives an access request sent by a client through a standardized interface, queries in the relational database according to the open geospatial information federation OGC standard and the access request, and automatically issues a result obtained by querying according to an issuing strategy.
According to the embodiment, the multi-source, multi-scale and multi-form meteorological data and satellite remote sensing data are divided into different types of data after being subjected to standardized processing, are uniformly stored, managed and accessed, are inquired by adopting the OGC standard, and can be effectively managed and accessed.
Further, on the basis of the above device embodiment, thedata storage module 402 is specifically configured to divide the standardized data into image data, vector data, and elevation data according to a data type, abstract the image data, the vector data, and the elevation data into each product object according to a data type, a data resource path, a palette resource path, and a data timeline resource path, and uniformly store, manage, and access each product object by using a file system and a relational database.
Further, on the basis of the above device embodiment, thedata query module 403 is specifically configured to construct an image pyramid from the query result according to an image pyramid technology, and automatically publish the query result according to the current resolution, the image pyramid, and a publication policy.
The meteorological data and satellite remote sensing data processing device described in this embodiment may be used to implement the above method embodiments, and the principle and technical effect are similar, which are not described herein again.
Referring to fig. 5, the electronic device includes: a processor (processor)501, a memory (memory)502, and abus 503;
wherein,
theprocessor 501 and thememory 502 are communicated with each other through thebus 503;
theprocessor 501 is used to call program instructions in thememory 502 to perform the methods provided by the above-described method embodiments.
The present embodiments disclose a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the methods provided by the above-described method embodiments.
The present embodiments provide a non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform the methods provided by the method embodiments described above.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
It should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.