Disclosure of Invention
In view of the above, the present invention provides a system and a method for cross-system file sharing based on an unstructured platform, so as to solve the problem that large file transmission affects system performance during cross-system transmission, and to achieve timely message delivery.
Based on the above purpose, the invention provides a cross-system file sharing system based on an unstructured platform, which comprises an unstructured data file;
the system A is used as a main system for storing, uploading and sending messages of unstructured data files;
the system B is a target object of unstructured data transmission;
the message storage platform comprises a headquarter message storage platform and a secondary message storage platform, and is used for a data exchange platform for message transmission to carry out two-stage data communication;
the unstructured platform comprises a headquarter unstructured platform and a secondary unstructured platform, and is used for transmitting, storing, managing and uniformly providing basic public services for the outside.
Preferably, the method comprises the steps of:
2.1, uploading the unstructured data file to a headquarter unstructured platform by the system A, and sending a data distribution instruction;
2.2, distributing the file to different secondary unstructured platforms by the headquarter unstructured platform;
2.3, after the distribution is successful, the unstructured platform returns a distribution success message to the system A;
2.4, the system A sends instruction information and attachment information to the message storage platform;
2.5, the system B receives the message, and when the uploading of a new file is finished, the system B consumes the message to a message storage platform;
2.6, the system B downloads corresponding unstructured files in the secondary unstructured platform according to the instruction message and the attachment information to finish one-time data sharing.
Preferably, the uploading of unstructured data in the system A comprises the following steps:
3.1, the unstructured platform supports the operation of uploading, downloading and deleting the unstructured data files;
3.2, after the unstructured data file is uploaded, storing the data in a headquarter unstructured platform;
3.3, distributing the data to a secondary unstructured platform by the headquarter unstructured platform;
and 3.4, after the operation is successful, the unstructured platform returns the processing result to the system A.
Preferably, the message passing and downloading process comprises the steps of:
4.1, when a new unstructured data file needs to be uploaded, sending the message to a message queue of a headquarter message storage platform by calling a secondary message storage platform;
4.2, the headquarter message storage platform sends the data to a second-level message storage platform message queue;
4.3, subscribing the message from the B system to the message storage platform;
4.4, when a new message arrives, the system B consumes the message to the secondary message storage platform and receives the through message;
4.5. and the system B downloads the corresponding file to the unstructured platform according to the consumed message to finish data sharing.
Preferably, the message storage platform message delivery mode is: the persistent subscription can complete consumption when the persistent subscriber is online again and can not lose messages even if the subscriber is not online currently and messages sent by a sender after the subscriber subscribes.
Preferably, the message storage platform message delivery mode is: queue mode, which implements a point-to-point communication mode, where each message can only be received by one consumer.
Preferably, an electronic device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method according to any one of claims 2 to 2 when executing the program.
Preferably, a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any of claims 2 to.
From the above, it can be seen that the present invention provides a cross-system file sharing system and method based on an unstructured platform, which has the following beneficial effects:
the invention combines the unstructured data management platform and the message storage platform to realize the sharing of large unstructured data. The system A needs to share data with the system B, large unstructured data are uploaded to an unstructured data management platform of a third party, instructions and attachment information are sent to a message storage platform, the system B subscribes messages in the message storage platform, and when new messages arrive, the system B consumes the messages in the message storage platform and downloads corresponding data. In the process of sharing the unstructured data, the transmission of the large file cannot be directly completed through the system A and the system B, so that the problem that the system performance is influenced because the transmission occupies system resources is solved. The data sharing is realized by taking the unstructured management platform as a data storage intermediate library and taking the message storage platform as a message transfer relay. Meanwhile, the setting and the use of the message storage platform can ensure that the message can be timely and accurately transmitted.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to specific embodiments and the accompanying drawings.
It is to be noted that technical terms or scientific terms used in the embodiments of the present invention should have the ordinary meanings as understood by those having ordinary skill in the art to which the present disclosure belongs, unless otherwise defined. The use of "first," "second," and similar terms in this disclosure is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly.
The invention is an embodiment: a cross-system file sharing system and method based on an unstructured platform.
The sharing system comprises an unstructured data file, which is referred to as a GIM in the embodiment, wherein the GIM is a Grid Information Model (GIM) concept which is provided by a national grid company according to the application of each stage in the information process of analyzing transmission and transformation projects from the perspective of information technology and engineering technology and mainly combined with the application condition of digital technology.
The system A is used as a main system for storing, uploading and message transferring of unstructured data files; and B, a target object of unstructured data transmission.
The message storage platform comprises a headquarter message storage platform and a secondary message storage platform, and a data exchange platform for message transmission is used for performing two-stage data communication. In this embodiment, the message storage platform refers to UEP, the headquarter message platform refers to headquarter UEP platform, and the secondary message storage platform refers to provincial UEP platform. UEP is a data exchange platform for message transmission by a power grid company, and performs two-stage data communication, and the instant message bus technology of UEP is mainly used herein.
The unstructured platform comprises a headquarter unstructured platform and a secondary unstructured platform, namely, the provincial network unstructured platform in the embodiment, and is used for transmitting, storing, managing and uniformly providing basic public services for the external world for unstructured data.
The following brief introduction to unstructured platform, GIM and UEP:
unstructured platform: data assets in an enterprise can be divided by type into structured data and unstructured data. Wherein structured data refers to data that can be represented in a two-dimensional table and processed using a relational database; unstructured data is data that cannot be represented using a two-dimensional table as structured data or processed using a relational database. Wherein the unstructured data accounts for approximately 70% of the total amount of enterprise data. The unstructured data management platform is a data center which integrates business application systems of unstructured data contained in a company, and is used for storing, circulating, managing and comprehensively applying all unstructured data uploaded by each business application system in a centralized manner.
Its advantages are as follows: 1. unified storage of unstructured data: through the unstructured data management platform, unstructured data storage of large data volume in an enterprise is achieved, a storage optimization strategy is customized, a storage structure is optimized, and the utilization rate of storage resources is improved. 2. Unified management of unstructured data: by formulating various unstructured data management standards, the unified management of unstructured data in enterprises is realized. The management of the metadata of the content in the enterprise is unified; the sharing and the exchange of data between the service application systems are unified. The management of the unstructured data is standardized, streamlined and automated. 3. The openness and unification provide basic public services to the outside: the method can provide basic public services including but not limited to content storage service, content access service, content management service, operation monitoring service and various tools (data analysis and mining tools and the like) for the outside, and can provide uniform unstructured data utilization service for related business application systems in enterprises. 4. The service application system supporting various development platforms is accessed: the platform has the capability of orderly and standardizing the intervention of diversified business systems in an enterprise, and can provide an integral access solution and relevant access specifications matched with the integral access solution for different types of business application systems.
GIM: the GIM is an electric network information model (GIM) concept which is provided by national network companies according to the application of information technology and engineering technology in each stage of the information process of the system analysis transmission and transformation engineering, and mainly combines the application condition of the digital technology. The GIM digitalizes the constituent elements of the power grid by means of a Geographic Information System (GIS), and realizes efficient, comprehensive and accurate application of information by taking an information model as a carrier, wherein the carrier of the power grid information model is a GIM file.
GIM has the following advantages: 1. virtualization: an entity model is established from the beginning of the design of the power transmission and transformation project and serves as an information carrier to promote the visualization and virtualization of the whole process of the project. The existing technology has the advantages that a digital three-dimensional virtual model is established, and meanwhile, the characteristics of a power grid need to realize layered display of model data related to a geographic information system and a GIS. 2. Relevance: according to different requirements, a unified model interface and information layering are determined to be established, information collaboration and sharing among parties with different information requirements in engineering construction are achieved, and collaborative design is promoted. As an open information carrier, the method can realize information transmission and sharing among units of design, management, material, operation and the like, solve data faults and realize one-time recording and multiple-time adoption of engineering data. And establishing a database for bearing information based on GIM (group information management), and realizing cross-platform sharing and information acquisition. 3. Simulation property: by adopting the GIM technology, the functions of power transmission and transformation project simulation construction, simulation emergency repair, simulation roaming and the like can be realized, and meanwhile, the system has an associated interface for online monitoring information and can provide structured and unstructured project data for production departments. 4. Full life periodicity: by utilizing the GIM technology, the data acquisition capacity can be greatly improved, and reliable basis can be provided for all decisions in the whole life cycle from purchase to scrapping of equipment by utilizing the shared knowledge information resources of the GIM. 5. Safety: according to the grade protection requirement of the power system, the GIM belongs to a non-control area, is used as an information template, carries out safety protection according to the principle of 'standard first, pilot application', and sets different query and application grades according to stages and subareas, thereby ensuring the safety and reliability of engineering information.
And (6) UEP: UEP is a data exchange platform for message transmission by a power grid company, and performs two-stage data communication, and the instant message bus technology of UEP is mainly used herein. The system is a middleware which meets JMS1.1 specification and faces message type data exchange, is used for sending messages among application programs or in a distributed system, and supports high-efficiency asynchronous communication with throughput of up to 80000 pieces/second. The system is generally used as an enterprise-level high-performance data integration bus, communication applications do not need to know physical positions of the communication applications, so that application integration is loosely coupled, how to establish communication among the communication applications is not needed, the complexity of communication of the application bottom layer is simplified, the communication applications do not need to be in a running state at the same time, reliable arrival of bus guarantee messages is avoided, and cross-platform and cross-system communication is achieved without running under the same operating system or network environment. The system supports three communication modes of one-to-one, one-to-many and request-response, and provides APIs of common programming languages such as java, c/c + + and the like. The system nodes can be deployed independently, and can also be deployed by distributed multi-level departments and networked to form a transverse and longitudinal data switching network.
The method for transmitting the GIM file from the system A to the system B comprises the following steps:
firstly, A, B complete process of data sharing between two systems: as shown in fig. 1
1. And uploading the GIM file to a headquarter unstructured platform by the system A, and sending a data distribution instruction.
2. And the headquarter unstructured platform distributes the files to different provincial network unstructured platforms.
3. And after the distribution is successful, the unstructured platform returns a distribution success message to the system A.
And 4. the A system sends instruction information and attachment information to the message storage platform.
And 5, subscribing the message by the system B, and consuming the message from the system B to the message storage platform when the uploading of the new file is finished.
And 6, the system B downloads corresponding unstructured files in the networked province unstructured platform according to the instruction message and the attachment information to finish one-time data sharing.
Secondly, file uploading process in the system A:
1. the unstructured platform provides support for uploading, downloading and deleting of data files.
And 2, after uploading the GIM file, storing the data in the headquarter unstructured platform.
3. And the headquarter unstructured platform distributes the data to different provincial network unstructured platforms.
4. And after the operation is successful, the unstructured platform returns a processing result to the system A.
Thirdly, message transmission and downloading processes: as shown in fig. 2
1. And when a new GIM file needs to be uploaded, calling a provincial UEP platform through a provincial application server, and sending the message to a headquarter UEP platform message queue.
2. And the headquarter UEP platform sends the data to a provincial UEP platform message queue.
B system-to-message storage platform subscription messages.
And 4, receiving the message in real time by the system B, and when a new message arrives, consuming the message from the system B to the message storage platform and receiving the through message.
5. And downloading the corresponding file to the unstructured platform to finish data sharing.
In addition, the message storage platform messaging mode in this embodiment is as follows:
a. the persistent subscription can complete consumption when the persistent subscriber is online again and can not lose messages even if the subscriber is not online currently and messages sent by a sender after the subscriber subscribes. Applicable scenarios are as follows: and the system A pushes data to the system B through UEP. Through the messaging model of publish-subscribe. Since the data involved is important: such as information about funds, transactions, stock prices. To ensure that the B system must receive the message sent by the a system, consider the abnormality such as power-off and restart of the B system, and set a persistent subscription mode. It can be guaranteed that after B subscribes to a topic, the subscriber state becomes inactive because of the power down. After the system B is restarted, the message can still be received.
b. Queue mode, which implements a point-to-point communication mode, where each message can only be received by one consumer. Applicable scenarios are as follows: in queue mode, a message can only be received by one consumer. For example, in an order system of a shopping website, when a foreground user places an order, a message is sent to a background processor through UEP, each order can be processed only once, and in this scenario, a point-to-point queue mode can be used.
Finally, the embodiment relates to an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the sharing method mentioned above. It also relates to a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the sharing method we have mentioned above. Both of the above mentioned types can be used as corresponding extensions of the present embodiment.
In summary, the unstructured data management platform and the message storage platform are combined for use, and large unstructured data sharing is achieved. The system A needs to share data with the system B, large unstructured data are uploaded to an unstructured data management platform of a third party, instructions and attachment information are sent to a message storage platform, the system B subscribes messages in the message storage platform, and when new messages arrive, the system B consumes the messages in the message storage platform and downloads corresponding data. In the process of sharing the unstructured data, the transmission of the large file cannot be directly completed through the system A and the system B, so that the problem that the system performance is influenced because the transmission occupies system resources is solved. The data sharing is realized by taking the unstructured management platform as a data storage intermediate library and taking the message storage platform as a message transfer relay. Meanwhile, the setting and the use of the message storage platform can ensure that the message can be timely and accurately transmitted.
Computer-readable media of the present embodiments, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the idea of the invention, also features in the above embodiments or in different embodiments may be combined, steps may be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity.
In addition, well known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown within the provided figures for simplicity of illustration and discussion, and so as not to obscure the invention. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the invention, and also in view of the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the present invention is to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the invention, it should be apparent to one skilled in the art that the invention can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic ram (dram)) may use the discussed embodiments.
The embodiments of the invention are intended to embrace all such alternatives, modifications and variances that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements and the like that may be made without departing from the spirit and principles of the invention are intended to be included within the scope of the invention.