Disclosure of Invention
In order to overcome the above-mentioned deficiencies of the prior art, an object of the present invention is to provide a mimo communication system and method for multimedia microservice, which is used to implement mimo communication for multimedia microservice based on mesh and object storage.
To achieve the above and other objects, the present invention provides a mimo communication system for multimedia microservice, comprising:
the service gateway correspondingly sends an http request to the upstream server, sends the http request containing the storage information to the downstream server after receiving the storage information returned by the storage server corresponding to the mesh component of the upstream server, and receives a calculation response result of the downstream server through the mesh component corresponding to the downstream server;
the Mesh component is arranged corresponding to each server, and when the Mesh component is taken as a Mesh component corresponding to an upstream server, the Mesh component receives an http request sent by the service gateway and sends the http request to the upstream server; receiving a multi-output message returned by an upstream server, transmitting packed data in the received multi-output message to a storage server for storage, acquiring storage information returned by the storage server and returning the storage information to a service gateway; when the gateway is used as a mesh component corresponding to a downstream server, receiving an http request carrying storage information sent by the service gateway and directly transmitting the http request to the downstream server, intercepting a data downloading request of the downstream server, acquiring corresponding data from the storage server according to the data downloading request, sending the corresponding data to the downstream server, and returning a calculation result fed back by the downstream server to the service gateway after receiving the calculation result;
the upstream server is used for receiving the http request sent by the mesh component, processing the http request to obtain a calculation result of a multi-output file structure, and sending the calculation result of the multi-output file structure to the mesh component through a multi-output message;
the storage server is used for receiving the data sent by the mesh component, and returning corresponding storage information to the mesh component after the received data is subjected to persistent storage;
and the downstream server acquires the http request which is forwarded by the mesh component and contains the storage information, initiates a corresponding data downloading request according to the http request, downloads and acquires corresponding data from the storage server through the mesh component, performs multi-input calculation after the data acquisition is completed, and returns a calculation result to the service gateway through the mesh component.
Preferably, the Mesh component further comprises:
and the upstream http request receiving and checking module is used for receiving the http request sent by the service gateway, checking whether the http request conforms to a preset rule, and sending the http request to the upstream server after checking that the http request conforms to the rule.
The multi-output message receiving and processing module is used for receiving an http response of a multi-output message which is returned by the upstream server and contains a plurality of data files and a summary file, and transmitting data in the http response to the storage server;
the storage result response module is used for receiving the storage information fed back by the storage server and returning the storage information to the service gateway as a response result of the http request;
a downstream http request receiving and processing module, configured to receive an http request including storage information sent by the service gateway, check whether the http request conforms to a preset rule, forward the http request to the downstream server after checking that the http request conforms to the rule, intercept a data download request of the downstream server for the storage information, forward the data download request to a certain storage service instance of the storage server through a balancing algorithm, obtain a summary file at the storage server, parse the summary file to obtain addresses of other data files, and obtain corresponding data from the storage server and send the data to the downstream server;
and the calculation result response module is used for receiving the calculation result response of the downstream server and feeding back the calculation result response to the service gateway.
Preferably, the calculation result of the http request by the upstream server is a multi-output file structure, the multi-output file structure includes a plurality of data files and a summary file of an application layer protocol, the data files and the summary file are in a unified directory hierarchy, and the summary file enumerates each used address to the data file by using an http relative path.
Preferably, the upstream server packs the multiple data files and the summary file according to a streaming multi-file packaging protocol, and returns the packed data files and the summary file as a request result to the mesh component.
Preferably, the http message returned by the upstream server includes a multi-output message identifier to tell that the message is a multi-output message and a summary file name thereof.
Preferably, the http message format returned by the upstream server includes http header, x-dora-multioutput-manifest: < summary file name >.
Preferably, after receiving the data sent by the mesh component, the storage server creates a virtual bucket, stores the received data in the virtual bucket, and returns the domain name corresponding to the virtual bucket to the mesh component after the storage is completed.
Preferably, the http request sent by the service gateway to the downstream server includes the domain name/summary file name fed back by the upstream server.
In order to achieve the above object, the present invention further provides a multimedia microservice mimo communication method, comprising the steps of:
step S1, the service gateway sends an http request to an upstream server;
step S2, the mesh component corresponding to the upstream server receives the http request sent by the service gateway and sends the http request to the upstream server, when receiving the multi-output message returned by the upstream server, the mesh component transfers the packed data in the received multi-output message to the storage server for storage, and obtains the storage information returned by the storage server and returns the storage information to the service gateway;
step S3, after receiving the storage information returned by the mesh component, the service gateway sends an http request containing the storage information to the downstream server;
step S4, corresponding to the mesh component of the downstream server, receiving the http request carrying the storage information sent by the service gateway and directly transmitting the http request to the downstream server, intercepting the data download request of the downstream server, obtaining corresponding data from the storage server according to the data download request, sending the corresponding data to the downstream server, and returning the calculation result fed back by the downstream server to the service gateway after receiving the calculation result.
Preferably, the step S2 further includes:
step S200, receiving an http request sent by a service gateway, checking whether the http request accords with a preset rule, and sending the http request to an upstream server after checking that the http request accords with the rule;
step S201, receiving an http response of a multi-output message which is returned by an upstream server and contains a plurality of data files and a summary file, and transmitting data in the http response to a storage server;
and step S202, receiving the storage information fed back by the storage server, and returning the storage information to the service gateway as a response result of the http request.
Compared with the prior art, the invention has the following beneficial effects:
according to the invention, when multiple output requirements exist, the multiple output files only need to be packaged and returned according to the protocol format, and the packaging result can be used by downstream calculation without paying attention to storage service, so that an application developer only needs to pay attention to the core algorithm of the application without paying attention to the operation environment and how to perform upstream cooperation specifically;
the invention is compatible with the http rpc protocol, and can enable the App service to expand the multi-output and multi-input capability under the original communication protocol
Thirdly, the invention adopts a stream type packaging protocol, and the storage service can rapidly store data
Fourthly, the communication between the services is ensured by adopting the mesh component, the packaged data only needs to be spit back when the App uploads, the common communication problems of the distributed system, such as which storage service instance is sent to and how to process if the storage is wrong, do not need to be concerned, the App downloads only need to be launched to the mesh component, and the common communication problems of the distributed system also do not need to be concerned
And fifthly, the object storage is adopted as the bottom storage in the invention, so that the horizontal expansion in the aspects of storage space, storage data access amount and the like can be realized, and the data in the object storage can be conveniently converted into an external serviceable http address through an s3 protocol, so that the App can conveniently read the data.
Detailed Description
Other advantages and capabilities of the present invention will be readily apparent to those skilled in the art from the present disclosure by describing the embodiments of the present invention with specific embodiments thereof in conjunction with the accompanying drawings. The invention is capable of other and different embodiments and its several details are capable of modification in various other respects, all without departing from the spirit and scope of the present invention.
Fig. 1 is a system architecture diagram of a mimo communication system for multimedia microservice according to the present invention. As shown in fig. 1, the present invention provides a mimo communication system for multimedia microservice, comprising:
theservice gateway 10 sends an http request to theupstream server 30, sends an http request containing storage information to thedownstream server 50 after receiving the storage information returned by thestorage server 40 returned by themesh component 20 corresponding to theupstream server 30, and receives a calculation response result of thedownstream server 50 through themesh component 20 corresponding to thedownstream server 50.
TheMesh component 20 is arranged corresponding to each server, and when the Mesh component is a Mesh component corresponding to an upstream server, receives an http request sent by theservice gateway 10 and sends the http request to theupstream server 30; receiving the multi-output message returned by theupstream server 30, transmitting the packed data in the received multi-output message to thestorage server 40 for storage, acquiring the storage information returned by thestorage server 40 and returning the storage information to theservice gateway 10; when the mesh component is a mesh component corresponding to the downstream server, the http request carrying the storage information sent by theservice gateway 10 is received and sent to thedownstream server 50, the data download request of thedownstream server 50 is intercepted, the corresponding data is obtained from thestorage server 40 according to the data download request and sent to thedownstream server 50, and the calculation result fed back by thedownstream server 50 is received and returned to theservice gateway 10.
Specifically, themesh component 20 further includes:
an upstream http request receiving and checking module, configured to receive an http request sent by aservice gateway 10, check whether the http request meets a preset rule, and send the http request to anupstream server 30 after checking that the http request meets the rule, where the http request generally includes data to be processed and a processing command, for example, under a seven-cow fop multimedia processing protocol, the http request is: POST/handlercmd xxx & url yyy wherein xxx after cmd represents a processing command and url represents a data address to be processed.
Specifically, after receiving the http request sent by theservice gateway 10, the upstream http request receiving and checking module first checks whether the http request is a valid http request, and when the http request is a valid http request, further checks whether the http request is a request sent to theupstream server 30, and if so, considers that the http request conforms to the rule and sends the rule to theupstream server 30. Preferably, the upstream http request receiving checking module may further check whether the initiator of the request has a right to send the request to the upstream server.
And the multi-output message receiving and processing module is configured to receive an http response of the multi-output message, which includes a plurality of data files and a summary file, returned by theupstream server 30, and transmit data in the http response to the storage server.
That is, after receiving the http request sent by the http request receiving and checking module, theupstream server 30 processes and calculates the http request, and the calculation result is a multi-output file structure including a plurality of data files and a summary file of an application layer protocol, for example, a video file in the format of mp4 is subjected to calculation of converting and packaging into the format of hls, or a video is subjected to screenshot according to a specified time interval so as to output a batch of screenshots, and the calculation result is packaged and a multi-output message is returned to themesh component 20.
And a storage result response module, configured to receive the storage information fed back by thestorage server 40, and return the storage information to theservice gateway 10 as a response result of the http request.
Specifically, after thestorage server 40 receives the data sent by theMesh component 20, the received data is persistently stored, the storage information is returned to theMesh component 20, theMesh component 20 returns the storage information to theservice gateway 10, so that the corresponding data file can be accessed through the storage information through thestorage server 40, for example, after thestorage server 40 persistently stores the data sent by the Mesh component, a domain name corresponding to the storage file is returned, such as http:// media.example.example.com// media.m.3u8, and theMesh component 20 returns a summary file address http:// media.example.com// sample.m.3u8 to theservice gateway 10
The downstream http request receiving and processing module is configured to receive an http request including storage information sent by theservice gateway 10, check whether the http request conforms to a preset rule, forward the http request to thedownstream server 50 after checking that the http request conforms to the rule, intercept a service request of the downstream server to the storage information, forward the service request to a certain storage service instance of thestorage server 40 through a specific balancing algorithm, obtain a summary file, and analyze the summary file to obtain addresses of other data files, thereby obtaining corresponding data from thestorage server 40, and send the corresponding data to thedownstream server 50.
For example, theservice gateway 10 sends an http request containing storage information to a downstream server through the Mesh component, when theMesh component 20 receives an http request containing storage information (such as address information) sent by theservice gateway 10, the http request is directly forwarded to the downstream server after checking that the http request meets a preset rule, the downstream server initiates a service request for the address information after receiving the http request containing the address information, theMesh component 20 intercepts a data download request of the downstream server for the address, the Mesh component ensures that such a request is correctly processed in a distributed system, such as load balancing and error retry, that is, theMesh component 20 forwards the data download request to a certain storage service instance through a specific load balancing algorithm, obtains a summary file and parses the summary file to obtain addresses of other data files, thereby obtaining corresponding data from thestorage server 40, and the data is sent to a downstream server, and when a network error occurs, the mesh component can identify retry, so that the reliability of data downloading is ensured.
In the embodiment of the present invention, the http request of the service gateway to thedownstream server 50 may include upstream output, that is, addresses of storage information "domain name/summary file name", for example, the request includes addresses of several summary files, that is, http:// media.example.com/example.m3u8, as input of calculation, theMesh component 20 forwards the http request to the downstream server, the downstream server initiates a data download request for the "domain name/summary file name", theMesh component 20 intercepts the data download request and forwards the data download request to a certain storage service instance through a load balancing algorithm, acquires a corresponding summary file from thestorage server 40 according to the "domain name/summary file name", and acquires addresses of other data files by analyzing the content of the summary file, thereby acquiring the data from thestorage server 40.
And a calculation result response module, configured to receive a calculation result response of thedownstream server 50, and feed back the calculation result response to theservice gateway 10.
That is, when thedownstream server 50 finishes acquiring the data, the multi-input calculation is completed, and the result is returned to theservice gateway 10 through themesh component 20.
Theupstream server 30 is configured to receive an http request sent by themesh component 20, process the http request to obtain a calculation result of a multi-output file structure, and send the calculation result of the multi-output file structure to themesh component 20 through a multi-output packet.
In an embodiment of the present invention, the calculation result of theupstream server 30 is a multi-output file structure, that is, a summary file including a plurality of data files and an application layer protocol, for example, in hls format, the data file is ts, the summary file is m3u8, and the summary file defined by the summary file application layer and in hls format, m3u8 summary file is taken as an example:
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXTINF:9.009,
/first.ts
#EXTINF:9.009,
/second.ts
#EXTINF:3.003,
/third.ts
it can be seen that there are three subfiles in this summary file, which are:
/first.ts
/second.ts
/third.ts
in the embodiment of the present invention, theupstream server 30 packages the plurality of data files and the summary file according to a streaming multi-file packaging protocol (such as tar), that is, packages the example.m3u8 file and the three ts files, and returns the packaged data files and the summary file as a request result to themesh component 30. In the embodiment of the present invention, the http message returned by theupstream server 30 may include a multi-output message identifier, such as an http header, an x-dora-multi-output-manifest < summary file name >, to tell the external that the message is a multi-output message and its summary file name.
And thestorage server 40 is configured to receive data sent by themesh component 20, perform persistent storage on the received data, and return corresponding storage information to themesh component 20.
Specifically, theMesh component 20 stores the 4 files into thestorage server 40 after receiving the files, and thestorage server 40 creates a virtual bucket after receiving the data sent by theMesh component 20 and stores the data in the virtual bucket; after the storage is finished, thestorage server 40 returns the domain name corresponding to the virtual bucket, such as http:// media. example. com, to themesh component 20, so that the summarized file can be accessed through thestorage server 40 by "domain name/summarized file name", and the data file can be accessed by "domain name/data file name". The mesh returns the summary file address http:// media. example. com/example. m. 3u8 to the gateway
In the present invention, since the data of theupstream server 20 is uploaded in a package by using a streaming protocol, thestorage server 40 can persist into the object store after a single subfile is stored, without waiting for all data to be received into the object store.
Thedownstream server 50 acquires the http request containing the storage information forwarded by themesh component 20, initiates a corresponding data downloading request according to the http request, downloads and acquires corresponding data from thestorage server 40 through themesh component 20, performs multi-input calculation after the data acquisition is completed, and returns a calculation result to theservice gateway 10 through themesh component 20.
FIG. 2 is a flowchart illustrating steps of a method for MIMO communication with multimedia microservice according to the present invention. As shown in fig. 2, the present invention provides a multimedia microservice mimo communication method, comprising the following steps:
step S1, the service gateway sends an http request to the upstream server.
Step S2, the mesh component corresponding to the upstream server receives the http request sent by the service gateway and sends the http request to the upstream server, and when receiving the multi-output packet returned by the upstream server, transmits the packed data in the received multi-output packet to the storage server for storage, and obtains the storage information returned by the storage server and returns the storage information to the service gateway.
Specifically, step S2 further includes:
and step S200, receiving an http request sent by the service gateway, checking whether the http request conforms to a preset rule, and sending the http request to an upstream server after checking that the http request conforms to the rule.
Step S201, receiving an http response of a multi-output packet including a plurality of data files and a summary file returned by the upstream server, and transmitting data in the http response to the storage server.
That is, after receiving the http request sent by the http request receiving and checking module, the upstream server processes and calculates the http request, and the calculation result is a multi-output file structure including a plurality of data files and a summary file of an application layer protocol, and packages the calculation result and returns a multi-output message to the mesh component.
And step S202, receiving the storage information fed back by the storage server, and returning the storage information to the service gateway as a response result of the http request.
And after the storage server receives the data sent by the mesh component, the received data is subjected to persistent storage, the storage information is returned to the mesh component, and the mesh component returns the storage information to the service gateway, so that the corresponding data file can be accessed through the storage information through the storage server.
Specifically, after receiving data sent by the mesh component, the storage server creates a virtual bucket and stores the data in the virtual bucket; after the storage is finished, thestorage server 40 returns the domain name corresponding to the virtual bucket to themesh component 20, so that the summarized file can be accessed through thestorage server 40 by using the "domain name/summarized file name", and the data file can be accessed by using the "domain name/data file name".
And step S3, after receiving the storage information returned by the mesh component, the service gateway sends an http request containing the storage information to the downstream server.
Step S4, corresponding to the mesh component of the downstream server, receiving the http request carrying the storage information sent by the service gateway, directly transmitting the request to the downstream server, intercepting the data download request of the downstream server, obtaining corresponding data from the storage server according to the data download request, sending the corresponding data to the downstream server, and returning the calculation result fed back by the downstream server to the service gateway after receiving the calculation result.
Examples
As shown in fig. 3, in this embodiment, an upstream server is App a, a downstream server is App B, and a mesh component is added to each service to perform traffic management between services, where the mesh component may complete traffic management of a normal http rpc request and may also serve a multi-output scene, and specifically, a communication process of a multi-input multi-output communication system of multimedia microservice is as follows:
step 1, a service gateway sends an http request to an AppA through a mesh component, the mesh component checks an entry request, and if a request is matched with a rule, the request is sent to the AppA;
step 2, after the AppA calculates according to the http request, according to the request requirement, the calculation result is a multi-output file structure, the multi-output file structure is a summary file of a plurality of data files and an application layer protocol, for example, under the hls format, the data files are ts, the summary file is m3u8, the data files and the summary file adopt http relative paths to enumerate the addresses of the used data files under the uniform directory level, the AppA packs the data files and the summary file according to a streaming multi-file packaging protocol (for example, tar) and returns the data files and the summary file as the request result to the mesh component, and the returned http message contains http header and x-dora-multioutput-manifest, the summary file name is used for telling that the external file is the name of the multi-output file and the summary file name thereof.
Take the m3u8 summary file in hls format defined by the summary file application layer as an example:
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXTINF:9.009,
/first.ts
#EXTINF:9.009,
/second.ts
#EXTINF:3.003,
/third.ts
it can be seen that there are three subfiles in this summary file, which are:
/first.ts
/second.ts
/third.ts
after downloading and parsing the summary m3u8 file, the downstream server can download and process these subfiles through the above addresses, for example, the addresses of the m3u8 file are http:// media.
http://media.example.com/first.ts
http://media.example.com/second.ts
http://media.example.com/third.ts
When the upstream server side AppA packs and outputs an example.m3u8 file and three ts files.
And 3, after receiving the multi-output message, the Mesh component transmits the packed data in the received http response (http response) to the storage service.
After receiving the data, the storage service creates a virtual socket and stores the data in the virtual socket; after the storage is finished, the storage service returns the domain name corresponding to the virtual bucket, so that the summary file can be accessed through the storage service by the 'domain name/summary file name'; the data file may be accessed by a "domain name/data file name". Because the data is uploaded in a streaming protocol package, the storage service can persist into the object store after a single subfile is stored, without waiting for all data to be received in the re-persisted track object store.
And 4, after the storage is finished, the mesh component returns the domain name/summary file name as a result to the service gateway.
Taking the above example as an example, the Mesh component stores the 4 files into the storage service after receiving the files, the storage service returns the domain name corresponding to the stored file, such as http:// media.example.com, and the Mesh component returns the summarized file address http:// media.example.com/example.m3u8 to the service gateway
And step 5, after receiving the result, the service gateway comprises an upstream output, namely a domain name/summary file name, in the request to the downstream App B, and the upstream output is used as the input of calculation.
The service gateway sends a request to a downstream server side App B through a mesh component, wherein the request comprises the addresses of several summary files of http:// media.
Andstep 6, App B can acquire the domain name/summary file name through the proxy service of the proxy Mesh component, and can acquire the addresses of other data files by analyzing the content of the file, so that the data can be read.
And 7, finally completing multi-input calculation by App B after reading the data, and returning the result to the service gateway through the mesh component.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Modifications and variations can be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the present invention. Therefore, the scope of the invention should be determined from the following claims.