Disclosure of Invention
In order to solve the existing problems, the invention provides a data interaction system based on micro-service, which comprises a service gateway, a service management system, a transaction coordination system and a basic service system, wherein,
the service gateway performs data interaction with the service management system and the transaction coordination system and is used for routing forwarding with the micro-service;
the service management system is used for carrying out service period management on the accessed micro-service, wherein the service period management comprises service standard management under a test environment and service running state unified management under an actual application environment;
the transaction coordination system is used for coordinating the registration, submission and rollback of the micro-service distributed transaction;
and the basic service system performs data interaction with the transaction coordination system and is used for realizing service application of the front-end system according to the message queue of the micro-service.
Preferably, the service gateway is implemented by a spring gateway technology, performs load balancing in a software manner, and is further used for user authentication, service routing, log recording request, fusing, and degradation processing.
Preferably, the service authentication of the service gateway is used for ensuring the security of the internal call of the service and the call between the interface and the third-party application, wherein the service authentication is realized by constructing a user authentication system by using the Oauth2 standard and providing authority control of the call between services by using jwt technology.
Preferably, the service management system at least comprises a service configuration unit, a service discovery unit and an authentication service unit;
the service configuration unit is used for providing a dynamic configuration updating function, reducing service redeployment and improving service dynamic updating capability;
the service discovery unit is used for reducing the coupling between the servers and simultaneously realizing dynamic capacity expansion and capacity reduction of the service under the condition that the service is not required to be adjusted;
and the authentication service unit is used for performing uniform security authentication on the upstream data provider and preventing the inflow of abnormal data.
Preferably, the service configuration unit uses apollo as a configuration center, and is used for at least realizing version management, gray release, version rollback, and multi-environment separation of service configuration.
Preferably, the service discovery unit provides a service discovery function by using a consul, performs remote invocation in cooperation with feign, and generates a coding experience consistent with that of invoking a local interface, so as to reduce coupling and implement dynamic capacity expansion and capacity reduction.
Preferably, the system further comprises a fusing current limiting module which interacts with the service gateway and the transaction coordination system respectively and is used for avoiding data blockage caused by service, and the service is subjected to current limiting, fusing and degrading by adopting Sentinel as a flow Sentinel.
Preferably, the transaction coordination system is a distributed transaction management system, and the distributed transaction is processed in a tcc mode, so that multi-service transactions are managed uniformly, and the data consistency requirement and the service idempotency are ensured.
Preferably, the basic service system comprises a basic service module and a log query module;
the basic service module is used for realizing the memory caching function, the relational database function, the object storage function and the message service function of the micro-service;
the log query module at least comprises a log analysis function, a log storage and retrieval function and a log query function;
the log query module is used for reducing the request pressure on the database and improving the service access speed by taking Redis as a module cache component;
the log query module is used as a log message receiving component through Kafka and used for receiving log messages;
and the log query module performs unified processing and storage of logs and establishes a retrieval platform through the ELK.
Preferably, the system further comprises a monitoring module, which interacts with the transaction coordination system;
the monitoring module is realized by adopting a combination mode of link monitoring, dimension monitoring and service monitoring and is used for helping operators to know the operation condition of the system.
The invention discloses the following technical effects:
compared with the prior art, the technical scheme of the invention has the following beneficial effects:
the method has the advantages of cross-platform, low cost and fast iteration requirement;
the invention has complete integral framework and function, achieves the level of production operation level, ensures that the change and adjustment of the later-stage business are more independent, and reduces the level of coupling among services.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1-2, the present invention provides a data interactive system based on microservice, the data interactive system includes a service gateway, a service management system, a transaction coordination system, and a basic service system, wherein,
the service gateway performs data interaction with the service management system and the transaction coordination system and is used for routing forwarding with the micro-service;
the service management system is used for carrying out service period management on the accessed micro-service, wherein the service period management comprises service standard management under a test environment and service running state unified management under an actual application environment;
the transaction coordination system is used for coordinating the registration, submission and rollback of the micro-service distributed transaction;
and the basic service system performs data interaction with the transaction coordination system and is used for realizing service application of the front-end system according to the message queue of the micro-service.
The service gateway is realized by a spring gateway technology, adopts a software mode to carry out load balancing, and is also used for user authentication, service routing, log recording request, fusing and degradation processing.
And the service authentication of the service gateway is used for ensuring the internal call security of the service and ensuring the call security between the interface and the third-party application, wherein the service authentication is realized by constructing a user authentication system by using the Oauth2 standard and providing authority control of the call between services by using jwt technology.
The service management system at least comprises a service configuration unit, a service discovery unit and an authentication service unit; the service configuration unit is used for providing a dynamic configuration updating function, reducing service redeployment and improving service dynamic updating capability; the service discovery unit is used for reducing the coupling between the servers and simultaneously realizing dynamic capacity expansion and capacity reduction of the service under the condition that the service is not required to be adjusted; and the authentication service unit is used for performing uniform security authentication on the upstream data provider and preventing the inflow of abnormal data.
The service configuration unit adopts apollo as a configuration center and is used for at least realizing version management, gray level release, version rollback and multi-environment separation of service configuration.
The service discovery unit provides a service discovery function by adopting the consul, remote calling is carried out by matching with the feign, and coding experience consistent with that of calling the local interface is generated for reducing coupling and realizing dynamic capacity expansion and reduction.
The system also comprises a fusing current limiting module which is respectively interacted with the service gateway and the transaction coordination system and used for avoiding data blockage caused by service, and the service is subjected to current limiting, fusing and degrading by adopting the Sentinel as a flow Sentinel.
The transaction coordination system is a distributed transaction management system, adopts a tcc mode to process distributed transactions, is used for uniformly managing multi-service transactions, and ensures the consistency requirement of data and the idempotency of service.
The basic service system comprises a basic service module and a log query module; the basic service module is used for realizing the memory caching function, the relational database function, the object storage function and the message service function of the micro-service; the log query module at least comprises a log analysis function, a log storage and retrieval function and a log query function; the log query module is used for reducing the request pressure on the database and improving the service access speed by taking Redis as a module cache component; the log query module is used as a log message receiving component through Kafka and used for receiving log messages; and the log query module performs unified processing and storage of logs and establishes a retrieval platform through the ELK.
The system also comprises a monitoring module which interacts with the transaction coordination system;
the monitoring module is realized by adopting a combination mode of link monitoring, dimension monitoring and service monitoring and is used for helping operators to know the operation condition of the system.
Example 1: the micro-service architecture based on the SpringCloud technology stack is adopted to design and research and develop the whole system, and based on rich research and development experience accumulated in the past system development process, various service components more suitable for the actual requirements of the project are customized, and the service components include but are not limited to the following components:
the service gateway: the spring gateway technology is adopted as a service gateway technology, and the spring gateway technology is used as a unique entry of a project, and provides user authentication, service routing, log recording request, fusing and degradation processing.
And (3) service authentication: a user authentication system is constructed by using the Oauth2 standard, authority control of calling between services is enhanced by using the jwt technology, safety of calling inside the services is guaranteed, and meanwhile technical support is provided for an interface to third-party application.
Service discovery: the cusul is adopted to provide a service discovery function, a specified remote service address is not needed in calling between services, coupling is effectively reduced, meanwhile, feign is matched to carry out remote calling, coding experience consistent with calling of a local interface is achieved, and meanwhile, dynamic capacity expansion and capacity reduction of the services can be achieved without service adjustment.
Service configuration: by adopting the apollo as a configuration center, the functions of configuration version management, gray level release, version rollback, multi-environment separation and the like are realized, and meanwhile, the dynamic configuration updating function is provided, the service redeployment is reduced, and the dynamic service updating capability is improved.
Fusing and current limiting: the method has the advantages that a micro-service framework is adopted, the condition that 'avalanche' is generated in service is avoided inevitably, the Sentinel is adopted as a flow Sentinel, the service is subjected to current limiting and fusing degradation service, and the phenomenon that 'avalanche' is generated in calling between services is avoided.
Distributed transaction: after the services are split, local transactions cannot be adopted for management like the original single application, one service may simultaneously relate to a plurality of services, the data consistency cannot be ensured, the distributed transactions in a tc mode are adopted for processing, the multi-service transactions are managed in a unified mode, and the data consistency requirement and the service idempotency are ensured; idempotent of data: a hash value generated by a core service field of the metadata is used as a unique key to ensure the idempotency of the upstream data, and the storage pressure caused by a large amount of repeated data when the upstream data is pushed in a full scale is reduced.
Message middleware: and the rabbitmq and the kafka are used as message middles to provide event-based cooperative processing capability among services and provide asynchronous processing capability of the services.
Monitoring: a set of complete monitoring solution is provided by adopting a combination mode of link monitoring, dimension monitoring and service monitoring. Effective optimization suggestions and reliable Shooting can be provided according to actual monitoring conditions, and meanwhile, operators can conveniently and comprehensively know the system operation conditions.
A log base component: redis is adopted as a system cache component, so that the request pressure to a database is reduced, and the service access speed is increased. And a Kafka log message receiving component is adopted, and a unified processing storage and retrieval platform of the log is performed through the ELK.
The data interaction system has several functions:
1. data reception
(1) And (3) metadata storage: and dynamically mapping the data provided by the external upstream and the structure of the internal data storage, and directly and dynamically routing the data provided by the upstream system to the corresponding data structure for storage according to the name and the type of the data provided by the upstream system.
(2) And (4) safety authentication: and carrying out uniform security authentication on the upstream data provider to prevent the inflow of abnormal data.
(3) Idempotent of data: a hash value generated by a core service field of the metadata is used as a unique key to ensure the idempotency of the upstream data, and the storage pressure caused by a large amount of repeated data when the upstream data is pushed in a full scale is reduced.
2. Data push
(1) Data configurability: the metadata is dynamically mapped and cleaned into customized data required by a service side through a Freemarker template when being pushed downstream, and the data required by the service can be conveniently and quickly online.
(2) Data push compensation: the effectiveness of data pushing is ensured through a timing compensation mechanism, and abnormal data can timely inform technicians to check problems of service or data through mails.
3. Business driver
To ensure real-time performance and performance for large amounts of upstream data storage, as well as downstream data consumption. By using the MQ as a bridge for data reception and pushing, the reception of upstream data and the pushing of downstream data are decoupled.
On the data push side, each service can obtain data of a format type required by the service according to the requirement of the service, and can also be used for bridging the existing metadata storage in a customized manner and aggregating to obtain the required service data.
In order to improve timeliness of data requirements, metadata receiving and sending and dynamic mapping of storage are achieved, and all storage operations are achieved in a self-defining mode on the basis of JDBC.
In order to deal with the change of business, according to the business characteristics of the main data synchronization process, the whole data circulation is designed based on the state mode, different data state processes can be executed to process the data according to different requirements, and the expandability is improved.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus once an item is defined in one figure, it need not be further defined and explained in subsequent figures, and moreover, the terms "first", "second", "third", etc. are used merely to distinguish one description from another and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the present invention in its spirit and scope. Are intended to be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.