Disclosure of Invention
The invention aims to provide an internet-based data service system for solving the defects in the background technology.
In order to achieve the above purpose, the invention provides the following technical scheme that the data service system based on the Internet comprises a data acquisition module, a network resource management module, a priority perception task scheduling module, a cloud processing and storage module, a comprehensive analysis module and an optimization module;
The data acquisition module is used for collecting original data from various distributed sensors in real time;
The network resource management module is used for monitoring and optimizing the data transmission path and carrying out real-time path adjustment by calculating the dynamic congestion degree;
the priority perception task scheduling module is used for dynamically adjusting the response priority of the task according to the importance of the task and evaluating the real-time performance of task execution by calculating service response priority deviation;
the cloud processing and storage module comprises a distributed storage system and a high-performance computing platform and is used for processing and storing the uploaded dynamic congestion degree data and response priority offset data;
The comprehensive analysis module is used for comprehensively analyzing the dynamic congestion degree and the service response priority deviation to determine the influence severity of the network congestion on the task response delay;
And the optimizing module is used for immediately switching the transmission of the high-priority task to the low DCL path if the severity is high, preferentially inserting the high-priority task into the task scheduling queue, and carrying out post-processing on the low-priority task, and reducing the data transmission rate of the low-priority task and improving the sequencing position of the high-priority task in the queue if the severity is low.
Preferably, in the network resource management module, n paths are set, and the network performance of each path is described by k indexes to form a data matrix: Wherein X is an n×k matrix, representing k network performance indexes of n paths, Xnk represents a kth index value of an nth path, and normalizing each column of data to zero mean and unit variance: wherein muj is the mean value of the jth index, sigmaj is the standard deviation of the jth index, and the normalized data matrix is Z: the covariance matrix Σ is used to describe the correlation between different indices: Wherein, sigma is a k×k matrix, element sigmaij in the matrix represents covariance of the ith index and the jth index, eigenvalue decomposition is carried out on the covariance matrix Sigma vi＝λivi, wherein, lambdai is the ith eigenvalue, which represents the interpretation variance proportion of the corresponding principal component, vi is the ith eigenvector, which represents the direction of the ith principal component, the eigenvector v1 corresponding to the largest eigenvalue is selected as the first principal component according to the arrangement of the eigenvalues from large to small, and the dynamic congestion degree is calculated by the expression: Where DCL is the dynamic congestion level and v1j is the j-th eigenvector component corresponding to the first principal component.
Preferably, in the network resource management module, a DCL threshold value is set, paths are classified according to a calculation result, namely, a low congestion path is DCL which is less than or equal to 0.5, a medium congestion path is DCL which is less than or equal to 0.5, a high congestion path is DCL which is more than 0.8, real-time DCL data of all available paths in the system are acquired, the DCL value is compared, a path with the lowest DCL is preferentially selected as a substitute, a high priority task is required to be allocated to the low congestion path, if no low congestion path exists, a standby network is started, a medium/low priority task is allocated to the medium congestion path or a suboptimal path, current limiting treatment is carried out on the high congestion path, and bandwidth resources are gradually released.
Preferably, in the priority aware task scheduling module, after analyzing the service response priority shift condition, a response priority shift index is generated, real-time performance of task execution is evaluated, and the method for acquiring the response priority shift index is as follows:
setting the actual response time as Tactual,i, representing the actual completion time of the task, setting the expected response time as Texpected,i, representing the target time when the task should be completed within the specified time, setting the service response offset value SRPDi, calculating the offset value calculated according to the task response time to obtain SRPDi＝Tactual,i-Texpected,i, setting the distribution rule of the priority P weight W as P=3 high priority, w=0.6, P=2 medium priority, w=0.3, P=1 low priority, w=0.1, and calculating the weight sum Wtotal of each task according to the task priority and the corresponding offset value, wherein the expression is as follows: Wherein wi is task priority weight, n is total task number, and the weighted offset values of all tasks are summed and divided by the weight sum to obtain a response priority offset index, and the expression is: Where SRPDI is the response priority offset index.
Preferably, in the comprehensive analysis module, comprehensive analysis is performed by combining the dynamic congestion degree and the service response priority offset, and the influence severity of the network congestion on the task response delay is determined, which specifically includes:
Converting the dynamic congestion degree and the response priority shift index into comprehensive feature vectors, taking the comprehensive feature vectors as input of a machine learning model, taking the influence severity value labels of each group of comprehensive feature vectors on the task response delay of the predicted network congestion of the machine learning model as a prediction target, taking the sum of prediction errors of the influence severity value labels of the minimum network congestion on the task response delay of the predicted network congestion as a training target, training the machine learning model until the sum of the prediction errors reaches convergence, stopping model training, and determining the influence severity value of the network congestion on the task response delay according to a model output result, wherein the machine learning model is a polynomial regression model.
Preferably, in the optimization module, the obtained influence severity value of the network congestion on the task response delay is compared with the influence severity reference threshold value set according to the historical data, if the influence severity value of the network congestion on the task response delay is greater than or equal to the set influence severity reference threshold value, the influence severity of the network congestion on the task response delay is high, an early warning signal is generated at the moment, the transmission of the high-priority task is immediately switched to a low DCL path, and the high-priority task is preferentially inserted into the task scheduling queue, and if the influence severity value of the network congestion on the task response delay is smaller than the set influence severity reference threshold value, the influence severity of the network congestion on the task response delay is low, the early warning signal is not generated at the moment, the data transmission rate of the low-priority task is reduced, and the ordering position of the high-priority task in the queue is improved.
Preferably, the transmission of the high-priority task is switched to the low DCL path, and the high-priority task is preferentially inserted into the task scheduling queue, specifically:
Switching the transmission path of the high-priority task to the path with the lowest dynamic congestion degree DCL to ensure that the transmission delay is minimized, wherein the formula is as follows: Wherein Popt is the optimal transmission path, P is the set of all available paths, DCL (P) is the dynamic congestion degree of path P, and the DCL (P) values of all paths in the current path set P are collected according to a common ruleThe path Popt with the lowest dynamic congestion degree is selected, and the data flow of the high-priority task Thigh is transferred to the path Popt:
Inserting a high-priority task Thigh into a priority position of a task scheduling queue Q to ensure execution instantaneity of the high-priority task, wherein the formula is that Q=insert (Q, Thigh, priority), Q is the task scheduling queue, Thigh is the high-priority task to be inserted, the priority is a priority attribute and used for sequencing adjustment, and a queue sequencing rule is that Sort (Q) =argmaxT∈Q priority (T), wherein the priority (T) is a priority value of the task T, the higher the value is, the higher the task priority is, the priority weight priority (T) is defined, and the priority weight is set for each task T: whigh>wlow indicates that the weight of the high-priority task is greater than that of the low-priority task, whigh、wlow indicates that the weights of the high-priority task and the low-priority task are respectively inserted into the priority positions in the queue according to the task priority, and the task queue Q is ordered from high to low according to the priority value.
Preferably, the data transmission rate of the low-priority task is reduced, and the ordering position of the high-priority task in the queue is improved, specifically:
The data transmission rate Rlow of the low-priority task is adjusted, so that the instantaneity of the high-priority task is ensured, and the new transmission rate of the low-priority task is ensured: in the formula,For an adjusted low priority task data transmission rate,In order to adjust the data transmission rate of the low-priority task before adjustment, alpha is a rate reduction coefficient, the value range is 0< alpha <1, the transmission rate of the low-priority task is dynamically adjusted according to a calculation formula, a high-priority task Thigh is moved to a front position through a reordering task scheduling queue Q, calculation resources and execution time are preferentially allocated, a scheduling queue reordering rule is that Sort (Q) =argmaxT∈Q priority (T), a task priority definition function is that: And defining the priority weight whigh、wmid、wlow of each task, sequencing the task queue Q according to the priority value, leading the high-priority task to be in front, and generating a new task scheduling queue.
In the technical scheme, the invention has the technical effects and advantages that:
1. According to the invention, the real-time acquisition of multi-source data is realized through the data acquisition module, the network resource management module is combined with the dynamic congestion Degree (DCL) to carry out real-time path adjustment, the priority sensing task scheduling module is used for dynamically adjusting the task response priority and evaluating the real-time performance according to the task importance, the cloud processing and storage module is used for ensuring the efficient processing and reliable storage of the data by adopting distributed storage and high-performance calculation, the comprehensive analysis module is combined with the DCL and response priority deviation index to analyze the influence severity of network congestion on task delay, and the optimization module is used for dynamically adjusting the transmission path and the task scheduling strategy according to the analysis result, so that the stability and the real-time performance of the system under a high concurrency scene are improved.
2. The invention obviously reduces the transmission delay and the data loss risk caused by network flow overload, ensures the timely response of the high-priority task, and optimizes the utilization rate of system resources and the fairness of task scheduling. The system can dynamically adapt to the change of network environment, effectively cope with peak load, and promote overall performance through multi-level data optimization strategies, thereby providing reliable and extensible technical support for the fields of smart cities, industrial Internet and the like, and having extremely high practical value and popularization prospect.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
An embodiment, referring to fig. 1, the data service system based on the internet in this embodiment includes a data acquisition module, a network resource management module, a priority aware task scheduling module, a cloud processing and storage module, a comprehensive analysis module and an optimization module;
The data acquisition module is used for collecting original data from various distributed sensors in real time;
The network resource management module is used for monitoring and optimizing the data transmission path and carrying out real-time path adjustment by calculating the dynamic congestion degree;
the priority perception task scheduling module is used for dynamically adjusting the response priority of the task according to the importance of the task and evaluating the real-time performance of task execution by calculating service response priority deviation;
the cloud processing and storage module comprises a distributed storage system and a high-performance computing platform and is used for processing and storing the uploaded dynamic congestion degree data and response priority offset data;
The comprehensive analysis module is used for comprehensively analyzing the dynamic congestion degree and the service response priority deviation to determine the influence severity of the network congestion on the task response delay;
And the optimizing module is used for immediately switching the transmission of the high-priority task to the low DCL path if the severity is high, preferentially inserting the high-priority task into the task scheduling queue, and carrying out post-processing on the low-priority task, and reducing the data transmission rate of the low-priority task and improving the sequencing position of the high-priority task in the queue if the severity is low.
In the data acquisition module, the sensor interface unit communicates with the distributed sensor, supporting different types of hardware interfaces and communication protocols. The supported protocols are lightweight protocols, namely MQTT and CoAP (suitable for low-power consumption equipment). Conventional network protocol HTTP/HTTPs (for general internet devices). Industrial internet protocol Modbus, zigbee, loRa (suitable for industrial environments and long distance low power consumption communications). The adapter is designed with a built-in protocol converter to ensure that data of different devices and protocols can be parsed by the system in a standardized format.
The data filtering unit performs preliminary screening on the collected original data, removes redundant data or obvious error data, and improves data quality. The method comprises data integrity check (such as checksum verification). Rule-based filtering (e.g., removing invalid data packets, noisy data). Preliminary validity detection (e.g., numerical range checking).
And the time synchronization unit ensures that the data collected from the distributed sensor has consistent time stamps and supports time sequence analysis and comparison of the data. The technology is realized by synchronizing with a sensor based on NTP (network time protocol). And a local time stamp is added during data transmission, so that the time consistency of the uploaded data is ensured.
The data coding unit performs standardized processing on the sensor data in different formats, and is convenient for subsequent transmission and analysis. JSON, protobuf (lightweight, cross-platform support). Custom binary format (suitable for efficient communication scenarios).
The data security unit encrypts the data acquisition and transmission process, and prevents the data from being leaked or tampered in the transmission process. The technology is realized by using TLS protocol for data link encryption. Data signing and verification (e.g., HMAC-based integrity verification).
And the device supports the simultaneous connection of various types of sensors (such as a traffic flow sensor, air quality monitoring equipment and an energy consumption metering instrument). The data streams from the plurality of sensors are combined to generate a unified real-time data stream. The millisecond-level data acquisition period is realized, and the high-real-time requirement scene (such as traffic light optimization) is satisfied. The acquisition frequency is supported to be adjusted as required so as to adapt to different scenes (such as high-frequency traffic data and low-frequency environment data).
After data are collected, partial non-core data are locally filtered and compressed, so that the data volume uploaded to the cloud is reduced, and the transmission efficiency is improved. Examples are extracting metadata of a traffic video stream (e.g., vehicle count) without uploading a complete video.
The data acquisition module establishes communication with the distributed sensor through the sensor interface unit. And receiving the real-time data packet sent by the sensor. The data filtering unit checks the data packet and filters out noise and invalid data. Ensuring that the uploaded data is complete, accurate and meaningful. And adding a uniform time stamp to the data to ensure the time sequence consistency of the multi-source data. Encoding the data into a standard format (e.g., JSON) facilitates subsequent transmission and processing. The data is encrypted and integrity verification information is appended. And uploading the encrypted data to a network layer.
The synchronous connection and data acquisition of millions of sensors are realized through an asynchronous I/O technology (such as libuv and Netty), and the performance of the system under a high concurrency scene is ensured. When the sensor connection is interrupted, the module supports automatic reconnection and resumes data acquisition. When the network is unstable, data is temporarily cached and uploaded after the network is recovered, so that data loss is avoided.
And the network resource management module is used for monitoring and optimizing the data transmission path and carrying out real-time path adjustment by calculating the dynamic congestion degree.
The calculation of the dynamic congestion degree can quantify the load condition of the network path in real time by comprehensively analyzing network performance indexes (such as flow utilization rate, time delay, packet loss rate and the like). Through the unified measurement value, the network resource management module can rapidly identify a high-load path, and avoid transmission delay and data loss caused by path congestion, thereby improving the transmission efficiency and response speed of the system. In addition, the dynamic congestion degree provides data support for the optimization of multipath transmission, so that the system can intelligently select the optimal path from a plurality of available paths, and the risks of network interruption and service quality degradation are effectively reduced.
The calculation of the dynamic congestion degree is particularly critical in a large-scale concurrency scene, and provides a quantitative basis for real-time flow regulation and resource allocation decisions. For example, in an intelligent traffic system, by dynamically adjusting the transmission path of high priority accident data by means of dynamic congestion, it is possible to ensure that critical data is transmitted preferentially on a low congestion path, reducing delay. Meanwhile, the dynamic congestion degree can be used for monitoring the long-term running state of the network, and by analyzing the congestion trend, an operator is helped to identify potential bottlenecks, the network architecture is optimized, and the service quality and reliability of the whole network are improved.
N paths are set, and the network performance of each path is described by k indexes to form a data matrix: Wherein X is an n X k matrix, which represents k network performance indexes of n paths, Xnk represents the kth index value of the nth path, and common indexes include X1 is link traffic utilization, X2 is link delay ratio, and X3 is packet loss rate P.
To eliminate the effects of different index dimensions, each column of data is normalized to zero mean and unit variance: wherein muj is the mean value of the jth index, sigmaj is the standard deviation of the jth index, and the normalized data matrix is Z: the covariance matrix Σ is used to describe the correlation between different indices: Wherein, sigma is a k×k matrix, element sigmaij in the matrix represents covariance of the ith index and the jth index, eigenvalue decomposition is carried out on the covariance matrix Sigma vi＝λivi, wherein, lambdai is the ith eigenvalue, which represents the interpretation variance proportion of the corresponding principal component, vi is the ith eigenvector, which represents the direction of the ith principal component, the eigenvector v1 corresponding to the largest eigenvalue is selected as the first principal component according to the arrangement of the eigenvalues from large to small, and the dynamic congestion degree is calculated by the expression: Where DCL is the dynamic congestion level and v1j is the j-th eigenvector component corresponding to the first principal component.
Setting a DCL threshold value, and classifying paths according to a calculation result, wherein the low congestion path is DCL less than or equal to 0.5, the medium congestion path is DCL less than or equal to 0.5 and less than or equal to 0.8, and the high congestion path is DCL more than 0.8;
Real-time DCL data of all available paths in the system is acquired. Comparing the DCL values, the path with the lowest DCL is preferentially selected as the alternative. High priority tasks must be assigned to low congestion paths. If there is no low congestion path, an alternate network (e.g., dedicated communication channel) may be enabled. Medium/low priority tasks-to medium congestion paths or sub-optimal paths. And carrying out current limiting treatment on the high-congestion path, and gradually releasing bandwidth resources.
And modifying the routing table by using a network controller (such as an SDN controller) to switch the task flow from the high-congestion path to the low-congestion path, namely modifying the forwarding table entry of the target path to forward the flow according to the new path. And the high-priority data packets are processed preferentially, so that the real-time performance of delay sensitive data is ensured. The remaining data flows are split and high priority tasks are assigned to either the backup path or the low congestion path. The low priority tasks employ a multipath transmission strategy to spread data across multiple paths. The data flow allocation proportion of each path is dynamically adjusted by using a load balancing algorithm (such as weighted polling and minimum connection number).
And reserving special bandwidth for tasks with different priorities, namely reserving more bandwidth for tasks with high priorities, and avoiding preemption by low-priority traffic. And the task with low priority limits the maximum bandwidth occupation and reduces the influence of the task on the path resources. And controlling the transmission rate of the high-congestion path by adopting a flow control algorithm (such as Token socket or leakage socket) to gradually relieve the congestion state. After the adjustment is completed, the DCL data of the path is continuously monitored, and the congestion relief effect is verified. If the desired effect is not achieved after the path switch, the other alternative paths are again evaluated. The path switching frequency and the traffic distribution strategy are dynamically adjusted. And storing the adjustment record into a system log for subsequent training of the machine learning model, and improving a path optimization strategy.
And the priority perception task scheduling module is used for dynamically adjusting the response priority of the task according to the importance of the task and evaluating the real-time performance of task execution through calculating service response priority deviation.
According to the importance and timeliness of the tasks, the tasks are divided into three types of high, medium and low priority tasks, namely, tasks with high priority, such as emergency handling, disaster early warning and the like, which need real-time response. Medium priority tasks, regular monitoring data (e.g. traffic flow monitoring). Low priority tasks, periodic or batch tasks (e.g., environmental data upload). And adjusting the priority of the task in real time according to the system load and the task characteristics.
The deviation of the actual response time and the expected response time is quantified, namely, the deviation is smaller, namely, the task scheduling is normal, and the response meets the requirement. The deviation is large, which indicates that task processing may be affected by network or resource bottleneck and needs optimization. System resources (bandwidth, computing resources and the like) are dynamically allocated according to task priorities, and execution of high-priority tasks is guaranteed preferentially. Throttling, deferred processing, or batch scheduling is performed on low priority tasks.
Static priority scheduling, namely determining the task priority when the task is submitted, and keeping the priority unchanged all the time during scheduling. Dynamic priority scheduling, namely adjusting the priority in real time according to the task response condition, time limit or system load, wherein the preemptive scheduling is that the high-priority task can interrupt the execution of the low-priority task. Non-preemptive scheduling, in which the executed task is not interrupted, but the high priority task preferentially allocates resources. The offset SRPD is calculated by SRPD =tactual-Texpected;Tactual, which is the actual response time of the task, and Texpected is the expected response time of the task.
The resources are dynamically adjusted according to the priority allocation proportion, for example, 60% of resources with high priority, 30% of resources with medium priority and 10% of resources with low priority.
After analyzing the service response priority deviation condition, generating a response priority deviation index, and evaluating the real-time performance of task execution, wherein the response priority deviation index acquiring method comprises the following steps:
The actual response time is set to be Tactual,i, which represents the actual completion time of the task, and the expected response time is set to be Texpected,i, which represents the target time for the task to complete within the prescribed time. Setting a service response offset value SRPDi, calculating an offset value according to the task response time, wherein the calculation expression is SRPDi＝Tactual,i-Texpected,i, and the higher the priority is, the larger the task weight is, so that the influence degree of the task weight on the overall performance of the system is reflected. The weight can be manually set according to the specific scene, and a dynamic adjustment strategy can also be adopted. The allocation rule of the priority P weight W is set to be P=3 (high priority), w=0.6, P=2 (medium priority), w=0.3, P=1 (low priority), w=0.1, and the weight sum Wtotal of each task is calculated according to the task priority and the corresponding offset value, wherein the expression is as follows: Wherein wi is task priority weight, reflects importance of tasks, n is total number of tasks, and the weighted offset values of all tasks are summed and divided by the weight sum to obtain a response priority offset index, and the expression is: Where SRPDI is the response priority offset index.
When the response priority shift index (SRPDI) is larger, it is indicated that the actual response time of the task in the system generally deviates from the expected time more, and especially the response delay of the high-priority task is significantly increased. This typically means that tasks are performed in poor real-time, possibly due to network congestion, resource maldistribution or imperfect task scheduling mechanisms. The larger offset index may also cause the overall efficiency of the system to decrease, especially in the scenario where the critical task occupies a relatively high area, the real-time problem may directly affect the stability and quality of service of task execution.
When the response priority shift index (SRPDI) is smaller, the actual response time of the task in the system is closer to the expected time, and particularly, the high-priority task is processed in time. This indicates that the real-time performance of task execution is better, and the system scheduling mechanism can effectively cope with multi-task concurrence and resource competition. Meanwhile, the smaller deviation index reflects the rationality of the resource allocation strategy, the delay of the high-priority task is less, the delay of the low-priority task is also in a controllable range, and the high-efficiency and stability of the system operation are maintained.
The cloud processing and storage module comprises a distributed storage system and a high-performance computing platform and is used for processing and storing the uploaded dynamic congestion degree data and response priority offset data.
And the distributed storage system is used for storing dynamic congestion degree data, response priority offset data and other related metadata. High availability, scalability and persistence of data are ensured. And storing multiple copies of each piece of data in different nodes to prevent single-point faults. Distributed file systems, such as HDFS (Hadoop Distributed FILE SYSTEM), ceph, are used to manage mass data. Key value stores such as APACHE CASSANDRA for quick storage and querying of dynamic index data (e.g., DCL and SRPD). And (3) layering and storing cold and hot data, namely storing DCL and SRPD data required by real-time analysis in a high-performance storage, and storing historical data in a low-cost storage device.
And a high-performance computing platform for processing the uploaded data in real time and computing new dynamic indexes (such as comprehensive DCL and SRPDI). Large-scale batch analysis (e.g., task delay trend prediction) is performed.
Streaming computing frameworks such as APACHE FLINK, APACHE KAFKA STREAMS are used to compute in real time the global average of dynamic congestion levels or the delay profile of high priority tasks. Parallel computing engines such as APACHE SPARK are used for batch analysis of large-scale task response offset data. And (3) containerized deployment, namely realizing dynamic expansion by using Kubernetes, and meeting the concurrent processing requirement.
DCL and SRPD data streams uploaded from the network resource management module and the priority scheduling module are received through message queues (e.g., APACHE KAFKA). Duplicate, erroneous or abnormal data is removed, ensuring data quality. The global DCL distribution is calculated and high congestion paths are identified. And counting SRPDI, and evaluating the real-time performance of the overall task of the system. The calculation result is temporarily stored in a high-performance memory database (such as Redis) for other modules to call in real time.
And analyzing the long-term delay trend of the high-priority task by combining the historical data, and providing a basis for optimizing the resource allocation strategy. Index aggregation, namely calculating average DCL and SRPDI of each path for evaluating the overall performance of the system. The latest DCL and SRPD data are written to a distributed database (e.g., APACHE CASSANDRA or MongoDB). Historical data is transferred to low cost storage (e.g., amazon S3 or Google Cloud Storage) for long term analysis and model training.
And the comprehensive analysis module is used for comprehensively analyzing the dynamic congestion degree and the service response priority deviation to determine the influence severity of the network congestion on the task response delay.
Converting the dynamic congestion degree and the response priority shift index into comprehensive feature vectors, taking the comprehensive feature vectors as input of a machine learning model, taking the influence severity value labels of each group of comprehensive feature vectors on the task response delay of the predicted network congestion of the machine learning model as a prediction target, taking the sum of prediction errors of the influence severity value labels of the minimum network congestion on the task response delay of the predicted network congestion as a training target, training the machine learning model until the sum of the prediction errors reaches convergence, stopping model training, and determining the influence severity value of the network congestion on the task response delay according to a model output result, wherein the machine learning model is a polynomial regression model.
The method for acquiring the severity value of the influence of the network congestion on the task response delay comprises the steps of acquiring a corresponding function expression LQ=F (DCL, SRPDI) from comprehensive feature vector training data of a machine learning model which is trained, wherein F is an output function of the model, DCL is dynamic congestion degree, SRPDI is response priority offset index, and LQ is the severity value of the influence of the network congestion on the task response delay.
And the optimizing module is used for immediately switching the transmission of the high-priority task to the low DCL path if the severity is high, preferentially inserting the high-priority task into the task scheduling queue, and carrying out post-processing on the low-priority task, and reducing the data transmission rate of the low-priority task and improving the sequencing position of the high-priority task in the queue if the severity is low.
Comparing the obtained influence severity value of the network congestion on the task response delay with a set influence severity reference threshold value according to historical data, if the influence severity value of the network congestion on the task response delay is greater than or equal to the set influence severity reference threshold value, indicating that the influence severity of the network congestion on the task response delay is high, generating an early warning signal at the moment, immediately switching the transmission of a high-priority task to a low DCL path, and preferentially inserting the high-priority task into a task scheduling queue, and if the influence severity value of the network congestion on the task response delay is smaller than the set influence severity reference threshold value, indicating that the influence severity of the network congestion on the task response delay is low, not generating the early warning signal at the moment, reducing the data transmission rate of the low-priority task and improving the ordering position of the high-priority task in the queue.
Switching the transmission of the high-priority task to the low DCL path, and preferentially inserting the high-priority task into the task scheduling queue, specifically:
Switching the transmission path of the high-priority task to the path with the lowest dynamic congestion degree DCL to ensure that the transmission delay is minimized, wherein the formula is as follows: Where Popt is the optimal transmission path, P is the set of all available paths, and DCL (P) is the dynamic congestion level of path P.
Collecting DCL (P) values of all paths in the current path set P according to a common ruleThe path Popt with the lowest dynamic congestion degree is selected, and the data flow of the high-priority task Thigh is transferred to the path Popt:
Inserting a high-priority task Thigh into a priority position of a task scheduling queue Q to ensure execution instantaneity of the high-priority task, wherein the formula is that Q=insert (Q, Thigh, priority), Q is the task scheduling queue, Thigh is the high-priority task to be inserted, the priority is a priority attribute and used for sequencing adjustment, and a queue sequencing rule is that Sort (Q) =argmaxT∈Q priority (T), wherein the priority (T) is a priority value of the task T, the higher the value is, the higher the task priority is, the priority weight priority (T) is defined, and the priority weight is set for each task T: whigh>wlow indicates that the weight of the high-priority task is greater than that of the low-priority task, whigh、wlow indicates that the weights of the high-priority task and the low-priority task are respectively inserted into the priority positions in the queue according to the task priority, and the task queue Q is ordered from high to low according to the priority value.
The method comprises the steps of reducing the data transmission rate of low-priority tasks and improving the ordering positions of high-priority tasks in a queue, and specifically comprises the following steps:
The data transmission rate Rlow of the low-priority task is adjusted, more bandwidth resources are reserved for the high-priority task, the instantaneity of the high-priority task is guaranteed, and the new transmission rate of the low-priority task is guaranteed: in the formula,For an adjusted low priority task data transmission rate,For low priority task data transmission rates prior to adjustment, α is a rate reduction coefficient, and the value range 0< α <1, α=0.5 represents halving the transmission rate. And dynamically adjusting the transmission rate of the low-priority task according to a calculation formula. By reordering the task scheduling queue Q, moving the high priority task Thigh to a forward position, computing resources and execution time are preferentially allocated, scheduling queue reordering rules Sort (Q) =argmaxT∈Q priority (T), task priority definition function: And defining the priority weight whigh、wmid、wlow of each task, sequencing the task queue Q according to the priority value, leading the high-priority task to be in front, and generating a new task scheduling queue.
In the embodiment, a data acquisition module is used for collecting original data from various distributed sensors in real time and providing basic data support for task execution, a network resource management module is used for calculating dynamic congestion Degree (DCL) and adjusting a transmission path in real time through monitoring a data transmission path to optimize network resource allocation, a priority sensing task scheduling module is used for dynamically adjusting response priority according to importance of the tasks and evaluating instantaneity of the tasks through calculating service response priority offset (SRPDI), a cloud processing and storage module is used for processing and storing uploaded dynamic congestion degree data and response priority offset data by combining a distributed storage system and a high-performance computing platform, reliability and high efficiency of the data are guaranteed, a comprehensive analysis module is used for comprehensively analyzing the dynamic congestion degree and the service response priority offset to determine influence severity of network congestion on task response delay, an optimization module is used for immediately switching high priority tasks to low DCL paths and inserting the high priority tasks into scheduling queues in advance and delaying the low priority tasks based on analysis results, and reducing transmission rates of the low priority tasks and improving the priority tasks and efficiency of the high priority tasks if the severity is low, and optimizing the dynamic resource allocation of the system is improved.
The above formulas are all formulas with dimensions removed and numerical values calculated, the formulas are formulas with a large amount of data collected for software simulation to obtain the latest real situation, and preset parameters in the formulas are set by those skilled in the art according to the actual situation.
The above embodiments may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with embodiments of the present application are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more sets of available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium may be a solid state disk.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application.