Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Examples
A first embodiment of the present application relates to a data storage and index calculation method. As shown in fig. 1, the method is performed by a management tool, and the method may include the steps of:
Step S101, discrete data are acquired;
the discrete data can be any data which needs to be stored and calculated by indexes. The data is discrete and may be, in particular, some data generated during the operation of a computer or during the transmission of network data.
In one embodiment, the discrete data is an interface response time.
Interface response time refers to the time elapsed from the client sending a request to the server returning a response.
The short response time of the interface can enable the user to feel that the system reacts rapidly, and the user satisfaction is improved. If the response time is too long, the user may feel restlessness and even forgo using the system. For applications with high real-time requirements, such as online gaming, financial transactions, etc., response time is critical and excessive response time can lead to serious consequences.
Interface response time is one of the important indicators for measuring system performance. By monitoring the response time of the interface, the performance of the system under different loads can be known, and the performance bottleneck can be found in time and optimized. Which helps to assess the scalability and stability of the system. As the number of users and traffic increases, the change in response time may reflect whether the system is able to withstand greater pressures.
According to the scheme, the interface response time is stored and counted in a new mode, so that the storage and processing efficiency of the interface response time can be improved, and data support is provided for more comprehensive and fine evaluation of the interface response time.
Step S102, storing the discrete data by using a first TreeMap and a second TreeMap, wherein the key value information of the first TreeMap is an index and the occurrence times of the indexed discrete data respectively, and the key value information of the second TreeMap is the maximum value of the index and the indexed discrete data respectively;
according to the scheme, interface response time data are stored in a low-cost mode, and TP value index calculation can be conveniently invoked from interfaces corresponding to different scenes.
Assuming that the interface response time follows a normal distribution, the total call number is 10W in 5 minutes, the mean μ (average response time) is 500ms, and the standard deviation σ is 125, the normal distribution diagram shown in fig. 2 is satisfied.
For this part of data, it is stored in three schemes, respectively including:
and 1, storing the interface response time in a certain time according to the sequence from big to small.
And 2, storing the interface response time in a certain time by using a data structure of the TreeMap. The interface response time within a certain time is stored by using a data structure of the TreeMap, the interface response time is used as a key, the number of times of repetition of the interface time is used as a value to perform full-scale storage, and the scheme is an optimized version of the scheme 1, but for a scene (for example, one hundred requests, the response time of 100 times is 1, 2 and 3.
Scheme 3. The technical scheme provided by the embodiment of the application.
Fig. 3 is a schematic diagram of the respective interval duty ratios in the case of standard normal distribution according to some embodiments of the present application. As shown in fig. 3, it is assumed that the interface response time follows a normal distributionThe total call count was 10W, the mean μ (average response time) was 500ms, and the standard deviation σ was 125 in 5 minutes.
Fig. 4 is a schematic diagram of another interval duty ratio in the case of standard normal distribution according to some embodiments of the present application. As shown in fig. 4, the simulation shown calculates the number of response times in each section:
10W 0.1% 2 about 200 for intervals of 0-125 and 875-1000;
125-250 and 750-875 intervals of 10W 2.1% 2 are about 4200;
intervals of 250-375 and 625-750 are 10W 13.6% 2 approximately equal to 27200;
Intervals 375 to 500 and 500 to 625 are 10W 34.1% 2 approximately equal to 68200;
the data size/b of the stored data of three schemes are calculated roughly (calculated by using a program simulation random number) from the data, and bytes:
0-125 and 875-1000 intervals, 200 response times;
scheme 1 occupies a byte number approximately equal to 913b;
scheme 2 occupies a byte number of approximately 894b;
the scheme of the application occupies the byte number approximately equal to 90b;
125-250 and 750-875 intervals, 4200 response times;
scheme 1 occupies a byte number of approximately 21000b;
Scheme 2 occupies a byte number of approximately 1997b;
The scheme of the application occupies the byte number approximately equal to 44b;
250-375 and 625-750 intervals, 27200 response times;
scheme 1 occupies bytes ≡ 136000b;
scheme 2 occupies a byte number of approximately 2205b;
The scheme of the application occupies the byte number approximately equal to 50b;
375-500 and 500-625 intervals, 68200 response times;
scheme 1 occupies bytes ≡ 341000b;
Scheme 2 occupies a byte number of 2250b;
the scheme of the application occupies the byte number approximately equal to 36b;
The arrangement statistical table is as follows (LL normal distribution diagram left section minimum value, LR normal distribution diagram left section maximum value, RL normal distribution diagram right section minimum value, RR normal distribution diagram right section maximum value)
Statistical scheme 1 results form
Statistical scheme 2 results form
Statistics of the results form of the inventive protocol
Summarizing byte sums of the three schemes;
scheme 1 total number of bytes 913+21000+136000+341000= 498913b;
scheme 2 total number of bytes 894+1997+2205+2250=7346 b;
The total number of bytes of the scheme is 90+44+50+36=220b;
from the simulated normal distribution data, the scheme 1 obviously has the serious condition of resource occupation, and 400+kb is required for only 10W interface calls, which is far higher than that of the scheme 2 and the scheme of the application.
The scheme of the application belongs to an optimized resource occupation scheme as scheme 2, but the resource occupation of the scheme of the application is obviously more advantageous.
While the effect of scheme 2 in the ideal case is good, in a practical scenario there may be a more extreme scenario, where the resources required for its storage may not be as good as scheme 1, or even higher, if the call time of the API is too thin.
The application designs a thinking of efficient storage and calculation. The method has the innovation points that the strategy for storing TP index data can be appointed according to the interface calling frequency in the actual scene as required, and the TP index approximation value is calculated efficiently.
In terms of data storage, two treemaps are used to store data.
The first TreeMap and key store the interface response time through the formulaThe calculated "index" value, value is the number of times the "index" value appears.
The second TreeMap and key store the interface response time through the formulaThe calculated "index" value, value is the maximum interface response time of the "index" value.
In this scheme, specifically, storing the discrete data using the first TreeMap and the second TreeMap includes:
Calculating the discrete data by using a logarithmic value, and rounding downwards to obtain indexes of a first TreeMap and a second TreeMap;
And accumulating each index to obtain the discrete data occurrence number of each index.
For the design of the first TreeMap, a data structure store of TreeMap may be used.
Wherein the key isIs referred to as an index.
The base x is a natural number not less than 2. The base directly affects the hash density of the actual stored data. Fig. 5 is a hash view of an index and the number of occurrences of the index with a response time of 2, and fig. 6 is a hash view of an index and the number of occurrences of the index with a response time of 3.
In this solution, specifically, the method further includes:
And determining the value of the bottom of the logarithmic value according to the quantity of the discrete data and the calculation accuracy requirement of the index data.
The smaller the value of x, the smaller the amount of data that can be carried, but the higher the accuracy of the calculated TP index approximation, fig. 7 is a trend chart of the base size and the TP index calculation accuracy.
The larger the value of x, the larger the amount of data that can be carried, but the calculated TP index approximation is relatively less accurate, fig. 8 is a trend plot of base size versus amount of data that can be carried.
Note that byThe logarithm calculated, i.e. the index, follows the "round down" principle. An example of a 2-based index is as follows.
Rounding down:≈2.32193=2;
≈3.16993=3;
≈6.16993=6;
and so on.
And y is the actual response time of the interface, which is in ms.
The value is the number of times of occurrence of the index corresponding to the response time, defaults to 0, and is accumulated sequentially along with the increase of the number of times of occurrence of the index.
Design for the second TreeMap:
data structure storage using TreeMap.
Wherein key is the same as the concept in the first TreeMap, meaning index. value is the maximum response time that occurs in the index. Fig. 9 is a hashed view of the second TreeMap.
Calculate (interface response time data generated in one minute) with a bottom TP value of 2:
1、3、7、15、29、30、32、48、56、64、77、88、99、100、120、130;
Index value of 1:=0;
Index value of 3:≈1.58496=1;
index value of 7:≈2.80735=2;
Index value of 15:≈3.90689=3;
index value of 29:≈4.85798=4;
......
Index value of 130:≈7.02237=7;
Fig. 10 is a hashed view of the first statistically derived TreeMap. The storage is {0:1, 1:1, 2:1, 3:1, 4:2, 5:3, 6:6, 7:1};
Fig. 11 is a hashed view of the statistically derived second TreeMap. The storage is {0:1, 1:3, 2:7, 3:15, 4:30, 5:56, 6:120, 7:130}.
Therefore, the scheme can greatly reduce the space for data storage through the arrangement.
Step S103, index data are obtained;
the index data may be TP99, TP999, etc., where TP99 is the value of the discrete data from small to large 99% among the discrete data, and TP999 is the value of the discrete data from small to large 99.9% among the discrete data.
Step S104, identifying the corresponding index of the index data according to the first TreeMap, and calculating the discrete data value of the index data according to the second TreeMap.
In this scheme, the discrete data value corresponding to the index data may be calculated based on the data recorded by the first TreeMap and the second TreeMap.
The method comprises the steps of obtaining discrete data, storing the discrete data by using a first TreeMap and a second TreeMap, wherein key value information of the first TreeMap is an index and the occurrence times of the indexed discrete data respectively, key value information of the second TreeMap is the maximum value of the index and the indexed discrete data respectively, obtaining index data, identifying the corresponding index of the index data according to the first TreeMap, and calculating the discrete data value of the index data according to the second TreeMap. By adopting the scheme, the first TreeMap and the second TreeMap are used for storing data, and the specific data value of the required data index can be efficiently calculated based on the stored data, so that the space consumption caused by data storage is reduced, the data calculation amount is reduced, and the calculation efficiency is improved.
In one embodiment, optionally, identifying the corresponding index of the index data according to the first TreeMap includes:
determining a target sequence value of the index data according to the quantity of the discrete data and the index data;
And determining the corresponding index of the target sequence value according to the occurrence times of the discrete data of each index in the first TreeMap.
The target sequence value of the index data may be a value of a position arranged from small to large, for example, TP99 is the 1980 th discrete data arranged from small to large among 2000 data.
After determining the target order value, it is possible to determine in which index the discrete data value corresponding to the index data is, based on the number of each index, and use it as the corresponding index.
In one embodiment, optionally, calculating the discrete data value of the index data according to the second TreeMap includes:
calculating a left critical value of a corresponding index of the target sequence value;
And calculating the discrete data value of the index data according to the left critical value and the maximum value of the discrete data of the corresponding index recorded by the second TreeMap.
After the corresponding index is defined, the discrete data value of the index data can be calculated according to the left critical value of the corresponding index and the maximum value of the discrete data of the corresponding index recorded by the second TreeMap.
For example, the target index has 4 discrete data, the left critical value is 8, and the maximum value is 11, the discrete data value corresponding to the index data can be estimated according to the two values, for example, the index data is the third value in the index, and the discrete data value of the index data can be determined to be 10.
In one embodiment, optionally, calculating the discrete data value of the index data according to the left critical value and the maximum value of the discrete data of the corresponding index of the second TreeMap record includes:
Assuming that in the corresponding index, between the left critical value and the maximum value of the discrete data of the corresponding index, each discrete data is uniformly distributed;
discrete data values of the index data are calculated according to the order of the index data in the corresponding index.
In this scheme, in calculating the discrete data value, it is assumed that, in the corresponding index, between the left critical value and the maximum value of the discrete data of the corresponding index, each discrete data is uniformly distributed.
The following describes the TP value calculation in connection with an actual scenario:
Fig. 12 is a hashed view of the first TreeMap of the simulated 2000 interface response times. The data store is {0:100, 1:550, 2:450, 3:600, 4:100, 5:80, 6:60, 7:45, 8:15};
fig. 13 is a hashed view of a second TreeMap of the simulated 2000 interface response times. The data store is {0:1, 1:3, 2:7, 3:14, 4:30, 5:62, 6:126, 7:238, 8:509}.
TP99 calculation:
Equation calculation variable definition:
Total number of calls X, given example 2000, i.e. x=2000;
TP index component B=0.99 (TP 99, B is 0.99; TP999, B is 0.999);
the position of the TP index value in the X response times is v=math.floor (x×b), note that math.floor is a function of the downward rounding, such as math.floor (1.9) =1, math.floor (21.2) =21, math.floor (100.98) =100.
The index value of TP index value (the value of key in the first and second TreeMap) N;
The index position of the TP index value in N is P;
The response time quantity of TP index value in N is C;
the maximum response time of the second TreeMap in N is M;
left critical response time of second TreeMap in N: z=;
Calculating TP index value R by final evaluation;
formula definition:
V=Math.floor(X*B);
substituting the formula can obtain V:
V=Math.floor(2000*0.99)=1980;
V=1980;
Available in combination with the first TreeMap in fig. 12;
N=7;
R=Z+(M-Z)*P/C;
The variable values obtained by substituting the data of fig. 12 and 13 into the formula:
Z===128;
M=238;
P=40;
C=45;
R=128+(238-128)*40/45=225.7777ms;
TP999 calculation:
Equation calculation variable definition:
Total number of calls X, given example 2000, i.e. x=2000;
TP index component B=0.999 (TP 99, B is 0.99; TP999, B is 0.999);
the position of the TP index value in the X response times is v=math.floor (x×b), note that math.floor is a function of the downward rounding, such as math.floor (1.9) =1, math.floor (21.2) =21, math.floor (100.98) =100.
Index value of TP index is N;
The index position of the TP index value in N is P;
The response time times of TP index value in N is C;
the maximum response time of the second TreeMap in N is M;
left critical response time of second TreeMap in N: z=;
TP index value is R;
formula definition:
V=Math.floor(X*B);
substituting the formula can obtain V:
V=math. Floor (2000 x 0.999) =1998, i.e. v=1998,
Available in combination with the first TreeMap in FIG. 12
TP index value n=8;
R=Z+(M-Z)*P/C;
Substituting the data of fig. 12 and 13 into a formula to obtain a variable value;
Z===256;
M=509;
P=12;
C=15;
R=256+(509-256)*12/15≈202.4ms;
The application can deal with the calculation of the interface TP indexes of different pressure scenes of different enterprises by providing the calculation formula of the TP indexes and the designed TP index storage structure.
The application can freely designate the conforming base number according to the interface magnitude of enterprises, maximally reduces the storage cost brought by TP data, and simultaneously greatly improves the calculation performance of TP index values.
The application can greatly improve the capability of business personnel for quickly integrating TP indexes and greatly improve the development efficiency of business in the aspect.
In addition, some embodiments of the application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and the like. The electronic device may also be various forms of mobile apparatus, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing apparatus.
The electronic device comprises one or more processors and a memory storing computer program instructions that, when executed, cause the processors to perform the steps of a method as provided in any one or more of the embodiments described above. Fig. 14 discloses an exemplary structural diagram of the electronic device. As shown in fig. 14, the electronic device includes one or more processors 1401, memory 1402, and interfaces for connecting the components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In some other embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple electronic devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). Wherein the components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the applications described and/or claimed herein.
The electronic device may further comprise input means 1403 and output means 1404. The processor 1401, memory 1402, input device 1403, and output device 1404 may be connected by a bus or otherwise, for example in fig. 14.
Input device 1403 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointer stick, one or more mouse buttons, a trackball, a joystick, and the like. The output devices 1404 may include a display device, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.
To provide for interaction with a user, the electronic device may be a computer. The computer has a display device (e.g., a Cathode Ray Tube (CRT) or LCD monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form (including acoustic input, speech input, or tactile input).
In an embodiment of the present application, a computer readable medium has stored thereon a computer program/instruction which, when executed by a processor, implements the steps of the method provided by any one or more of the embodiments described above. The computer readable medium may be contained in the electronic device described in the above embodiment or may exist alone without being incorporated in the device. The computer-readable medium carries one or more computer-readable instructions.
Memory 1402 may be used as a non-transitory computer readable storage medium storing non-transitory software programs, non-transitory computer executable programs, and modules. The processor 1401 executes various functional applications of the server and data processing by running non-transitory software programs, instructions and modules stored in the memory 1402 to implement program instructions/modules corresponding to the methods provided by any one or more of the embodiments of the present application.
Memory 1402 may include a storage program area that may store an operating system, application programs required for at least one function, and a storage data area that may store data created according to the use of the electronic device, etc. Further, memory 1402 can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 1402 optionally includes memory remotely located relative to processor 1401, which may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The computer readable medium according to the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access Memory (RAM, random Access Memory), a Read-Only Memory (ROM), an erasable programmable Read-Only Memory (EPROM, erasable Programmable Read-Only Memory, or flash Memory), an optical fiber, a portable compact disc Read-Only Memory (CD-ROM, compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
Computer-readable media include both permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase-Change Memory (PRAM), static Random-Access Memory (SRAM), dynamic Random-Access Memory (DRAM), other types of Random-Access Memory (RAM), read-Only Memory (ROM), electrically erasable programmable read-Only Memory (EEPROM, ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory), flash Memory or other Memory technology, compact disc read-Only Memory (CD-ROM), digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device.
Computer program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN, local Area Network) or a wide area network (WAN, wide Area Network), or may be connected to an external computer (e.g., through the internet using an internet service provider).
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. For example, it may be implemented using an Application-specific integrated Circuit (ASIC), a general purpose computer, or any other similar hardware device. In some embodiments, the software program of the present application may be executed by a processor to implement the above steps or functions. Likewise, the software programs of the present application (including associated data structures) may be stored on a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. In addition, some steps or functions of the present application may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions.
Embodiments of the present application provide a computer program product comprising one or more computer programs/instructions which, when executed by a processor, produce, in whole or in part, a process or function in accordance with embodiments of the present application. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL, digital Subscriber Line), or wireless (e.g., infrared, wireless, microwave, etc.) means, the computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server, data center, etc., that contains an integration of one or more available media, the available media may be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid state disk, SSD, solid STATE DISK), etc.
The flowchart or block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The scope of the application is indicated by the appended claims rather than by the foregoing description, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the apparatus claims can also be implemented by means of one unit or means in software or hardware. The words "first," "second," and the like are used merely to distinguish between descriptions and do not indicate any particular order, nor are they to be construed as indicating or implying relative importance.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art can easily mention variations or alternatives within the scope of the present application. The present application is therefore to be considered in all respects as illustrative and not restrictive, and the scope of the application is indicated by the appended claims.