Disclosure of Invention
The embodiment of the invention provides a file collection method, which is used for reducing the workload of research personnel and reducing the operation cost and comprises the following steps:
performing custom processing on the acquired data source information;
formulating a file acquisition rule for acquiring various types of file information according to a user-defined processing result of the data source information;
and acquiring the information of the file to be processed according to the file acquisition rule.
Optionally, the method further includes:
and judging whether to carry out shunting processing on the file information to be processed according to the file acquisition rule, and if so, carrying out shunting processing on the file information to be processed according to a preset shunting processing rule.
Optionally, the method further includes:
configuring storage directories corresponding to the multiple storage types according to configuration rules;
and storing the file information to be processed into the configured storage directory.
Optionally, the method further includes:
and clearing the file information in the configured storage directory according to a preset time interval.
The embodiment of the invention also provides a file collecting device, which is used for reducing the workload of research personnel and reducing the operation cost, and comprises:
the user-defined module is used for performing user-defined processing on the acquired data source information;
the rule making module is used for making a file acquisition rule for acquiring various types of file information according to a user-defined processing result of the data source information;
and the file acquisition module is used for acquiring the information of the file to be processed according to the file acquisition rule.
Optionally, the apparatus further comprises:
and the shunting processing module is used for judging whether to carry out shunting processing on the file information to be processed according to the file acquisition rule, and if so, carrying out shunting processing on the file information to be processed according to a preset shunting processing rule.
Optionally, the apparatus further comprises:
the configuration module is used for configuring the storage directories corresponding to the multiple storage types according to the configuration rule;
and the storage module is used for storing the file information to be processed into the configured storage directory.
Optionally, the apparatus further comprises:
and the cleaning module is used for cleaning the file information in the configured storage directory according to a preset time interval.
The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method when executing the computer program.
An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program for executing the above method is stored.
In the embodiment of the invention, the acquired data source information is subjected to custom processing, the file acquisition rule for acquiring various types of file information is formulated according to the custom processing result of the data source information, and the acquisition of the file information to be processed can be completed according to the file acquisition rule without research and development of various acquisition logics by research and development personnel, so that the workload of the research and development personnel is reduced, and the operation cost is reduced.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
Fig. 1 shows a flowchart of a file collection method according to an embodiment of the present invention, and as shown in fig. 1, the method includes:
step 101, performing custom processing on the acquired data source information.
In this embodiment, the data source information includes: parameters such as server IP address, port, user, password, transmission mode and the like, and the self-defining processing of the data source information refers to: the various parameters described above are stored in a configuration file that can be modified, for example. The data source information includes: whether to delete the source file, yes or no; a user name xxxxx; the password is xxxxxx; xxx.xxx.xxx.xxx; a server port: xx. And if the source file is deleted, modifying the parameters corresponding to the user name, the password, the server address and the server port.
And 102, formulating a file acquisition rule for acquiring various types of file information according to the self-defined processing result of the data source information.
In this embodiment, the file collection rule includes:
remote directories supporting the configuration of a corresponding directory per each file, for example, type 1: remote path 1; type 2: remote path 2.
Local directories, which support configuring locally stored directories per file, for example, type 1: a local path 1; type 2: local path 2.
File name matching regularization: and configuring a regular expression of the file name to be acquired, and once the regular expression is matched with the file name, indicating that the file meets the acquisition requirement.
Multi-device acquisition: one server supports simultaneous acquisition of a plurality of network element devices, configures network element variables in a configuration directory, and then replaces names of the network element devices to be acquired one by one during acquisition to realize acquisition of data files of the plurality of devices at one time.
Breakpoint rules: the rule is mainly used for recording the time point of each acquisition, and the data can be continuously acquired from the time point when the data are acquired in the next period, so that the data cannot be repeated, and the effect of incremental acquisition is achieved.
Breakpoint query command: the rule is used for inquiring the latest files needing to be collected, all the files do not need to be listed, and the collection efficiency can be improved.
And (3) breakpoint file configuration: the rule is used for configuring the storage position of the breakpoint file, and the definable configuration is beneficial to the centralized management of the file.
And 103, acquiring information of the file to be processed according to the file acquisition rule.
As can be seen from fig. 1, in the file collection method provided in the embodiment of the present invention, the obtained data source information is subjected to the customized processing, the file collection rule for collecting various types of file information is formulated according to the customized processing result of the data source information, and the collection of the file information to be processed can be completed according to the file collection rule, so that research and development personnel do not need to research and develop various collection logics, the workload of the research and development personnel is reduced, and the operation cost is reduced.
Fig. 2 shows another flowchart of the file collection method provided in the embodiment of the present invention, which can process files simultaneously in a scenario of mass data to meet the requirement of service expansion, and as shown in fig. 2, the method further includes:
step 201, judging whether to perform distribution processing on the file information to be processed according to the file acquisition rule, and if so, performing distribution processing on the file information to be processed according to a preset distribution processing rule.
In specific implementation, whether to perform the splitting processing on the file information to be processed is determined according to the "local directory" information in the file acquisition rule instep 102, for example, if 3 local directories are configured, the acquired files are evenly distributed to the 3 directories in sequence.
Fig. 3 is a flowchart illustrating a file information storage process in an embodiment of the present invention, and as shown in fig. 3, the method further includes:
step 301, configuring storage directories corresponding to multiple storage types according to configuration rules.
In this embodiment, the configuration rule is: after a plurality of local peer directories are configured, a module is taken from a file, and the file is evenly distributed to each shunting directory to achieve data balance. Wherein, the module in the module taking is the directory position for storing the file. The storage directory corresponding to the storage type may be: local directory, temporary directory, backup directory, etc.
In specific implementation, the data balance can be realized by modulus taking according to the file sequence number under the condition of mass data files, after shunting, subsequent processing can be simultaneously carried out by a plurality of servers, so that the capability of transverse expansion is realized, and when the processing capability reaches the bottleneck, the service requirement can be well met by the shunting processing.
Step 302, storing the file information to be processed in the configured storage directory.
In specific implementation, before the file is stored in the local directory, a temporary file is generated, stored in the temporary directory and downloaded, and named as a formal file. After the file is stored in the local directory, whether the file needs to be compressed or not can be judged according to the size of the file, if compression is carried out, the file needs to be decompressed subsequently, when decompression is carried out, if the performance of the server is good, the file can be decompressed in batches, and if the performance of the server is poor, the file can be decompressed one by one, so that the peak staggering effect is achieved, and all files can be smoothly decompressed.
Fig. 4 is a flowchart illustrating a file information cleaning process in an embodiment of the present invention, and as shown in fig. 4, the method further includes:
step 401, clearing file information in the configured storage directory according to a preset time interval.
In this embodiment, the "time interval" may be year, month, day, etc., that is, a time variable is configured in the local directory, and in the specific implementation, the time interval is 1 day, for example, a day granularity variable is configured in the local directory, so that the fetched files are stored by day, after the fetching is finished, the data clearing command is executed, and according to the configured time, the expired data is deleted, so as to reduce the loss of the storage space of the server.
Based on the same inventive concept, the embodiment of the invention also provides a file collecting device, as described in the following embodiments. Because the principle of solving the problems of the file acquisition device is similar to that of the file acquisition method, the implementation of the file acquisition device can refer to the implementation of the file acquisition method, and repeated parts are not described again. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 5 is a schematic structural diagram of a document collection apparatus according to an embodiment of the present invention, and as shown in fig. 5, the apparatus includes:
and thecustom module 501 is configured to perform custom processing on the obtained data source information.
Therule making module 502 is configured to make a file collection rule for collecting multiple types of file information according to a user-defined processing result of the data source information.
And thefile acquisition module 503 is configured to acquire information of the file to be processed according to the file acquisition rule.
Fig. 6 is a schematic structural diagram of a file collecting device according to an embodiment of the present invention, and as shown in fig. 6, the device further includes:
the shuntingprocessing module 601 is configured to determine whether to perform shunting processing on the file information to be processed according to the file acquisition rule, and if the determination result is yes, perform shunting processing on the file information to be processed according to a preset shunting processing rule.
Fig. 7 is a schematic diagram illustrating a storage structure of file information in an embodiment of the present invention, and as shown in fig. 7, the apparatus further includes:
aconfiguration module 701, configured to configure storage directories corresponding to multiple storage types according to a configuration rule;
thestorage module 702 is configured to store the file information to be processed in the configured storage directory.
Fig. 8 is a schematic diagram illustrating a structure for clearing file information in an embodiment of the present invention, and as shown in fig. 8, the apparatus further includes:
acleaning module 801, configured to clean file information in the configured storage directory according to a preset time interval.
The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above method when executing the computer program.
An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program for executing the above method is stored.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.