Movatterモバイル変換


[0]ホーム

URL:


CN111125004B - File collection method and device - Google Patents

File collection method and device
Download PDF

Info

Publication number
CN111125004B
CN111125004BCN201911219780.8ACN201911219780ACN111125004BCN 111125004 BCN111125004 BCN 111125004BCN 201911219780 ACN201911219780 ACN 201911219780ACN 111125004 BCN111125004 BCN 111125004B
Authority
CN
China
Prior art keywords
file
acquisition
rule
directory
files
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911219780.8A
Other languages
Chinese (zh)
Other versions
CN111125004A (en
Inventor
黄建平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Unihub China Information Technology Co Ltd
Original Assignee
Unihub China Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Unihub China Information Technology Co LtdfiledCriticalUnihub China Information Technology Co Ltd
Priority to CN201911219780.8ApriorityCriticalpatent/CN111125004B/en
Publication of CN111125004ApublicationCriticalpatent/CN111125004A/en
Application grantedgrantedCritical
Publication of CN111125004BpublicationCriticalpatent/CN111125004B/en
Activelegal-statusCriticalCurrent
Anticipated expirationlegal-statusCritical

Links

Images

Classifications

Landscapes

Abstract

The invention discloses a file collection method and a device, wherein the method comprises the following steps: performing custom processing on the acquired data source information; formulating a file acquisition rule for acquiring various types of file information according to a user-defined processing result of the data source information; and acquiring information of the file to be processed according to the file acquisition rule. According to the invention, the acquired data source information is subjected to custom processing, the file acquisition rule for acquiring various types of file information is formulated according to the custom processing result of the data source information, and the acquisition of the file information to be processed can be completed according to the file acquisition rule, so that research and development personnel are not required to research and develop various acquisition logics, the workload of the research and development personnel is reduced, and the operation cost is reduced.

Description

File collection method and device
Technical Field
The invention relates to the technical field of communication, in particular to a file collection method and device.
Background
In the communication field, various protocols are used for data sharing, wherein one of the more common protocols is a file transfer protocol, and data sharing can be performed by defining interface specifications by two data interaction parties.
In the prior art, a network management system generally serves as a data user and needs to collect files through the file transfer protocol, but the network management system needs to be connected with various types of data in an operation process, file interface information between different systems and different devices is various, different collection logics are needed for different services of collected files, so that the workload of research and development personnel is greatly increased, a large amount of time is needed for researching and developing different collection logics, and the operation cost is increased.
Disclosure of Invention
The embodiment of the invention provides a file collection method, which is used for reducing the workload of research personnel and reducing the operation cost and comprises the following steps:
performing custom processing on the acquired data source information;
formulating a file acquisition rule for acquiring various types of file information according to a user-defined processing result of the data source information;
and acquiring the information of the file to be processed according to the file acquisition rule.
Optionally, the method further includes:
and judging whether to carry out shunting processing on the file information to be processed according to the file acquisition rule, and if so, carrying out shunting processing on the file information to be processed according to a preset shunting processing rule.
Optionally, the method further includes:
configuring storage directories corresponding to the multiple storage types according to configuration rules;
and storing the file information to be processed into the configured storage directory.
Optionally, the method further includes:
and clearing the file information in the configured storage directory according to a preset time interval.
The embodiment of the invention also provides a file collecting device, which is used for reducing the workload of research personnel and reducing the operation cost, and comprises:
the user-defined module is used for performing user-defined processing on the acquired data source information;
the rule making module is used for making a file acquisition rule for acquiring various types of file information according to a user-defined processing result of the data source information;
and the file acquisition module is used for acquiring the information of the file to be processed according to the file acquisition rule.
Optionally, the apparatus further comprises:
and the shunting processing module is used for judging whether to carry out shunting processing on the file information to be processed according to the file acquisition rule, and if so, carrying out shunting processing on the file information to be processed according to a preset shunting processing rule.
Optionally, the apparatus further comprises:
the configuration module is used for configuring the storage directories corresponding to the multiple storage types according to the configuration rule;
and the storage module is used for storing the file information to be processed into the configured storage directory.
Optionally, the apparatus further comprises:
and the cleaning module is used for cleaning the file information in the configured storage directory according to a preset time interval.
The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method when executing the computer program.
An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program for executing the above method is stored.
In the embodiment of the invention, the acquired data source information is subjected to custom processing, the file acquisition rule for acquiring various types of file information is formulated according to the custom processing result of the data source information, and the acquisition of the file information to be processed can be completed according to the file acquisition rule without research and development of various acquisition logics by research and development personnel, so that the workload of the research and development personnel is reduced, and the operation cost is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
FIG. 1 is a flow chart of a file collection method in an embodiment of the invention;
FIG. 2 is a flowchart of a file collection method according to an embodiment of the present invention;
FIG. 3 is a flowchart of a file information storage process according to an embodiment of the present invention;
FIG. 4 is a flowchart of a file information cleaning process according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a file collection apparatus according to an embodiment of the present disclosure;
FIG. 6 is a schematic structural diagram of a file collection apparatus according to an embodiment of the present disclosure;
FIG. 7 is a diagram illustrating a storage structure of file information according to an embodiment of the present invention;
fig. 8 is a schematic diagram of a structure for clearing file information in the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
Fig. 1 shows a flowchart of a file collection method according to an embodiment of the present invention, and as shown in fig. 1, the method includes:
step 101, performing custom processing on the acquired data source information.
In this embodiment, the data source information includes: parameters such as server IP address, port, user, password, transmission mode and the like, and the self-defining processing of the data source information refers to: the various parameters described above are stored in a configuration file that can be modified, for example. The data source information includes: whether to delete the source file, yes or no; a user name xxxxx; the password is xxxxxx; xxx.xxx.xxx.xxx; a server port: xx. And if the source file is deleted, modifying the parameters corresponding to the user name, the password, the server address and the server port.
And 102, formulating a file acquisition rule for acquiring various types of file information according to the self-defined processing result of the data source information.
In this embodiment, the file collection rule includes:
remote directories supporting the configuration of a corresponding directory per each file, for example, type 1: remote path 1; type 2: remote path 2.
Local directories, which support configuring locally stored directories per file, for example, type 1: a local path 1; type 2: local path 2.
File name matching regularization: and configuring a regular expression of the file name to be acquired, and once the regular expression is matched with the file name, indicating that the file meets the acquisition requirement.
Multi-device acquisition: one server supports simultaneous acquisition of a plurality of network element devices, configures network element variables in a configuration directory, and then replaces names of the network element devices to be acquired one by one during acquisition to realize acquisition of data files of the plurality of devices at one time.
Breakpoint rules: the rule is mainly used for recording the time point of each acquisition, and the data can be continuously acquired from the time point when the data are acquired in the next period, so that the data cannot be repeated, and the effect of incremental acquisition is achieved.
Breakpoint query command: the rule is used for inquiring the latest files needing to be collected, all the files do not need to be listed, and the collection efficiency can be improved.
And (3) breakpoint file configuration: the rule is used for configuring the storage position of the breakpoint file, and the definable configuration is beneficial to the centralized management of the file.
And 103, acquiring information of the file to be processed according to the file acquisition rule.
As can be seen from fig. 1, in the file collection method provided in the embodiment of the present invention, the obtained data source information is subjected to the customized processing, the file collection rule for collecting various types of file information is formulated according to the customized processing result of the data source information, and the collection of the file information to be processed can be completed according to the file collection rule, so that research and development personnel do not need to research and develop various collection logics, the workload of the research and development personnel is reduced, and the operation cost is reduced.
Fig. 2 shows another flowchart of the file collection method provided in the embodiment of the present invention, which can process files simultaneously in a scenario of mass data to meet the requirement of service expansion, and as shown in fig. 2, the method further includes:
step 201, judging whether to perform distribution processing on the file information to be processed according to the file acquisition rule, and if so, performing distribution processing on the file information to be processed according to a preset distribution processing rule.
In specific implementation, whether to perform the splitting processing on the file information to be processed is determined according to the "local directory" information in the file acquisition rule instep 102, for example, if 3 local directories are configured, the acquired files are evenly distributed to the 3 directories in sequence.
Fig. 3 is a flowchart illustrating a file information storage process in an embodiment of the present invention, and as shown in fig. 3, the method further includes:
step 301, configuring storage directories corresponding to multiple storage types according to configuration rules.
In this embodiment, the configuration rule is: after a plurality of local peer directories are configured, a module is taken from a file, and the file is evenly distributed to each shunting directory to achieve data balance. Wherein, the module in the module taking is the directory position for storing the file. The storage directory corresponding to the storage type may be: local directory, temporary directory, backup directory, etc.
In specific implementation, the data balance can be realized by modulus taking according to the file sequence number under the condition of mass data files, after shunting, subsequent processing can be simultaneously carried out by a plurality of servers, so that the capability of transverse expansion is realized, and when the processing capability reaches the bottleneck, the service requirement can be well met by the shunting processing.
Step 302, storing the file information to be processed in the configured storage directory.
In specific implementation, before the file is stored in the local directory, a temporary file is generated, stored in the temporary directory and downloaded, and named as a formal file. After the file is stored in the local directory, whether the file needs to be compressed or not can be judged according to the size of the file, if compression is carried out, the file needs to be decompressed subsequently, when decompression is carried out, if the performance of the server is good, the file can be decompressed in batches, and if the performance of the server is poor, the file can be decompressed one by one, so that the peak staggering effect is achieved, and all files can be smoothly decompressed.
Fig. 4 is a flowchart illustrating a file information cleaning process in an embodiment of the present invention, and as shown in fig. 4, the method further includes:
step 401, clearing file information in the configured storage directory according to a preset time interval.
In this embodiment, the "time interval" may be year, month, day, etc., that is, a time variable is configured in the local directory, and in the specific implementation, the time interval is 1 day, for example, a day granularity variable is configured in the local directory, so that the fetched files are stored by day, after the fetching is finished, the data clearing command is executed, and according to the configured time, the expired data is deleted, so as to reduce the loss of the storage space of the server.
Based on the same inventive concept, the embodiment of the invention also provides a file collecting device, as described in the following embodiments. Because the principle of solving the problems of the file acquisition device is similar to that of the file acquisition method, the implementation of the file acquisition device can refer to the implementation of the file acquisition method, and repeated parts are not described again. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 5 is a schematic structural diagram of a document collection apparatus according to an embodiment of the present invention, and as shown in fig. 5, the apparatus includes:
and thecustom module 501 is configured to perform custom processing on the obtained data source information.
Therule making module 502 is configured to make a file collection rule for collecting multiple types of file information according to a user-defined processing result of the data source information.
And thefile acquisition module 503 is configured to acquire information of the file to be processed according to the file acquisition rule.
Fig. 6 is a schematic structural diagram of a file collecting device according to an embodiment of the present invention, and as shown in fig. 6, the device further includes:
the shuntingprocessing module 601 is configured to determine whether to perform shunting processing on the file information to be processed according to the file acquisition rule, and if the determination result is yes, perform shunting processing on the file information to be processed according to a preset shunting processing rule.
Fig. 7 is a schematic diagram illustrating a storage structure of file information in an embodiment of the present invention, and as shown in fig. 7, the apparatus further includes:
aconfiguration module 701, configured to configure storage directories corresponding to multiple storage types according to a configuration rule;
thestorage module 702 is configured to store the file information to be processed in the configured storage directory.
Fig. 8 is a schematic diagram illustrating a structure for clearing file information in an embodiment of the present invention, and as shown in fig. 8, the apparatus further includes:
acleaning module 801, configured to clean file information in the configured storage directory according to a preset time interval.
The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above method when executing the computer program.
An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program for executing the above method is stored.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims (6)

CN201911219780.8A2019-12-032019-12-03File collection method and deviceActiveCN111125004B (en)

Priority Applications (1)

Application NumberPriority DateFiling DateTitle
CN201911219780.8ACN111125004B (en)2019-12-032019-12-03File collection method and device

Applications Claiming Priority (1)

Application NumberPriority DateFiling DateTitle
CN201911219780.8ACN111125004B (en)2019-12-032019-12-03File collection method and device

Publications (2)

Publication NumberPublication Date
CN111125004A CN111125004A (en)2020-05-08
CN111125004Btrue CN111125004B (en)2021-09-17

Family

ID=70497204

Family Applications (1)

Application NumberTitlePriority DateFiling Date
CN201911219780.8AActiveCN111125004B (en)2019-12-032019-12-03File collection method and device

Country Status (1)

CountryLink
CN (1)CN111125004B (en)

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication numberPriority datePublication dateAssigneeTitle
CN105205687A (en)*2015-08-242015-12-30浪潮通用软件有限公司Mass data acquisition method
CN105868169B (en)*2016-04-062019-04-30西安电子科技大学 A data acquisition device, data acquisition method and system
CN108881350A (en)*2017-05-122018-11-23深圳中软华泰信息技术有限公司The authentic data acquisition method of facing cloud platform Trusting eBusiness
CN107643932B (en)*2017-08-312020-12-25深圳天珑无线科技有限公司Data acquisition method, acquisition device and storage device
CN107766508B (en)*2017-10-232021-06-15深圳市中润四方信息技术有限公司Method and system for collecting and distributing data files
US20190306170A1 (en)*2018-03-302019-10-03Yanlin WangSystems and methods for adaptive data collection using analytics agents
CN110457348B (en)*2018-05-022022-05-10北京三快在线科技有限公司Data processing method and device
CN109766206A (en)*2018-12-292019-05-17北京中电普华信息技术有限公司 Method and system for collecting logs
CN110489391B (en)*2019-07-252022-06-28深圳壹账通智能科技有限公司Data processing method and related equipment

Also Published As

Publication numberPublication date
CN111125004A (en)2020-05-08

Similar Documents

PublicationPublication DateTitle
CN109284334A (en)Real-time database synchronization method and device, electronic equipment and storage medium
CN113360554A (en)Method and equipment for extracting, converting and loading ETL (extract transform load) data
CN112711612B (en) Data processing methods, devices, storage media and electronic devices
CN109818934A (en)A kind of method, apparatus and calculating equipment of automation daily record processing
CN111880966A (en)Database data recovery method and device
CN111367686A (en)Service interface calling method and device, computer equipment and storage medium
CN107622064A (en) A data reading method and system
CN112417050A (en) Data synchronization method and device, system, storage medium and electronic device
CN115328928B (en)kudu table updating method, kudu table updating device, kudu table updating equipment and storage medium
CN114238703A (en)Event flow arrangement method, device and application
CN108959657B (en)Data change processing method and device
CN110928681A (en)Data processing method and device, storage medium and electronic device
CN105656643B (en) A method and device for compatible management based on network configuration protocol
CN113839796B (en) Slice service processing method, device, network device and readable storage medium
CN112954082B (en)Domain name registration IP address query method and device
CN113055490B (en)Data storage method and device
CN111125004B (en)File collection method and device
CN106919574B (en)Method for processing remote synchronous file in real time
CN116032849B (en)Data exchange method, device, system and electronic equipment
CN108897497B (en)Centerless data management method and device
CN107968798A (en)A kind of network management resources label acquisition method, cache synchronization method, apparatus and system
CN106469086B (en)Event processing method and device
CN104484174A (en)Processing method and processing device for compressed file with RAR (Roshal A Rchive) format
CN111130921B (en)Method and device for processing performance index of core network element
CN113282651A (en)Data processing method and device, storage medium and electronic device

Legal Events

DateCodeTitleDescription
PB01Publication
PB01Publication
SE01Entry into force of request for substantive examination
SE01Entry into force of request for substantive examination
GR01Patent grant
GR01Patent grant
CP02Change in the address of a patent holder
CP02Change in the address of a patent holder

Address after:Room 702-2, No. 4811, Cao'an Highway, Jiading District, Shanghai

Patentee after:CHINA UNITECHS

Address before:100872 5th floor, Renmin culture building, 59 Zhongguancun Street, Haidian District, Beijing

Patentee before:CHINA UNITECHS


[8]ページ先頭

©2009-2025 Movatter.jp