CN110619083B - Data filtering method and device - Google Patents
Data filtering method and deviceDownload PDFInfo
- Publication number
- CN110619083B CN110619083BCN201910762591.9ACN201910762591ACN110619083BCN 110619083 BCN110619083 BCN 110619083BCN 201910762591 ACN201910762591 ACN 201910762591ACN 110619083 BCN110619083 BCN 110619083B
- Authority
- CN
- China
- Prior art keywords
- message data
- message
- data
- preset
- key information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9536—Search customisation based on social or collaborative filtering
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
Description
Technical Field
The application belongs to the technical field of data processing, and particularly relates to a data filtering method and device.
Background
In the course of import and export trade, with the development of computer technology, the original paper face spelling is gradually replaced by electronic documents. A large amount of information between related departments such as a ship company, a dock, a yard, a customhouse, and the like, or between a company and other units is exchanged and processed by Electronic Data Interchange (EDI). However, since the information generated in the trading process changes frequently, the system for receiving the EDI message needs to process a lot of invalid data additionally, and in a serious case, even a processing channel of normal data is blocked, thereby causing a problem of low data transmission efficiency.
Disclosure of Invention
The embodiment of the application provides a data filtering method and device, and can solve the problem that data transmission efficiency is low due to data blockage easily caused in the existing data exchange process.
In a first aspect, an embodiment of the present application provides a data filtering method, including:
acquiring message data and acquiring the message type of the message data;
analyzing key information of the message data through a preset analysis mode, wherein the preset analysis mode has a corresponding relation with the message type, and the key information comprises version information;
if the version information of the message data is not overdue and the message data is judged to have no preset abnormality, marking the message data as transmittable and storing the message data in a database.
It should be understood that if the version information of the message data is expired, the message data is marked as invalid to indicate that the same type of unsent message data corresponding to the message data is not sent any more.
In the embodiment, the key information containing the version information is analyzed through a preset analysis mode corresponding to the message type of the message data, and when the version information of the message data is not expired and no preset exception exists, the message data is marked as transmittable, so that the message data to be transmitted is guaranteed to be valid data, a processing channel of the valid data is prevented from being blocked, the data transmission efficiency is further improved, and the problem of low data transmission efficiency caused by data blocking in the existing data exchange process is solved. And marking the message data with the overdue version information as invalid, and then subsequently not sending the same type of message data corresponding to the message data, thereby not causing data blockage, improving the data transmission efficiency and ensuring the normal operation of the system.
In a second aspect, an embodiment of the present application provides a data filtering apparatus, including:
the acquisition module is used for acquiring message data and acquiring the message type of the message data;
the analysis module is used for analyzing key information of the message data through a preset analysis mode, wherein the preset analysis mode and the message type have a corresponding relation, and the key information comprises version information;
and the marking module is used for marking the message data as transmittable if the version information of the message data is not overdue and the message data is judged to have no preset exception, and storing the message data into a database.
In a third aspect, an embodiment of the present application provides a network device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the data filtering method described in any one of the above first aspects when executing the computer program.
In a fourth aspect, the present application provides a computer-readable storage medium, where a computer program is stored, where the computer program is implemented, when executed by a processor, to implement the data filtering method in any one of the above first aspects.
In a fifth aspect, an embodiment of the present application provides a computer program product, which, when run on a network device, causes the network device to execute the data filtering method described in any one of the above first aspects.
It is understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description of the first aspect, and are not described herein again.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a schematic diagram of a system provided by an embodiment of the present application;
FIG. 2 is a schematic flow chart diagram illustrating a data filtering method according to an embodiment of the present application;
FIG. 3 is a schematic flow chart diagram illustrating a data filtering method according to another embodiment of the present application;
FIG. 4 is a schematic structural diagram of a data filtering apparatus provided in an embodiment of the present application;
fig. 5 is a schematic structural diagram of a network device according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
Referring to fig. 1, fig. 1 is a schematic diagram of a system according to an embodiment of the present disclosure. The system comprises: afirst terminal 101, anetwork device 102 and asecond terminal 103. Thefirst terminal 101, thenetwork device 102 and thesecond terminal 103 communicate through a wireless network.
Thefirst terminal 101 is a terminal of a sender, thesecond terminal 103 is a terminal of a target recipient, the terminals include but are not limited to a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and the specific type of the terminal is not limited in this embodiment.
Thenetwork device 102 is a service device for forwarding the EDI message data, such as a server of an EDI center, where the service device includes, but is not limited to, a cluster server, a database server, a WEB server, and the like, and the specific type of the service device is not limited in this embodiment of the present application.
The processor may include one or more of a central processing unit (cpu), an Application Processor (AP), a baseband processor, and the like. The processor may be a neural hub and a command center of the wireless router. The processor can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and instruction execution. The memory may be used to store computer-executable program code, which includes instructions. The processor executes various functional applications of the network device and data processing by executing instructions stored in the memory. The memory may include a program storage area and a data storage area, such as data storing a sound signal to be played, and the like. For example, the memory may be a double data rate synchronous dynamic random access memory DDR or a Flash memory Flash, etc.
The communication module may provide a solution for communication applied to a network device, including Wireless Local Area Network (WLAN) (e.g., Wi-Fi network), bluetooth, Zigbee, mobile communication network, Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), and the like. The communication module may be one or more devices integrating at least one communication processing module. The communication module may include an antenna, and the antenna may have only one array element, or may be an antenna array including a plurality of array elements. The communication module can receive electromagnetic waves through the antenna, frequency-modulate and filter electromagnetic wave signals, and send the processed signals to the processor. The communication module can also receive a signal to be sent from the processor, frequency-modulate and amplify the signal, and convert the signal into electromagnetic waves through the antenna to radiate the electromagnetic waves.
The power management module may receive input from the battery and/or charger to power the processor, memory, and communication module, etc.
It should be noted that thenetwork device 102 may include more or fewer components, or combine some components, or different components, such as a display screen, indicator lights, motors, controls (e.g., buttons), gyroscopic sensors, acceleration sensors, etc.
Fig. 2 shows a schematic flow chart of the data filtering method provided in the present application, which can be applied to thenetwork device 102 described above by way of example and not limitation.
S201, acquiring message data and acquiring the message type of the message data;
s202, analyzing key information of the message data through a preset analysis mode, wherein the preset analysis mode and the message type have a corresponding relation, and the key information comprises version information;
s203, if the version information of the message data is not overdue and the message data is judged to have no preset abnormality, marking the message data as transmittable, and storing the message data in a database.
In S201, the message data includes message data of multiple service information sent by different senders, where the senders include a shipping company, a customs, a wharf company, and the like in the shipping process, and the service information includes a cabin booking service, a ship distribution service, a ship loading service, a container service, and the like.
The message type and the service information have a corresponding relationship, for example, the message type corresponding to the cabin booking service information is a cabin booking message, a ship distribution message corresponding to the ship distribution service information, a ship loading message corresponding to the ship loading service information, and a suitcase corresponding to the suitcase service information. The file name formats, message headers and contents of specified positions in the messages of the message data corresponding to different message types are also different. It should be understood that the message types described above may also be message types corresponding to various information in train, road, and pipeline transportation, and are only used for illustration and not as a limitation of the present application.
In S202, optionally, the key information includes, but is not limited to, version information, a sender, an intended recipient, a service unique identifier, and a message file unique number. The version information may be only in the header of a set of message data, or may be in the content of each message type of a set of message data, where the message version number (i.e., version information) is represented by a field name "FILEDSP =", for example. It should be understood that, in other embodiments, the version information may also be composed of other part information in the key information, for example, the version information is composed of the service unique identifier and the message file unique number, which is not described herein again.
The preset parsing mode may be to identify field names in the message data and obtain data corresponding to the field names, where the identified field names are field names corresponding to key information such as version information, a sender, a target receiver, a unique service identifier, and a unique message file number. For example: one segment of data in the message data is: FTX + TDT +1+ + SHIMENT FOR SHARM SHINZHEN TO HAMBURG BY VESSEL THEN WITH TRANSIT TO TEPLICE' means that the ship is used TO transfer from Shenzhen TO hamburger TO Toeplier harvest, wherein FTX + TDI is the field name.
The message data and the field names are only used for illustration, and are not limited to the specific embodiments of the present application.
In S203, the predetermined exception may be that the target recipient cannot identify the message data. For example, in a group of message data of a shipping service, a piece of message data is sent to a target receiver, but the target receiver does not have information such as a service unique identifier of the message data, that is, the target receiver user does not record the shipping service into an EDI receiving system, so that the system cannot identify the message data, and thus, the identification fails and is abnormal.
Specifically, if the version information of a group of message data is not expired and there is no preset exception in a previously sent piece of message data, it is indicated that the group of message data can be sent, the group of message data is marked as transmittable and the message data is stored in the database, a task table in the database is scanned at regular time by waiting for background monitoring, and the task table is executed to send the corresponding message data marked as transmittable to a target receiver. If the version information of a group of message data is expired, it indicates that the service information corresponding to the group of message data has been updated, and the group of message data does not need to be transmitted.
In the embodiment, the key information containing the version information is analyzed through a preset analysis mode corresponding to the message type of the message data, and when the version information of the message data is not expired and no preset exception exists, the message data is marked as transmittable, so that the message data to be transmitted is guaranteed to be valid data, a processing channel of the valid data is prevented from being blocked, the data transmission efficiency is further improved, and the problem of low data transmission efficiency caused by data blocking in the existing data exchange process is solved. And marking the message data with the overdue version information as invalid, and then subsequently not sending the same type of message data corresponding to the message data, thereby not causing data blockage, improving the data transmission efficiency and ensuring the normal operation of the system.
On the basis of the embodiment shown in fig. 2, the present application also proposes another embodiment of a data filtering method. It should be noted that the steps that are the same as those in the embodiment of fig. 2 are not repeated, and refer to the foregoing embodiment.
Optionally, the S202 includes:
acquiring a preset analysis mode corresponding to the message type;
and analyzing the key information of the message data according to the preset analysis mode.
In this embodiment, according to the industry EDI message specification, the message file formats of different message types are different, and the field names of each data in the message are also different, so that the key information of the message data of the corresponding message type is analyzed by using a preset analysis mode corresponding to the message type.
For example, the voyage information of a ship includes ship information, voyage throughput, ship stop time information, passenger information, and cargo information, i.e., a set of message data of the voyage information includes data of 5 message types. The field names of the message data in the ship information, the voyage throughput, the ship stopping time information, the passenger information and the cargo information are different, so that the message data of other information such as the passenger information cannot be analyzed by the analysis mode of analyzing the message data of the ship information, and therefore at least one analysis mode is corresponding to each message type.
Optionally, the step S203 includes:
if the message data meets a preset condition, marking the message data as transmittable message data needing to be preprocessed, wherein the preset condition is that a target receiver of the message data is a preset receiver, and key information of the message data also comprises the target receiver;
and if the message data do not accord with the preset conditions, marking the message data as transmittable.
In this embodiment, some target recipients have requirements for the message data, so the message data of the required target recipients are preprocessed, and the message data of the target recipients without requirements are not preprocessed.
Specifically, the above-mentioned preprocessing includes merging or splitting. Because the message processing capacities of different target receiver systems are different, some systems are good at processing a large amount of small files and some systems are good at processing a single large file; therefore, the preset receivers comprise a first preset receiver and a second preset receiver so as to preprocess the corresponding message data; wherein the first intended recipient is an intended recipient good at handling a large number of small files and the second intended recipient is an intended recipient good at handling a single large file.
For example, the message data of one voyage information includes message data of ship information, voyage throughput, ship stop time information, passenger information, and cargo information. When the target receiver is a first preset receiver, the message data such as the ship information is marked as the transmittable message data which needs to be split, so that the message data of the ship information is split into a plurality of message data such as a voyage code, a ship type, the number of crew members, the number of hatches, a ship company and the like before being transmitted, and the message data of the voyage number, the information of times of ship stopping, the passenger information and the cargo information can also be split into a plurality of message data. When the target receiver is a second preset receiver, the message data of the ship information, the voyage throughput, the ship stopping time information, the passenger information and the cargo information are marked as transmittable message data which need to be merged, so that the message data of the ship information, the voyage throughput, the ship stopping time information, the passenger information and the cargo information are merged into one message data of the ship term information before being transmitted. When the target receiver is not the preset receiver, the message data of the ship information, the voyage handling capacity, the ship stopping time information, the passenger information and the cargo information are marked as transmittable, so that the message data can be directly transmitted.
Optionally, after S203, the method further includes:
acquiring message data marked as transmittable in the database;
and sending the message data to a target receiver according to the key information, wherein the key information also comprises the target receiver.
In this embodiment, after the message data is filtered, the transmittable message data is transmitted to the corresponding target receiving party, so that data blocking caused by transmitting a large amount of invalid or abnormal preset message data is avoided, and the data transmission efficiency is improved.
Specifically, the message data marked as transmittable is stored in a task table in a database, a background monitors and scans the task table at regular time to acquire corresponding message data, groups the acquired message data according to a sender and a target receiver, and transmits the message data to the target receiver according to the groups. And further, checking whether the message data in the same group needs to be preprocessed or not, and if so, preprocessing the message data and then sending the preprocessed message data, wherein the preprocessing comprises merging or splitting.
On the basis of the embodiment shown in fig. 2, fig. 3 shows a flowchart for implementing another data filtering method proposed in the present application. As shown in fig. 3, S301 and S302 are also included before step S203 in the embodiment shown in fig. 2. It should be noted that the steps that are the same as those in the embodiment of fig. 2 are not repeated herein, and refer to the foregoing embodiment.
S301, according to the key information, acquiring a message receipt of the same type of sent message data corresponding to the message data;
s302, if the message receipt is that the sent similar message data is successfully received, judging that the message data has no preset abnormity.
In this embodiment, the similar message data is all message data of the same service information, and the same service information has the same key information such as the service unique identifier, so that other similar message data corresponding to one message data can be searched according to the key information, and thus, the problem that the similar message data cannot be searched due to the disordered transmission sequence of the similar message data caused by various reasons can be avoided.
The message receipt is used for feeding back the receiving condition of the target receiver of the previously sent similar message data, and the receiving condition comprises that the sent similar message data is successfully received or unsuccessfully received. When the sent similar message data is successfully received, it is indicated that all the message data corresponding to the service are normal, and no preset exception exists.
Optionally, after S301, the method further includes:
s3011, if the message receipt indicates that the sent similar message data has failed to be received, marking the message data as temporarily unsent, and marking unsent similar message data corresponding to the message data as temporarily unsent according to the key information.
In this embodiment, when one of all the message data of the same service information is failed to be received, the other message data of the service information that has not been sent may also be failed to be received. For example, in marine EDI, if a ship date message is not maintained in the target receiver system, all message data corresponding to the ship date message will fail because the target receiver system cannot identify the ship date message, and thus even if the same type of message data that has not been sent is sent to the target receiver, an error will still occur, so that the same type of message data that has not been sent is marked as not being sent temporarily, so as to avoid reducing data transmission efficiency due to blocking of a normal data processing channel.
Optionally, after S3011, the method further includes:
after the preset time, retransmitting first message data, wherein the first message data is one of the sent same type of message data which fails to be received;
acquiring a message receipt of the first message data;
and if the message receipt of the first message data is that the first message data is successfully received, marking all similar message data which are marked as temporarily unsent and correspond to the first message data as unsessable.
In this embodiment, the first message data may be any one of the same type of message data sent with failed reception, or may be the same type of message data sent with failed reception. For example, when the message receipt of the first sent message data indicates that the same type of message data sent has failed to be received, the message data is marked as delayed retry, that is, the message data is resent within a preset time.
In the embodiment, whether the target receiving party maintains the corresponding service information in the corresponding system is verified by continuously retransmitting the first message data which is failed to be received, and after the target receiving party maintains the corresponding service information in the corresponding system, the message receipt of the first message data is successfully received, so that all the message data of the service information can be transmitted to the target receiving party without the occurrence of the preset exception, and the data transmission efficiency is improved.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Fig. 4 shows a block diagram of a data filtering apparatus provided in an embodiment of the present application, corresponding to the data filtering method described in the foregoing embodiment, and only shows portions related to the embodiment of the present application for convenience of description.
Referring to fig. 4, theapparatus 400 includes:
an obtainingmodule 401, configured to obtain message data and obtain a message type of the message data;
ananalysis module 402, configured to analyze key information of the packet data in a preset analysis manner, where the preset analysis manner and the packet type have a corresponding relationship, and the key information includes version information;
amarking module 403, configured to mark the message data as transmittable if the version information of the message data is not expired and it is determined that the message data has no preset exception, and store the message data in a database.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
Fig. 5 is a schematic structural diagram of a network device according to an embodiment of the present application. As shown in fig. 5, thenetwork device 5 of this embodiment includes: at least one processor 50 (only one shown in fig. 5), amemory 51, and acomputer program 52 stored in thememory 51 and executable on the at least oneprocessor 50, wherein theprocessor 50 executes thecomputer program 52 to implement the steps of any of the various data filtering method embodiments described above.
Thenetwork device 5 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The network device may include, but is not limited to, aprocessor 50, amemory 51. Those skilled in the art will appreciate that fig. 5 is merely an example of thenetwork device 5, and does not constitute a limitation of thenetwork device 5, and may include more or less components than those shown, or combine some of the components, or different components, such as input output devices, network access devices, etc.
TheProcessor 50 may be a Central Processing Unit (CPU), and theProcessor 50 may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Thememory 51 may in some embodiments be an internal storage unit of thenetwork device 5, such as a hard disk or a memory of thenetwork device 5. Thememory 51 may also be an external storage device of thenetwork device 5 in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on thenetwork device 5. Further, thememory 51 may also include both an internal storage unit and an external storage device of thenetwork device 5. Thememory 51 is used for storing an operating system, an application program, a BootLoader (BootLoader), data, and other programs, such as program codes of the computer program. Thememory 51 may also be used to temporarily store data that has been output or is to be output.
The embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps in the above-mentioned method embodiments.
The embodiments of the present application provide a computer program product, which when running on a mobile terminal, enables the mobile terminal to implement the steps in the above method embodiments when executed.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or apparatus capable of carrying computer program code to a network device, recording medium, computer Memory, Read-Only Memory (ROM), Random-Access Memory (RAM), electrical carrier wave signals, telecommunications signals, and software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other ways. For example, the above-described apparatus/network device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (8)
1. A method of filtering data, comprising:
acquiring message data and acquiring the message type of the message data;
analyzing the key information of the message data in a preset analysis mode, wherein the key information comprises the following steps: acquiring a preset analysis mode corresponding to the message type; analyzing key information of the message data according to the preset analysis mode, wherein the preset analysis mode and the message type have a corresponding relation, and the key information comprises version information;
if the version information of the message data is not overdue and the message data is judged to have no preset abnormality, marking the message data as transmittable and storing the message data in a database; before marking the message data as transmittable, the method further includes: acquiring a message receipt of the sent similar message data corresponding to the message data according to the key information; and if the message receipt is that the sent similar message data is successfully received, judging that the message data has no preset abnormity.
2. The data filtering method according to claim 1, wherein after obtaining the message receipt of the same type of message data corresponding to the message data, further comprising:
if the message receipt is that the sent similar message data is failed to be received, marking the message data as temporary non-sending, and marking the non-sent similar message data corresponding to the message data as temporary non-sending according to the key information.
3. The data filtering method according to claim 2, wherein after marking the same type of unsent packet data corresponding to the packet data as temporarily unsent, further comprising:
after the preset time, retransmitting first message data, wherein the first message data is one of the sent same type of message data which fails to be received;
acquiring a message receipt of the first message data;
and if the message receipt of the first message data is that the first message data is successfully received, marking all similar message data which are marked as temporarily unsent and correspond to the first message data as unsessable.
4. The data filtering method of claim 1, wherein said marking said message data as transmittable comprises:
if the message data meets a preset condition, marking the message data as transmittable message data needing to be preprocessed, wherein the preset condition is that a target receiver of the message data is a preset receiver, and key information of the message data also comprises the target receiver;
and if the message data do not accord with the preset conditions, marking the message data as transmittable.
5. The data filtering method of claim 1, wherein after marking the message data as transmittable, further comprising:
acquiring message data marked as transmittable in the database;
and sending the message data to a target receiver according to the key information, wherein the key information also comprises the target receiver.
6. A data filtering device, comprising:
the acquisition module is used for acquiring message data and acquiring the message type of the message data;
the analysis module is used for analyzing the key information of the message data in a preset analysis mode, and comprises: acquiring a preset analysis mode corresponding to the message type; analyzing key information of the message data according to the preset analysis mode, wherein the preset analysis mode and the message type have a corresponding relation, and the key information comprises version information;
the marking module is used for marking the message data as transmittable if the version information of the message data is not overdue and the message data is judged to have no preset exception, and storing the message data into a database; before marking the message data as transmittable, the method further includes: acquiring a message receipt of the sent similar message data corresponding to the message data according to the key information; and if the message receipt is that the sent similar message data is successfully received, judging that the message data has no preset abnormity.
7. A network device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 5 when executing the computer program.
8. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910762591.9ACN110619083B (en) | 2019-08-19 | 2019-08-19 | Data filtering method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910762591.9ACN110619083B (en) | 2019-08-19 | 2019-08-19 | Data filtering method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110619083A CN110619083A (en) | 2019-12-27 |
| CN110619083Btrue CN110619083B (en) | 2022-04-19 |
Family
ID=68921915
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910762591.9AActiveCN110619083B (en) | 2019-08-19 | 2019-08-19 | Data filtering method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110619083B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113839859B (en) | 2020-06-23 | 2023-05-23 | 华为技术有限公司 | Message notification method and related device |
| CN115529267A (en)* | 2021-06-25 | 2022-12-27 | 华为技术有限公司 | A communication method and related equipment |
| CN116248781A (en)* | 2023-03-08 | 2023-06-09 | 中国工商银行股份有限公司 | Message processing method, device, equipment and storage medium based on transaction information |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103036648A (en)* | 2012-12-13 | 2013-04-10 | 福建星网锐捷网络有限公司 | Control and provisioning of wireless access point (CAPWAP) message processing method and processing device |
| CN107026821A (en)* | 2016-02-01 | 2017-08-08 | 阿里巴巴集团控股有限公司 | The processing method and processing device of message |
| CN107222936A (en)* | 2017-06-26 | 2017-09-29 | 广东欧珀移动通信有限公司 | A kind of data processing method, device and terminal |
| CN107580076A (en)* | 2017-11-07 | 2018-01-12 | 山东网智物联网科技有限公司 | Internet of Things communication means and device |
| CN108959514A (en)* | 2018-06-27 | 2018-12-07 | 中国建设银行股份有限公司 | A kind of data processing method and device |
| CN109597805A (en)* | 2018-11-07 | 2019-04-09 | 平安科技（深圳）有限公司 | A kind of data processing method, electronic equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9229611B2 (en)* | 2013-03-05 | 2016-01-05 | Oracle International Corporation | Automatic management of client data consistency |
- 2019
- 2019-08-19CNCN201910762591.9Apatent/CN110619083B/enactiveActive
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103036648A (en)* | 2012-12-13 | 2013-04-10 | 福建星网锐捷网络有限公司 | Control and provisioning of wireless access point (CAPWAP) message processing method and processing device |
| CN107026821A (en)* | 2016-02-01 | 2017-08-08 | 阿里巴巴集团控股有限公司 | The processing method and processing device of message |
| CN107222936A (en)* | 2017-06-26 | 2017-09-29 | 广东欧珀移动通信有限公司 | A kind of data processing method, device and terminal |
| CN107580076A (en)* | 2017-11-07 | 2018-01-12 | 山东网智物联网科技有限公司 | Internet of Things communication means and device |
| CN108959514A (en)* | 2018-06-27 | 2018-12-07 | 中国建设银行股份有限公司 | A kind of data processing method and device |
| CN109597805A (en)* | 2018-11-07 | 2019-04-09 | 平安科技（深圳）有限公司 | A kind of data processing method, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 数据预处理之数据清洗(Data Cleansing);Orange_Spotty_Cat;《CSDN》;20180817;第2页* |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110619083A (en) | 2019-12-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110619083B (en) | Data filtering method and device | |
| CA3120888C (en) | Enforcing data consistency in a transportation network | |
| US20090127327A1 (en) | Method and apparatus for delivery tracking correlated with barcode images | |
| WO2014122445A1 (en) | A data processing method and apparatus | |
| CN116634494A (en) | Multi-component heterogeneous data coding and framing method based on Beidou short message | |
| CN109088933A (en) | High-volume list transfer approach, acquisition methods and corresponding device, electronic equipment | |
| CN109743133B (en) | Data reconciliation method and device | |
| CN108833443A (en) | A kind of method for message transmission and system, computer equipment | |
| MX2007006289A (en) | System for message delivery to field personnel. | |
| CN117499334B (en) | Intelligent vehicle communication method, terminal equipment and storage medium | |
| CN110662191B (en) | Communication mode selection method and device and electronic equipment | |
| CN113094002B (en) | Message processing method, device, electronic equipment and computer medium | |
| CN118363053A (en) | Satellite data processing method, device, computer equipment and storage medium | |
| CN114819030B (en) | Electronic waybill, mobile terminal, device, method, apparatus and storage medium | |
| CN111045723A (en) | Method and device for notifying code change between associated systems | |
| CN114710422A (en) | Luggage message format checking method and related device | |
| CN116388834A (en) | Packet loss response method based on Beidou short message and terminal equipment | |
| CN111274831B (en) | Information circulation system, method, equipment and storage medium based on RFID | |
| KR20140127036A (en) | Server and method for spam filtering | |
| JP2000285389A (en) | Mobile operation management system | |
| CN113485412A (en) | Method and system for distinguishing return by using multilink data signal source by unmanned aerial vehicle | |
| US20200193369A1 (en) | System and method for managing shipping processes and personnel | |
| CN113762841A (en) | Waybill processing method and device, electronic equipment and storage medium | |
| CN106408988A (en) | Bus station broadcasting system, method and device | |
| US12143363B2 (en) | CANBUS cybersecurity firewall |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |