US8150814B2

Movatterモバイル変換

Info

Publication number: US8150814B2
Application number: US12/419,811
Authority: US
Inventors: Steven E. Felsheim
Original assignee: Business Objects Software Ltd
Current assignee: Business Objects Software Ltd
Priority date: 2009-04-07
Filing date: 2009-04-07
Publication date: 2012-04-03
Also published as: JP2010244539A; US20100257145A1; JP5744415B2

Abstract

In one embodiment the present invention includes a computer-implemented method for data cleansing using rule based formatting. The method includes tokenizing and parsing a first input data and a second input data. The method further includes including a first token in a first output data if a first formatting rule component in a formatting rule is a first valid index to said first tokenized input data. The method further includes including a second token in a second output data if said first formatting rule component in the formatting rule is a second valid index to said second tokenized input data. The method further includes formatting said first output data and said second output data according to the formatting rule.

Description

BACKGROUND

The present invention relates to data cleansing, and in particular, to a system and method of data cleansing using rule based formatting.

Extract, transform, and load (ETL) may be some processes that are performed as part of managing databases. A subset of desired data may be extracted from various data sources as part of the extract component. The transform component may convert the extracted data into a suitable state. Finally, the load component of ETL may include transferring the transformed data to a target data source like another database, a data mart, or a data warehouse, for example. Thus, ETL allows data that is extracted from various data sources to be converted into some desirable format and transferred to another data source.

Data cleansing may be a process that is performed in the transform component of ETL. Data cleansing may include the detection of incorrect data, which may then be corrected or removed, and the formatting of data. Moreover, the detection of data may be accomplished by tokenizing the data and parsing the data according to some predetermined rules. One technique of parsing data is to use rules (i.e., rule-based parsing). When formatting the output data, it may be desirable to control how the tokens may be ordered or what strings may delimit the tokens. Thus, it may be desirable to tokenize and parse data. However, when using a rule-based parsing technique to parse data, it may be difficult to control how the parsed components may be ordered or what strings may delimit the parsed components.

Thus, there is a need for improved data cleansing that allows control of the formatting of output data when rule-based parsing is used. The present invention solves these and other problems by providing a system and method of data cleansing using rule based formatting.

SUMMARY

Embodiments of the present invention improve data cleansing by allowing control of the formatting of output data. In one embodiment the present invention includes a computer-implemented method for data cleansing using rule based formatting. The method includes obtaining a first input data and a second input data, wherein said first input data is tokenized according to a data dictionary, wherein said second input data is tokenized according to said data dictionary. The method further includes parsing said first input data and said second input data using a predefined parsing rule. The method further includes obtaining a formatting rule, wherein said formatting rule includes one or more formatting rule components. The method further includes including a first token in a first output data if a first formatting rule component in the formatting rule is a first valid index to said first tokenized input data, wherein said first token is associated with said first valid index, and including a first string literal in said first output data if said first formatting rule component in the formatting rule is a string literal. The method further includes including a second token in a second output data if said first formatting rule component in the formatting rule is a second valid index to said second tokenized input data, wherein said second token is associated with said second valid index and including a second string literal in said second output data if said first formatting rule component in the formatting rule is the string literal. The method further includes formatting said first output data and said second output data according to the formatting rule.

The following detailed description and accompanying drawings provide a better understanding of the nature and advantages of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a flow diagram of a method of formatting output data according to one embodiment of the present invention.

FIG. 1B illustrates a flow diagram showing an example of a process for conditionally including a token in output data according to one embodiment of the present invention.

FIG. 2A illustrates example entries of a data dictionary according to one embodiment of the present invention.

FIG. 2B illustrates an example parsing rule and formatting rule according one embodiment of the present invention.

FIG. 3 illustrates a table of example input and output data according to one embodiment of the present invention.

FIG. 4 illustrates an example of discrete field input formatting according to one embodiment of the present invention.

FIG. 5 is a block diagram that illustrates a system according to one embodiment of the present invention.

FIG. 6 is a block diagram of an example computer system and network for implementing embodiments of the present invention.

DETAILED DESCRIPTION

Described herein are techniques for data cleansing using rule based formatting. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present invention. It will be evident, however, to one skilled in the art that the present invention as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

FIG. 1A illustrates a flow diagram of a method of formatting output data according to one embodiment of the present invention. The method may be implemented as one or more computer programs that are executed on a hardware system (seeFIG. 5 for more details). Moreover, the method ofFIG. 1A may allow input data, regardless of how it may be formatted, to be processed, and flexibly outputted in a standardized format.

Inbox101, data may be obtained from a data source. Data stored in a data source may have a variety of different values and may contain various bits of information. The meaning of the data may depend on how the data may be defined by a user of the data. For example, a data source may store a piece of data related to agriculture with the value of [LOC 334 75 BEANS G-VG], which may be defined to mean that plant #334 in building 75 processed beans of a quality level of “very good.” Therefore, a piece of data can take on various values according to what a user may define the data to represent. In certain embodiments, the tokenized and parsed data may be obtained from one or more data sources.

In addition, data may be tokenized inbox101. A token may be an atomic piece of the data classified by a dictionary. Tokens may also contain standard versions of the tokens text for each parsed context the token may be contained in. Standard versions of tokens and contexts may be defined by the user. Furthermore, tokens may be a result of lexical analysis of the data. Data may be tokenized by breaking the data into segments (i.e., tokens) based on a selected breaking strategy and how they are classified by a data dictionary. Classification by the data dictionary may be defined by the user. Breaking strategies may include, but are not limited to, breaking on whitespace, and breaking on punctuation, for example. Breaking strategies may be specified by the user, e.g. that the user specifies break keys. Continuing with the above example data value of [LOC 334 75 BEANS G-VG], if the breaking strategy applied is to break on whitespace, the resulting tokens may be [LOC], [334], [75], [BEANS], and [G-VG].

Inbox102, data may be parsed using on or more parsing rules. A parsing rule, which may be defined by a user, may be defined to specify patterns of classifications defined in the data dictionary (e.g., the tokens) to recognize (i.e., a match). Therefore, a match may occur when one of the token's classifications satisfies the correct condition of a specified classification pattern in a parsing rule. Further, a parse may result when the tokenized data satisfies a parsing rule (i.e., the tokenized data matches all of the classification conditions of the specified pattern in the parsing rule). Thus, parsing the data (i.e., the tokens) may be a way of ensuring that the data is in an expected form, and the expected form may be based on the definition of the parsing rule (which may be defined by the user). In certain embodiments, one or more parsing rules may be included in a rule file.

The tokens of the resulting parse may be referenced by using indexes in format syntax of the rule. An index may indicate a particular component of the parsing rule indexed from left to right. Accordingly, for a given tokenized data, an index may reference a matched token that may be associated with a particular component of the parsing rule. For example, if a parsing rule is defined to be a string followed by two numbers and two more strings, the indexes for the above example data of [LOC 334 75 BEANS G-VG] may be 1 for [LOC], 2 for [334], 3 for [75], 4 for [BEANS], and 5 for [G-VG].

In some embodiments, a parsing rule may use the * operator to match one or more tokens that may match a particular token's classification. Because there may be an unknown number of matches when using the * operator, the one or more tokens that may match the particular pattern portion of a rule cannot be individually indexed. Therefore, the one or more tokens that may match the particular pattern portion of a rule may have one index. For example, if the parsing rule pattern is defined to be a string followed by one or more numbers (i.e., using the * operator) and two more strings, the indexes for the example data of [LOC 334 75 BEANS G-VG] may be 1 for [LOC], 2 for [334] and [75], 3 for [BEANS], and 4 for [G-VG]. In other embodiments of the present invention, a [ON*“some string”] operator, where some string may be a string, may be used to control what may be inserted between the one or more tokens that may match a particular classification. Continuing with the example, a [ON*“ ”] operator performed on the one or more tokens that match a number may result inindex 2 having a value of [33475] since an empty string was specified in the operator. Accordingly, if a [ON*“X”] operator is performed instead,index 2 may have a value of [334X75]. Therefore, the [ON*“some string”] operator may be used to control what string, if any, may delimit the one or more tokens that may match a particular rule pattern portion.

In certain embodiments of the present invention, a parsing rule may use the “?” operator to indicate that a particular index may be optional. That is, a parse (i.e., a tokenized data that satisfies a parsing rule) may still occur if a particular index that may have the “?” operator applied does not have a token assigned to it. For example, a parsing rule may be an optional string (i.e., using the “?” operator) followed by two numbers and two strings. Using the above example of [LOC 334 75 BEANS G-VG], the resulting tokenized data of [LOC], [334], [75], [BEANS], and [G-VG] may be a parse of this parsing rule. Similarly, a variation of the example of [334 75 BEANS G-VG], the resulting tokenized data of [334], [75], [BEANS], and [G-VG] may also be a parse of this parsing rule because the first string may be optional.

Inbox104, a token may be included in the output data if the formatting rule component is a valid index to the tokenized data. An index to the tokenized data may be a valid index if the index is specified in the parsing rule. For example, a parsing rule that may be defined to be 5 consecutive strings may have 5 valid indexes (e.g., 1-5). In this case, an index of 6 is not a valid index, and, thus, may not be included in the output data, because the parsing rule is defined to match five consecutive strings (i.e., tokens). (Further details ofbox104 are provided below with reference toFIG. 1B.)

Inbox105, a string literal may be included in the output data if the formatting rule component is a string literal. As previously mentioned above, a string literal may be enclosed with double quotes.

FIG. 1B illustrates a flow diagram showing more details ofbox104 ofFIG. 1A according to one embodiment of the present invention. The method may be implemented as one or more computer programs that are executed on a hardware system (seeFIG. 5 for more details). There may be cases when a user may want to include string literals if one or both of the tokens that surround the string literal exist in the parse. These cases may arise when a parsing rule may have optional components in the rule (i.e., using the “?” operator). To handle this, two conditional format operators may be used to indicate when a string literal may be included in the parse. The left-facing conditional format operator “<” (also referred to as the “<” operator) may indicate that the left side may be required. The right-facing conditional format operator “>” (also referred to as the “>” operator) may indicate that the right side may be required. Accordingly, the operators may generate four potential conditions. These may be handled as per the four processing operations shown inFIG. 1B (which may occur in any order).

Atbox111, a string literal may be included in the output data. This condition may occur when the parsing rule does not include the “<” and “>” operators. For example, a formatting rule may be defined as the following: 1+“some string”+2. Since the “<” and “>” operators are not included in the formatting rule, the string literal of “some string” may be included in the output data.

Atbox112, a string literal may be included in the output data if the token immediately to the left of the “<” operator (i.e., the left-facing conditional format operator) exists. For example, a formatting rule may be defined as the following: 1+<“some string”+2. Therefore, “some string” may be included in the output data if index 1 (i.e., the index to the token immediately to the left of the “<” operator) exists. If the parsing rule indicates thatindex 1 may be optional (i.e., using the “?” operator) and a particular tokenized data does not have a token assigned toindex 1, “some string” may not be included in the output data.

Atbox113, a string literal may be included in the output data if the token immediately to the right of the “>” operator (i.e., the right-facing conditional format operator) exists. For example, a formatting rule may be defined as the following: 1+“some string”>+2. Thus, “some string” may be included in the output data if index 2 (i.e., the index to the token immediately to the right of the “>” operator) exists. If the parsing rule indicates thatindex 2 may be optional (i.e., using the “?” operator) and a particular tokenized data does not have a token assigned toindex 2, “some string” may not be included in the output data.

Atbox114, a string literal may be included in the output data if the token immediately to the left of the “<” operator and the token immediately to the right of the “>” operator both exists. For example, a formatting rule may be defined as the following: 1+<“some string”>+2. Hence, “some string” may be included in the output data if index 1 (i.e., the index to the token immediately to left of the “<” operator) and index 2 (i.e., the index to the token immediately to the right of the “>” operator) both exist. If the parsing rule indicates that bothindex 1 andindex 2 may be optional (i.e., using the “?” operator) and a particular tokenized data does not have a token assigned toindex 1 andindex 2, “some string” may not be included in the output data.

EXAMPLE IMPLEMENTATION

The following is a detailed explanation of an example implementation according to one embodiment of the present invention. The example implementation may be implemented as one or more computer programs that are executed on a hardware system (seeFIG. 5 for more details). Furthermore, the example implementation may implement all or part of the method ofFIG. 1A and method ofFIG. 1B described above. In this example, it is assumed that the input data represents production information from several plants that process field crops, that each of the plants has a different way to record the processing information, and that a single representation for a master data warehouse is desired, which may be a format a custom analysis tool expects, for example. The input data contains a lot of different information, but it is assumed that where the crop was processed (PROCESSING_FACILITY), what kind of crop it is (PRODUCT), and the quality of processed crop (QUALITY) are the information of interest. For purposes of explanation, the example input data previously used above with the value of [LOC 334 75 BEANS G-VG] is used. This data represents that plant #334 in building 75 processed beans of a quality level 1 (“very good”).

The selected scheme that is used to tokenize the input data is to break on any whitespace or punctuation. The resulting tokens after the example input data has been tokenized is the following: [LOC], [334], [75], [BEANS], [G], [-], and [VG]. A selected parsing rule and a data dictionary may be applied to the tokenized input to determine whether a parse exists. A data dictionary may include entries that define the possible values for a particular classification, and what the possible values for the particular classification may represent.FIG. 2A illustrates example entries of a data dictionary according to one embodiment of the present invention.FIG. 2B illustrates an example250 parsing rule and formatting rule according one embodiment of the present invention. For purposes of illustration, the data dictionary ofFIG. 2A and the parsing rule252 (i.e. “crop_rule”) ofFIG. 2B are applied to the example input data. The first index of theparsing rule252 expects a PLANT_ID classification, which may be “Plant,” “Pl,” “P,” “Plnt,” “Loc,” or “location” according to the data dictionary ofFIG. 2A. Here, the first token matches the PLANT_ID classification because the first token is [LOC]. Since the first token matches the first index, the first index will have a value of “F” because the data dictionary indicates that a classification of PLANT_ID with a value of [LOC] represents “F.” Note that the “?” operator is applied to the first index. Therefore, a parse may still occur even if the first token does not match the first index since the “?” operator indicates that the index is optional.

The second index of theparsing rule252 expects a NUMBER classification with a “*” applied. Thus, according to the data dictionary, the second index expects one or more tokens that consist of any number. The next token is [334], which matches the second index. Similarly, the third token is [75], which also matches the second index. However, the fourth token is [BEANS], which does not match the second index. Therefore, the second index consists of the tokens [334] and [75].

The third index of theparsing rule252 expects a PRODUCT_ID classification. The next token, [BEANS], matches the third index because [BEANS] is a value included in the PRODUCT_ID classification. Accordingly, [BEAN] is the value of the third index because [BEANS] represents [BEAN]. The fourth index of theparsing rule252 expects a GRADE classification whose values may be “G,” “grade,” “grd,” “qual,” and “quality.” The next token is [G], which represents “Q” in the data dictionary. Therefore, the fourth index has a value of “Q.” The fifth index of theparsing rule252 expects any optional PUNCTUATION because the “?” is applied to it. In this example, the next token is [-], a punctuation. Thus, the fifth index has a value of [-]. Finally, the sixth and final index of theparsing rule252 expects a GRADE_LEVEL classification. The last token has a value of [VG], which represents the value “1” according to the data dictionary. Hence, the sixth index of theparsing rule252 has a value of “1.” Since the tokenized input data satisfies theparsing rule252, this particular tokenized input data is a parse.

The action lines254 indicate the information of interest in the input data. Withing theaction lines254, the following line illustrates the application of the [ON*] operator according to one embodiment of the present invention: “CROP=1: PROCESSING_FACILITY: 2: ON* “ ”.” As discussed above, the “*” operator was applied to the second index. Therefore, to control what may delimit the one or more token matches, the [ON*] operator may be used. Here, an empty is used to delimit the one or more token matches associated with the second index. Since the token matches are [334] and [75] as described previously, the value of the second index is “33475” after applying the [ON*] operator.

The formatting rules256 depicted inFIG. 2B (e.g., “format=”) may define how matching token indexes are ordered, what string, if any, may be used to delimit the indexes, and which parts of the parse may be included in the output data. For example, the first of the formatting rules256 inFIG. 2B is defined as follows: “format=CROP: CROP: “|”+3+“:”+1+2+“GRADE:”+6+“|”;”. A formatting line may begin by specifying “format=”. This indicates that a formatting rule may follow. In some embodiments, all formatting rules are the last lines in the action section of the rule. The “CROP” following the “format=” may indicate what is the top level category for this formatting rule. In other embodiments, like our example, “CROP” may match the action type of the rule. A top level category may be followed by a “:” to indicate the end of the identifier. Following a top level category may be a sub category for which the formatting rule may define. For a top level category formatting rule, what follows the top level category may be the same as the previous identifier, like in this example (i.e., “CROP” follows “CROP:”). Continuing with the example, the first formatting rule defines the top level category formatting to begin with a “|” string literal followed by the value of the third index followed by a “:” string literal, and so forth. This portion of the format lines tells Data Cleanse what the top level category for this format is. The resulting output for according to the example input data described above is [|BEAN:F33475GRADE:1|]. Likewise,FIG. 3 illustrates a table of example input and output data according to this example implementation.

The second of the formatting rules256 defines a format for the PROCESSING_FACILITY sub category. Similarly, the third of the formatting rules256 defines a format for the QUALITY category. In one embodiment, a formatting rule for a sub category may apply when the particular sub category is outputted and does not affect the building of the top level component. The sub category identifier may be followed by a “:” to indicate the end of the identifier.

Discrete Field Input

In one embodiment of the present invention, input data on discrete input fields may not be parsed by the parsing rules, and may function to standardize the input and produce a particular output field. Because there may not be a rule associated with a parse that results from discrete input fields, formatting information may be specified differently. In some embodiments, this may be implemented by adding a new section to a rule file. The section may control how output that may be generated from discrete input fields may be formatted, and how multiple tokens of input may be separated when the [ON*] may be used. Accordingly, formatting may be controlled by specifying the output sub category. The syntax may be similar to the formatting rules described above. In addition, string literals may function the same way (e.g., conditional inclusion of string literals as described above).

EXAMPLE IMPLEMENTATION

The following is an example implementation of discrete field input according to one embodiment of the present invention.FIG. 4 illustrates an example of discrete field input formatting according to one embodiment of the present invention. The example implementation may be a section within a rule file like the rule file depicted inFIG. 2B, for example. The “discrete_parser_options_start;” in the first line inFIG. 4 may be an identifier that may be used to indicate that the discrete input formatting options may follow. Similarly, the “discrete_parser_options_end;” in the last line may be an identifier that may be used to indicate that the discrete input formatting options may be finished. The “DISCRETE_NAME_FORMAT” may be an identifier that may indicate what the format should be for output generated from discrete name input fields. The output sub categories may be used to arrange the order of the NAME output created. The string literal syntax may be the same as the formatting rule as previously described. The “DISCRETE_NAME_ON*” may be an identifier that may indicate what, if any, string to use between multiple tokens that occur in the same discrete name input field. This may function similar to the [ON*] operator discussed above. Furthermore, the “DISCRETE_FIRM_ON*” may be an identifier that may indicate what, if any, string to use between multiple tokens that occur in the same discrete firm input field. The “DISCRETE_FIRM_FORMAT” may be an identifier that may indicate what the format should be for output generated from discrete firm input fields. The output sub categories may be used to arrange the order of the FIRM output created. The string literal syntax may be the same as the formatting rule as previously described.

FIG. 5 is a block diagram that illustrates a system according to one embodiment of the present invention.System500 may include data cleansesoftware application501, data repository505 (i.e., data source1), data repository506 (i.e., data source2), data repository507 (i.e., data target3), data repository508 (i.e., data dictionary repository), and data repository509 (i.e., rule repository). Datacleanse software application501 may obtain input data from

data repositories

505 and506, which may be processed by data cleanse software application. Accordingly,

data repositories

505 and506 may be configured to store input data, and may each comprise one or more databases, for example. Datacleanse software application501 may transmit output data todata repository507. Thus,data repository507 may be configured to store output data, and may comprise one or more databases, for example. As an example, thedata repository507 may reside in the same physical database as one or more of the

data repositories

505 and506, as cleansed data.

In certain embodiments of the present invention, datacleanse software application501 may includetokenizing module502, rule-basedparsing module503, andformatting module504.Tokenizing module502 may be configured to tokenize data. For example,tokenizing module502 may tokenize input data from

data repositories

505 and506. In some embodiments of the present invention,tokenizing module502 may tokenize input data according to a data dictionary, which may be stored indata dictionary repository508. In addition, rule-basedparsing module503 may be configured to parse input data. Rule-basedparsing module503 may parse input data that may have been tokenized bytokenizing module502, for example. According to an embodiment, the content of one or more of thedata repository508 and thedata repository509 may reside in a working memory readily accessible to a processor (seeFIG. 6). In one embodiment, the parsing rules used by rule-basedparsing module503 may be stored inrule repository509.Formatting module504 may be configured to format data and generate an output data. For example,formatting module504 may format data that may have been tokenized bytokenizing module502 and parsed by rule-basedparsing module503. In an embodiment of the present invention, the formatting rules used by formattingmodule504 may be stored inrule repository508. In some embodiments, datacleanse software application501 may implement all or part of the method ofFIG. 1A and the method ofFIG. 1B.Tokenizing module502 may implement all or part ofbox101 ofFIG. 1A, as described above. Rule-basedparsing module503 may implement all or part ofbox102 ofFIG. 1A. Additionally,formatting module504 may implement all or part of boxes103-105 ofFIG. 1A and boxes111-114 ofFIG. 1B.

FIG. 6 is a block diagram of an example computer system andnetwork600 for implementing embodiments of the present invention.Computer system610 includes abus605 or other communication mechanism for communicating information, and aprocessor601 coupled withbus605 for processing information.Computer system610 also includes amemory602 coupled tobus605 for storing information and instructions to be executed byprocessor601, including information and instructions for performing the techniques described above. This memory may also be used for storing variables or other intermediate information during execution of instructions to be executed byprocessors601. Possible implementations of this memory may be, but are not limited to, random access memory (RAM), read only memory (ROM), or both. Astorage device603 is also provided for storing information and instructions. Common forms of storage devices include, for example, a hard drive, a magnetic disk, an optical disk, a CD-ROM, a DVD, a flash memory, a USB memory card, or any other medium from which a computer can read (e.g., a “computer-readable medium”).Storage device603 may include source code, binary code, or software files for performing the techniques or embodying the constructs above, for example.

Computer system

610 may be coupled viabus605 to an output device, such as adisplay612, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. Aninput device611 such as a keyboard and/or mouse is coupled tobus605 for communicating information and command selections from the user toprocessor601. The combination of these components allows the user to communicate with the system. In some systems,bus605 may be divided into multiple specialized buses.

Computer system

610 also includes anetwork interface604 coupled withbus605.Network interface604 may provide two-way data communication betweencomputer system610 and thelocal network620. Thenetwork interface604 may be a digital subscriber line (DSL) or a modem to provide data communication connection over a telephone line, for example. Another example of the network interface is a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links using radio frequency communications are another example. In any such implementation,network interface604 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

Computer system

610 can send and receive information, including messages or other interface actions, through thenetwork interface604 to an Intranet or theInternet630. In the Internet example, software components or services may reside on multipledifferent computer systems610 or servers631-635 across the network. The processes described above may be implemented on one or more servers, for example. Aserver631 may transmit actions or messages from one component, throughInternet630,local network620, andnetwork interface604 to a component oncomputer system610. Different processes may be implemented on any computer system and send and/or receive information across a network, for example. In one embodiment, the techniques describe above may be implemented by software services on one or more servers631-635, for example. As an example, the

data sources

505,506 and507, thedata dictionary repository508, and therule repository509 may be stored by one or more of the servers631-635, for example in their respective memories (compare the memory602) or storage devices (compare the storage device603).

According to one embodiment, datacleanse software application501 may be implemented bycomputer system610. In some embodiments,tokenizing module502, rule-basedparsing module503, andformatting module504 may be implemented byprocessor601,memory602, andstorage device603. In other embodiments,data repository505 may be implemented byserver614,data repository506 may be implemented byserver615, anddata repository507 may be implemented byserver616.

According to another embodiment, all or part of the method ofFIG. 1A may be implemented bycomputer system610. The data sources ofbox101 may be stored on one or more servers614-616. The tokenizing of data ofbox101 may be implemented byprocessor601 andmemory602. The parsing inbox102 may be implemented byprocessor601 andmemory602 where the parsing rule used for parsing may be stored instorage device603. The data processing in

box

104 and105 may be implemented byprocessor601 andmemory602 where the formatting rule used to process the data may be stored instorage device603. According to an embodiment, all or part of the method ofFIG. 1B may be implemented byprocessor601 andmemory602. Thememory602 may implement thedata dictionary repository508 or the rule repository509 (seeFIG. 5).

According to an embodiment, agricultural information may be cleansed. Other types of data may also be cleansed. Name information may be cleansed. Address information may be cleansed. Non-English language information may be cleansed, for example, Japanese language information. Some languages may not use spaces (or other symbols) to delimit the components in information, such as for address and name information in Japanese; an embodiment of the present invention may be used to parse and standardize such information.

The above description illustrates various embodiments of the present invention along with examples of how aspects of the present invention may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present invention as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the invention as defined by the claims.

Claims

What is claimed is:

1. A computer-implemented method for data cleansing using rule based formatting, comprising:

obtaining a first input data from a first data source and a second input data from a second data source, wherein said first input data is tokenized according to a data dictionary, wherein said second input data is tokenized according to said data dictionary;

parsing, by a rule-based parsing module implemented by a hardware server, said first input data and said second input data using a predefined parsing rule including an option operator, wherein the option operator indicates that a particular index defined in the predefined parsing rule is optional;

obtaining a formatting rule, wherein said formatting rule includes one or more formatting rule components including at least one conditional format operator, wherein the at least one conditional format operator indicates whether to include a particular string literal in an output data based on the existence of a particular token;

including a first token in a first output data if a first formatting rule component in the formatting rule is a first valid index to said first tokenized input data, wherein said first token is associated with said first valid index, and including a first string literal in said first output data if said first formatting rule component in the formatting rule is a string literal;

including a second token in a second output data if said first formatting rule component in the formatting rule is a second valid index to said second tokenized input data, wherein said second token is associated with said second valid index and including a second string literal in said second output data if said first formatting rule component in the formatting rule is the string literal;

formatting, by a formatting module implemented by the hardware server, said first output data and said second output data according to the formatting rule; and

outputting said first output data and said second output data having been formatted.

2. The method ofclaim 1 further comprising storing, in a target data source, said first output data and said second output data having been outputted.

3. The method ofclaim 1 wherein said first string literal is included in said first output data if a token associated with a second formatting rule component to the immediate left of said first formatting rule component exists.

4. The method ofclaim 1 wherein said first string literal is included in said first output data if a token associated with a second formatting rule component to the immediate right of said first formatting rule component exists.

5. The method ofclaim 1 wherein a first token included in said first output data is associated with an index, wherein a second token included in said first output data is associated with the index.

6. The method ofclaim 5 wherein said first token included in said first output data and said second token included in said first output data are separated by a predetermined string.

7. A non-transitory computer-readable medium containing instructions for controlling a computer system to perform a method for data cleansing using rule based formatting, the method comprising:

obtaining a first input data and a second input data, wherein said first input data is tokenized according to a data dictionary, wherein said second input data is tokenized according to the data dictionary;

parsing said first input data and said second input data using a predefined parsing rule including an option operator, wherein the option operator indicates that a particular index defined in the predefined parsing rule is optional;

including a second token in a second output data if said first formatting rule component in the formatting rule is a second valid index to said second tokenized input data, wherein said second token is associated with said second valid index and including a second string literal in said second output data if said first formatting rule component in the formatting rule is the string literal; and

formatting said first output data and said second output data according to the formatting rule.

8. The non-transitory computer-readable medium ofclaim 7 wherein the first tokenized input data is obtained from a first data source, and the second tokenized input data is obtained from a second data source.

9. The non-transitory computer-readable medium ofclaim 7 further comprising storing said first output data in a target data source, and storing said second output data in said target data source.

10. The non-transitory computer-readable medium ofclaim 7 wherein said first string literal is included in said first output data if a token associated with a second formatting rule component to the immediate left of said first formatting rule component exists.

11. The non-transitory computer-readable medium ofclaim 7 wherein said first string literal is included in said first output data if a token associated with a second formatting rule component to the immediate right of said first formatting rule component exists.

12. The non-transitory computer-readable medium ofclaim 7 wherein said first string literal is included in said first output data if a first token associated with a second formatting rule component to the immediate left of said first formatting rule component and a second token associated with a third formatting rule component to the immediate right of said first formatting rule component both exist.

13. The non-transitory computer-readable medium ofclaim 7 wherein a first token included in said first output data is associated with an index, wherein a second token included in said first output data is associated with the index.

14. The non-transitory computer-readable medium ofclaim 13 wherein said first token included in said first output data and said second token included in said first output data are separated by a predetermined string.

15. A system for data cleansing using rule based formatting, comprising:

a hardware server that implements the system;

a tokenizing module, implemented by the hardware server, that tokenizes a first input data according to a data dictionary, and tokenizes a second input data according to said data dictionary;

a rule-based parsing module, implemented by the hardware server, that parses said first input data and said second tokenized input data using a predefined parsing rule including an option operator, wherein the option operator indicates that a particular index defined in the predefined parsing rule is optional;

a formatting module, implemented by the hardware server, that receives the said first tokenized input data and said second tokenized input data,

wherein a first token is included in a first output data if a first formatting rule component in a formatting rule is a first valid index to said first tokenized input data, wherein said first token is associated with said first valid index, wherein a first string literal is included in said first output data if said first formatting rule component in the formatting rule is a string literal, and wherein said formatting rule includes an immediate at least one conditional format operator, wherein the at least one conditional format operator indicates whether to include a particular string literal in an output data based on the existence of a particular token,

wherein a second token is included in a second output data if said first formatting rule component in the formatting rule is a second valid index to said second tokenized input data, wherein said second token is associated with said second valid index, wherein a second string literal is included in said second output data if said first formatting rule component in the formatting rule is a string literal, and

wherein said first output data and said second output data are formatted according to the formatting rule;

a first data source that stores said first input data;

a second data source that stores said second input data; and

a third data source that stores said first output data and said second output data.

16. The system ofclaim 15 further comprising a rule repository that stores said formatting rule and said predefined parsing rule.

17. The system ofclaim 16, further comprising a memory that stores at least one of said formatting rule and said predefined parsing rule.

18. The system ofclaim 15 further comprising a data dictionary repository that stores said data dictionary.

19. A computer-implemented method for data cleansing using rule based formatting, comprising:

parsing, by a rule-based parsing module implemented by a hardware server, said first input data and said second input data using a predefined parsing rule;

obtaining a formatting rule, wherein said formatting rule includes one or more formatting rule components;

outputting said first output data and said second output data having been formatted,

wherein said first string literal is included in said first output data if a first token associated with a second formatting rule component to the immediate left of said first formatting rule component and a second token associated with a third formatting rule component to the immediate right of said first formatting rule component both exist, and

wherein said second formatting rule component corresponds to a left-facing conditional format operator, and wherein said third formatting rule component corresponds to a right-facing conditional format operator.