Document AI generates evaluation metrics, such as precision and recall, to helpyou determine the predictive performance of your processors.

These evaluation metrics are generated by comparing the entities returned by theprocessor (the predictions) against the annotations in the test documents. Ifyour processor does not have a test set, then you must firstcreate a dataset andlabel the test documents.

Run an evaluation

An evaluation is automatically run whenever you train or uptrain a processorversion.

You can also manually run an evaluation. This is required to generate updatedmetrics after you've modified the test set, or if you are evaluating apretrained processor version.

fromgoogle.api_core.client_optionsimportClientOptionsfromgoogle.cloudimportdocumentai# type: ignore# TODO(developer): Uncomment these variables before running the sample.# project_id = 'YOUR_PROJECT_ID'# location = 'YOUR_PROCESSOR_LOCATION' # Format is 'us' or 'eu'# processor_id = 'YOUR_PROCESSOR_ID'# processor_version_id = 'YOUR_PROCESSOR_VERSION_ID'# gcs_input_uri = # Format: gs://bucket/directory/defevaluate_processor_version_sample(project_id:str,location:str,processor_id:str,processor_version_id:str,gcs_input_uri:str,)->None:# You must set the api_endpoint if you use a location other than 'us', e.g.:opts=ClientOptions(api_endpoint=f"{location}-documentai.googleapis.com")client=documentai.DocumentProcessorServiceClient(client_options=opts)# The full resource name of the processor version# e.g. `projects/{project_id}/locations/{location}/processors/{processor_id}/processorVersions/{processor_version_id}`name=client.processor_version_path(project_id,location,processor_id,processor_version_id)evaluation_documents=documentai.BatchDocumentsInputConfig(gcs_prefix=documentai.GcsPrefix(gcs_uri_prefix=gcs_input_uri))# NOTE: Alternatively, specify a list of GCS Documents## gcs_input_uri = "gs://bucket/directory/file.pdf"# input_mime_type = "application/pdf"## gcs_document = documentai.GcsDocument(#     gcs_uri=gcs_input_uri, mime_type=input_mime_type# )# gcs_documents = [gcs_document]# evaluation_documents = documentai.BatchDocumentsInputConfig(#     gcs_documents=documentai.GcsDocuments(documents=gcs_documents)# )#request=documentai.EvaluateProcessorVersionRequest(processor_version=name,evaluation_documents=evaluation_documents,)# Make EvaluateProcessorVersion request# Continually polls the operation until it is complete.# This could take some time for larger filesoperation=client.evaluate_processor_version(request=request)# Print operation details# Format: projects/PROJECT_NUMBER/locations/LOCATION/operations/OPERATION_IDprint(f"Waiting for operation{operation.operation.name} to complete...")# Wait for operation to completeresponse=documentai.EvaluateProcessorVersionResponse(operation.result())# After the operation is complete,# Print evaluation ID from operation responseprint(f"Evaluation Complete:{response.evaluation}")

Get results of an evaluation

fromgoogle.api_core.client_optionsimportClientOptionsfromgoogle.cloudimportdocumentai# type: ignore# TODO(developer): Uncomment these variables before running the sample.# project_id = 'YOUR_PROJECT_ID'# location = 'YOUR_PROCESSOR_LOCATION' # Format is 'us' or 'eu'# processor_id = 'YOUR_PROCESSOR_ID' # Create processor before running sample# processor_version_id = 'YOUR_PROCESSOR_VERSION_ID'# evaluation_id = 'YOUR_EVALUATION_ID'defget_evaluation_sample(project_id:str,location:str,processor_id:str,processor_version_id:str,evaluation_id:str,)->None:# You must set the api_endpoint if you use a location other than 'us', e.g.:opts=ClientOptions(api_endpoint=f"{location}-documentai.googleapis.com")client=documentai.DocumentProcessorServiceClient(client_options=opts)# The full resource name of the evaluation# e.g. `projects/{project_id}/locations/{location}/processors/{processor_id}/processorVersions/{processor_version_id}`evaluation_name=client.evaluation_path(project_id,location,processor_id,processor_version_id,evaluation_id)# Make GetEvaluation requestevaluation=client.get_evaluation(name=evaluation_name)create_time=evaluation.create_timedocument_counters=evaluation.document_counters# Print the Evaluation Information# Refer to https://cloud.google.com/document-ai/docs/reference/rest/v1beta3/projects.locations.processors.processorVersions.evaluations# for more information on the available evaluation dataprint(f"Create Time:{create_time}")print(f"Input Documents:{document_counters.input_documents_count}")print(f"\tInvalid Documents:{document_counters.invalid_documents_count}")print(f"\tFailed Documents:{document_counters.failed_documents_count}")print(f"\tEvaluated Documents:{document_counters.evaluated_documents_count}")

List all evaluations for a processor version

fromgoogle.api_core.client_optionsimportClientOptionsfromgoogle.cloudimportdocumentai# type: ignore# TODO(developer): Uncomment these variables before running the sample.# project_id = 'YOUR_PROJECT_ID'# location = 'YOUR_PROCESSOR_LOCATION' # Format is 'us' or 'eu'# processor_id = 'YOUR_PROCESSOR_ID' # Create processor before running sample# processor_version_id = 'YOUR_PROCESSOR_VERSION_ID'deflist_evaluations_sample(project_id:str,location:str,processor_id:str,processor_version_id:str)->None:# You must set the api_endpoint if you use a location other than 'us', e.g.:opts=ClientOptions(api_endpoint=f"{location}-documentai.googleapis.com")client=documentai.DocumentProcessorServiceClient(client_options=opts)# The full resource name of the processor version# e.g. `projects/{project_id}/locations/{location}/processors/{processor_id}/processorVersions/{processor_version_id}`parent=client.processor_version_path(project_id,location,processor_id,processor_version_id)evaluations=client.list_evaluations(parent=parent)# Print the Evaluation Information# Refer to https://cloud.google.com/document-ai/docs/reference/rest/v1beta3/projects.locations.processors.processorVersions.evaluations# for more information on the available evaluation dataprint(f"Evaluations for Processor Version{parent}")forevaluationinevaluations:print(f"Name:{evaluation.name}")print(f"\tCreate Time:{evaluation.create_time}\n")

Evaluation metrics for all labels

Metrics forAll labels are computed based on the number of true positives,false positives, and false negatives in the dataset across all labels, and thus,are weighted by the number of times each label appears in the dataset. Fordefinitions of these terms, seeEvaluation metrics for individual labels.

• Precision: the proportion of predictions that match the annotations inthe test set. Defined asTrue Positives / (True Positives + FalsePositives)

• Recall: the proportion of annotations in the test set that are correctlypredicted. Defined asTrue Positives / (True Positives + False Negatives)

• F1 score: the harmonic mean of precision and recall, which combinesprecision and recall into a single metric, providing equal weight to both.Defined as2 * (Precision * Recall) / (Precision + Recall)

• True Positives: the predicted entities that match an annotation in thetest document. For more information, seematching behavior.

• False Positives: the predicted entities that don't match any annotationin the test document.

• False Negatives: the annotations in the test document that don't matchany of the predicted entities.

  • False Negatives (Below Threshold): the annotations in the testdocument that would have matched a predicted entity, but the predictedentity's confidence value is below the specified confidence threshold.

Confidence threshold

The evaluation logic ignores any predictions with confidence below the specifiedConfidence Threshold, even if the prediction is correct. Document AIprovides a list ofFalse Negatives (Below Threshold), which are theannotations that would have a match if the confidence threshold were set lower.

Document AI automatically computes theoptimal threshold, which maximizesthe F1 score, and by default, sets the confidence threshold to this optimalvalue.

The optimal threshold for the parent is theconfidence threshold value that, when applied to allchildren, yields the maximum F1 score for the parent.

Matching behavior

A predicted entity matches an annotation if:

• the type of the predicted entity(entity.type)matches the annotation's label name
• the value of the predicted entity(entity.mention_textorentity.normalized_value.text)matches the annotation's text value, subject tofuzzy matching if it is enabled.

Note that type and text value are all that is used for matching. Otherinformation, such as text anchors and bounding boxes (with the exception oftabular entities described below) are not used.

Single- versus multi-occurrence labels

Single-occurrence labels have one value per document (for example, invoice ID)even if that value is annotated multiple times in the same document (forexample, the invoice ID appears in every page of the same document). Even if themultiple annotations have different text, they are considered equal. In otherwords, if a predicted entity matches any of the annotations, it counts as amatch. The extra annotations are consideredduplicate mentions and don'tcontribute towards any of the true positive, false positive, or false negativecounts.

Multi-occurrence labels can have multiple, different values. Thus, eachpredicted entity and annotation is considered and matched separately. If adocument contains N annotations for a multi-occurrence label, then there can beN matches with the predicted entities. Each predicted entity and annotation areindependently counted as a true positive, false positive, or false negative.

Fuzzy Matching

TheFuzzy Matching toggle lets you tighten or relax some of the matchingrules to decrease or increase the number of matches.

For example, without fuzzy matching, the stringABC does not matchabc dueto capitalization. But with fuzzy matching, they match.

When fuzzy matching is enabled, here are the rule changes:

• Whitespace normalization: removes leading-trailing whitespace andcondenses consecutive intermediate whitespaces (including newlines) intosingle spaces.

• Leading/trailing punctuation removal: removes the followingleading-trailing punctuation characters!,.:;-"?|.

• Case-insensitive matching: converts all characters to lowercase.

• Money normalization: For labels with the data typemoney, remove theleading-trailing currency symbols.

1. In the Google Cloud console, go to theProcessors page and choose yourprocessor.

   Go to the Processors page

2. In theEvaluate & Test tab, clickDownload Metrics, to download theevaluation metrics as a JSON file.

Monitoring dashboard

The monitoring dashboard in the Google Cloud console provides a useful way to createyour own monitoring visualizations for different metrics and resources used inDocument AI processors.

List of metrics and their fields:Number of successfully processed pages andTotal number of docs sent to processing both have the following fields:location,processor type,processor_id,processor_version_id,status,andprocessing_type.

Sync processing latency,Batch processing latency per document, andBatch processing operation count have the following fields:location,processor_type,processor_id,processor_version_id, andstatus.

Create a monitoring view

You can monitor your processors individually and collectively, with a perprocessor view and a per project view. The steps for these are the same.

1. In the Google Cloud console, in the Document AI section, go to theProcessors page.

   Go to the Processors page

2. Optional: If you want to monitor a specific processor, select it from the list.

3. Use the navigation pane to select theMonitoringmonitoring option.

4. Set theTime range.

5. Fill the checkboxes for the selected metric and resource labels, and selectOk.

   Tip: You can hold the pointer over the trendlines and series legends to highlight them.

6. Optional: You can see more options for each graph with theMenumore_vert option.