- Notifications
You must be signed in to change notification settings - Fork126
Add functionality for export of latency logs via telemetry#608
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
…r operationsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
…ze and get telemetry clientSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
…pTelemetryClient class with NOOP_TELEMETRY_CLIENT singleton, updated tests accordinglySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Thanks for your contribution! To satisfy the DCO policy in ourcontributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Thanks for your contribution! To satisfy the DCO policy in ourcontributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Thanks for your contribution! To satisfy the DCO policy in ourcontributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
vikrantpuppala approved these changesJul 3, 2025
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Thanks for your contribution! To satisfy the DCO policy in ourcontributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
jprakash-db approved these changesJul 3, 2025
Contributor
jprakash-db left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM, Thanks for the making the changes
Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Thanks for your contribution! To satisfy the DCO policy in ourcontributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
0dfe0f4 intotelemetry 23 of 27 checks passed
Uh oh!
There was an error while loading.Please reload this page.
saishreeeee added a commit that referenced this pull requestJul 22, 2025
* send telemetry to unauth endpoint in case of connection/auth errorsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added unit test for send_connection_error_telemetrySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* retry errorsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* Add functionality for export of latency logs via telemetry (#608)* added functionality for export of failure logsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* changed logger.error to logger.debug in exc.pySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* Fix telemetry loss during Python shutdownSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* unit tests for export_failure_logSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* try-catch blocks to make telemetry failures non-blocking for connector operationsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed redundant try/catch blocks, added try/catch block to initialize and get telemetry clientSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* skip null fields in telemetry requestSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed dup import, renamed func, changed a filter_null_values to lamdaSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed unnecassary class variable and a redundant try/except blockSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* public functions defined at interface levelSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* changed export_event and flush to private functionsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* changed connection_uuid to thread local in thrift backendSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* made errors more specificSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* revert change to connection_uuidSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* reverting change in close in telemetry clientSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* JsonSerializableMixinSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* isdataclass check in JsonSerializableMixinSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* convert TelemetryClientFactory to module-level functions, replace NoopTelemetryClient class with NOOP_TELEMETRY_CLIENT singleton, updated tests accordinglySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* renamed connection_uuid as session_id_hexSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added NotImplementedError to abstract class, added unit testsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added PEP-249 link, changed NoopTelemetryClient implementationSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed unused importSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* made telemetry client close a module-level functionSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* unit tests verboseSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* debug logs in unit testsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* debug logs in unit testsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed ABC from mixin, added try/catch block around executor shutdownSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* checking stuffSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* finding out* finding out more* more more finding out more nice* locks are useless anyways* haha* normal* := looks like walrus horizontally* one more* walrus again* old stuff without walrus seems to fail* manually do the walrussing* change 3.13t, v2Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formatting, added walrusSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed walrus, removed test before stalling testSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* changed order of stalling testSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed debugging, added TelemetryClientFactorySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* remove more debuggingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* latency logs funcitionalitySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* fixed type of return value in get_session_id_hex() in thrift backendSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* debug on TelemetryClientFactory lockSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* type notation for _waitersSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* called connection.close() in test_arraysize_buffer_size_passthroughSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* run all unit testsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* more debuggingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed the connection.close() from that test, put debug statement before and after TelemetryClientFactory lockSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* more debugSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* more more moreSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* whySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* whywhySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* thread nameSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added teardown to all tests except finalizer test (gc collect)Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added the get_attribute functions to the classesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed tearDown, added connection.close() to first testSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* finallySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* remove debuggingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added test for export_latency_log, made mock of thrift backend with retry policySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added multi threaded testsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added TelemetryExtractor, removed multithreaded testsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* fixes in testSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* fix in telemetry extractorSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added doc strings to latency_logger, abstracted export_telemetry_logSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* statement type, unit test fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* unit test fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* statement type changesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* test_fetches fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added mocks to resolve the errors caused by log_latency decorator in testsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed function in test_fetches cuz it is only used onceSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* added _safe_call which returns None in case of errors in the get functionsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed the changes in test_client and test_fetchesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed the changes in test_fetchesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* test_telemetrySigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed testSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>---------Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* MaxRetryDurationErrorSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* main changesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* formattingSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* import jsonSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* without the max retry errorsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* unauth telemetry clientSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* remove duplicate code setting telemetry_enabledSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* removed unused errorsSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* merge with main changesSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* testSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* without try/catch blockSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* -Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* error log for auth provider, ThriftDatabricksClientSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* error log for session.openSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* retry tests fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* test connection failure logSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* check types fixSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* testSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>* rephrase importSigned-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>---------Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading.Please reload this page.
What type of PR is this?
Description
Add functionality for export of latency logs via telemetry
Example latency log:
{
"frontend_log_event_id": "30565427-1299-4fbc-bec0-25378cfe912b",
"context": {
"client_context": {
"timestamp_millis": 1750592858108,
"user_agent": "PyDatabricksSqlConnector/4.0.4"
}
},
"entry": {
"sql_driver_log": {
"session_id": "01f04f5e-b1a8-1dd1-962a-e40efd092bdd",
"system_configuration": {
"driver_version": "4.0.4",
"os_name": "Darwin",
"os_version": "24.5.0",
"os_arch": "arm64",
"runtime_name": "Python 3.13.3",
"runtime_version": "3.13.3",
"runtime_vendor": "CPython",
"driver_name": "Databricks SQL Python Connector",
"char_set_encoding": "utf-8",
"locale_name": "en_US"
},
"driver_connection_params": {
"http_path": "/sql/1.0/warehouses/864004c1b3961382",
"mode": "THRIFT",
"host_info": {
"host_url": "e2-dogfood.staging.cloud.databricks.com",
"port": 443
},
"auth_mech": "PAT"
},
"sql_statement_id": "01f04f5e-b1d6-1a1e-afe4-e99e1ccb8805",
"sql_operation": {
"statement_type": "sql",
"is_compressed": true,
"execution_result": "inline_arrow",
"retry_count": 0
},
"operation_latency_ms": 518
}
}
}
How is this tested?
Related Tickets & Documents
PECOBLR-554