- Notifications
You must be signed in to change notification settings - Fork126
E2E POC for python telemetry for connect logs#581
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Merged
Changes fromall commits
Commits
Show all changes
269 commits Select commitHold shift + click to select a range
46af92e [ES-402013] Close cursors before closing connection (#38)
a9e6828 Bump version to 2.0.5 and improve CHANGELOG (#40)
d244b0b fix dco issue
moderakh69a40c8 fix dco issue
moderakh402f2ca dco tunning
moderakh2b638af dco tunning
moderakh7d5906b Github workflows: run checks on pull requests from forks (#47)
dbe5ced OAuth implementation (#15)
moderakh8e9c763 Automate deploys to Pypi (#48)
5cb62c4 [PECO-205] Add functional examples (#52)
508cf57 Bump version to 2.1.0 (#54)
4c34b19 [SC-110400] Enabling compression in Python SQL Connector (#49)
mohitsingla-db4f930d1 Add tests for parameter sanitisation / escaping (#46)
680809e Bump thrift dependency to 0.16.0 (#65)
2b8aba6 Bump version to 2.2.0 (#66)
bc7c122 Support Python 3.11 (#60)
c137fc5 Bump version to 2.2.1 (#70)
3c135a4 Add none check on _oauth_persistence in DatabricksOAuthProvider (#71)
jackyhu-db73432dc Support custom oauth client id and redirect port (#75)
jackyhu-db2f6a818 Bump version to 2.2.2 (#76)
jackyhu-dba41810c Merge staging ingestion into main (#78)
37d5454 Bump version to 2.3.0 and update changelog (#80)
7e46c89 Add pkgutil-style for the package (#84)
lu-wang-dlcbd10f2 Add SQLAlchemy Dialect (#57)
10fa788 Bump to version 2.4.0(#89)
c1fc318 Fix syntax in examples in root readme. (#92)
shea-parkesff45a2b Less strict numpy and pyarrow dependencies (#90)
d04fa2d Update example in docstring so query output is valid Spark SQL (#95)
bab7175 Bump version to 2.4.1 (#96)
dfd3322 Update CODEOWNERS (#97)
moderakhf9d4566 Add Andre to CODEOWNERS (#98)
yunbodeng-db9f9801a Add external auth provider + example (#101)
andrefurlan-db0a04cb7 Retry on connection timeout (#103)
andrefurlan-dbe1ba222 [PECO-244] Make http proxies work (#81)
09e3ed6 Bump to version 2.5.0 (#104)
2453efd Fix changelog release date for version 2.5.0
dc825b7 Relax sqlalchemy requirement (#113)
4411ee2 Update to version 2.5.1 (#114)
e6c8841 Fix SQLAlchemy timestamp converter + docs (#117)
600932d Relax pandas and alembic requirements (#119)
cf4df43 Bump to version 2.5.2 (#118)
281fed8 Use urllib3 for thrift transport + reuse http connections (#131)
2a9ec09 Default socket timeout to 15 min (#137)
mattdeekayc71b68d Bump version to 2.6.0 (#139)
7855ce9 Fix: some thrift RPCs failed with BadStatusLine (#141)
8c93500 Bump version to 2.6.1 (#142)
7da5eda [ES-706907] Retry GetOperationStatus for http errors (#145)
9e1eb01 Bump version to 2.6.2 (#147)
ae079ac [PECO-626] Support OAuth flow for Databricks Azure (#86)
jackyhu-db9f8f9a3 Use a separate logger for unsafe thrift responses (#153)
86c9adb Improve e2e test development ergonomics (#155)
a9eb4dd Don't raise exception when closing a stale Thrift session (#159)
996cda4 Bump to version 2.7.0 (#161)
363a82b Cloud Fetch download handler (#127)
mattdeekay62ba2b8 Cloud Fetch download manager (#146)
mattdeekay38f2ee0 Cloud fetch queue and integration (#151)
mattdeekayf4f0510 Cloud Fetch e2e tests (#154)
mattdeekay4b99316 Update changelog for cloudfetch (#172)
mattdeekayb826c7f Improve sqlalchemy backward compatibility with 1.3.24 (#173)
5cc4a21 OAuth: don't override auth headers with contents of .netrc file (#122)
1e5a948 Fix proxy connection pool creation (#158)
sebbegg11090a9 Relax pandas dependency constraint to allow ^2.0.0 (#164)
itsdani6d33d80 Use hex string version of operation ID instead of bytes (#170)
1493347 SQLAlchemy: fix has_table so it honours schema= argument (#174)
7f45fef Fix socket timeout test (#144)
mattdeekay20bd3b0 Disable non_native_boolean_check_constraint (#120)
bkyryliuk8bd6367 Remove unused import for SQLAlchemy 2 compatibility (#128)
WilliamGentryc3f7107 Bump version to 2.8.0 (#178)
133d2cc Fix typo in python README quick start example (#186)
dbarrundia-tigerb69604f Configure autospec for mocked Client objects (#188)
1b20055 Use urllib3 for retries (#182)
450c1bd Bump version to 2.9.0 (#189)
b23d5bd Explicitly add urllib3 dependency (#191)
jacobus-hermana3ad6c4 Bump to 2.9.1 (#195)
de9be67 Make backwards compatible with urllib3~=1.0 (#197)
c325a9c Convenience improvements to v3 retry logic (#199)
21d616d Bump version to 2.9.2 (#201)
b7de520 Github Actions Fix: poetry install fails for python 3.7 tests (#208)
57aa100 Make backwards compatible with urllib3~=1.0 [Follow up #197] (#206)
a765748 Bump version to 2.9.3 (#209)
98e002f Add note to sqlalchemy example: IDENTITY isn't supported yet (#212)
9f20316 [PECO-1029] Updated thrift compiler version (#216)
nithinkdb19bd63a [PECO-1055] Updated thrift defs to allow Tsparkparameters (#220)
nithinkdbbb33511 Update changelog to indicate that 2.9.1 and 2.9.2 have been yanked. (…
31e2c6d Fix changelog typo: _enable_v3_retries (#225)
997c266 Introduce SQLAlchemy reusable dialog tests (#125)
unj1m398a08b [PECO-1026] Add Parameterized Query support to Python (#217)
nithinkdbe508773 Parameterized queries: Add e2e tests for inference (#227)
0f3105c [PECO-1109] Parameterized Query: add suport for inferring decimal typ…
8e78fb4 SQLAlchemy 2: reorganise dialect files into a single directory (#231)
f73d610 [PECO-1083] Updated thrift files and added check for protocol version…
nithinkdbbe4e054 [PECO-840] Port staging ingestion behaviour to new UC Volumes (#235)
ab21a42 Query parameters: implement support for binding NoneType parameters (…
c75f6fe SQLAlchemy 2: Bump dependency version and update e2e tests for existi…
45011b1 Revert "[PECO-1083] Updated thrift files and added check for protocol…
d4db4fa SQLAlchemy 2: add type compilation for all CamelCase types (#238)
51c99cb SQLAlchemy 2: add type compilation for uppercase types (#240)
f6b317b SQLAlchemy 2: Stop skipping all type tests (#242)
fd11df8 [PECO-1134] v3 Retries: allow users to bound the number of redirects …
9608db9 Parameters: Add type inference for BIGINT and TINYINT types (#246)
596f971 SQLAlchemy 2: Stop skipping some non-type tests (#247)
f3657ee SQLAlchemy 2: implement and refactor schema reflection methods (#249)
b65d72c Add GovCloud domain into AWS domains (#252)
jackyhu-db4b15bde SQLAlchemy 2: Refactor __init__.py into base.py (#250)
2a34ade SQLAlchemy 2: Finish implementing all of ComponentReflectionTest (#251)
b8919a3 SQLAlchemy 2: Finish marking all tests in the suite (#253)
3a8ef6d SQLAlchemy 2: Finish organising compliance test suite (#256)
d799418 SQLAlchemy 2: Fix failing mypy checks from development (#257)
f0dea6c Enable cloud fetch by default (#258)
c505cea [PECO-1137] Reintroduce protocol checking to Python test fw (#248)
nithinkdbee23215 sqla2 clean-up: make sqlalchemy optional and don't mangle the user-ag…
36850e7 SQLAlchemy 2: Add support for TINYINT (#265)
06267ba Add OAuth M2M example (#266)
jackyhu-db06bd616 Native Parameters: reintroduce INLINE approach with tests (#267)
378f395 Document behaviour of executemany (#213)
martinitus936641c SQLAlchemy 2: Expose TIMESTAMP and TIMESTAMP_NTZ types to users (#268)
ac7544f Drop Python 3.7 as a supported version (#270)
cb91a2a GH Workflows: remove Python 3.7 from the matrix for _all_ workflows (…
c2a0cac Add README and updated example for SQLAlchemy usage (#273)
00e76b4 Rewrite native parameter implementation with docs and tests (#281)
4c43fa0 Enable v3 retries by default (#282)
c1c017f security: bump pyarrow dependency to 14.0.1 (#284)
6e06bd1 Bump package version to 3.0.0 (#285)
83e8565 Fix docstring about default parameter approach (#287)
Falydoorbe68e2f [PECO-1286] Add tests for complex types in query results (#293)
56779de sqlalchemy: fix deprecation warning for dbapi classmethod (#294)
e8a274b [PECO-1297] sqlalchemy: fix: can't read columns for tables containing…
a7a7cce Prepared 3.0.1 release (#297)
9d12913 Make contents of `__init__.py` equal across projects (#304)
pietern211b986 Fix URI construction in ThriftBackend (#303)
NodeJSmithf1213ff [sqlalchemy] Add table and column comment support (#329)
55b48aa Pin pandas and urllib3 versions to fix runtime issues in dbt-databric…
benc-dbb1eea30 SQLAlchemy: TINYINT types didn't reflect properly (#315)
TimTheinAtTabs577d1e9 [PECO-1435] Restore `tests.py` to the test suite (#331)
26b81c1 Bump to version 3.0.2 (#335)
a697b2a Update some outdated OAuth comments (#339)
jackyhu-db01a8473 Redact the URL query parameters from the urllib3.connectionpool logs …
mkazia-db76c16fc Bump to version 3.0.3 (#344)
jackyhu-dba7d0f6e [PECO-1411] Support Databricks OAuth on GCP (#338)
jackyhu-db4cc8225 [PECO-1414] Support Databricks native OAuth in Azure (#351)
jackyhu-db82d1e39 Prep for Test Automation (#352)
benc-dba4a7ff8 Update code owners (#345)
yunbodeng-db23e56a9 Reverting retry behavior on 429s/503s to how it worked in 2.9.3 (#349)
benc-dbae14739 Bump to version 3.1.0 (#358)
jackyhu-dbd040f32 [PECO-1440] Expose current query id on cursor object (#364)
kravets-levko74a7c48 Add a default for retry after (#371)
benc-db1dbf808 Fix boolean literals (#357)
aholyokeaad4439 Don't retry network requests that fail with code 403 (#373)
0971a3f Bump to 3.1.1 (#374)
benc-db0a0404d Fix cookie setting (#379)
benc-dba208f55 Fixing a couple type problems: how I would address most of #381 (#382)
wyattscarpenter6e877bf fix the return types of the classes' __enter__ functions (#384)
wyattscarpentereca1485 Add Kravets Levko to codeowners (#386)
kravets-levkoa9590b8 Prepare for 3.1.2 (#387)
benc-db4a82f2e Update the proxy authentication (#354)
amir-haroun34b0d2a Fix failing tests (#392)
kravets-levkod40eda5 Relax `pyarrow` pin (#389)
dhirschfeld72cbc75 Fix log error in oauth.py (#269)
susodapop50c6891 Enable `delta.feature.allowColumnDefaults` for all tables (#343)
dhirschfelde530c53 Fix SQLAlchemy tests (#393)
kravets-levkoc7cbd50 Add more debug logging for CloudFetch (#395)
kravets-levko8e74098 Update Thrift package (#397)
m1n01a863d9 Prepare release 3.2.0 (#396)
kravets-levko91481cc move py.typed to correct places (#403)
wyattscarpenter9d72daf Upgrade mypy (#406)
wyattscarpenter46a8989 Do not retry failing requests with status code 401 (#408)
Hodnebo9822ba3 [PECO-1715] Remove username/password (BasicAuth) auth option (#409)
jackyhu-dbf8d2630 [PECO-1751] Refactor CloudFetch downloader: handle files sequentially…
kravets-levkobb84016 Fix CloudFetch retry policy to be compatible with all `urllib3` versi…
kravets-levko3a7d7a4 Disable SSL verification for CloudFetch links (#414)
kravets-levko7e5155a Prepare relese 3.3.0 (#415)
kravets-levko6c39c30 Fix pandas 2.2.2 support (#416)
kfollesdal24eca04 [PECO-1801] Make OAuth as the default authenticator if no authenticat…
jackyhu-db1a598cd [PECO-1857] Use SSL options with HTTPS connection pool (#425)
kravets-levko906bae8 Prepare release v3.4.0 (#430)
kravets-levko354a7c4 [PECO-1926] Create a non pyarrow flow to handle small results for the…
jprakash-db72a77be [PECO-1961] On non-retryable error, ensure PySQL includes useful info…
shivam2680a8323c5 Reformatted all the files using black (#448)
jprakash-db46f7e06 Prepare release v3.5.0 (#457)
jackyhu-dbb424e2e [PECO-2051] Add custom auth headers into cloud fetch request (#460)
jackyhu-db67cb66a Prepare release 3.6.0 (#461)
jackyhu-dbb81a10e [ PECO - 1768 ] PySQL: adjust HTTP retry logic to align with Go and N…
jprakash-dbff7c284 [ PECO-2065 ] Create the async execution flow for the PySQL Connector…
jprakash-db0f1c7b8 Fix for check_types github action failing (#472)
jprakash-dbe9a82ea Remove upper caps on dependencies (#452)
arredondca410cc Updated the doc to specify native parameters in PUT operation is not …
jprakash-db78ad2e0 Incorrect rows in inline fetch result (#479)
jprakash-db17b56a3 Bumped up to version 3.7.0 (#482)
jprakash-db99782c3 PySQL Connector split into connector and sqlalchemy (#444)
jprakash-db1fd7bd9 Removed CI CD for python3.8 (#490)
jprakash-dbfe8a609 Added CI CD upto python 3.12 (#491)
jprakash-db1b925cc Merging changes from v3.7.1 release (#488)
jprakash-db9f8b89d Bumped up to version 4.0.0 (#493)
jprakash-db420878d Updated action's version (#455)
newwingbird4d84361 Support Python 3.13 and update deps (#510)
dhirschfeldcda81bb Improve debugging + fix PR review template (#514)
samikshya-db59ee2ac Forward porting all changes into 4.x.x. uptil v3.7.3 (#529)
jprakash-db3c9576f Updated the CODEOWNERS (#531)
jprakash-dba947750 Add version check for urllib3 in backoff calculation (#526)
shivam26801031637 [ES-1372353] make user_agent_header part of public API (#530)
shivam2680a6fbb83 Updates runner used to run DCO check to use databricks-protected-runn…
madhav-db5bf56aa Support multiple timestamp formats in non arrow flow (#533)
jprakash-dbc0d687c prepare release for v4.0.1 (#534)
shivam26809718ae6 Relaxed bound for python-dateutil (#538)
jprakash-db8959bba Bumped up the version for 4.0.2 (#539)
jprakash-db5326cb7 Added example for async execute query (#537)
jprakash-db4f195f8 Added urllib3 version check (#547)
jprakash-dbb9b9498 Bump version to 4.0.3 (#549)
jprakash-dbc43d480 Cleanup fields as they might be deprecated/removed/change in the futu…
vikrantpuppala9b0638e Refactor decimal conversion in PyArrow tables to use direct casting (…
jayantsing-db3cd70e2 [PECOBLR-361] convert column table to arrow if arrow present (#551)
shivam2680cc09200 Update CODEOWNERS (#562)
jprakash-db6354c97 Enhance Cursor close handling and context manager exception managemen…
madhav-dbd527a9d PECOBLR-86 improve logging on python driver (#556)
saishreeeeeb5d8c2b Update github actions run conditions (#569)
jprakash-db0dc49ff New Complex type test table + Github Action changes (#575)
jprakash-db24c48f1 Added classes required for telemetry (#572)
saishreeeee7d3ee9d telemetry client and logs
saishreeeee8c1dbeb fixed error args
saishreeeee01f19e1 fixed type of client_app_name
saishreeeee18e4cd0 formatting
saishreeeee71a4232 TelemetryManager
saishreeeee3232079 fixed statement_id in latency_logger
saishreeeeeba19a90 updated latency_logger
saishreeeee3b246e4 only added initial telemetry logs
saishreeeeed02c57a removed error logs logic
saishreeeee1e6246f removed user logging for telemetry errors
saishreeeee9887f9d removed no-op telemetry client
saishreeeee1ceb658 changed order of args in telemetry client
saishreeeeed8674e4 shifted thread pool executor to telemetry manager
saishreeeee3c76876 formatting
saishreeeee73953a1 Revert "shifted thread pool executor to telemetry manager"
saishreeeee8ec7db1 telemetry client changes
saishreeeeed1fcad4 Merge remote-tracking branch 'origin/telemetry' into PECOBLR-457
saishreeeee1256bb6 fixed spelling error
saishreeeee1c1c1be restructured telemetry client
saishreeeee44fade7 unit test for telemetry client
saishreeeee8a871e3 modified get_telemetry_client and some comments
saishreeeeefa6274e changed driver system config
saishreeeee5f8dff5 added log statements
saishreeeee906c187 changed TelemetryClientFactory to a static class and made changes in …
saishreeeeef6bd9ad added lock in TelemetryClientFactory and specified dict type in _client
saishreeeeec47bc73 changed type of attribute _executor in TelemetryClientFactory
saishreeeee4a2386b removed discovery_url
saishreeeeee4c05d1 removed unused params from driver connection params, initialize_telem…
saishreeeeebf92c6c renamed other as client_cert in auth_mech
saishreeeeee218ef9 lazy logging, renamed private variables, defined telemetry endpoints …
saishreeeee9615519 changed unittest to pytest
saishreeeeec084846 shifted EnumEncoder to telemetry/utils.py
saishreeeee7394295 changed type of _clients in TelemetryClientFactory
saishreeeeeFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
There are no files selected for viewing
36 changes: 36 additions & 0 deletionssrc/databricks/sql/client.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -49,6 +49,15 @@ | ||
| TSparkParameter, | ||
| TOperationState, | ||
| ) | ||
| from databricks.sql.telemetry.telemetry_client import ( | ||
| TelemetryClientFactory, | ||
| TelemetryHelper, | ||
| ) | ||
| from databricks.sql.telemetry.models.enums import DatabricksClientType | ||
| from databricks.sql.telemetry.models.event import ( | ||
| DriverConnectionParameters, | ||
| HostDetails, | ||
| ) | ||
| logger = logging.getLogger(__name__) | ||
| @@ -294,6 +303,31 @@ def read(self) -> Optional[OAuthToken]: | ||
| kwargs.get("use_inline_params", False) | ||
| ) | ||
| TelemetryClientFactory.initialize_telemetry_client( | ||
| telemetry_enabled=self.telemetry_enabled, | ||
| connection_uuid=self.get_session_id_hex(), | ||
| auth_provider=auth_provider, | ||
| host_url=self.host, | ||
| ) | ||
| self._telemetry_client = TelemetryClientFactory.get_telemetry_client( | ||
| connection_uuid=self.get_session_id_hex() | ||
| ) | ||
| driver_connection_params = DriverConnectionParameters( | ||
| http_path=http_path, | ||
| mode=DatabricksClientType.THRIFT, | ||
| host_info=HostDetails(host_url=server_hostname, port=self.port), | ||
| auth_mech=TelemetryHelper.get_auth_mechanism(auth_provider), | ||
| auth_flow=TelemetryHelper.get_auth_flow(auth_provider), | ||
| socket_timeout=kwargs.get("_socket_timeout", None), | ||
| ) | ||
| self._telemetry_client.export_initial_telemetry_log( | ||
| driver_connection_params=driver_connection_params, | ||
saishreeeee marked this conversation as resolved. Show resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| user_agent=useragent_header, | ||
| ) | ||
| def _set_use_inline_params_with_warning(self, value: Union[bool, str]): | ||
| """Valid values are True, False, and "silent" | ||
| @@ -430,6 +464,8 @@ def _close(self, close_cursors=True) -> None: | ||
| self.open = False | ||
| self._telemetry_client.close() | ||
| def commit(self): | ||
| """No-op because Databricks does not support transactions""" | ||
| pass | ||
9 changes: 4 additions & 5 deletionssrc/databricks/sql/telemetry/models/enums.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
31 changes: 12 additions & 19 deletionssrc/databricks/sql/telemetry/models/event.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
9 changes: 5 additions & 4 deletionssrc/databricks/sql/telemetry/models/frontend_logs.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.