Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit759401c

Browse files
authored
Cloud Fetch e2e tests (#154)
* Cloud Fetch e2e testsSigned-off-by: Matthew Kim <11141331+mattdeekay@users.noreply.github.com>* Test case works for e2-dogfood shared unity catalogSigned-off-by: Matthew Kim <11141331+mattdeekay@users.noreply.github.com>* Moving test to LargeQueriesSuite and setting catalog to hive_metastoreSigned-off-by: Matthew Kim <11141331+mattdeekay@users.noreply.github.com>* Align default value of buffer_size_bytes in driver testsSigned-off-by: Matthew Kim <11141331+mattdeekay@users.noreply.github.com>* Adding comment to specify what's needed to run successfullySigned-off-by: Matthew Kim <11141331+mattdeekay@users.noreply.github.com>---------Signed-off-by: Matthew Kim <11141331+mattdeekay@users.noreply.github.com>
1 parent5a34a4a commit759401c

File tree

1 file changed

+33
-1
lines changed

1 file changed

+33
-1
lines changed

‎tests/e2e/test_driver.py‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
importitertools
12
fromcontextlibimportcontextmanager
23
fromcollectionsimportOrderedDict
34
importdatetime
@@ -52,6 +53,7 @@ def __init__(self, method_name):
5253
# If running in local mode, just use environment variables for params.
5354
self.arguments=os.environifget_args_from_envelse {}
5455
self.arraysize=1000
56+
self.buffer_size_bytes=104857600
5557

5658
defconnection_params(self,arguments):
5759
params= {
@@ -84,7 +86,7 @@ def connection(self, extra_params=()):
8486
@contextmanager
8587
defcursor(self,extra_params=()):
8688
withself.connection(extra_params)asconn:
87-
cursor=conn.cursor(arraysize=self.arraysize)
89+
cursor=conn.cursor(arraysize=self.arraysize,buffer_size_bytes=self.buffer_size_bytes)
8890
try:
8991
yieldcursor
9092
finally:
@@ -104,6 +106,36 @@ def get_some_rows(self, cursor, fetchmany_size):
104106
else:
105107
returnNone
106108

109+
@skipUnless(pysql_supports_arrow(),'needs arrow support')
110+
deftest_cloud_fetch(self):
111+
# This test can take several minutes to run
112+
limits= [100000,300000]
113+
threads= [10,25]
114+
self.arraysize=100000
115+
# This test requires a large table with many rows to properly initiate cloud fetch.
116+
# e2-dogfood host > hive_metastore catalog > main schema has such a table called store_sales.
117+
# If this table is deleted or this test is run on a different host, a different table may need to be used.
118+
base_query="SELECT * FROM store_sales WHERE ss_sold_date_sk = 2452234 "
119+
fornum_limit,num_threads,lz4_compressioninitertools.product(limits,threads, [True,False]):
120+
withself.subTest(num_limit=num_limit,num_threads=num_threads,lz4_compression=lz4_compression):
121+
cf_result,noop_result=None,None
122+
query=base_query+"LIMIT "+str(num_limit)
123+
withself.cursor({
124+
"use_cloud_fetch":True,
125+
"max_download_threads":num_threads,
126+
"catalog":"hive_metastore"
127+
})ascursor:
128+
cursor.execute(query)
129+
cf_result=cursor.fetchall()
130+
withself.cursor({
131+
"catalog":"hive_metastore"
132+
})ascursor:
133+
cursor.execute(query)
134+
noop_result=cursor.fetchall()
135+
assertlen(cf_result)==len(noop_result)
136+
foriinrange(len(cf_result)):
137+
assertcf_result[i]==noop_result[i]
138+
107139

108140
# Exclude Retry tests because they require specific setups, and LargeQueries too slow for core
109141
# tests

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp