Commitb804373

Jim Fulton

authored

fix: use a larger chunk size when loading data (#799)

* The chunk size used for data uploads was too small (1MB). Now it's 100MB.* fix: The chunk size used for data uploads was too small

1 parentf0990f2 commitb804373Copy full SHA for b804373

File tree

+21

-1

lines changed

+21

-1

lines changed

Lines changed: 1 addition & 1 deletion

Original file line number	Diff line number	Diff line change
`@@ -98,7 +98,7 @@`
`98`	`98`	`fromgoogle.cloud.bigquery.tableimportRowIterator`
`99`	`99`
`100`	`100`
`101`		`-_DEFAULT_CHUNKSIZE=1048576#1024 * 1024B = 1 MB`
	`101`	`+_DEFAULT_CHUNKSIZE=10010241024# 100 MB`
`102`	`102`	`_MAX_MULTIPART_SIZE=510241024`
`103`	`103`	`_DEFAULT_NUM_RETRIES=6`
`104`	`104`	`_BASE_UPLOAD_TEMPLATE="{host}/upload/bigquery/v2/projects/{project}/jobs?uploadType="`

Lines changed: 20 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -8076,3 +8076,23 @@ def test_schema_to_json_with_file_object(self):`
`8076`	`8076`
`8077`	`8077`	`client.schema_to_json(schema_list,fake_file)`
`8078`	`8078`	`assertfile_content==json.loads(fake_file.getvalue())`
	`8079`	`+`
	`8080`	`+`
	`8081`	`+deftest_upload_chunksize(client):`
	`8082`	`+withmock.patch("google.cloud.bigquery.client.ResumableUpload")asRU:`
	`8083`	`+upload=RU.return_value`
	`8084`	`+`
	`8085`	`+upload.finished=False`
	`8086`	`+`
	`8087`	`+deftransmit_next_chunk(transport):`
	`8088`	`+upload.finished=True`
	`8089`	`+result=mock.MagicMock()`
	`8090`	`+result.json.return_value= {}`
	`8091`	`+returnresult`
	`8092`	`+`
	`8093`	`+upload.transmit_next_chunk=transmit_next_chunk`
	`8094`	`+f=io.BytesIO()`
	`8095`	`+client.load_table_from_file(f,"foo.bar")`
	`8096`	`+`
	`8097`	`+chunk_size=RU.call_args_list[0][0][1]`
	`8098`	`+assertchunk_size==100* (1<<20)`

Comments

(0)