Amusingly, it currently says that the file sizes are off by one: it sees one fewer bytes than Files.size() reports. This is also noticeably slower than gsutil (perhaps because it's single-threaded?).

Sample output:

$ target/appassembler/bin/CountBytes <redacted>/dbsnp_138.b37.1.1-65M.vcf<redacted>/dbsnp_138.b37.1.1-65M.vcf: 237432747 bytes.Reading the whole file...Read all 237432746 bytes in 8s. (6 calls to chan.read)Wait, this doesn't match! We saw 237432746 bytes, yet the file size is listed at 237432747 bytes.

jean-philippe-martin added2 commits

April 27, 2016 10:04

CountBytes

f7b5724

Larger buffer improves performance

3943e17

googlebot added the cla: yesThis human has signed the Contributor License Agreement. label

May 2, 2016

Copy link

Contributor

mziccard commentedMay 3, 2016

Let me start with general comments:

Amusingly, it currently says that the file sizes are off by one: it sees one fewer bytes than Files.size() reports.

This is not true. This error comes from the way you loop and computetotal size:

longtotal =0;intreadCalls =0;for (intread =1;read >0; ) {readCalls ++;read =chan.read(buf);buf.flip();total +=read;// On the last call you read -1 and sum it to total size, thus the mismatch}

In general the for loop seems a bit strange to me, I would have used something like (not tested):

while (chan.read(buf) >0) {readCalls++;total +=buf.position();buf.flip();}readCalls++;// We must count the last call

This is also noticeably slower than gsutil (perhaps because it's single-threaded?).

What does this mean? Is it slower at reading the size metadata or at downloading the file? Can you share some numbers?

mziccard reviewed

May 3, 2016

View reviewed changes

gcloud-java-examples/src/main/java/com/google/cloud/examples/nio/CountBytes.java

		@@ -0,0 +1,82 @@
		package com.google.cloud.examples.nio;

This comment was marked as spam.

Copy link

Author

jean-philippe-martin commentedMay 3, 2016•
edited
Loading

Amusingly, it currently says that the file sizes are off by one: it sees one fewer bytes than Files.size() reports.
This is not true. This error comes from the way you loop and compute total size:

Thank you! Indeed it's obvious now.

This is also noticeably slower than gsutil (perhaps because it's single-threaded?).
What does this mean? Is it slower at reading the size metadata or at downloading the file? Can you share some numbers?

Downloading the file. Here's what I see:

$ time gsutil cp gs://<redacted>/dbsnp_138.b37.256m.vcf /tmp/Copying gs://<redacted>/dbsnp_138.b37.256m.vcf...Downloading file:///tmp/dbsnp_138.b37.256m.vcf:                  214.89 MiB/214.89 MiB    Downloading file:///tmp/dbsnp_138.b37.256m.vcf:                  214.89 MiB/214.89 MiB    Downloading file:///tmp/dbsnp_138.b37.256m.vcf:                  214.89 MiB/214.89 MiB    Downloading file:///tmp/dbsnp_138.b37.256m.vcf:                  214.89 MiB/214.89 MiB    real    0m11.984suser    0m7.618ssys 0m4.513s$ target/appassembler/bin/CountBytes gs://<redacted>dbsnp_138.b37.256m.vcfgs://<redacted>/dbsnp_138.b37.256m.vcf: 901305932 bytes.Reading the whole file...Read all 901305931 bytes in 42s. (19 calls to chan.read)

We can see that gsutil takes 1/4th of the time. The way it prints it suggests that it did 4 parallel downloads (each of 1/4th the size).

Fix byte counting

a9b294c

Added licence text, plus some esthetic changes.

mziccard reviewed

May 3, 2016

View reviewed changes

gcloud-java-examples/src/main/java/com/google/cloud/examples/nio/CountBytes.java Outdated

		/**
		* CountBytes will read through the whole file given as input.
		*
		* <p>It's meant for testing that NIO doesn't read too slowly.

This comment was marked as spam.

Comment improvement

53d7b41

Copy link

Author

jean-philippe-martin commentedMay 4, 2016

Are we OK to merge?

mziccard merged commit7ea0fc8 intogoogleapis:gcs-nio

May 4, 2016

Copy link

Contributor

mziccard commentedMay 4, 2016

@jean-philippe-martin just merged. As usual: thanks!

Copy link

Author

jean-philippe-martin commentedMay 4, 2016

You are welcome!

jean-philippe-martin deleted the nio_countbytes branch

May 4, 2016 20:19

jean-philippe-martin mentioned this pull request

May 4, 2016

Add ParallelCountBytes#983

Merged

mziccard pushed a commit to mziccard/gcloud-java that referenced this pull request

Jun 29, 2016

NIO CountBytes example (googleapis#975)

767fbae

mziccard pushed a commit to mziccard/gcloud-java that referenced this pull request

Jun 30, 2016

NIO CountBytes example (googleapis#975)

9e94824

github-actionsbot pushed a commit to suztomo/google-cloud-java that referenced this pull request

Jun 29, 2022

feat: add ListSavedQueries rpc to aiplatform v1 dataset_service.proto (…

ab9ba69

…googleapis#975)- [ ] Regenerate this pull request now.feat: add saved_query.proto to aiplatform v1feat: add saved_query_id to InputDataConfig in aiplatform v1 training_pipeline.protoPiperOrigin-RevId: 456872211Source-Link:googleapis/googleapis@88f3973Source-Link:https://github.com/googleapis/googleapis-gen/commit/daf8d1cb20fadabdd8914581da1ceeef048a34f7Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZGFmOGQxY2IyMGZhZGFiZGQ4OTE0NTgxZGExY2VlZWYwNDhhMzRmNyJ9feat: add ListSavedQueries rpc to aiplatform v1beta1 dataset_service.protofeat: add saved_query.proto to aiplatform v1beta1feat: add saved_query_id to InputDataConfig in aiplatform v1beta1 training_pipeline.protoPiperOrigin-RevId: 456872207Source-Link:googleapis/googleapis@aeb384aSource-Link:https://github.com/googleapis/googleapis-gen/commit/c5c8894abbfcd1f6b9d38d9e9fd70a18f15cf0ccCopy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzVjODg5NGFiYmZjZDFmNmI5ZDM4ZDllOWZkNzBhMThmMTVjZjBjYyJ9

github-actionsbot pushed a commit to suztomo/google-cloud-java that referenced this pull request

Jul 1, 2022