![48マルチGPU学習のパフォーマンスNVIDIA DGX-1, Chainer 1.17.0 with multi-process patch0123456780 1 2 3 4 5 6 7 8Speed-upto1GPUNumber of GPUsAlexNet VGG-D ResNet[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:1200.511.522.51 2 4 8Relativetimeto1GPUNumber of GPUsTime per one batch (VGG-D)UpdateAllreduceBackwardForwardDGX-1’s NVLink is not well utilized.Chainer’s all-reduce implementationis naïve “gather and broadcat”.](/image.pl?url=https%3a%2f%2fimage.slidesharecdn.com%2f20170421tensorflowusergroup-170425052716%2f75%2f20170421-tensor-flowusergroup-43-2048.jpg&f=jpg&w=240)
![49マルチGPU学習のパフォーマンス(NCCL使用なし) )NVIDIA DGX-1, Chainer 1.17.0 with multi-process patch024680 2 4 6 8Number of GPUsScalabilityResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers)024680 2 4 6 8024680 2 4 6 8Gather & Bcast[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12](/image.pl?url=https%3a%2f%2fimage.slidesharecdn.com%2f20170421tensorflowusergroup-170425052716%2f75%2f20170421-tensor-flowusergroup-44-2048.jpg&f=jpg&w=240)
![53NCCLの実装• 1 CPU and 4 GPUs (PCIe)Ring AlgorithmMost collectives amenable to bandwidth-optimalimplementation on rings, and many topologyies can beinterpreted as one or more rings [P. Patarasuk and X. Yuan]](/image.pl?url=https%3a%2f%2fimage.slidesharecdn.com%2f20170421tensorflowusergroup-170425052716%2f75%2f20170421-tensor-flowusergroup-48-2048.jpg&f=jpg&w=240)
![54NCCLの実装• 2 CPUs and 8 GPUs (QPI and PCIe)Ring AlgorithmMost collectives amenable to bandwidth-optimalimplementation on rings, and many topologyies can beinterpreted as one or more rings [P. Patarasuk and X. Yuan]](/image.pl?url=https%3a%2f%2fimage.slidesharecdn.com%2f20170421tensorflowusergroup-170425052716%2f75%2f20170421-tensor-flowusergroup-49-2048.jpg&f=jpg&w=240)
![56Multi-GPU performance w/o NCCLNVIDIA DGX-1, Chainer 1.17.0 with multi-process patch024680 2 4 6 8Number of GPUsScalabilityResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers)024680 2 4 6 8024680 2 4 6 8Gather & Bcast[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12](/image.pl?url=https%3a%2f%2fimage.slidesharecdn.com%2f20170421tensorflowusergroup-170425052716%2f75%2f20170421-tensor-flowusergroup-51-2048.jpg&f=jpg&w=240)
![57Multi-GPU performance with NCCLNVIDIA DGX-1, Chainer 1.17.0 with NCCL patch024680 2 4 6 8Number of GPUsScalabilityResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers)024680 2 4 6 8024680 2 4 6 8NCCL (4-ring)NCCL (1-ring)Gather & Bcast[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12](/image.pl?url=https%3a%2f%2fimage.slidesharecdn.com%2f20170421tensorflowusergroup-170425052716%2f75%2f20170421-tensor-flowusergroup-52-2048.jpg&f=jpg&w=240)
Recommended
![[db analytics showcase Sapporo 2017] B14: GPU コンピューティング最前線 by エヌビディア 佐々木邦暢](/image.pl?url=https%3a%2f%2fcdn.slidesharecdn.com%2fss_thumbnails%2f20170630dbassprnvidia-170707074715-thumbnail.jpg%3fwidth%3d640%26amp%3bheight%3d640%26amp%3bfit%3dbounds&f=jpg&w=240)
20170421 tensor flowusergroup
- 1.エヌビディア合同会社ディープラーニング ソリューションアーキテクト兼CUDAエンジニア 村上真奈TensorFlowUser Group ハード部 #2TensorFlow+GPUディープラーニング
- 2.2自己紹介村上真奈(むらかみまな) / mmurakami@nvidia.com•CUDAエンジニア＋ディープラーニングSA• ディープラーニング・CUDA技術サポートとか、いろいろ埼玉県さいたま市早稲田大学教育学部理学科数学⇒システム計画研究所⇒サムスン日本研究所⇒エヌビディア画像処理(主に静止画)、ソフトの最適化とか、プリクラとか放送機器とかテレビとか2010年頃に初めてCUDAにふれる(CUDA1.XXとかの時代)NVIDIAGPUComputingNVIDIAJapan@NVIDIAJapan
- 3.
- 4.
- 5.
- 6.72012 20142008 20102016 20184836120246072TeslaFermiKeplerMaxwellPascal混合精度演算倍精度演算3D メモリNVLinkVoltaGPU ロードマップSGEMM/W
- 7.8Tesla P100 SXM2(CC 6.0)3584 CUDA CoresFP64: 5.3 TFFP32: 10.6 TFFP16: 21.2 TFINT8: …HBM24096 bit width16 GB732 GB/s
- 8.9Tesla P100 SXM2(CC 6.0)3584 CUDA CoresFP64: 5.3 TFFP32: 10.6 TFFP16: 21.2 TFINT8: …HBM24096 bit width16 GB732 GB/sSMCUDAコア．．．
- 9.10Tesla P100 SXM2(CC 6.0)3584 CUDA CoresFP64: 5.3 TFFP32: 10.6 TFFP16: 21.2 TFINT8: …HBM24096 bit width16 GB732 GB/s
- 10.11GPUアーキテクチャ概要PCI I/Fホスト接続インタフェースGiga ThreadEngineSMに処理を割り振るスケジューラDRAM (384-bit, GDDR5)全SM、PCI I/Fからアクセス可能なメモリ (デバイスメモリ, フレームバッファ)L2 cache (1.5MB)全SMからアクセス可能なR/WキャッシュSM (Streaming Multiprocessor)「並列」プロセッサPascal GP100
- 11.12SM (streaming multiprocessor)▪CUDA core▪ GPUスレッドはこの上で動作▪ Pascal: 64個▪ Other units▪ DP, LD/ST, SFU▪ Register File (65,536 x 32bit)▪ Shared Memory/L1 Cache(64KB)▪ Read-Only Cache(48KB)
- 12.13Register FileSchedulerDispatchSchedulerDispatchLoad/Store Unitsx 16Special Func Units x 4Interconnect Network64K ConfigurableCache/Shared MemUniform CacheCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreCoreInstruction CacheCompute Capability (CC)KeplerCC 3.5192 cores / SMXMaxwellCC 5.0128 cores / SMMFermiCC 2.032 cores / SMPascalCC 6.064 cores / SMMhttps://developer.nvidia.com/cuda-gpus
- 13.
- 14.
- 15.
- 16.17Image Recognition(Microsoft)認識精度向上のためモデルはよりディープに、データはより大きく強力な計算パワーが必要に2012AlexNet8 Layers1.4GFLOP~16% Error152 Layers22.6 GFLOP~3.5% Error2015ResNet16XModel2014Deep Speech 180 GFLOP7,000 hrs of Data~8% Error465 GFLOP12,000 hrs of Data~5% Error2015Deep Speech 210XTraining OpsSpeech Recognition(Baidu)
- 17.様々な問題への応用18Semantic segmentaionGANMedical imageanalysis using 3D convolution
- 18.GPUがディープラーニングに向いている理由1. 汎用アクセラレータである。しかも計算が高速▪ 様々なアプリケーションを走らせる事が可能。しかも高速に計算する事が出来る。2.開発時間が短い。最適化が簡単。▪ CUDA C/C++というC/C++の拡張言語で記述可能。学習コストも低く、簡単にアプリケーション開発が出来る。▪ 開発ツールが充実しており、プロファイラやデバッガを使えば、バグ修正や最適化が容易3. ハードウェアがディープラーニング向けに進化▪ NVLINKという高速なデータ転送インターコネクトをサポート(マルチGPUでの学習がより高速に)▪ GPU搭載メモリ量の増加(より複雑なネットワークを大きなバッチサイズで計算可能に)▪ その他いろいろ19
- 19.20K80 M40 M4P100(SXM2)P100(PCIE)P40P4GPU 2x GK210 GM200 GM206 GP100 GP100 GP102 GP104CUDA core 4992(2496*2) 3072 1024 3584 3584 3840 2560PEAK FP64 (TFLOPs) 2.9 NA NA 5.3 4.7 NA NAPEAK FP32 (TFLOPs) 8.7 7 2.2 10.6 9.3 12 5.5PEAK FP16 (TFLOPs) NA NA NA 21.2 18.7 NA NAPEAK TIOPs NA NA NA NA NA 47 22Memory Size 2x 12GB GDDR5 24 GB GDDR5 4 GB GDDR5 16 GB HBM2 16/12 GB HBM2 24 GB GDDR5 8 GB GDDR5Memory BW 480 GB/s 288 GB/s 80 GB/s 732 GB/s 732/549 GB/s 346 GB/s 192 GB/sInterconnect PCIe Gen3 PCIe Gen3 PCIe Gen3NVLINK +PCIe Gen3PCIe Gen3 PCIe Gen3 PCIe Gen3ECC Internal + GDDR5 GDDR5 GDDR5 Internal + HBM2 Internal + HBM2 GDDR5 GDDR5Form Factor PCIE Dual Slot PCIE Dual Slot PCIE LP SXM2 PCIE Dual Slot PCIE Dual Slot PCIE LPPower 300 W 250 W 50-75 W 300 W 250 W 250 W 50-75 WTesla製品一覧たくさんあるが、どれを使えばよいのか?
- 20.
- 21.
- 22.
- 23.24K80 M40 M4P100(SXM2)P100(PCIE)P40P4GPU 2x GK210 GM200 GM206 GP100 GP100 GP102 GP104CUDA core 4992(2496*2) 3072 1024 3584 3584 3840 2560PEAK FP64 (TFLOPs) 2.9 NA NA 5.3 4.7 NA NAPEAK FP32 (TFLOPs) 8.7 7 2.2 10.6 9.3 12 5.5PEAK FP16 (TFLOPs) NA NA NA 21.2 18.7 NA NAPEAK TIOPs NA NA NA NA NA 47 22Memory Size 2x 12GB GDDR5 24 GB GDDR5 4 GB GDDR5 16 GB HBM2 16/12 GB HBM2 24 GB GDDR5 8 GB GDDR5Memory BW 480 GB/s 288 GB/s 80 GB/s 732 GB/s 732/549 GB/s 346 GB/s 192 GB/sInterconnect PCIe Gen3 PCIe Gen3 PCIe Gen3NVLINK +PCIe Gen3PCIe Gen3 PCIe Gen3 PCIe Gen3ECC Internal + GDDR5 GDDR5 GDDR5 Internal + HBM2 Internal + HBM2 GDDR5 GDDR5Form Factor PCIE Dual Slot PCIE Dual Slot PCIE LP SXM2 PCIE Dual Slot PCIE Dual Slot PCIE LPPower 300 W 250 W 50-75 W 300 W 250 W 250 W 50-75 WTesla製品一覧推論 推論学習 学習
- 24.25K80 M40 M4P100(SXM2)P100(PCIE)P40P4GPU 2x GK210 GM200 GM206 GP100 GP100 GP102 GP104CUDA core 4992(2496*2) 3072 1024 3584 3584 3840 2560PEAK FP64 (TFLOPs) 2.9 NA NA 5.3 4.7 NA NAPEAK FP32 (TFLOPs) 8.7 7 2.2 10.6 9.3 12 5.5PEAK FP16 (TFLOPs) NA NA NA 21.2 18.7 NA NAPEAK TIOPs NA NA NA NA NA 47 22Memory Size 2x 12GB GDDR5 24 GB GDDR5 4 GB GDDR5 16 GB HBM2 16/12 GB HBM2 24 GB GDDR5 8 GB GDDR5Memory BW 480 GB/s 288 GB/s 80 GB/s 732 GB/s 732/549 GB/s 346 GB/s 192 GB/sInterconnect PCIe Gen3 PCIe Gen3 PCIe Gen3NVLINK +PCIe Gen3PCIe Gen3 PCIe Gen3 PCIe Gen3ECC Internal + GDDR5 GDDR5 GDDR5 Internal + HBM2 Internal + HBM2 GDDR5 GDDR5Form Factor PCIE Dual Slot PCIE Dual Slot PCIE LP SXM2 PCIE Dual Slot PCIE Dual Slot PCIE LPPower 300 W 250 W 50-75 W 300 W 250 W 250 W 50-75 WTesla製品一覧
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.39benchmark(DGX-1)• Two fullyconnected quads,connected at corners• 160GB/s per GPU bidirectional to Peers• Load/store access to Peer Memory• Full atomics to Peer GPUs• High speed copy engines for bulk datacopy• PCIe to/from CPUDGX-1Dual 20-core Intel® Xeon® E5-2698 v4 2.2 GHz8x Tesla GP100
- 36.40TensorFlowDeep Learning TrainingAnopen-source software library for numericalcomputation using data flow graphs.VERSION1.0ACCELERATED FEATURESFull framework acceleratedSCALABILITYMulti-GPU and multi-nodeMore Informationhttps://www.tensorflow.org/TensorFlow Deep Learning FrameworkTraining on 8x P100 GPU Server vs 8 x K80 GPU Server-1.02.03.04.05.0Speedupvs.Serverwith8xK80AlexNet GoogleNet ResNet-50 ResNet-152 VGG162.5xAvg. Speedup3xAvg. SpeedupGPU Servers: Single Xeon E5-2690 v4@2.6GHz with GPUs configs as shownUbuntu 14.04.5, CUDA 8.0.42, cuDNN 6.0.5; NCCL 1.6.1, data set: ImageNet;batch sizes: AlexNet (128), GoogleNet (256), ResNet-50 (64), ResNet-152 (32), VGG-16 (32)Server with 8x P10016GB NVLinkServer with 8x P100PCIe 16GB
- 37.
- 38.
- 39.44データ並列(同期型)w w wLayer1 Layer 2Inputs Layer NLossFuncLossFuncGPU 1GPU 2“cat”Labels“monkey”w w wCopy Model, Assigne different data
- 40.45データ並列(同期型)wx ywx ywxyLayer 1“dog”Layer 2Inputs OutputsLayer NLossFunc“human”LossFuncGPU 1GPU 2“cat”Labels“monkey”error⊿y⊿x⊿y⊿x⊿y⊿w⊿w⊿wwx ywx ywx y⊿y⊿x⊿y⊿x⊿y⊿w⊿w⊿werrorForward & Backward Independently
- 41.46データ並列(同期型)wx ywx ywxyLayer 1“dog”Layer 2Inputs OutputsLayer NLossFunc“human”LossFuncGPU 1GPU 2“cat”Labels“monkey”error⊿y⊿x⊿y⊿x⊿y⊿w⊿w⊿wwx ywx ywx y⊿y⊿x⊿y⊿x⊿y⊿w⊿w⊿werrorCombine ⊿w over multi-GPU⊿w⊿w⊿w⊿w⊿w⊿wAll-reduceAll-reduceAll-reduce
- 42.47データ並列(同期型)wx ywx ywxyLayer 1“dog”Layer 2Inputs OutputsLayer NLossFunc“human”LossFuncGPU 1GPU 2“cat”Labels“monkey”error⊿y⊿x⊿y⊿x⊿y⊿w⊿w⊿wwx ywx ywx y⊿y⊿x⊿y⊿x⊿y⊿w⊿w⊿werrorUpdate Weights Independentlyw w ww w w
- 43.48マルチGPU学習のパフォーマンスNVIDIA DGX-1, Chainer1.17.0 with multi-process patch0123456780 1 2 3 4 5 6 7 8Speed-upto1GPUNumber of GPUsAlexNet VGG-D ResNet[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:1200.511.522.51 2 4 8Relativetimeto1GPUNumber of GPUsTime per one batch (VGG-D)UpdateAllreduceBackwardForwardDGX-1’s NVLink is not well utilized.Chainer’s all-reduce implementationis naïve “gather and broadcat”.
- 44.49マルチGPU学習のパフォーマンス(NCCL使用なし) )NVIDIA DGX-1,Chainer 1.17.0 with multi-process patch024680 2 4 6 8Number of GPUsScalabilityResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers)024680 2 4 6 8024680 2 4 6 8Gather & Bcast[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12
- 45.
- 46.51NCCL(NVIDIA Collective CollectionLibrary)マルチGPU集合通信ライブラリ• 最新リリースはv1.2.3• https://github.com/NVIDIA/ncclall-gather, reduce, broadcast など標準的な集合通信の処理をバンド幅が出るように最適化シングルプロセスおよびマルチプロセスで使用する事が可能ディープラーニング SDK
- 47.52NCCL(NVIDIA Collective CollectionLibrary)NCCLの集合通信処理
- 48.53NCCLの実装• 1 CPUand 4 GPUs (PCIe)Ring AlgorithmMost collectives amenable to bandwidth-optimalimplementation on rings, and many topologyies can beinterpreted as one or more rings [P. Patarasuk and X. Yuan]
- 49.54NCCLの実装• 2 CPUsand 8 GPUs (QPI and PCIe)Ring AlgorithmMost collectives amenable to bandwidth-optimalimplementation on rings, and many topologyies can beinterpreted as one or more rings [P. Patarasuk and X. Yuan]
- 50.55NCCL パフォーマンスBandwidth atdifferent problem sizes (4 Maxwell GPUs)All-GatherAll-ReduceReduce-ScatterBroadcast
- 51.56Multi-GPU performance w/oNCCLNVIDIA DGX-1, Chainer 1.17.0 with multi-process patch024680 2 4 6 8Number of GPUsScalabilityResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers)024680 2 4 6 8024680 2 4 6 8Gather & Bcast[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12
- 52.57Multi-GPU performance withNCCLNVIDIA DGX-1, Chainer 1.17.0 with NCCL patch024680 2 4 6 8Number of GPUsScalabilityResNet (152 layers)VGG-D (16 layers)AlexNet (7 layers)024680 2 4 6 8024680 2 4 6 8NCCL (4-ring)NCCL (1-ring)Gather & Bcast[Batch size per GPU] AlexNet:768, VGG-D:32, ResNet:12
- 53.58Multi-GPU performance withNCCLNVIDIA DGX-1, Chainer 1.17.0 with NCCL patch00.511.522.5…G&BNCCL(1-ring)NCCL(4-ring)G&BNCCL(1-ring)NCCL(4-ring)G&BNCCL(1-ring)NCCL(4-ring)1 GPU 2 GPUs 4 GPUs 8 GPUsRelativetimeto1GPUTime per one batch (VGG-D)UpdateAllreduceBackwardForward
- 54.
- 55.60nvidia-docker+コンテナでアプリケーションを起動GPU2 GPU3 GPU4GPU6 GPU7NVIDIA CUDA DriverDockerエンジンGPU5GPU0 GPU1ホストPCGPU0 GPU1CUDA LibrariesDockerコンテナ1CUDA 7.5 Runtimeアプリケーション1GPU0 GPU1 GPU2CUDA LibrariesDockerコンテナ2CUDA 8.0 Runtimeアプリケーション2GPU0 GPU1 GPU2CUDA LibrariesDockerコンテナ3CUDA 7.0 Runtimeアプリケーション3
- 56.61明示的にどのGPUを使うか指定して起動sudo NV_GPU=‘1,2’ nvidia-dockerrun –rm gcr.io/tensorflow/tensorflow:1.0.1nvidia-dockerでコンテナの起動GPU 1と２を使用(GPU0は使用しない) dockerコンテナ名
- 57.