Movatterモバイル変換

May 13th, 2019Tatsuya ShirakawaSeeing Unseens with Machine Learning ⾒えていないものを⾒出す機械学習Tech-on MeetUp#06 — What can “AI (I)” do?

Tatsuya Shirakawa2ABEJA, Inc. (Researcher)- Deep Learning (CV, Graph, NLP, )- Machine LearningGithub https://github.com/TatsuyaShirakawaNTT Data Mathematical Systems Inc.- Mathematical Optimization- Machine Learning / Deep LearningMath.Tech blog http://tech-blog.abeja.asia/- 異空間への埋め込み！Poincare Embeddingsが拓く表現学習の新展開- 機は熟した！グラフ構造に対するDeep Learning、Graph Convolutionのご紹介- 双曲空間でのMachine Learningの最近の進展- より良い機械学習のためのアノテーションの機械学習Now

Researchers at ABEJA31.2.3.4.先に⾒つけるシンプルに解く先に失敗する• 最新テクノロジーのキャッチアップ• 技術視点を交えた新しいビジネス構想• 独⾃技術の開発・検証• ⾼難易度タスクのコアロジックの構築• 技術ソリューションの提案• プロダクトの根本的な精度改善• アイデアの検証• 既存のやり⽅/考え⽅の再検討 視点を与える

AIの特性ふりかえり4MLについて⾔いたいことAIMLDL

Daniel KehnemanThere are two modes of thoughtSystem 1（勘・直感）fast, instinctive and emotionalSystem 2（論理的思考）Slower, more deliberative, and more logical5MLはコッチ

6(Andrew Ng, “AI Transformation Playbook”)AIは運⽤することで 改善する

AIは⼈間の代替ではない7Human AI同じ作業を続ける ✔スケールさせる ✔未知な状況への適応 ✔複雑な作業 ✔適材適所が⼤事

Today’s Talk1. Software 2.0 2. Bigger, Deeper and Better 3. Discovery 2.081. Software 2.0 2. Bigger, Deeper and Better 3. Discovery 2.0

Software 2.0Software 1.0 — Write a program that works- Explicit instructions to the computer which identifies a specific point inprogram space with some desirable behaviorSoftware 2.0 — Find a desirable program that fits to data- A rough skelton of the code (e.g. NNs) that identifies a subset of programspace to search- Search this space for a program that works10

Why Software 2.0?"it is significantly easier to collect the data (or more generally, identify adesirable behavior) than to explicitly write the program”11

Dogs and CatsWhy dogs are dogs and cats are cats?12

Gender, Age, Recognition is Not Trivial !13

Paradigm ChangeThings which is hard to define/code can be learn implicitly from data1400110110110 1101010101011010111011 1011011010001010111011Coding LearnSoftware

Today’s Talk1. Software 2.0 2. Bigger, Deeper and Better 3. Discovery 2.015

Bigger, Deeper and Better16Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018.9) BigGAN — 巨⼤な計算リソースで学習された巨⼤なモデルで⾼解像度画像の⽣成に成功。GPipe: Eﬃcient Training of Giant Neural Networks using Pipeline Parallelism (2018.11) GPipe — 巨⼤なNNを効率的に学習するための分散学習ライブラリ。ImageNetで新SOTA。BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018.10) BERT — 巨⼤なモデルを巨⼤なデータで教師なしすることで⾔語理解系タスクにたいする強⼒な初期モデルを獲得Language Models are Unsupervised Multitask Learners (2019.2)  GPT-2 — 巨⼤な⾔語モデルをクリーンで巨⼤なデータで学習し、⽂書⽣成系タスクをゼロショットで⾼精度にこなせるモデルを獲得

BigGAN — Class Conditionalな⾼解像度画像⽣成既存のSOTA⼿法（SA-GAN）に対して、バッチサイズやチャンネル数を増やし、各種⼯夫を加えることで、512x512のClass Conditionalな⾼精度画像⽣成に成功。既存SOTAを⼤きく上回るスコアを達成。17“Large Scale GAN Training for High Fidelity Natural Image Synthesis ”

GPipe — 巨⼤なNNの学習に最適化された分散学習ライブラリ複数のGPUを活⽤してForward/Backward計算をスケーラブルかつ効率的に⾏うライブラリ。これを⽤いて学習された巨⼤なモデルはImageNetで新たなSOTAを達成。18(“GPipe: Eﬃcient Training of Giant Neural Networks using Pipeline Parallelism”)

BERT — ⾔語理解系タスクでの強⼒な教師なし事前学習⼿法強⼒なモデル（BERT）を教師なしで構成できる下記の2タスクで事前学習することで さまざまな⾔語理解系タスクでSOTAを⼤幅更新19The cat [MASK] on the matsat1. 単語の⽳埋めGLUE test results (論⽂より）1. The man went to [MASK] store2. He bought a gallon [MASK] milk→ IsNext / NotNext?2. ⼆⽂が連続⽂かどうかの判定

GPT-2 — クリーンで多様なデータで学習された巨⼤な⾔語モデル信頼性の⾼いWebページをクローリングして得たクリーンで多様なコーパス（WebText）上で強⼒な⾔語モデル（GPT-2）を教師なし学習（尤度最⼤化）。 ⽂書⽣成系のさまざまなタスクのZero-shot学習でSOTAを更新20

21Winning WayLarger (Cleaner) Datasets  + Deeper Neural Networks

Today’s Talk1. Software 2.0 2. Bigger, Deeper and Better 3.Discovery 2.022

Can You See Gender/Age from Ears?230 10 20 30 40 50 60 70 80Age

Can You See Gender, Age and BMI from Eyes (Fundus)? How about Heart / Brain Diseases?240 10 20 30 40 50 60 70 80Age

DNNs Can See Gender/Age from EarsD. Yaman+, “Age and Gender Classification from Ear Images”, IWBF201825Age: 18-28 / 29-38 / 39-48 / 49-58 / 58-68+

DNNs Can See Gender, Age, BMI and even  Brain/Heart Diseases from EyesR. Poplin+, “Prediction of cardiovascular risk factors from retinal fundus  photographs via deep learning”, Nature Biomedical Engineering 201826

What happens?DNNs can see what human cannot see or recognize.27…

Discovery 2.0Discovery 1.0 — Fully Utilizing Domain Knowledge- explicit construction of hypothesis is constructed mainly fromdomain knowledge or deep understanding of the domainDiscovery 2.0 — Seeing by Training- capture some aspects of data by training models on it- not new but should be emphasized again28※ serendipity could be another source of discovery :)

Discovery 2.0 — Seeing by Training1. Seeing Predictability / Correlation29

Discovery 2.0 — Seeing by Training1. Seeing Predictability / CorrelationBeyond human imagination - Every data should be connect to create new connectionsCorrelation first- Correlation finding is the first goal- Causality should be checked post-hook if possibleRelatively cheap to apply if data exists- Models should have weak domain dependence (e.g. NNs)30

Graph Convolutional Neural Networks (GCNNs)A specific type of neural networks which is  designed for processing connectivity of data well31Tech blog http://tech-blog.abeja.asia/- 異空間への埋め込み！Poincare Embeddingsが拓く表現学習の新展開- 機は熟した！グラフ構造に対するDeep Learning、Graph Convolutionのご紹介- 双曲空間でのMachine Learningの最近の進展- より良い機械学習のためのアノテーションの機械学習

Task Relations — TaskonomyRelation = Transferability A. R. Zamir+, “Taskonomy: Disentangling Task Transfer Learning”, CVPR201832AutoencodingObject Class.Scene Class.CurvatureDenoisingOcclusion EdgesEgomotionCam. Pose (fix)2D Keypoint3D KeypointCam. Pose (nonfix)MatchingReshadingDistanceZ-DepthNormalsLayout2.5D Segm.2D Segm.Semantic Segm.Vanishing Pts.Novel Task 1Novel Task 2Novel Task 3AutoencodingObject Class.Scene Class.CurvatureDenoisingOcclusion EdgesEgomotionCam. Pose (fix)2D Keypoint3D KeypointCam. Pose (nonfix)MatchingReshadingDistanceZ-DepthNormalsLayout2.5D Segm.2D Segm.Semantic Segm.Vanishing Pts.Novel Task 1Novel Task 2Novel Task 3https://storage.googleapis.com/taskonomy_slides/taskonomy_slides.html

Discovery 2.0 — Seeing by Training1. Seeing Predictability / Correlation 2. Representation Learning / Embeddings33T. Mikolov+, “Distributed representation of words and phrases and their  compositionality, NeurIPS2013https://github.com/facebookresearch/poincare-embeddings

Hyperbolic Space• Manifolds with positive constant sectional curvature• Tree structure is naturally aligned in the space → automatic tree structure detection!34Tech blog http://tech-blog.abeja.asia/- 異空間への埋め込み！Poincare Embeddingsが拓く表現学習の新展開- 機は熟した！グラフ構造に対するDeep Learning、Graph Convolutionのご紹介- 双曲空間でのMachine Learningの最近の進展- より良い機械学習のためのアノテーションの機械学習「異空間散歩！双曲空間を歩いてみよう。」

Hyperbolic Embeddings35[M. Nickel+] Poincaré Embeddings ~17’ 05 18’ 04[C. D. Sa+] Representation Tradeoﬀ ~ (Near-exact tree embs., h-MDS)[O. Ganea+] Hyperbolic Entailment Cones (Poincare embs. + Order embs.)18’ 05[C. Gulcehre+] Hyperbolic Attention Networks (Einstein Mid. Point)[O. Ganea+] Hyperbolic Neural Networks[M. Nickel+] Learning Continuous Hierarchies in the Lorentz Model~[A. Tifrea+] Poincaré Glove ~(Poincaré Glove)18’ 06 18’ 10 …19’ 2[R. Suzuki+] Hyperbolic Disk Embeddings[A. Gu+] Mixed-Curvature  Representations18’ 9

Mixed-Curvature RepresentationsA. Gu+, “Learning Mixed-Curvature Representations in Products of model Spaces”, ICLR2019ユークリッド空間、球⾯、双曲空間の積空間への埋め込みを構成することで、様々な（断⾯）曲率の空間への埋込を可能にした。36

37データの構造が ⾒えた！

Discovery 2.0 — Seeing by Training1. Seeing Predictability / Correlation 2. Representation Learning / Embeddings※ Off course, domain/scientific knowledge is crucial for efficient/meaningful exploration38

What can AI (I) do?3900110110110 1101010101011010111011 1011011010001010111011Coding LearnSoftwareDiscoveryDomainKnowledgeLearnSoftware 1.0 Software 2.0Discovery 1.0Discovery 2.0

What can AI (I) do?40Psychology  x AI

⼈格⼼理学（Personality Psychology）Personality Psychology is a scientific study which aims to show how peopleare individually different due to psychological forces (wikipedia).41Personality Traits（特性） FeaturesPersonality Types（類型）Clustering / Classificationあなたは◯◯タイプ！ex) ex)

Big 5（Five Factor Model, FFM）421. Openness（経験への開放性） is a general appreciation for art, emotion, adventure,  unusual ideas, imagination, curiosity, and variety of experience2. Conscientiousness（誠実性） is a tendency to display self-discipline, act dutifully, and strive  for achievement against measures or outside expectations3. Extraversion（外向性） is characterized by breadth of activities (as opposed to depth), surgency  from external activity/situations, and energy creation from external means4. Agreeableness（協調性） trait reflects individual differences in general concern for social harmony5. Neuroticism（神経症的傾向） is the tendency to experience negative emotions, such as anger, anxiety, or depression(wikipedia)

Big5はすごい！• さまざまな研究で提案されたパーソナリティ特性との相関性が⾼い（事実上のデファクトスタンダード） • 英語辞書中のパーソナリティに関する単語と既存パーソナリティテストの結果を総合して作られた、けっこうデータ駆動な作られ⽅• 双⼦の研究によると、Big5の変動のだいたい50%は遺伝で、残りの50%は環境で決まっている（分散分析）• Big5のうち、Agreeablenessをのぞいた4つは、年をとってもあまり変化しない43

What can I do?1. Personality embeddings (in hyperbolic spaces?)2. Multimodal analysis (facial expressions, psychological measures, )3. More efficient assessor (like Akinator?)   脳科学、遺伝学、進化論、⼼理学、…、機械学習の融合領域！44

Movatterモバイル変換

Change Language

Seeing Unseens with Machine Learning --  見えていないものを見出す機械学習

Recommended

More Related Content

What's hot

Similar to Seeing Unseens with Machine Learning --  見えていないものを見出す機械学習

More from Tatsuya Shirakawa

Seeing Unseens with Machine Learning --  見えていないものを見出す機械学習

Movatterモバイル変換

Seeing Unseens with Machine Learning -- 見えていないものを見出す機械学習

Recommended

More Related Content

What's hot

Similar to Seeing Unseens with Machine Learning -- 見えていないものを見出す機械学習

More from Tatsuya Shirakawa

Seeing Unseens with Machine Learning -- 見えていないものを見出す機械学習

Seeing Unseens with Machine Learning --  見えていないものを見出す機械学習

Similar to Seeing Unseens with Machine Learning --  見えていないものを見出す機械学習

Seeing Unseens with Machine Learning --  見えていないものを見出す機械学習