Movatterモバイル変換


[0]ホーム

URL:


Skip to main content
Cornell University

Monday, May 5: arXiv will be READ ONLY at 9:00AM EST for approximately 30 minutes. We apologize for any inconvenience.

We gratefully acknowledge support from the Simons Foundation,member institutions, and all contributors.Donate
arxiv logo>cs> arXiv:2209.03695
arXiv logo
Cornell University Logo

Computer Science > Machine Learning

arXiv:2209.03695 (cs)
[Submitted on 8 Sep 2022 (v1), last revised 15 Jan 2023 (this version, v3)]

Title:Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes

View PDF
Abstract:A fundamental property of deep learning normalization techniques, such as batch normalization, is making the pre-normalization parameters scale invariant. The intrinsic domain of such parameters is the unit sphere, and therefore their gradient optimization dynamics can be represented via spherical optimization with varying effective learning rate (ELR), which was studied previously. However, the varying ELR may obscure certain characteristics of the intrinsic loss landscape structure. In this work, we investigate the properties of training scale-invariant neural networks directly on the sphere using a fixed ELR. We discover three regimes of such training depending on the ELR value: convergence, chaotic equilibrium, and divergence. We study these regimes in detail both on a theoretical examination of a toy example and on a thorough empirical analysis of real scale-invariant deep learning models. Each regime has unique features and reflects specific properties of the intrinsic loss landscape, some of which have strong parallels with previous research on both regular and scale-invariant neural networks training. Finally, we demonstrate how the discovered regimes are reflected in conventional training of normalized networks and how they can be leveraged to achieve better optima.
Comments:Published in NeurIPS 2022. First three authors contributed equally
Subjects:Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as:arXiv:2209.03695 [cs.LG]
 (orarXiv:2209.03695v3 [cs.LG] for this version)
 https://doi.org/10.48550/arXiv.2209.03695
arXiv-issued DOI via DataCite

Submission history

From: Ekaterina Lobacheva Ms [view email]
[v1] Thu, 8 Sep 2022 10:30:05 UTC (21,514 KB)
[v2] Mon, 17 Oct 2022 18:19:17 UTC (25,252 KB)
[v3] Sun, 15 Jan 2023 13:52:37 UTC (24,980 KB)
Full-text links:

Access Paper:

  • View PDF
  • TeX Source
  • Other Formats
Current browse context:
cs.LG
Change to browse by:
export BibTeX citation

Bookmark

BibSonomy logoReddit logo

Bibliographic and Citation Tools

Bibliographic Explorer(What is the Explorer?)
Connected Papers(What is Connected Papers?)
scite Smart Citations(What are Smart Citations?)

Code, Data and Media Associated with this Article

CatalyzeX Code Finder for Papers(What is CatalyzeX?)
Hugging Face(What is Huggingface?)
Papers with Code(What is Papers with Code?)

Demos

Hugging Face Spaces(What is Spaces?)

Recommenders and Search Tools

Influence Flower(What are Influence Flowers?)
CORE Recommender(What is CORE?)
IArxiv Recommender(What is IArxiv?)

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?Learn more about arXivLabs.

Which authors of this paper are endorsers? |Disable MathJax (What is MathJax?)

[8]ページ先頭

©2009-2025 Movatter.jp