• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels

we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm, while yielding good performance in a wide range of noisy label scenarios.

• Visualizing the Loss Landscape of Neural Nets

In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature, and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.

• A Probabilistic U-Net for Segmentation of Ambiguous Images

a group of graders typically produces a set of diverse but plausible segmentations. We consider the task of learning a distribution over segmentations given an input. To this end we propose a generative segmentation model based on a combination of a U-Net with a conditional variational autoencoder that is capable of efficiently producing an unlimited number of plausible hypotheses.

• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model.

• Step Size Matters in Deep Learning

Step size of the algorithm plays a critical role in these behaviors: it determines the subset of the local optima that the algorithm can converge to, and it specifies the magnitude of the oscillations if the algorithm converges to an orbit. To elucidate the effects of the step size on the training of neural networks, we study the gradient descent algorithm as a discrete-time dynamical system, and by analyzing the Lyapunov stability of different solutions

• Reducing Network Agnostophobia

We also introduce a new evaluation metric that focuses on comparing the performance of multiple approaches in scenarios where such unseen classes or unknowns are encountered. Our major contributions are simple yet effective Entropic Open-Set and Objectosphere losses that train networks using negative samples from some classes. These novel losses are designed to maximize entropy for unknown inputs while increasing separation in deep feature space by modifying magnitudes of known and unknown samples.

• Natasha 2: Faster Non-Convex Optimization Than SGD

We design a stochastic algorithm to find \varepsilon-approximate local minima of any smooth nonconvex function in rate O(\varepsilon^{-3.25}), with only oracle access to stochastic gradients. The best result before this work was O(\varepsilon^{-4}) by stochastic gradient descent (SGD).

• Norm matters: efficient and accurate normalization schemes in deep networks

In this work, we present a novel view on the purpose and function of normalization methods and weight-decay, as tools to decouple weights' norm from the underlying optimized objective. This property highlights the connection between practices such as normalization, weight decay and learning-rate adjustments.

• Contour location via entropy reduction leveraging multiple information sources
• Evidential Deep Learning to Quantify Classification Uncertainty

We provide a preliminary analysis on how the peculiarities of our new loss function drive improved uncertainty estimation. We observe that our method achieves unprecedented success on detection of out-of-distribution queries and endurance against adversarial perturbations.

• Predictive Uncertainty Estimation via Prior Networks

• FishNet: A Versatile Backbone for Image, Region, and Pixel Level Prediction