应兰州大学数学与统计学院邀请, 江西师范大学计算机学院曾锦山教授将于2021年5月23日在城关校区西区学生活动中心506会议室举办专题学术报告.

报告题目：On ADMM in Deep Learning: Convergence and Saturation-Avoidance

时 间：2021年5月23日(星期天)上午11:10-11:50

线下地点: 兰州大学城关校区西区学生活动中心506会议室

报告摘要：

In this talk, we develop an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called sigmoid-ADMM pair), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of deep neural networks with sigmoid-type activations (called deep sigmoid nets) over their rectified linear unit (ReLU) counterparts (called deep ReLU nets) in terms of approximation. In particular, we prove that the approximation capability of deep sigmoid nets is not worse than deep ReLU nets by showing that ReLU activation fucntion can be well approximated by deep sigmoid nets with two hidden layers and finitely many free parameters but not vice-verse. We also establish the global convergence of the proposed ADMM for the nonlinearly constrained formulation of the deep sigmoid nets training from arbitrary initial points to a Karush-Kuhn-Tucker (KKT) point at a rate of order O(1/k). Besides sigmoid activation, such a convergence theorem holds for a general class of smooth activations. Compared with the widely used stochastic gradient descent (SGD) algorithm for the deep ReLU nets training (called ReLU-SGD pair), the proposed sigmoid-ADMM pair is practically stable with respect to the algorithmic hyperparameters including the learning rate, initial schemes and the pro-processing of the input data. Moreover, we find that to approximate and learn simple but important functions the proposed sigmoid-ADMM pair numerically outperforms the ReLU-SGD pair.

曾锦山教授简介

曾锦山,系江西师范大学计算机信息工程学院于2015年引进的优秀海归博士,2018、2020年两次获得世界华人数学家大会（ICCM）最佳论文奖并受邀作45分钟学术报告.曾锦山老师多年来聚焦于人工智能应用中的优化算法理论研究,在相关研究领域的重要期刊和会议上发表高水平论文30余篇,其中SCI论文20余篇,IEEE Transactions系列论文10篇,ESI热点论文1篇,CCF A类论文3篇.论文近五年被引用近千次,单篇最高引用逾450次.

甘肃省高校应用数学与复杂系统省级重点实验室

兰州大学数学与统计学院

兰州大学萃英学院

2021年5月20日