- 原书名：System Parameter Identification: Information Criteria and Algorithms
- 原出版社： Elsevier
Yu Zhu received the B.S. degree in radio electronics in 1983 from Beijing Normal University, and the M.S. degree in computer applications in 1993 and the Ph.D. degree in mechanical design and theory in 2001, both from China University of Mining and Technology. He is currently a professor with the Department of Mechanical Engineering, Tsinghua University. His research field mainly covers IC manufacturing equil ment development strategy, ultra-precision air/maglev stage machinery design theory and technology, ultra precision measurement theory and technology, and pre cision motion control theory and technology. He has more than 140 research papers and 100 (48 awarded) invention patents.
Jinchun Hu, associate professor, born in 1972, graduated from Nanjing University of Science and Technology. He received the B.E. and Ph.D. degrees in control science and engineering in 1994 and 1998, respectively. Currently, he works at the Department of Mechanical Engineering, Tsinghua University. His current research interests include modem control theory and control systems, ultra-precision measurement princi ples and methods, micro/nano motion control system analysis and realization, special driver technology and device for precision motion systems, and super precision measurement and control.
Jose C. Principe is a distinguished professor of electrical and computer engineering and biomedical engineering at the University of Florida where he teaches advanced signal processing, machine learning, and artificial neural networks modeling. He is BellSouth Professor and the founding director of the University of Florida Computational NeuroEngineering Laboratory. His primary research interests are in advanced signal processing with information theoretic criteria (entropy and mutual information) and adaptive models in repro ducing kernel Hilbert spaces, and the application of these advanced algorithms to brain machine interfaces. He is a Fellow of the IEEE, ABME, and AIBME. He is the past editor in chief of the IEEE Transactions on Biomedical Engineering, past chair of the Technical Committee on Neural Networks of the 1EEE Signal Processing Society, and past President of the International Neural Network Society. He received the IEEE EMBS Career Award and the IEEE Neural Network Pioneer Award. He has more than 600 publications and 30 patents (awarded or filed).
About the Authors i
Symbols and Abbreviations v
1 Introduction 1
1.1 Elements of System Identification 1
1.2 Traditional Identification Criteria 3
1.3 Information Theoretic Criteria 4
1.3.1 MEE Criteria 6
1.3.2 Minimum Information Divergence Criteria 7
1.3.3 Mutual Information-Based Criteria 7
1.4 Organization of This Book 8
Appendix A: Unifying Framework of ITL 9
2 Information Measures 13
2.1 Entropy 13
2.2 Mutual Information 19
2.3 Information Divergence 21
2.4 Fisher Information 23
2.5 Information Rate 24
Appendix B: α-Stable Distribution 26
System identification is a common method for building the mathematical model of a physical plant, which is widely utilized in practical engineering situations. In general, the system identification consists of three key elements, i.e., the data, the model, and the criterion. The goal of identification is then to choose one from a set of candidate models to fit the data best according to a certain criterion. The criterion function is a key factor in system identification, which evaluates the con-sistency of the model to the actual plant and is, in general, an objective function for developing the identification algorithms. The identification performances, such as the convergence speed, steady-state accuracy, robustness, and the computational complexity, are directly related to the criterion function.
Well-known identification criteria mainly include the least squares (LS) crite-rion, minimum mean square error (MMSE) criterion, and the maximum likelihood (ML) criterion. These criteria provide successful engineering solutions to most practical problems, and are still prevalent today in system identification. However, they have some shortcomings that limit their general use. For example, the LS and MMSE only consider the second-order moment of the error, and the identification performance would become worse when data are non-Gaussian distributed (e.g., with multimodal, heavy-tail, or finite range). The ML criterion requires the knowledge of the conditional probability density function of the observed samples, which is not available in many practical situations. In addition, the computational complexity of the ML estimation is usually high. Thus, selecting a new criterion beyond second-order statistics and likelihood function is attractive in problems of system identification.
In recent years, criteria based on information theoretic descriptors of entropy and dissimilarity (divergence, mutual information) have attracted lots of attentions and become an emerging area of study in signal processing and machine learning domains. Information theoretic criteria (or briefly, information criteria) can capture higher order statistics and information content of signals rather than simply their energy. Many studies suggest that information criteria do not suffer from the limita-tion of Gaussian assumption and can improve performance in many realistic sce-narios. Combined with nonparametric estimators of entropy and divergence, many adaptive identification algorithms have been developed, including the practical gradient-based batch or recursive algorithms, fixed-point algorithms (no step-size), or other advanced search algorithms. Although many elegant results and techniques have been developed over the past few years, till now there is no book devoted to a systematic study of system identification under information theoretic criteria. The
primary focus of this book is to provide an overview of these developments, with emphasis on the nonparametric estimators of information criteria and gradient-based identification algorithms. Most of the contents of this book originally appeared in the recent papers of the authors.
The book is divided into six chapters: the first chapter is the introduction to the information theoretic criteria and the state-of-the-art techniques; the second chapter presents the definitions and properties of several important information measures; the third chapter gives an overview of information theoretic approaches to parameter estimation; the fourth chapter discusses system identification under minimum error entropy criterion; the fifth chapter focuses on the minimum infor-mation divergence criteria; and the sixth chapter changes the focus to the mutual information-based criteria.
It is worth noting that the information criteria can be used not only for system parameter identification but also for system structure identification (e.g., model selection). The Akaike’s information criterion (AIC) and the minimum description length (MDL) are two famous information criteria for model selection. There have been several books on AIC and MDL, and in this book we don’t discuss them in detail. Although most of the methods in this book are developed particularly for system parameter identification, the basic principles behind them are universal. Some of the methods with little modification can be applied to blind source sepa-ration, independent component analysis, time series prediction, classification and pattern recognition.
This book will be of interest to graduates, professionals, and researchers who are interested in improving the performance of traditional identification algorithms and in exploring new approaches to system identification, and also to those who are interested in adaptive filtering, neural networks, kernel methods, and online machine learning.
The authors are grateful to the National Natural Science Foundation of China and the National Basic Research Program of China (973 Program), which have funded this book. We are also grateful to the Elsevier for their patience with us over the past year we worked on this book. We also acknowledge the support and encouragement from our colleagues and friends.
P.R. China March 2013