DSP Laboratory - DIEI

Gaussian processes (GPs) are versatile tools that have been successfully employed to solve nonlinear estimation problems in machine learning but are rarely used in signal processing. In this tutorial, we present GPs for regression as a natural nonlinear extension to optimal Wiener filtering. After establishing their basic formulation, we discuss several important aspects and extensions, including recursive and adaptive algorithms for dealing with nonstationarity, low-complexity solutions, non-Gaussian noise models, and classification scenarios. Furthermore, we provide a selection of relevant applications to wireless digital communications.

Gaussian process-based machine learning is a powerful Bayesian paradigm for nonparametric nonlinear regression and classification. In this article, we discuss connections of Gaussian process regression with Kalman filtering and present methods for converting spatiotemporal Gaussian process regression problems into infinite-dimensional state-space models. This formulation allows for use of computationally efficient infinite-dimensional Kalman filtering and smoothing methods, or more general Bayesian filtering and smoothing methods, which reduces the problematic cubic complexity of Gaussian process regression in the number of time steps into linear time complexity. The implication of this is that the use of machine-learning models in signal processing becomes computationally feasible, and it opens the possibility to combine machine-learning techniques with signal processing methods.

Over the last decade, nonlinear kernel-based learning methods have been widely used in the sciences and in industry for solving, e.g., classification, regression, and ranking problems. While their users are more than happy with the performance of this powerful technology, there is an emerging need to additionally gain better understanding of both the learning machine and the data analysis problem to be solved. Opening the nonlinear black box, however, is a notoriously difficult challenge. In this review, we report on a set of recent methods that can be universally used to make kernel methods more transparent. In particular, we discuss relevant dimension estimation (RDE) that allows to assess the underlying complexity and noise structure of a learning problem and thus to distinguish high/low noise scenarios of high/low complexity respectively. Moreover, we introduce a novel local technique based on RDE for quantifying the reliability of the learned predictions. Finally, we report on techniques that can explain the individual nonlinear prediction. In this manner, our novel methods not only help to gain further knowledge about the nonlinear signal processing problem itself, but they broaden the general usefulness of kernel methods in practical signal processing applications.

Signal processing methods have significantly changed over the last several decades. Traditional methods were usually based on parametric statistical inference and linear filters. These frameworks have helped to develop efficient algorithms that have often been suitable for implementation on digital signal processing (DSP) systems. Over the years, DSP systems have advanced rapidly, and their computational capabilities have been substantially increased. This development has enabled contemporary signal processing algorithms to incorporate more computations. Consequently, we have recently experienced a growing interaction between signal processing and machine-learning approaches, e.g., Bayesian networks, graphical models, and kernel-based methods, whose computational burden is usually high.

Kernel-based methods provide a rich and elegant framework for developing nonparametric detection procedures for signal processing. Several recently proposed procedures can be simply described using basic concepts of reproducing kernel Hilbert space (RKHS) embeddings of probability distributions, mainly mean elements and covariance operators. We propose a unified view of these tools and draw relationships with information divergences between distributions.

Many modern applications of signal processing and machine learning, ranging from computer vision to computational biology, require the analysis of large volumes of high-dimensional continuous-valued measurements. Complex statistical features are commonplace, including multimodality, skewness, and rich dependency structures. Such problems call for a flexible and robust modeling framework that can take into account these diverse statistical features. Most existing approaches, including graphical models, rely heavily on parametric assumptions. Variables in the model are typically assumed to be discrete valued or multivariate Gaussians; and linear relations between variables are often used. These assumptions can result in a model far different from the data generating process.

Signal processing tasks as fundamental as sampling, reconstruction, minimum mean-square error interpolation, and prediction can be viewed under the prism of reproducing kernel Hilbert spaces (RKHSs). Endowing this vantage point with contemporary advances in sparsity-aware modeling and processing promotes the nonparametric basis pursuit advocated in this article as the overarching framework for the confluence of kernel-based learning (KBL) approaches leveraging sparse linear regression, nuclear-norm regularization, and dictionary learning. The novel sparse KBL toolbox goes beyond translating sparse parametric approaches to their nonparametric counterparts to incorporate new possibilities such as multikernel selection and matrix smoothing. The impact of sparse KBL to signal processing applications is illustrated through test cases from cognitive radio sensing, microarray data imputation, and network traffic prediction.

Kernel-based learning (KBL) methods have recently become prevalent in many engineering applications, notably in signal processing and communications. The increased interest is mainly driven by the practical need of being able to develop efficient nonlinear algorithms, which can obtain significant performance improvements over their linear counterparts at the price of generally higher computational complexity. In this article, an overview of applying various KBL methods to statistical signal processing-related open issues in cognitive radio networks (CRNs) is presented. It is demonstrated that KBL methods provide a powerful set of tools for CRNs and enable rigorous formulation and effective solutions to both long-standing and emerging design problems.

Tensors (also called multiway arrays) are a generalization of vectors and matrices to higher dimensions based on multilinear algebra. The development of theory and algorithms for tensor decompositions (factorizations) has been an active area of study within the past decade, e.g., [1] and [2]. These methods have been successfully applied to many problems in unsupervised learning and exploratory data analysis. Multiway analysis enables one to effectively capture the multilinear structure of the data, which is usually available as a priori information about the data. Hence, it might provide advantages over matrix factorizations by enabling one to more effectively use the underlying structure of the data. Besides unsupervised tensor decompositions, supervised tensor subspace regression and classification formulations have been also successfully applied to a variety of fields including chemometrics, signal processing, computer vision, and neuroscience.

Over the last decade, several positive-definite kernels have been proposed to treat spike trains as objects in Hilbert space. However, for the most part, such attempts still remain a mere curiosity for both computational neuroscientists and signal processing experts. This tutorial illustrates why kernel methods can, and have already started to, change the way spike trains are analyzed and processed. The presentation incorporates simple mathematical analogies and convincing practical examples in an attempt to show the yet unexplored potential of positive definite functions to quantify point processes. It also provides a detailed overview of the current state of the art and future challenges with the hope of engaging the readers in active participation.

In any estimation problem, there is always a need to find the bias and mean square error (MSE) of an estimator. These values are then compared against their sample averages obtained from simulation to confirm the theoretical development, and/or the Cram?r-Rao lower bound (CRLB) [1] to assess the optimality of the estimator. When the estimator is a nonlinear function of the measurements, it is rather difficult to derive exact expressions for the bias and MSE. Based on Taylor series expansion (TSE) of the estimator cost function near the true value, [2] provides a generic approximation for these performance measures. In [3], equations for bias and variance are obtained by a direct TSE of the estimator function. Their difference is that [2] is a TSE of the estimator cost function, while [3] is a TSE of the estimator itself. We shall review the bias and MSE formulas obtained from these two approaches, provide several representative application examples, and compare their results. It will be explained that for linear parameter estimation problems, both techniques give identical and exact bias and MSE expressions. However, the former has a wider applicability over the latter for nonlinear estimation, particularly when the estimate is not an explicit function of the measurements.

Resulting from the synergy between the sequential Monte Carlo (SMC) method [1] and interval analysis [2], box particle filtering is an approach that has recently emerged [3] and is aimed at solving a general class of nonlinear filtering problems. This approach is particularly appealing in practical situations involving imprecise stochastic measurements that result in very broad posterior densities. It relies on the concept of a box particle that occupies a small and controllable rectangular region having a nonzero volume in the state space. Key advantages of the box particle filter (box-PF) against the standard particle filter (PF) are its reduced computational complexity and its suitability for distributed filtering. Indeed, in some applications where the sampling importance resampling (SIR) PF may require thousands of particles to achieve accurate and reliable performance, the box-PF can reach the same level of accuracy with just a few dozen box particles. Recent developments [4] also show that a box-PF can be interpreted as a Bayes? filter approximation allowing the application of box-PF to challenging target tracking problems [5].

Video is a booming industry: content is embedded on many Web sites, delivered over the Internet, and streamed to mobile devices. Content providers own vast quantities of studio-quality video (i.e., produced to the quality standards of a television studio), but legal contracts between actors, producers, and owners limit how and where others can use such video. As a result, finding and getting rights to use relevant video remains an obstacle to addressing relevant research problems. The Consumer Digital Video Library (CDVL) Web site (http//:www.cdvl.org) attempts to address this obstacle. The CDVL makes high-quality, uncompressed video clips available for free download. Content owners have granted permission for use, and a use agreement protects owners? rights. The clips are ideal for use by the education, research, and product development communities. Developing objective video quality models and testing emergency telemedicine systems are two applications enabled by CDVL content.

In the article ??Dynamic Network Cartography?? by G. Mateos and K. Rajawat [1], IEEE Signal Processing Magazine,vol. 30, no. 3, pp. 129??143, Figures 3 and 6 were incorrect due to a production error. The subfigures within Figure 3 were misplaced. Part (a) should be swapped with (c), and (b) should be swapped with (d). In the legend of Figure 6(a), the fourth row should read ??Estimator (10)?? instead of ??Estimator (17).?? The correct figures are printed.

Frequency-Response Masking (FRM) technique has been widely used in all kinds of applications where FIR filters with extremely narrow transition band are needed. Thus, researchers have invested much effort to find ways to increase the efficiency of the FRM algorithm. In this paper, a novel 2-stage FRM structure was proposed. The three interpolation factors of the subfilters in the second stage can be chosen independently. The band-edge shaping filter synthesized by the second stage has a non-periodical frequency response, which is quite different from conventional FRM structures. Various filters were designed to test the performance of the proposed method. Experiments show that the proposed structure achieves lower complexity, in terms of the number of multipliers, compared to the FRM algorithms such as 1-stage FRM, 2-stage FRM, IFIR-FRM, SFFM-FRM, serial-masking FRM and some current FRM structures.

Manifold learning is widely used in machine learning and pattern recognition. However, manifold learning only considers the similarity of samples belonging to the same class and ignores the within-class variation of data, which will impair the generalization and stableness of the algorithms. For this purpose, we construct an adjacency graph to model the intraclass variation that characterizes the most important properties, such as diversity of patterns, and then incorporate the diversity into the discriminant objective function for linear dimensionality reduction. Finally, we introduce the orthogonal constraint for the basis vectors and propose an orthogonal algorithm called stable orthogonal local discriminate embedding. Experimental results on several standard image databases demonstrate the effectiveness of the proposed dimensionality reduction approach.

For images, gradient domain composition methods like Poisson blending offer practical solutions for uncertain object boundaries and differences in illumination conditions. However, adapting Poisson image blending to video presents new challenges due to the added temporal dimension. In video, the human eye is sensitive to small changes in blending boundaries across frames and slight differences in motions of the source patch and target video. We present a novel video blending approach that tackles these problems by merging the gradient of source and target videos and optimizing a consistent blending boundary based on a user-provided blending trimap for the source video. Our approach extends mean-value coordinates interpolation to support hybrid blending with a dynamic boundary while maintaining interactive performance. We also provide a user interface and source object positioning method that can efficiently deal with complex video sequences beyond the capabilities of alpha blending.

We develop new metrics for texture similarity that accounts for human visual perception and the stochastic nature of textures. The metrics rely entirely on local image statistics and allow substantial point-by-point deviations between textures that according to human judgment are essentially identical. The proposed metrics extend the ideas of structural similarity and are guided by research in texture analysis-synthesis. They are implemented using a steerable filter decomposition and incorporate a concise set of subband statistics, computed globally or in sliding windows. We conduct systematic tests to investigate metric performance in the context of “known-item search,” the retrieval of textures that are “identical” to the query texture. This eliminates the need for cumbersome subjective tests, thus enabling comparisons with human performance on a large database. Our experimental results indicate that the proposed metrics outperform peak signal-to-noise ratio (PSNR), structural similarity metric (SSIM) and its variations, as well as state-of-the-art texture classification metrics, using standard statistical measures.

The tracking and recognition of facial activities from images or videos have attracted great attention in computer vision field. Facial activities are characterized by three levels. First, in the bottom level, facial feature points around each facial component, i.e., eyebrow, mouth, etc., capture the detailed face shape information. Second, in the middle level, facial action units, defined in the facial action coding system, represent the contraction of a specific set of facial muscles, i.e., lid tightener, eyebrow raiser, etc. Finally, in the top level, six prototypical facial expressions represent the global facial muscle movement and are commonly used to describe the human emotion states. In contrast to the mainstream approaches, which usually only focus on one or two levels of facial activities, and track (or recognize) them separately, this paper introduces a unified probabilistic framework based on the dynamic Bayesian network to simultaneously and coherently represent the facial evolvement in different levels, their interactions and their observations. Advanced machine learning methods are introduced to learn the model based on both training data and subjective prior knowledge. Given the model and the measurements of facial motions, all three levels of facial activities are simultaneously recognized through a probabilistic inference. Extensive experiments are performed to illustrate the feasibility and effectiveness of the proposed model on all three level facial activities.