ApDeepSense: Deep Learning Uncertainty Estimation Without the Pain for IoT Applications

Author:

Shuochao Yao, Yiran Zhao, Huajie Shao, Chao Zhang, Aston Zhang, Dongxin Liu, Shengzhong Liu, Lu Su, Tarek F. Abdelzaher

Intro:

As a result of the recent advancement of deep learning applications, growing attention from the IoT community has been attracted to this technique. Different from other applications, the one with IoT areas require the learning method provide outputs as well as the uncertainty estimates of the outputs. The current state-of-art solution involves intensive sampling procedure, or even re-train the network, which is too expensive for many small applications. This paper presents ApDeepSense; a method saves around 90% of energy and time consumptions while producing more accurate uncertainty estimates.

Methodology:

The current state-of-art used dropout training to estimate the uncertainty and proved that the equivalence between dropout training and the approximate inference in a deep Gaussian process. This paper, however, showed that the internal-layer representation of deep neural networks could be represented by an internal-layer representation of multivariate Gaussian distributions. And by applying the matrix multiplication and activation functions to the probabilistic distribution input, we can get not only the expected value of output but also the probability distribution of that output. 

Authors also tried to define special operations to deal the probabilistic distribution input, which includes using element-wise multiplication to represent the matrix form of expected value and variance of output and use piece-wise linear functions as the closed-form solution for non-linear activation function.

Critique:

1. This paper's result is amazing, it provides good result while using less time and energy. You get the output for the expected value and the uncertainty of the output from the variance. The basic idea here is you can use the mean and variance from the previous layer to get the mean and variance for the next layer. I think the only problem with this approach is that it assumes the input always has a normal distribution, it will be problematic when the input has some other underlying distribution.

2. One limitation of this paper is that it only considered fully-connected layers, limited the type of network this paper could apply. But I don't think this would be really big problem.

That's all I have for this paper. I cannot find any other flaws I can attack. Even the proof is good, it's right amount of math that you don't spend hours to go through yet is long enough that you feel you are smart.

--Yifan Hu

Comments

  1. jon_weissmanApril 16, 2019 at 6:47 AM

    I'm impressed that you worked through some of the mathematics! The problem of uncertainty estimation appears to be critical in human-in-the-loop applications such as many IoT scenarios. Assumption of Gaussian/Normal seem fine given the physical nature of the data in the IoT scenarios.

    ReplyDelete

Post a Comment

Popular posts from this blog

A Machine Learning Approach to Live Migration Modeling

Authors:  Changyeon Jo, Youngsu Cho, Bernhard Egger Seoul National University Summary:  This paper proposed a machine learning approach to predict the system metrics for different VM migration algorithms. These system metrics are affected by migration algorithm, host machine status and user applications and feasible for machine learning approach. The prediction model takes 21 system/application input features (including which algorithm), and generates 6 metrics as the output. As a result, it can accurately estimate each metrics under different application, VM and algorithms. One of the application, the paper provide a semi-automated algorithm selection based on this machine learning model and user-provided SLA (service level agreements) constraints. Questions about Machine Learning:  Why machine learning is applied: The combination of algorithm, VM status and current running applications status, is a huge searching space. Machine learning features: which algorithm, 20 featu
Read more

NetBouncer: Active Device and Link Failure Localization in Data Center Networks

Authors : Cheng Tan, Ze Jin, Chuanxiong Guo, Tianrong Zhang, Haitao Wu, Karl Deng, Dongming Bi, and Dong Xiang Introduction : Availability of large scale data centers ensures that all the operations within the center's network function smoothly. The biggest challenge for such data centers is the ability to accurately locate and contain device and link failures, across millions of servers and thousands of network devices. The authors introduce NetBouncer, a failure localization system which is capable of detecting both device and link failures. The authors introduce the framework's algorithm for high accuracy link failure inference and use domain knowledge to make it robust to real-world data inconsistencies.  Pros : The NetBouncer framework makes use of Clos network architectures, which were introduced by Bell Labs in 1950s and made famous again with the onset of Ethernet switches. Many software companies now make use of multi-tier Clos network-based architecture
Read more

Deep Learning for Entity Matching: A Design Space Exploration

Authors - Sidharth Mudgal, Han Li, Theodoros Rekatsinas, AnHai Doan, Youngchoon Park, Ganesh Krishnan, Rohit Deep, Esteban Arcaute, Vijay Raghavendra Motivation/Abstract: Introduces entity matching, deep learning separately. Categorizes all the Deep Learning solutions in similar situations(matching tasks in NLP), and then works towards solving Entity Matching. Talks about the involvement of Deep Learning in EM in the past (one model). Finds out different fields where Deep Learning might perform better, does testing and reports results and conclusion. Main Points: There are three criteria/context where the discussion(comparing Deep Learning model vs traditional model) takes place - 1. Structured Data - SQL type stored data 2. Textual Data - Consists of text blobs which consist of huge sentences 3. Dirty Data - Erroneous structured data where text fields have values all mixed up Four representative models are picked and the Deep Learning process is explained through them. 
Read more