Technology Trends in Data Science, Machine Learning, and Artificial Intelligence Sebastian Raschka 1,*,†, Joshua Patterson 2 and Corey Nolet 2,3 1 Department of Statistics, University of Wisconsin-Madison, Madison, WI 53575, USA 2 NVIDIA, Santa Clara, CA 95051, USA; [email protected] (J.P.); [email protected] (C.N.) 3 Department of Comp Sci & Electrical Engineering, University of Maryland, Baltimore County, Baltimore, MD 21250, USA * Correspondence: [email protected] † Current address: 1300 University Ave, Medical Sciences Building, Madison, WI 53706, USA. Received: 6 February 2020; Accepted: 31 March 2020; Published: 4 April 2020 Abstract: Smarter applications are making better use of the insights gleaned from data, having an impact on every industry and research discipline. At the core of this revolution lies the tools and the methods that are driving it, from processing the massive piles of data generated each day to learning from and taking useful action. Deep neural networks, along with advancements in classical machine learning and scalable general-purpose graphics processing unit (GPU) computing, have become critical components of artificial intelligence, enabling many of these astounding breakthroughs and lowering the barrier to adoption. Python continues to be the most preferred language for scientific See “Machine Learning with Python,” a special issue of Information (ISSN 2078-2489) https://www.mdpi.com/journal/information/special_issues/ML_Python
Analytics Scikit-Learn Machine Learning Network-X Graph Analytics PyTorch Chainer MxNet Deep Learning Matplotlib Seaborn Visualization GPU Memory Figure 1. The standard Python ecosystem for machine learning, data science, and scientific computing. While both NumPy and Pandas [7] (Figure 1) provide abstractions over a collection of data points 1 with operations that work on the dataset as a whole, Pandas extends NumPy by providing a data 2 frame-like object supporting heterogeneous column types and row and column metadata. In recent 3 years, Pandas library has become the de-facto format for representing tabular data in Python for extract, 4 Sebastian Raschka, Joshua Patterson, and Corey Nolet (2020) Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence Information 2020, 11, 193
cuIO Analytics cuML Machine Learning cuGraph Graph Analytics PyTorch Chainer MxNet Deep Learning cuXfilter <> pyViz Visualization Figure 4. RAPIDS is an open source effort to support and grow the ecosystem of GPU-accelerated Python tools for data science, machine learning, and scientific computing. RAPIDS supports existing libraries, fills gaps by providing open source libraries with crucial components that are missing from the Python community, and promotes cohesion across the ecosystem by supporting interoperability across the libraries. 4.3. NDArray and Vectorized Operations 556 While NumPy is capable of invoking a BLAS implementation to optimize SIMD operations, its 557 capability of vectorizing functions is limited, providing little to no performance benefits. The Numba 558 Sebastian Raschka, Joshua Patterson, and Corey Nolet (2020) Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence Information 2020, 11, 193
following websites: http://www.geforce.com/hardware/desktop-gpus/geforce-gtx-980- ti/specifications http://ark.intel.com/products/82930/Intel-Core-i7-5960X- Processor-Extreme-Edition-20M-Cache-up-to-3_50-GHz (date: August 20, 2015) At 70 percent of the price of a modern CPU, we can get a GPU that has 450 times PRUHFRUHVDQGLVFDSDEOHRIDURXQGWLPHVPRUHÁRDWLQJSRLQWFDOFXODWLRQVSHU second. So, what is holding us back from utilizing GPUs for our machine learning tasks? The challenge is that writing code to target GPUs is not as trivial as executing Python code in our interpreter. There are special packages such as CUDA and OpenCL that allow us to target the GPU. However, writing code in CUDA or OpenCL is probably not the most convenient environment for implementing and running machine learning algorithms. The good news is that this is what Theano was developed for! [ 422 ] If we recall from Chapter 12, ,PSOHPHQWLQJD0XOWLOD\HU$UWLÀFLDO1HXUDO1HWZRUN from Scratch, where we implemented a very simple multilayer perceptron with only one hidden layer consisting of 100 units, we had to optimize approximately 80,000 weight parameters (> @> @ ) to learn a PRGHOIRUDYHU\VLPSOHLPDJHFODVVLÀFDWLRQWDVN7KHLPDJHVLQ01,67DUHUDWKHU small (28 x 28 pixels), and we can only imagine the explosion in the number of parameters if we want to add additional hidden layers or work with images that have higher pixel densities. Such a task would quickly become unfeasible for a single processing unit. The TXHVWLRQWKHQEHFRPHV³KRZFDQZHWDFNOHVXFKSUREOHPVPRUHHIIHFWLYHO\" The obvious solution to this problem is to use GPUs, which are real work horses. <RXFDQWKLQNRIDJUDSKLFVFDUGDVDVPDOOFRPSXWHUFOXVWHULQVLGH\RXUPDFKLQH Another advantage is that modern GPUs are relatively cheap compared to the state-of-the-art CPUs, as we can see in the following overview: Chapter 13 The sources for the information in the table are the following websites (Date: October https://www.theverge.com/2015/5/31/8695075/nvidia-geforce-gtx-980-ti-announced Image: https://www.amazon.com/Nvidia-GEFORCE-GTX-1080-Ti/dp/B06XH5ZCLP Image: https://www.nvidia.com/en-in/geforce/graphics-cards/rtx-2080-ti/ Sebastian Raschka. Python Machine Learning. Birmingham, UK: Packt Publishing, 2015 Sebastian Raschka and Vahid Mirjalili. Python Machine Learning 2nd Ed. Birmingham, UK: Packt Publishing, 2017 Sebastian Raschka and Vahid Mirjalili. Python Machine Learning 3rd Ed. Birmingham, UK: Packt Publishing, 2019
paper published in a major ML conference over the last few years. Papers were categorized based on whether they mention PyTorch or TensorFlow, excluding papers with authors affiliated with either Google or Facebook, as well as those that mention both TensorFlow and PyTorch
ML academia); PyTorch is by a very slim margin faster than TensorFlow 2.0 in our experiences when you run TensorFlow in non-Eager mode. However, since Eager mode is now enabled by default in TensorFlow 2.0; PyTorch is significantly faster." 18 https://www.reddit.com/r/MachineLearning/comments/f19dj4/d_tensorflow_20_v_pytorch_performance_question/
static computation graph programming paradigm enabled by dynamic graphs in PyTo Torch 7 was eventually superseded by PyTorch in 2017 799 Python wrapper around Torch 7’s lower-level C/C++ cod 800 Python-based deep learning frameworks, such as Chainer 801 an imperative programming style instead of using graph m 802 attractive to researchers, as it provides a familiar interface fo 803 debugging, and is directly compatible with other Python- 804 Defining the graph Initializing and evaluating the graph b a In: In: Out: Out: b In: Out: Out: Figure 7. Comparison between (a) a static computation graph in TensorFlow 1.15 and (b) an imperative programming paradigm enabled by dynamic graphs in PyTorch 1.4. Torch 7 was eventually superseded by PyTorch in 2017 [101], which started out as a user-friendly 99 Python wrapper around Torch 7’s lower-level C/C++ code. Inspired by pioneers in dynamic and 00 Python-based deep learning frameworks, such as Chainer [77] and DyNet [102], PyTorch embraces 01 an imperative programming style instead of using graph meta-programming21. This is particularly 02 Sebastian Raschka, Joshua Patterson, and Corey Nolet (2020) Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence Information 2020, 11, 193
https://thegradient.pub/state-of-ml- frameworks-2019-pytorch-dominates-research- tensorflow-dominates-industry/ Two ways for turning a PyTorch model into a static graph for optimization and deployment: a) Tracing b) Scripting DL Frameworks are converging
Ben Nassi1, Dudi Nassi1, Raz Ben-Netanel1, Yisroel Mirsky1,2, Oleg Drokin3, Yuval Elovici1 Video Demonstration - https://youtu.be/1cSw4fXYqWI {nassib,nassid,razx,yisroel,elovici}@post.bgu.ac.il, [email protected] 1 Ben-Gurion University of the Negev, 2 Georgia Tech,3 Independent Tesla Researcher ABSTRACT The absence of deployed vehicular communication sys- tems, which prevents the advanced driving assistance systems (ADASs) and autopilots of semi/fully autonomous cars to validate their virtual perception regarding the physical en- vironment surrounding the car with a third party, has been exploited in various attacks suggested by researchers. Since the application of these attacks comes with a cost (exposure of the attacker’s identity), the delicate exposure vs. application balance has held, and attacks of this kind have not yet been encountered in the wild. In this paper, we investigate a new perceptual challenge that causes the ADASs and autopi- lots of semi/fully autonomous to consider depthless objects (phantoms) as real. We show how attackers can exploit this perceptual challenge to apply phantom attacks and change the abovementioned balance, without the need to physically approach the attack scene, by projecting a phantom via a a b Fig. 1: Perceptual Challenge: Would you consider the projec- tion of the person (a) and road sign (b) real? Telsa considers (a) a real person and Mobileye 630 PRO considers (b) a real road sign. exchanging information between vehicles and roadside units,
en- party, has been searchers. Since a cost (exposure e vs. application d have not yet we investigate a ASs and autopi- epthless objects can exploit this cks and change ed to physically phantom via a a b Fig. 1: Perceptual Challenge: Would you consider the projec- tion of the person (a) and road sign (b) real? Telsa considers (a) a real person and Mobileye 630 PRO considers (b) a real road sign.
UAP [180] no no yes yes no DeepFool [181] yes yes yes yes yes NewtonFool [182] no yes yes no no JSMA [183] yes yes yes yes yes CW/CW2 [184] yes yes yes yes yes PGD [185] yes no yes yes yes OM [186] no no no yes no EAD [187] yes yes yes yes no Boundary Attack [188] no yes yes no no HopSkipJumpAttack [189] yes yes yes no no MaxConf [190] yes no no no no Inversion attack [191] yes yes no no no SparseL1 [192] yes yes no no no SPSA [193] yes no no no no HCLU [194] no no yes no no ADef [195] no yes no no no DDNL2 [196] no yes no no no Local Search [197] no yes no no no Pointwise attack [198] no yes no no no GenAttack [199] no yes no no no Defense mechanisms Feature Squeezing [200] no no yes no yes Spatial Smoothing [200] no no yes no yes Label Smoothing [200] no no yes no yes Gaussian Augmentation [201] no no yes no yes Adversarial Training [185] no no yes yes yes Thermometer Encoding [202] no no yes yes yes NAT [203] no no no yes no EAT [177] no no no yes no DD [204] no no no yes no IGR [205] no no no yes no EIT [206] no no yes yes no RT [207] no no no yes no PixelDefend [208] no no yes yes no Regr.-based classfication [209] no no no yes no JPEG compression [210] no no yes no no 8. Conclusions 1137 This article reviewed some of the most notable advances in machine learning, data science, an 1138 scientific computing. It provided a brief background into major topics, while investigating the variou 1139 challenges and current state of solutions for each. There are several more specialized application an 1140 research areas that are outside the scope of this article. For example, attention-based Transforme 1141 Table 1. Selection of evasion attack and defense mechanisms that are implemented in adversarial learning toolkits. Note that ART also implements methods for poisoning and extraction attacks (not shown). Cleverhans v3.0.1 FoolBox v2.3.0 ART v1.1.0 DEEPSEC (2019) AdvBox v0.4.1 Supported frameworks TensorFlow yes yes yes no yes MXNet yes yes yes no yes PyTorch no yes yes yes yes PaddlePaddle no no no no yes (Evasion) attack mechanisms BLB [163] yes no no yes no AMD [170] yes no no no no ZOO [171] no no yes no no VA [172] yes yes yes no no AP [173] no no yes no no STA [174] no yes yes no no DTA [175] no no yes no no FGSM [176] yes yes yes yes yes R+FGSM [177] no no no yes no R+LLC [177] no no no yes no U-MI-FGSM [178] yes yes no yes no T-MI-FGSM [178] yes yes no yes no BIM [179] no yes yes yes yes LLC / ILLC [179] no yes no yes no UAP [180] no no yes yes no DeepFool [181] yes yes yes yes yes NewtonFool [182] no yes yes no no JSMA [183] yes yes yes yes yes CW/CW2 [184] yes yes yes yes yes PGD [185] yes no yes yes yes OM [186] no no no yes no EAD [187] yes yes yes yes no Boundary Attack [188] no yes yes no no HopSkipJumpAttack [189] yes yes yes no no MaxConf [190] yes no no no no Inversion attack [191] yes yes no no no SparseL1 [192] yes yes no no no SPSA [193] yes no no no no HCLU [194] no no yes no no ADef [195] no yes no no no DDNL2 [196] no yes no no no Local Search [197] no yes no no no Pointwise attack [198] no yes no no no GenAttack [199] no yes no no no Defense mechanisms Feature Squeezing [200] no no yes no yes Spatial Smoothing [200] no no yes no yes Label Smoothing [200] no no yes no yes Gaussian Augmentation [201] no no yes no yes Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence (2020). Sebastian Raschka, Joshua Patterson, and Corey Nolet
Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. InAdvances in neural information processing systems 2017 (pp. 5998-6008). Figure 1: The Transformer - model architecture. 3.1 Encoder and Decoder Stacks
conversational skills. The bot, named Meena, is a 2.6 billion parameter language model trained on 341GB of text data, filtered from public domain social media conversations. Meena uses a seq2seq model (the same sort of technology that powers Google's "Smart Compose" feature in gmail), paired with an Evolved Transformer encoder and decoder - it's interesting to see something like this depend so much on a component developed via neural architecture search." https://mailchi.mp/jack-clark/import-ai-183-curve-fitting-conversation-with- meena-gans-paint-a-climate-change-future-and-what-compute-data-arbitrage- means-for-simulators
old, Monday blues...what are you up to? Meena: I'm procrastinating on a paper I'm supposed to be writing. Human: are you thinking of pulling an all nighter? Meena: Possibly. What's your paper on? Human: Nah, i just do the grunge work around here...one day I wish to write a paper that will change the world! Meena: What's the topic? Human: not sure, any suggestions? Meena: Well what's the assignment on? Human: breathing under water https://github.com/google-research/ google-research/blob/master/meena/ meena.txt
tasks, which depend on the relationship ies of input and output data: scuss the different relationship categories between input and output data, ZHUHGHSLFWHGLQWKHSUHYLRXVÀJXUHLQPRUHGHWDLO,IQHLWKHUWKHLQSXWQRU data represents sequences, then we are dealing with standard data, and we Despite LSTM, it may be hard to memorize long sequences/ sentences (e.g., for language translation) Recurrent Neural Networks
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. "... allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word ..." Published as a conference paper at ICLR 2015 0 10 20 30 40 50 60 Sentence length 0 5 10 15 20 25 30 BLEU score RNNsearch-50 RNNsearch-30 RNNenc-50 RNNenc-30 Figure 2: The BLEU scores of the generated translations on the test set with respect to the lengths of the sen- tences. The results are on the full test set which in- cludes sentences having un- known words to the models. "traditional" encoder+decoder RNN
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. "... allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word ..." Assign attention weight to each word to know how much "attention" the model should pay to each word (i.e., for each word, the network learns a "context")
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. Assign attention weight to each word to know how much "attention" the model should pay to each word (i.e., for each word, the network learns a "context") 3 LEARNING TO ALIGN AND TRANSLATE In this section, we propose a novel architecture for neural machine translation. The new architecture consists of a bidirectional RNN as an encoder (Sec. 3.2) and a decoder that emulates searching through a source sentence during decoding a translation (Sec. 3.1). 3.1 DECODER: GENERAL DESCRIPTION Figure 1: The graphical illus- tration of the proposed model trying to generate the t-th tar- get word yt given a source sentence (x1, x2, . . . , xT ). In a new model architecture, we define each conditional probability in Eq. (2) as: p(yi |y1, . . . , yi 1, x) = g(yi 1, si, ci), (4) where si is an RNN hidden state for time i, computed by si = f(si 1, yi 1, ci). It should be noted that unlike the existing encoder–decoder ap- proach (see Eq. (2)), here the probability is conditioned on a distinct context vector ci for each target word yi . The context vector ci depends on a sequence of annotations (h1, · · · , hTx ) to which an encoder maps the input sentence. Each annotation hi contains information about the whole input sequence with a strong focus on the parts surrounding the i-th word of the input sequence. We explain in detail how the annotations are com- puted in the next section. The context vector ci is, then, computed as a weighted sum of these annotations hi : ci = Tx X j=1 ↵ijhj. (5) hidden state attention weight
hB,2 hB,1 { Bidirectional RNN { Added attention c1 S0 S1 α1,1 α1,2 α1,T−1 ... c1 = T ∑ t=1 α1,t ht where the context vector is defined as c1 ... ... ̂ y1 (looks like a standard RNN but with context vectors as in-/output) RNN Attention Mechanism
Google Brain [email protected] Noam Shazeer⇤ Google Brain [email protected] Niki Parmar⇤ Google Research [email protected] Jakob Uszkoreit⇤ Google Research [email protected] Llion Jones⇤ Google Research [email protected] Aidan N. Gomez⇤ † University of Toronto [email protected] Łukasz Kaiser⇤ Google Brain [email protected] Illia Polosukhin⇤ ‡ [email protected] Abstract The dominant sequence transduction models are based on complex recurrent or cs.CL] 6 Dec 2017 https://arxiv.org/abs/1706.03762 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008).
L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008). Figure 1: The Transformer - model architecture.
Zhang Zheng Zhang Alexander J. Smola Amazon Web Services {chgwang, yeziha, astonz, zhaz, smola}@amazon.com Abstract Transformer has been widely used thanks to its ability to capture sequence information in an efficient way. However, recent develop- ments, such as BERT and GPT-2, deliver only heavy architectures with a focus on effective- ness. In this paper, we explore three carefully- designed light Transformer architectures to fig- ure out whether the Transformer with less computations could produce competitive re- sults. Experimental results on language model benchmark datasets hint that such trade-off is promising, and the light Transformer reduces 70% parameters at best, while obtains compet- itive perplexity compared to standard Trans- former. The source code is publicly avail- able 1. 1 Introduction Transformer has shown its effectiveness in model- ing sequence information due to the combination of size of the model. Therefore a light version of the standard Transformer architecture is expected to relieve the heavy computation issue and compress the model to ease the deployment in real world applications. In this paper, we carefully design several light Transformer architectures. The intuition behind the light Transformers is: preserving the Transformer connections that are useful to capture the essential sequence information, while omitting the ones with less impact. In particular, we explore along two directions: 1) better preserve the connections that are useful for capturing long-range dependency. We adapt the idea of dilated convolutions (Yu and Koltun, 2015) to preserve the Transformer connec- tions that are useful to extend the effective history of the context, and 2) better preserve the connec- tions that are essential in capturing local context. We leverage cascade connections that are capable to intensively incorporate the local context infor- 70v1 [cs.CL] 14 Feb 2020 (a) Full Transformer. (b) Dilated Transformer. (c) Dilated Transformer with memory. (d) Cascade Transformer.
ATTENTION NETWORKS Petar Veliˇ ckovi´ c⇤ Department of Computer Science and Technology University of Cambridge [email protected] Guillem Cucurull⇤ Centre de Visi´ o per Computador, UAB [email protected] Arantxa Casanova⇤ Centre de Visi´ o per Computador, UAB [email protected] Adriana Romero Montr´ eal Institute for Learning Algorithms [email protected] Pietro Li` o Department of Computer Science and Technology University of Cambridge [email protected] Yoshua Bengio Montr´ eal Institute for Learning Algorithms [email protected] ABSTRACT We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods’ features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix op- eration (such as inversion) or depending on knowing the graph structure upfront. https://arxiv.org/abs/1710.10903 Published as a conference paper at ICLR 2018 ↵ij ~ a softmaxj W~ hi W~ hj ~ h1 ~ h2 ~ h3 ~ h4 ~ h5 ~ h6 ~ ↵16 ~ ↵11 ~ ↵12 ~ ↵13 ~ ↵14 ~ ↵15 ~ h0 1 concat/avg Figure 1: Left: The attention mechanism a(W~ hi, W~ hj) employed by our model, parametrized by a weight vector ~ a 2 R2F 0 , applying a LeakyReLU activation. Right: An illustration of multi- head attention (with K = 3 heads) by node 1 on its neighborhood. Different arrow styles and colors denote independent attention computations. The aggregated features from each head are concatenated or averaged to obtain ~ h0 1 . applying a nonlinearity, ): ~ h 0 i = 0 @ X j2Ni ↵ijW~ hj 1 A . (4) To stabilize the learning process of self-attention, we have found extending our mechanism to em- ploy multi-head attention to be beneficial, similarly to Vaswani et al. (2017). Specifically, K inde- pendent attention mechanisms execute the transformation of Equation 4, and then their features are concatenated, resulting in the following output feature representation: 0 1
1 . . . 0 0 1 3D voxelization SMILES string Molecular fingerprint Graph encoding + + = A B C D O C O Figure 4: Summary of commonly used molecular representation methods based on the example of Aspirin (shown in the center). (A) A molecular fingerprint encodes structural motifs into a sparse bit vector. (B) A SMILES string encoding structural information of the molecule as well as its stereochemistry. (C) A visualization of the 3D voxelization concept. Note that information about which atoms occupy which voxels would be encoded in a 4th dimension which is omitted in this visualization. (D) Illustration of how information is passed to an atom in a simple graph neural network. Note that the graph-structural information will be passed from more distant atoms when the summation is repeated (not shown). Sebastian Raschka and Benjamin Kaufman (2020) Machine learning and AI-based approaches for bioactive ligand discovery and GPCR-ligand recognition arXiv:2001.06545
AA. Colorful image colorization. InEuropean conference on computer vision 2016 Oct 8 (pp. 649-666). Springer, Cham. 2 Zhang, Isola, Efros Fig. 1. Example input grayscale photos and output colorizations from our algo- rithm. These examples are cases where our model works especially well. Please visit http://richzhang.github.io/colorization/ to see the full range of results and to try our model and code. Best viewed in color (obviously). we leverage large-scale data. Predicting color has the nice property that training data is practically free: any color photo can be used as a training example, simply by taking the image’s L channel as input and its ab channels as the supervisory signal. Others have noted the easy availability of training data, and previous works have trained convolutional neural networks (CNNs) to predict color on large datasets [1,2]. However, the results from these previous attempts tend to look desaturated. One explanation is that [1,2] use loss functions that encourage conservative predictions. These losses are inherited from standard regression problems, where the goal is to minimize Euclidean error between an estimate and the ground truth. We instead utilize a loss tailored to the colorization problem. As pointed out by [3], color prediction is inherently multimodal – many objects can take on several plausible colorizations. For example, an apple is typically red, green, or yellow, but unlikely to be blue or orange. To appropriately model the multimodal 2 Vondrick, Shrivastava, Fathi, Guadarrama, Murphy Input Frame Target Colors Reference Frame Reference Colors Pointer Copy Fig. 1. Self-supervised Tracking: We capitalize on large amounts of unlabeled video to learn a self-supervised model for tracking. The model learns to predict the target colors for a gray-scale input frame by pointing to a colorful reference frame, and copying the color channels. Although we train without ground-truth labels, experiments and visualizations suggest that tracking emerges automatically in this model. Vondrick C, Shrivastava A, Fathi A, Guadarrama S, Murphy K. Tracking emerges by colorizing videos. InProceedings of the European Conference on Computer Vision (ECCV) 2018 (pp. 391-408). Larsson G, Maire M, Shakhnarovich G. Learning representations for automatic colorization. InEuropean Conference on Computer Vision 2016 Oct 8 (pp. 577-593). Springer, Cham.
Deepak Pathak Philipp Kr¨ ahenb¨ uhl Jeff Donahue Trevor Darrell Alexei A. Efros University of California, Berkeley {pathak,philkr,jdonahue,trevor,efros}@cs.berkeley.edu Abstract We present an unsupervised visual feature learning algo- rithm driven by context-based pixel prediction. By analogy with auto-encoders, we propose Context Encoders – a con- volutional neural network trained to generate the contents of an arbitrary image region conditioned on its surround- ings. In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s). When training context encoders, we have experi- mented with both a standard pixel-wise reconstruction loss, as well as a reconstruction plus an adversarial loss. The latter produces much sharper results because it can better handle multiple modes in the output. We found that a con- text encoder learns a representation that captures not just appearance but also the semantics of visual structures. We quantitatively demonstrate the effectiveness of our learned features for CNN pre-training on classification, detection, and segmentation tasks. Furthermore, context encoders can be used for semantic inpainting tasks, either stand-alone or as initialization for non-parametric methods. (a) Input context (b) Human artist (c) Context Encoder (L2 loss) (d) Context Encoder (L2 + Adversarial loss) .07379v2 [cs.CV] 21 Nov 2016 Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA. Context encoders: Feature learning by inpainting. InProceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 2536-2544).
Favaro P. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision 2016 Oct 8 (pp. 69-84). Springer, Cham. 2 M. Noroozi and P. Favaro (a) (b) (c) Fig. 1: Learning image representations by solving Jigsaw puzzles. (a) The image from which the tiles (marked with green lines) are extracted. (b) A puzzle ob- tained by shu✏ing the tiles. Some tiles might be directly identifiable as object parts, but others are ambiguous (e.g., have similar patterns) and their identi- fication is much more reliable when all tiles are jointly evaluated. In contrast, with reference to (c), determining the relative position between the central tile and the top two tiles from the left can be very challenging [10]. While it is true that biological agents typically make use of multiple images and also integrate additional sensory information, such as ego-motion, it is also true that single snapshots may carry more information than we have been able to ex- tract so far. This work shows that this is indeed the case. We introduce a novel Doersch C, Gupta A, Efros AA. Unsupervised visual representation learning by context prediction. InProceedings of the IEEE International Conference on Computer Vision 2015 (pp. 1422-1430). Unsupervised Visual Representation Learning by Context Prediction Carl Doersch1,2 Abhinav Gupta1 Alexei A. Efros2 1 School of Computer Science 2 Dept. of Electrical Engineering and Computer Science Carnegie Mellon University University of California, Berkeley Abstract This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, we extract random pairs of patches from each image and train a convolutional neural net to predict the po- sition of the second patch relative to the first. We argue that doing well on this task requires the model to learn to recog- nize objects and their parts. We demonstrate that the fea- ture representation learned using this within-image context indeed captures visual similarity across images. For exam- ple, this representation allows us to perform unsupervised visual discovery of objects like cats, people, and even birds from the Pascal VOC 2011 detection dataset. Furthermore, we show that the learned ConvNet can be used in the R- CNN framework [19] and provides a significant boost over a randomly-initialized ConvNet, resulting in state-of-the- art performance among algorithms which use only Pascal- _ _ ? ? Example: Question 1: Question 2: Figure 1. Our task for learning patch representations involves ran- domly sampling a patch (blue) and then one of eight possible neighbors (red). Can you guess the spatial configuration for the two pairs of patches? Note that the task is much easier once you have recognized the object! Answer key: Q1: Bottom right Q2: Top center
feature learning by learning to spot artifacts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 (pp. 2733-2742). Self-Supervised Feature Learning by Learning to Spot Artifacts Simon Jenni Paolo Favaro University of Bern, Switzerland {jenni,favaro}@inf.unibe.ch Abstract We introduce a novel self-supervised learning method based on adversarial training. Our objective is to train a discriminator network to distinguish real images from im- ages with synthetic artifacts, and then to extract features from its intermediate layers that can be transferred to other data domains and tasks. To generate images with artifacts, we pre-train a high-capacity autoencoder and then we use a damage and repair strategy: First, we freeze the autoen- coder and damage the output of the encoder by randomly dropping its entries. Second, we augment the decoder with a repair network, and train it in an adversarial manner against the discriminator. The repair network helps gen- erate more realistic images by inpainting the dropped fea- ture entries. To make the discriminator focus on the arti- facts, we also make it predict what entries in the feature were dropped. We demonstrate experimentally that features learned by creating and spotting artifacts achieve state of Figure 1. A mixture of real images (green border) and images with synthetic artifacts (red border). Is a good object representation necessary to tell them apart? Towards this goal, we propose to learn features by clas- sifying images as real or with artifacts (see Figure 1). We aim at creating image artifacts, such that a model capable
Networks for Multi-attribute Face Privacy arXiv:2001.00561 4 Generator Discriminator Attribute Classifier Auxiliary Face Matcher Target Labels Input Image !" # #′ Output Image Match score Gender, Age, Race Real / Synthesized LD <latexit sha1_base64="7+0IjgJn4DAPQ+LbgVg/SZ8/Sno=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6ChYUFYoaEiVSXQke/dFeLyUTE1g/Y8lXiD/gLt0pPwkKoTnffOvecU3Xv9ZLAzxRjbyVrYHBoeGR0bHxicmp6pjw7187iPBWyJeIgTo88nsnAj2RL+SqQR0kqeegF8tC72tT5wxuZZn4cHahuIjshv4j8c19wRVDnJOTqUvCgt3N3unVarjCHrdXYet1mTnW1UVtxKVhlbnWF2a7DzKqgWLtx+RUnOEMMgRwhJCIoigNwZPQcwwVDQlgHPcJSinyTl7jDOGlzYklicEKv6HtBu+MCjWivPTOjFnRKQG9KShtLpImJl1KsT7NNPjfOGv3Nu2c89d269PcKr5BQhUtC/9L1mf/V6VoUztEwNfhUU2IQXZ0oXHLTFX1z+1tVihwSwnR8RvmUYmGU/T7bRpOZ2nVvucm/G6ZG9V4U3Bwf+pY04P4U7d+DdtVxmePu1SrNjWLUo1jAIpZpnnU0sY1dtMj7Gk94xou1b91a99bDF9UqFZp5/FjW4yfjiJTW</latexit> <latexit sha1_base64="7+0IjgJn4DAPQ+LbgVg/SZ8/Sno=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6ChYUFYoaEiVSXQke/dFeLyUTE1g/Y8lXiD/gLt0pPwkKoTnffOvecU3Xv9ZLAzxRjbyVrYHBoeGR0bHxicmp6pjw7187iPBWyJeIgTo88nsnAj2RL+SqQR0kqeegF8tC72tT5wxuZZn4cHahuIjshv4j8c19wRVDnJOTqUvCgt3N3unVarjCHrdXYet1mTnW1UVtxKVhlbnWF2a7DzKqgWLtx+RUnOEMMgRwhJCIoigNwZPQcwwVDQlgHPcJSinyTl7jDOGlzYklicEKv6HtBu+MCjWivPTOjFnRKQG9KShtLpImJl1KsT7NNPjfOGv3Nu2c89d269PcKr5BQhUtC/9L1mf/V6VoUztEwNfhUU2IQXZ0oXHLTFX1z+1tVihwSwnR8RvmUYmGU/T7bRpOZ2nVvucm/G6ZG9V4U3Bwf+pY04P4U7d+DdtVxmePu1SrNjWLUo1jAIpZpnnU0sY1dtMj7Gk94xou1b91a99bDF9UqFZp5/FjW4yfjiJTW</latexit> <latexit sha1_base64="7+0IjgJn4DAPQ+LbgVg/SZ8/Sno=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6ChYUFYoaEiVSXQke/dFeLyUTE1g/Y8lXiD/gLt0pPwkKoTnffOvecU3Xv9ZLAzxRjbyVrYHBoeGR0bHxicmp6pjw7187iPBWyJeIgTo88nsnAj2RL+SqQR0kqeegF8tC72tT5wxuZZn4cHahuIjshv4j8c19wRVDnJOTqUvCgt3N3unVarjCHrdXYet1mTnW1UVtxKVhlbnWF2a7DzKqgWLtx+RUnOEMMgRwhJCIoigNwZPQcwwVDQlgHPcJSinyTl7jDOGlzYklicEKv6HtBu+MCjWivPTOjFnRKQG9KShtLpImJl1KsT7NNPjfOGv3Nu2c89d269PcKr5BQhUtC/9L1mf/V6VoUztEwNfhUU2IQXZ0oXHLTFX1z+1tVihwSwnR8RvmUYmGU/T7bRpOZ2nVvucm/G6ZG9V4U3Bwf+pY04P4U7d+DdtVxmePu1SrNjWLUo1jAIpZpnnU0sY1dtMj7Gk94xou1b91a99bDF9UqFZp5/FjW4yfjiJTW</latexit> <latexit sha1_base64="G30nvnJkyKBlKcCymNnrqTxJgV4=">AAACtXicjVLLSgMxFD0dX7VWrWs3g0VwVTJudCnowmUF+4BaZCZNa+y8TDJCKf6AWz9O/AP9C2/iCGoRzTAzJ+fec5KbmyiPpTaMvVS8peWV1bXqem2jXtvc2m7UuzorFBcdnsWZ6kehFrFMRcdIE4t+rkSYRLHoRdNTG+/dC6Vlll6aWS6GSThJ5Vjy0BDVvm40WYu54S+CoARNlCNrPOMKI2TgKJBAIIUhHCOEpmeAAAw5cUPMiVOEpIsLPKBG2oKyBGWExE7pO6HZoGRTmltP7dScVonpVaT0sU+ajPIUYbua7+KFc7bsb95z52n3NqN/VHolxBrcEPuX7jPzvzpbi8EYx64GSTXljrHV8dKlcKdid+5/qcqQQ06cxSOKK8LcKT/P2Xca7Wq3Zxu6+KvLtKyd8zK3wJvdJfU3+NnNRdA9bAWsFVwwVLGLPRxQG49wgnO00SHLER7x5J15t97dxz3wKuWF2MG34el34YWM3A==</latexit> <latexit sha1_base64="4MrC+jrKE9Y6Uixdn4yn3AW3+G8=">AAACxXicjVLLSsNAFD2Nr1qrVrdugkVwVRI3uhR04cJFFfuAtshkOm1D83IyEaWI+BVu9avEP9C/8M6YglpEJyQ5c+49Z+bOHS8J/FQ5zmvBmptfWFwqLpdWyqtr65WNcjONM8lFg8dBLNseS0XgR6KhfBWIdiIFC71AtLzxkY63roVM/Ti6ULeJ6IVsGPkDnzNFVK8bMjXiLJic3l0eX1aqTs0xw54Fbg6qyEc9rrygiz5icGQIIRBBEQ7AkNLTgQsHCXE9TIiThHwTF7hDibQZZQnKYMSO6TukWSdnI5prz9SoOa0S0CtJaWOHNDHlScJ6NdvEM+Os2d+8J8ZT7+2W/l7uFRKrMCL2L9008786XYvCAAemBp9qSgyjq+O5S2ZORe/c/lKVIoeEOI37FJeEuVFOz9k2mtTUrs+WmfibydSsnvM8N8O73iU12P3ZzlnQ3Ku5Ts09c1DEFraxS23cxyFOUEeDLK/wiCc8W+fWjXX/eRWsQn4nNvFtWA8f6vCTXg==</latexit> <latexit sha1_base64="bVHLM6rdJ3hojfyxBqy7/nN4hpM=">AAACxXicjVLJTsMwFJyGfS9cuUQgJE6R01KWGxIcOHAARBepVMgxhkZkI3EQVVUhvoIrfBXiD+AveDapBIcKHCUZz3sz9vOzlwR+phh7L1lj4xOTU9Mzs3PzC4tL5eX5RhbnqZB1EQdx2vJ4JgM/knXlq0C2klTy0Atk07s90PHmvUwzP47OVS+RnZDfRP61L7giqnMRctUVPOgfDy4PL8vrzGHbW2xvx2ZOpba7VXUJ1JhbqTLbdZgZ6yjGSVx+wwWuEEMgRwiJCIpwAI6MnjZcMCTEddAnLiXkm7jEALOkzSlLUgYn9pa+NzRrF2xEc+2ZGbWgVQJ6U1La2CBNTHkpYb2abeK5cdbsKO++8dR769HfK7xCYhW6xP6lG2b+V6drUbjGrqnBp5oSw+jqROGSm1PRO7d/VKXIISFO4yuKp4SFUQ7P2TaazNSuz5ab+IfJ1KyeiyI3x6feJTV42EV7NGhUHJc57inDNFaxhk1q4w72cYQT1MnyDs94wat1Zj1Yj99XwSoVd2IFv4b19AVzU5Oa</latexit> <latexit sha1_base64="PKmVZwp+bG9nGxjnb1GtIBSU4rk=">AAAC0HicjVHLTsMwEJyGd3kVOHKJqJA4RU553yrgwIFDQbQgQYUcYyBqXiQOoqoQ4soPcIWvQvwB/AVrk0pwQOAoyXp2Zuzd9ZLAzxRjbyVrYHBoeGR0rDw+MTk1XZmZbWVxngrZFHEQp8cez2TgR7KpfBXI4ySVPPQCeeR1tnX+6EammR9Hh6qbyHbILyP/whdcEdQ+Dbm6Ejzo7d2d7ZxVqsxhaytsc91mTm11Y2XZpWCVubVlZrsOM6uKYjXiyitOcY4YAjlCSERQFAfgyOg5gQuGhLA2eoSlFPkmL3GHMmlzYklicEI79L2k3UmBRrTXnplRCzoloDclpY1F0sTESynWp9kmnxtnjf7m3TOe+m5d+nuFV0iowhWhf+n6zP/qdC0KF9gwNfhUU2IQXZ0oXHLTFX1z+1tVihwSwnR8TvmUYmGU/T7bRpOZ2nVvucm/G6ZG9V4U3Bwf+pY04P4U7d+DVs1xmePus2p9qxj1KOaxgCWa5zrq2EUDTfK+xhOe8WIdWLfWvfXwRbVKhWYOP5b1+AniSJTS</latexit> <latexit sha1_base64="7+0IjgJn4DAPQ+LbgVg/SZ8/Sno=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6ChYUFYoaEiVSXQke/dFeLyUTE1g/Y8lXiD/gLt0pPwkKoTnffOvecU3Xv9ZLAzxRjbyVrYHBoeGR0bHxicmp6pjw7187iPBWyJeIgTo88nsnAj2RL+SqQR0kqeegF8tC72tT5wxuZZn4cHahuIjshv4j8c19wRVDnJOTqUvCgt3N3unVarjCHrdXYet1mTnW1UVtxKVhlbnWF2a7DzKqgWLtx+RUnOEMMgRwhJCIoigNwZPQcwwVDQlgHPcJSinyTl7jDOGlzYklicEKv6HtBu+MCjWivPTOjFnRKQG9KShtLpImJl1KsT7NNPjfOGv3Nu2c89d269PcKr5BQhUtC/9L1mf/V6VoUztEwNfhUU2IQXZ0oXHLTFX1z+1tVihwSwnR8RvmUYmGU/T7bRpOZ2nVvucm/G6ZG9V4U3Bwf+pY04P4U7d+DdtVxmePu1SrNjWLUo1jAIpZpnnU0sY1dtMj7Gk94xou1b91a99bDF9UqFZp5/FjW4yfjiJTW</latexit> <latexit sha1_base64="7+0IjgJn4DAPQ+LbgVg/SZ8/Sno=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6ChYUFYoaEiVSXQke/dFeLyUTE1g/Y8lXiD/gLt0pPwkKoTnffOvecU3Xv9ZLAzxRjbyVrYHBoeGR0bHxicmp6pjw7187iPBWyJeIgTo88nsnAj2RL+SqQR0kqeegF8tC72tT5wxuZZn4cHahuIjshv4j8c19wRVDnJOTqUvCgt3N3unVarjCHrdXYet1mTnW1UVtxKVhlbnWF2a7DzKqgWLtx+RUnOEMMgRwhJCIoigNwZPQcwwVDQlgHPcJSinyTl7jDOGlzYklicEKv6HtBu+MCjWivPTOjFnRKQG9KShtLpImJl1KsT7NNPjfOGv3Nu2c89d269PcKr5BQhUtC/9L1mf/V6VoUztEwNfhUU2IQXZ0oXHLTFX1z+1tVihwSwnR8RvmUYmGU/T7bRpOZ2nVvucm/G6ZG9V4U3Bwf+pY04P4U7d+DdtVxmePu1SrNjWLUo1jAIpZpnnU0sY1dtMj7Gk94xou1b91a99bDF9UqFZp5/FjW4yfjiJTW</latexit> <latexit sha1_base64="7+0IjgJn4DAPQ+LbgVg/SZ8/Sno=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6ChYUFYoaEiVSXQke/dFeLyUTE1g/Y8lXiD/gLt0pPwkKoTnffOvecU3Xv9ZLAzxRjbyVrYHBoeGR0bHxicmp6pjw7187iPBWyJeIgTo88nsnAj2RL+SqQR0kqeegF8tC72tT5wxuZZn4cHahuIjshv4j8c19wRVDnJOTqUvCgt3N3unVarjCHrdXYet1mTnW1UVtxKVhlbnWF2a7DzKqgWLtx+RUnOEMMgRwhJCIoigNwZPQcwwVDQlgHPcJSinyTl7jDOGlzYklicEKv6HtBu+MCjWivPTOjFnRKQG9KShtLpImJl1KsT7NNPjfOGv3Nu2c89d269PcKr5BQhUtC/9L1mf/V6VoUztEwNfhUU2IQXZ0oXHLTFX1z+1tVihwSwnR8RvmUYmGU/T7bRpOZ2nVvucm/G6ZG9V4U3Bwf+pY04P4U7d+DdtVxmePu1SrNjWLUo1jAIpZpnnU0sY1dtMj7Gk94xou1b91a99bDF9UqFZp5/FjW4yfjiJTW</latexit> <latexit sha1_base64="7+0IjgJn4DAPQ+LbgVg/SZ8/Sno=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6ChYUFYoaEiVSXQke/dFeLyUTE1g/Y8lXiD/gLt0pPwkKoTnffOvecU3Xv9ZLAzxRjbyVrYHBoeGR0bHxicmp6pjw7187iPBWyJeIgTo88nsnAj2RL+SqQR0kqeegF8tC72tT5wxuZZn4cHahuIjshv4j8c19wRVDnJOTqUvCgt3N3unVarjCHrdXYet1mTnW1UVtxKVhlbnWF2a7DzKqgWLtx+RUnOEMMgRwhJCIoigNwZPQcwwVDQlgHPcJSinyTl7jDOGlzYklicEKv6HtBu+MCjWivPTOjFnRKQG9KShtLpImJl1KsT7NNPjfOGv3Nu2c89d269PcKr5BQhUtC/9L1mf/V6VoUztEwNfhUU2IQXZ0oXHLTFX1z+1tVihwSwnR8RvmUYmGU/T7bRpOZ2nVvucm/G6ZG9V4U3Bwf+pY04P4U7d+DdtVxmePu1SrNjWLUo1jAIpZpnnU0sY1dtMj7Gk94xou1b91a99bDF9UqFZp5/FjW4yfjiJTW</latexit> <latexit sha1_base64="7+0IjgJn4DAPQ+LbgVg/SZ8/Sno=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6ChYUFYoaEiVSXQke/dFeLyUTE1g/Y8lXiD/gLt0pPwkKoTnffOvecU3Xv9ZLAzxRjbyVrYHBoeGR0bHxicmp6pjw7187iPBWyJeIgTo88nsnAj2RL+SqQR0kqeegF8tC72tT5wxuZZn4cHahuIjshv4j8c19wRVDnJOTqUvCgt3N3unVarjCHrdXYet1mTnW1UVtxKVhlbnWF2a7DzKqgWLtx+RUnOEMMgRwhJCIoigNwZPQcwwVDQlgHPcJSinyTl7jDOGlzYklicEKv6HtBu+MCjWivPTOjFnRKQG9KShtLpImJl1KsT7NNPjfOGv3Nu2c89d269PcKr5BQhUtC/9L1mf/V6VoUztEwNfhUU2IQXZ0oXHLTFX1z+1tVihwSwnR8RvmUYmGU/T7bRpOZ2nVvucm/G6ZG9V4U3Bwf+pY04P4U7d+DdtVxmePu1SrNjWLUo1jAIpZpnnU0sY1dtMj7Gk94xou1b91a99bDF9UqFZp5/FjW4yfjiJTW</latexit> <latexit sha1_base64="7+0IjgJn4DAPQ+LbgVg/SZ8/Sno=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6ChYUFYoaEiVSXQke/dFeLyUTE1g/Y8lXiD/gLt0pPwkKoTnffOvecU3Xv9ZLAzxRjbyVrYHBoeGR0bHxicmp6pjw7187iPBWyJeIgTo88nsnAj2RL+SqQR0kqeegF8tC72tT5wxuZZn4cHahuIjshv4j8c19wRVDnJOTqUvCgt3N3unVarjCHrdXYet1mTnW1UVtxKVhlbnWF2a7DzKqgWLtx+RUnOEMMgRwhJCIoigNwZPQcwwVDQlgHPcJSinyTl7jDOGlzYklicEKv6HtBu+MCjWivPTOjFnRKQG9KShtLpImJl1KsT7NNPjfOGv3Nu2c89d269PcKr5BQhUtC/9L1mf/V6VoUztEwNfhUU2IQXZ0oXHLTFX1z+1tVihwSwnR8RvmUYmGU/T7bRpOZ2nVvucm/G6ZG9V4U3Bwf+pY04P4U7d+DdtVxmePu1SrNjWLUo1jAIpZpnnU0sY1dtMj7Gk94xou1b91a99bDF9UqFZp5/FjW4yfjiJTW</latexit> LA <latexit sha1_base64="ljXmF37Llrl81rV1fQ78Ns6CQk4=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6wsbBAzJAwkepS6OiX7moxmYjY+gFbvkr8AX/hVulJWAjV6e5b555zqu69XhL4mWLsrWQNDA4Nj4yOjU9MTk3PlGfn2lmcp0K2RBzE6ZHHMxn4kWwpXwXyKEklD71AHnpXWzp/eCPTzI+jA9VNZCfkF5F/7guuCOqchFxdCh70du5ON07LFeawtRpbr9vMqa42aisuBavMra4w23WYWRUUazcuv+IEZ4ghkCOERARFcQCOjJ5juGBICOugR1hKkW/yEncYJ21OLEkMTugVfS9od1ygEe21Z2bUgk4J6E1JaWOJNDHxUor1abbJ58ZZo79594ynvluX/l7hFRKqcEnoX7o+8786XYvCORqmBp9qSgyiqxOFS266om9uf6tKkUNCmI7PKJ9SLIyy32fbaDJTu+4tN/l3w9So3ouCm+ND35IG3J+i/XvQrjouc9y9WqW5WYx6FAtYxDLNs44mtrGLFnlf4wnPeLH2rVvr3nr4olqlQjOPH8t6/ATcaJTT</latexit> <latexit sha1_base64="ljXmF37Llrl81rV1fQ78Ns6CQk4=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6wsbBAzJAwkepS6OiX7moxmYjY+gFbvkr8AX/hVulJWAjV6e5b555zqu69XhL4mWLsrWQNDA4Nj4yOjU9MTk3PlGfn2lmcp0K2RBzE6ZHHMxn4kWwpXwXyKEklD71AHnpXWzp/eCPTzI+jA9VNZCfkF5F/7guuCOqchFxdCh70du5ON07LFeawtRpbr9vMqa42aisuBavMra4w23WYWRUUazcuv+IEZ4ghkCOERARFcQCOjJ5juGBICOugR1hKkW/yEncYJ21OLEkMTugVfS9od1ygEe21Z2bUgk4J6E1JaWOJNDHxUor1abbJ58ZZo79594ynvluX/l7hFRKqcEnoX7o+8786XYvCORqmBp9qSgyiqxOFS266om9uf6tKkUNCmI7PKJ9SLIyy32fbaDJTu+4tN/l3w9So3ouCm+ND35IG3J+i/XvQrjouc9y9WqW5WYx6FAtYxDLNs44mtrGLFnlf4wnPeLH2rVvr3nr4olqlQjOPH8t6/ATcaJTT</latexit> <latexit sha1_base64="ljXmF37Llrl81rV1fQ78Ns6CQk4=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6wsbBAzJAwkepS6OiX7moxmYjY+gFbvkr8AX/hVulJWAjV6e5b555zqu69XhL4mWLsrWQNDA4Nj4yOjU9MTk3PlGfn2lmcp0K2RBzE6ZHHMxn4kWwpXwXyKEklD71AHnpXWzp/eCPTzI+jA9VNZCfkF5F/7guuCOqchFxdCh70du5ON07LFeawtRpbr9vMqa42aisuBavMra4w23WYWRUUazcuv+IEZ4ghkCOERARFcQCOjJ5juGBICOugR1hKkW/yEncYJ21OLEkMTugVfS9od1ygEe21Z2bUgk4J6E1JaWOJNDHxUor1abbJ58ZZo79594ynvluX/l7hFRKqcEnoX7o+8786XYvCORqmBp9qSgyiqxOFS266om9uf6tKkUNCmI7PKJ9SLIyy32fbaDJTu+4tN/l3w9So3ouCm+ND35IG3J+i/XvQrjouc9y9WqW5WYx6FAtYxDLNs44mtrGLFnlf4wnPeLH2rVvr3nr4olqlQjOPH8t6/ATcaJTT</latexit> <latexit sha1_base64="ljXmF37Llrl81rV1fQ78Ns6CQk4=">AAAC0HicjVHLLgRBFD3T3u/B0qZjIrHqVI9hxm6wsbBAzJAwkepS6OiX7moxmYjY+gFbvkr8AX/hVulJWAjV6e5b555zqu69XhL4mWLsrWQNDA4Nj4yOjU9MTk3PlGfn2lmcp0K2RBzE6ZHHMxn4kWwpXwXyKEklD71AHnpXWzp/eCPTzI+jA9VNZCfkF5F/7guuCOqchFxdCh70du5ON07LFeawtRpbr9vMqa42aisuBavMra4w23WYWRUUazcuv+IEZ4ghkCOERARFcQCOjJ5juGBICOugR1hKkW/yEncYJ21OLEkMTugVfS9od1ygEe21Z2bUgk4J6E1JaWOJNDHxUor1abbJ58ZZo79594ynvluX/l7hFRKqcEnoX7o+8786XYvCORqmBp9qSgyiqxOFS266om9uf6tKkUNCmI7PKJ9SLIyy32fbaDJTu+4tN/l3w9So3ouCm+ND35IG3J+i/XvQrjouc9y9WqW5WYx6FAtYxDLNs44mtrGLFnlf4wnPeLH2rVvr3nr4olqlQjOPH8t6/ATcaJTT</latexit> LM <latexit sha1_base64="Z2C6iZioE2mvy0K3nmyxeWXg0lg=">AAAC0HicjVHLLgRBFD3T3uM1WNp0TCRWnerxaHbCxoIEMUbCZFJdCh390l0tZCJi6wds+SrxB/yFW6UnYSFUp7tvnXvOqbr3+mkY5Iqxt4rV1z8wODQ8Uh0dG5+YrE1NH+ZJkQnZFEmYZEc+z2UYxLKpAhXKozSTPPJD2fIvN3W+dS2zPEjiA3WbynbEz+PgLBBcEdQ+ibi6EDzsbt91djq1OnO8lSVvddFmzuIyazCXArbWcD3Pdh1mVh3l2k1qrzjBKRIIFIggEUNRHIIjp+cYLhhSwtroEpZRFJi8xB2qpC2IJYnBCb2k7zntjks0pr32zI1a0CkhvRkpbcyTJiFeRrE+zTb5wjhr9DfvrvHUd7ulv196RYQqXBD6l67H/K9O16JwhlVTQ0A1pQbR1YnSpTBd0Te3v1WlyCElTMenlM8oFkbZ67NtNLmpXfeWm/y7YWpU70XJLfChb0kD7k3R/j04bDguc9y9pfr6RjnqYcxiDgs0Tw/r2MIumuR9hSc848Xat26se+vhi2pVSs0Mfizr8RMUd5Tr</latexit> <latexit sha1_base64="Z2C6iZioE2mvy0K3nmyxeWXg0lg=">AAAC0HicjVHLLgRBFD3T3uM1WNp0TCRWnerxaHbCxoIEMUbCZFJdCh390l0tZCJi6wds+SrxB/yFW6UnYSFUp7tvnXvOqbr3+mkY5Iqxt4rV1z8wODQ8Uh0dG5+YrE1NH+ZJkQnZFEmYZEc+z2UYxLKpAhXKozSTPPJD2fIvN3W+dS2zPEjiA3WbynbEz+PgLBBcEdQ+ibi6EDzsbt91djq1OnO8lSVvddFmzuIyazCXArbWcD3Pdh1mVh3l2k1qrzjBKRIIFIggEUNRHIIjp+cYLhhSwtroEpZRFJi8xB2qpC2IJYnBCb2k7zntjks0pr32zI1a0CkhvRkpbcyTJiFeRrE+zTb5wjhr9DfvrvHUd7ulv196RYQqXBD6l67H/K9O16JwhlVTQ0A1pQbR1YnSpTBd0Te3v1WlyCElTMenlM8oFkbZ67NtNLmpXfeWm/y7YWpU70XJLfChb0kD7k3R/j04bDguc9y9pfr6RjnqYcxiDgs0Tw/r2MIumuR9hSc848Xat26se+vhi2pVSs0Mfizr8RMUd5Tr</latexit> <latexit sha1_base64="Z2C6iZioE2mvy0K3nmyxeWXg0lg=">AAAC0HicjVHLLgRBFD3T3uM1WNp0TCRWnerxaHbCxoIEMUbCZFJdCh390l0tZCJi6wds+SrxB/yFW6UnYSFUp7tvnXvOqbr3+mkY5Iqxt4rV1z8wODQ8Uh0dG5+YrE1NH+ZJkQnZFEmYZEc+z2UYxLKpAhXKozSTPPJD2fIvN3W+dS2zPEjiA3WbynbEz+PgLBBcEdQ+ibi6EDzsbt91djq1OnO8lSVvddFmzuIyazCXArbWcD3Pdh1mVh3l2k1qrzjBKRIIFIggEUNRHIIjp+cYLhhSwtroEpZRFJi8xB2qpC2IJYnBCb2k7zntjks0pr32zI1a0CkhvRkpbcyTJiFeRrE+zTb5wjhr9DfvrvHUd7ulv196RYQqXBD6l67H/K9O16JwhlVTQ0A1pQbR1YnSpTBd0Te3v1WlyCElTMenlM8oFkbZ67NtNLmpXfeWm/y7YWpU70XJLfChb0kD7k3R/j04bDguc9y9pfr6RjnqYcxiDgs0Tw/r2MIumuR9hSc848Xat26se+vhi2pVSs0Mfizr8RMUd5Tr</latexit> <latexit sha1_base64="Z2C6iZioE2mvy0K3nmyxeWXg0lg=">AAAC0HicjVHLLgRBFD3T3uM1WNp0TCRWnerxaHbCxoIEMUbCZFJdCh390l0tZCJi6wds+SrxB/yFW6UnYSFUp7tvnXvOqbr3+mkY5Iqxt4rV1z8wODQ8Uh0dG5+YrE1NH+ZJkQnZFEmYZEc+z2UYxLKpAhXKozSTPPJD2fIvN3W+dS2zPEjiA3WbynbEz+PgLBBcEdQ+ibi6EDzsbt91djq1OnO8lSVvddFmzuIyazCXArbWcD3Pdh1mVh3l2k1qrzjBKRIIFIggEUNRHIIjp+cYLhhSwtroEpZRFJi8xB2qpC2IJYnBCb2k7zntjks0pr32zI1a0CkhvRkpbcyTJiFeRrE+zTb5wjhr9DfvrvHUd7ulv196RYQqXBD6l67H/K9O16JwhlVTQ0A1pQbR1YnSpTBd0Te3v1WlyCElTMenlM8oFkbZ67NtNLmpXfeWm/y7YWpU70XJLfChb0kD7k3R/j04bDguc9y9pfr6RjnqYcxiDgs0Tw/r2MIumuR9hSc848Xat26se+vhi2pVSs0Mfizr8RMUd5Tr</latexit> Target Labels Input Image !" # #′ Output Image Original Labels !; A) B) <=>? <@""> A B Fig. 2: Schematic representation of the architecture of PrivacyNet for deriving perturbations to obfuscate three attribute classifiers, gender, age, and race, while allowing biometric face matchers to perform well. (A) Different components of the PrivacyNet: generator, source discriminator, attribute classifier, and auxiliary face matcher. (B) Cycle-consistency constraint applied to the generator by transforming an input face image to a target label and reconstructing the original version. Gender Age Race PrivacyNet M: G: A: R: M: G: A: R: M: G: A: R: M: G: A: R: M: G: A: R: M: G: A: R: M: G: A: R: M: G: A: R: Original image Gender Age Race M: G: A: R: M: G: A: R: M: G: A: R: M: G: A: R: M: G: A: R: M: G: A: R: M: G: A: R: M: G: A: R: Original image PrivacyNet: Transforming face images for obfuscating soft-biometric attributes Fig. 1: Illustration of the overall objective of this work: transforming an input face image across three orthogonal axes for imparting multi-attribute privacy selectively while retaining recognition utility. The abbreviated letters are M: Matching, G: Gender, A: Age, and R: Race. for predicting the gender from face images has resulted in models with almost perfect prediction accuracy [28], [30], [31], [32], [33]. Methods for estimating the apparent age from face images are similarly well studied, and current-state of the art methods can predict the apparent age of a person with a prediction error below three years on average [26], [27], [34], [35]. While tremendous progress has been made towards the man observer with high con generating su posed, and th DNN-based m versarial attac [41], [42], [43 attacks raises chine learning [39], [46], [47 robustness of the soft-biome sarial attacks. perturbations investigated t bations for im This scheme to conceal m these perturba derive advers classifier, the p across unseen application, g seen attribute Recently, privacy throu sentation vect ~2018-present: Increasing focus on user privacy
learning (pre-train on related dataset) One- and few-shot learning (train on many tasks with few examples per class) Human-in- the-loop Self-supervised learning (like transfer learning but pre-train on related task) Synthetic data (GAN, autoencoder)
rasbt
takabow
cyberagentdevelopers
iselegant
kimujun
soudai
hakkadaikon
amixedcolor
megascus
furuton
kneath
malarkey
garrettdimon
kastner
bcantrill
aarron
tammyeverts
samlambert
chriscoyier
bkeepers
marcelosomers
eileencodes
![30 LLC / ILLC [179] no yes no yes no](https://files.speakerdeck.com/presentations/860f0d8254a6456eb27eef0a8be96947/slide_29.jpg){kind=link}
