Awais Rauf

Selected Publications

1. TPAMI

   Foundational Models Defining a New Era in Vision: A Survey and Outlook

   Awais, Muhammad, Muzammal, Naseer, Khan, Salman, Anwer, Rao, Cholakkal, Hisham, Shah, Mubarak, Yang, Ming-Hsuan, and Khan, Fahad Shahbaz

   IEEE Transactions on Pattern Analysis and Machine Intelligence 2024

   PDF Code
2. arXiv

   Leveraging Pre-trained Models for Efficient Cross-task Robustness Transfer for Object Detection

   Awais, Muhammad, Weiming, Zhuang, Lingjuan, Lyu, and Sung-Ho, Bae

   Under Review (arXiv) 2024

   Abs PDF

   Object detection is an important task in computer vision and plays an essential role in many security-critical systems. Despite significant recent advancements, state-of-the-art object detection models remain vulnerable to small adversarial perturbations, which malicious actors can exploit to manipulate the model’s output. Unlike classification, the robustness of object detection models has received relatively limited attention, with most existing work focusing on adapting adversarial training. In this paper, we take a different approach, leveraging classification-based robust models to enhance the robustness of object detection. However, directly incorporating a classification-based robust backbone into object detection training can result in catastrophic forgetting of robust features. To address this issue, we propose efficient modifications to the robust backbone, enabling adversarial robustness with standard training. Additionally, we introduce a theoretically grounded, efficient two-phase training strategy that achieves state-of-the-art robustness. Our method first trains on normal examples while preserving robust features in the backbone, followed by adversarial training on single-step adversarial examples in the final stages of training. Our approach achieves state-of-the-art robustness and significantly improves efficiency compared to traditional adversarial training methods. Extensive experiments on the MS-COCO and Pascal VOC datasets validate the effectiveness of our proposed approach.

3. ECCV

   Efficient 3D-Aware Facial Image Editing via Attribute-Specific Prompt Learning

   Kumar, Amandeep*, Awais, Muhammad*, Narayan, Sanath, Cholakkal, Hisham, Khan, Salman, and Anwer, Rao

   European Conference on Computer Vision 2024

   HTML Slides
4. MICCAI

   BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning

   Hanif, Asif, Shamshad, Fahad, Awais, Muhammad, Naseer, Muzammal, Khan, Fahad Shahbaz, Nandakumar, Karthik, Khan, Salman, and Anwer, Rao

   2024

   Abs HTML PDF Code

   Medical foundation models are gaining prominence in the medical community for their ability to derive general representations from extensive collections of medical image-text pairs. Recent research indicates that these models are susceptible to backdoor attacks, which allow them to classify clean images accurately but fail when specific triggers are introduced. However, traditional backdoor attacks necessitate a considerable amount of additional data to maliciously pre-train a model. This requirement is often impractical in medical imaging applications due to the usual scarcity of data. Inspired by the latest developments in learnable prompts, this work introduces a method to embed a backdoor into the medical foundation model during the prompt learning phase. By incorporating learnable prompts within the text encoder and introducing imperceptible learnable noise trigger to the input images, we exploit the full capabilities of the medical foundation models (Med-FM). Our method, BAPLe, requires only a minimal subset of data to adjust the noise trigger and the text prompts for downstream tasks, enabling the creation of an effective backdoor attack. Through extensive experiments with four medical foundation models, each pre-trained on different modalities and evaluated across six downstream datasets, we demonstrate the efficacy of our approach. BAPLe achieves a high backdoor success rate across all models and datasets, outperforming the baseline backdoor attack methods. Our work highlights the vulnerability of Med-FMs towards backdoor attacks and strives to promote the safe adoption of Med-FMs before their deployment in real-world applications. We believe that our work will help the medical community understand the potential risks associated with deploying Med-FMs and encourage the development of robust and secure models.

5. NeurIPS

   ZooD: Exploiting model zoo for out-of-distribution generalization

   Qishi, Dong*, Awais, Muhammad*, Zhou, Fengwei, Xie, Chuanlong, Hu, Tianyang, Yang, Yongxin, Sung-Ho, Bae, and Li, Zhenguo

   Conference on Neural Information Processing Systems (NeurIPS) 2022

   PDF Video
6. NeurIPS

   MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Maps

   Awais, Muhammad*, Fengwei, Zhou*, Chuanlong, Xie*, Jiawei, Li, Sung-Ho, Bae, and Zhenguo, Li

   Conference on Neural Information Processing Systems (NeurIPS) 2021

   Abs PDF Poster Video

   Deep neural networks are susceptible to adversarially crafted, small and imperceptible changes in the natural inputs. The most effective defense mechanism against these examples is adversarial training which constructs adversarial examples during training by iterative maximization of loss. The model is then trained to minimize the loss on these constructed examples. This min-max optimization requires more data, larger capacity models, and additional computing resources. It also degrades the standard generalization performance of a model. Can we achieve robustness more efficiently? In this work, we explore this question from the perspective of knowledge transfer. First, we theoretically show the transferability of robustness from an adversarially trained teacher model to a student model with the help of mixup augmentation. Second, we propose a novel robustness transfer method called Mixup-Based Activated Channel Maps (MixACM) Transfer. MixACM transfers robustness from a robust teacher to a student by matching activated channel maps generated without expensive adversarial perturbations. Finally, extensive experiments on multiple datasets and different learning scenarios show our method can transfer robustness while also improving generalization on natural images.

7. ICCV

   Adversarial Robustness for Unsupervised Domain Adaptation

   Awais, Muhammad, Fengwei, Zhou, Hang, Xu, Lanqing, Hong, Ping, Luo, Sung-Ho, Bae, and Zhenguo, Li

   Internatioanl Conference on Computer Vision (ICCV) 2021

   Abs HTML PDF

   Extensive Unsupervised Domain Adaptation (UDA) studies have shown great success in practice by learning transferable representations across a labeled source domain and an unlabeled target domain with deep models. However, previous works focus on improving the generalization ability of UDA models on clean examples without considering the adversarial robustness, which is crucial in real-world applications. Conventional adversarial training methods are not suitable for the adversarial robustness on the unlabeled target domain of UDA since they train models with adversarial examples generated by the supervised loss function. In this work, we leverage intermediate representations learned by multiple robust ImageNet models to improve the robustness of UDA models. Our method works by aligning the features of the UDA model with the robust features learned by ImageNet pre-trained models along with domain adaptation training. It utilizes both labeled and unlabeled domains and instills robustness without any adversarial intervention or label requirement during domain adaptation training. Experimental results show that our method significantly improves adversarial robustness compared to the baseline while keeping clean accuracy on various UDA benchmarks.

8. TNNLS

   Revisiting Internal Covariant Shift for Batch Normalization

   Awais, Muhammad, Md. Tauhid Bin, Iqbal, and Sung-Ho, Bae

   IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2020

   Abs HTML

   Despite the success of Batch Normalization (BatchNorm) and a plethora of its variants, the exact reasons for its success are still shady. The original BatchNorm paper explained it as a mechanism that reduces the Internal Covariate Shift (ICS), i.e., the distribution shifts in the input of the layers during training. Recently, some papers manifested skepticism on this hypothesis and provided \textitalternative explanations for the success of BatchNorm, such as the applicability of very high learning rates and the ability to smooth the landscape in optimization. In this work, we counter this textit alternative arguments by demonstrating the importance of reduction in ICS following an empirical approach. We demonstrated various ways to achieve the above-mentioned alternative properties without any performance boost. In this light, we explored the importance of different BatchNorm parameters (i.e., batch statistics, affine transformation parameters) by visualizing their effectiveness in the performance and analyzed their connections with ICS. Afterward, we showed a different normalization scheme that fulfills all alternative explanations except reduction in ICS. Despite having all the alternative properties, we observed its poor performance, which nullifies the alternative claims rather signifies the importance of the ICS reduction. We performed comprehensive experiments on many variants of BatchNorm, finding that all of them similarly reduce ICS.