🔬research

Selected Publications

1. TPAMI

   Foundational Models Defining a New Era in Vision: A Survey and Outlook

   Awais, Muhammad, Muzammal, Naseer, Khan, Salman, Anwer, Rao, Cholakkal, Hisham, Shah, Mubarak, Yang, Ming-Hsuan, and Khan, Fahad Shahbaz

   IEEE Transactions on Pattern Analysis and Machine Intelligence 2024

   PDF Code
2. arXiv

   Leveraging Pre-trained Models for Efficient Cross-task Robustness Transfer for Object Detection

   Awais, Muhammad, Weiming, Zhuang, Lingjuan, Lyu, and Sung-Ho, Bae

   Under Review (arXiv) 2024

   Abs PDF

   Object detection is an important task in computer vision and plays an essential role in many security-critical systems. Despite significant recent advancements, state-of-the-art object detection models remain vulnerable to small adversarial perturbations, which malicious actors can exploit to manipulate the model’s output. Unlike classification, the robustness of object detection models has received relatively limited attention, with most existing work focusing on adapting adversarial training. In this paper, we take a different approach, leveraging classification-based robust models to enhance the robustness of object detection. However, directly incorporating a classification-based robust backbone into object detection training can result in catastrophic forgetting of robust features. To address this issue, we propose efficient modifications to the robust backbone, enabling adversarial robustness with standard training. Additionally, we introduce a theoretically grounded, efficient two-phase training strategy that achieves state-of-the-art robustness. Our method first trains on normal examples while preserving robust features in the backbone, followed by adversarial training on single-step adversarial examples in the final stages of training. Our approach achieves state-of-the-art robustness and significantly improves efficiency compared to traditional adversarial training methods. Extensive experiments on the MS-COCO and Pascal VOC datasets validate the effectiveness of our proposed approach.

3. ECCV

   Efficient 3D-Aware Facial Image Editing via Attribute-Specific Prompt Learning

   Kumar, Amandeep*, Awais, Muhammad*, Narayan, Sanath, Cholakkal, Hisham, Khan, Salman, and Anwer, Rao

   European Conference on Computer Vision 2024

   HTML Slides
4. MICCAI

   BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning

   Hanif, Asif, Shamshad, Fahad, Awais, Muhammad, Naseer, Muzammal, Khan, Fahad Shahbaz, Nandakumar, Karthik, Khan, Salman, and Anwer, Rao

   2024

   Abs HTML PDF Code

   Medical foundation models are gaining prominence in the medical community for their ability to derive general representations from extensive collections of medical image-text pairs. Recent research indicates that these models are susceptible to backdoor attacks, which allow them to classify clean images accurately but fail when specific triggers are introduced. However, traditional backdoor attacks necessitate a considerable amount of additional data to maliciously pre-train a model. This requirement is often impractical in medical imaging applications due to the usual scarcity of data. Inspired by the latest developments in learnable prompts, this work introduces a method to embed a backdoor into the medical foundation model during the prompt learning phase. By incorporating learnable prompts within the text encoder and introducing imperceptible learnable noise trigger to the input images, we exploit the full capabilities of the medical foundation models (Med-FM). Our method, BAPLe, requires only a minimal subset of data to adjust the noise trigger and the text prompts for downstream tasks, enabling the creation of an effective backdoor attack. Through extensive experiments with four medical foundation models, each pre-trained on different modalities and evaluated across six downstream datasets, we demonstrate the efficacy of our approach. BAPLe achieves a high backdoor success rate across all models and datasets, outperforming the baseline backdoor attack methods. Our work highlights the vulnerability of Med-FMs towards backdoor attacks and strives to promote the safe adoption of Med-FMs before their deployment in real-world applications. We believe that our work will help the medical community understand the potential risks associated with deploying Med-FMs and encourage the development of robust and secure models.

5. NeurIPS

   ZooD: Exploiting model zoo for out-of-distribution generalization

   Qishi, Dong*, Awais, Muhammad*, Zhou, Fengwei, Xie, Chuanlong, Hu, Tianyang, Yang, Yongxin, Sung-Ho, Bae, and Li, Zhenguo

   Conference on Neural Information Processing Systems (NeurIPS) 2022

   PDF Video
6. WACV

   AgroGPT: Efficient Agricultural Vision-Language Model with Expert Tuning

   Awais, Muhammad, Alharthi, Ali Husain Salem Abdulla, Kumar, Amandeep, Cholakkal, Hisham, and Anwer, Rao Muhammad

   IEEE/CVF Winter Conference on Applications of Computer Vision 2025

   Abs

   Significant progress has been made in advancing large multimodal conversational models (LMMs), capitalizing on vast repositories of image-text data available online. Despite this progress, these models often encounter substantial domain gaps, hindering their ability to engage in complex conversations across new domains. Recent efforts have aimed to mitigate this issue, albeit relying on domain-specific image-text data to curate instruction-tuning data. However, many domains, such as agriculture, lack such vision-language data. In this work, we propose an approach to construct instruction-tuning data that harnesses vision-only data for the agriculture domain. We utilize diverse agricultural datasets spanning multiple domains, curate class-specific information, and employ large language models (LLMs) to construct an expert-tuning set, resulting in a 70k expert-tuning dataset called AgroInstruct. Subsequently, we expert-tuned and created AgroGPT, an efficient LMM that can hold complex agriculture-related conversations and provide useful insights. We also develop AgroEvals for evaluation and compare AgroGPT’s performance with large open and closed-source models. AgroGPT excels at identifying fine-grained agricultural concepts, can act as an agriculture expert, and provides helpful information for multimodal agriculture questions. The code, datasets, and models are available at this https URL.

7. NeurIPS

   MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Maps

   Awais, Muhammad*, Fengwei, Zhou*, Chuanlong, Xie*, Jiawei, Li, Sung-Ho, Bae, and Zhenguo, Li

   Conference on Neural Information Processing Systems (NeurIPS) 2021

   Abs PDF Poster Video

   Deep neural networks are susceptible to adversarially crafted, small and imperceptible changes in the natural inputs. The most effective defense mechanism against these examples is adversarial training which constructs adversarial examples during training by iterative maximization of loss. The model is then trained to minimize the loss on these constructed examples. This min-max optimization requires more data, larger capacity models, and additional computing resources. It also degrades the standard generalization performance of a model. Can we achieve robustness more efficiently? In this work, we explore this question from the perspective of knowledge transfer. First, we theoretically show the transferability of robustness from an adversarially trained teacher model to a student model with the help of mixup augmentation. Second, we propose a novel robustness transfer method called Mixup-Based Activated Channel Maps (MixACM) Transfer. MixACM transfers robustness from a robust teacher to a student by matching activated channel maps generated without expensive adversarial perturbations. Finally, extensive experiments on multiple datasets and different learning scenarios show our method can transfer robustness while also improving generalization on natural images.

8. ICCV

   Adversarial Robustness for Unsupervised Domain Adaptation

   Awais, Muhammad, Fengwei, Zhou, Hang, Xu, Lanqing, Hong, Ping, Luo, Sung-Ho, Bae, and Zhenguo, Li

   Internatioanl Conference on Computer Vision (ICCV) 2021

   Abs HTML PDF

   Extensive Unsupervised Domain Adaptation (UDA) studies have shown great success in practice by learning transferable representations across a labeled source domain and an unlabeled target domain with deep models. However, previous works focus on improving the generalization ability of UDA models on clean examples without considering the adversarial robustness, which is crucial in real-world applications. Conventional adversarial training methods are not suitable for the adversarial robustness on the unlabeled target domain of UDA since they train models with adversarial examples generated by the supervised loss function. In this work, we leverage intermediate representations learned by multiple robust ImageNet models to improve the robustness of UDA models. Our method works by aligning the features of the UDA model with the robust features learned by ImageNet pre-trained models along with domain adaptation training. It utilizes both labeled and unlabeled domains and instills robustness without any adversarial intervention or label requirement during domain adaptation training. Experimental results show that our method significantly improves adversarial robustness compared to the baseline while keeping clean accuracy on various UDA benchmarks.

9. TNNLS

   Revisiting Internal Covariant Shift for Batch Normalization

   Awais, Muhammad, Md. Tauhid Bin, Iqbal, and Sung-Ho, Bae

   IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2020

   Abs HTML

   Despite the success of Batch Normalization (BatchNorm) and a plethora of its variants, the exact reasons for its success are still shady. The original BatchNorm paper explained it as a mechanism that reduces the Internal Covariate Shift (ICS), i.e., the distribution shifts in the input of the layers during training. Recently, some papers manifested skepticism on this hypothesis and provided \textitalternative explanations for the success of BatchNorm, such as the applicability of very high learning rates and the ability to smooth the landscape in optimization. In this work, we counter this textit alternative arguments by demonstrating the importance of reduction in ICS following an empirical approach. We demonstrated various ways to achieve the above-mentioned alternative properties without any performance boost. In this light, we explored the importance of different BatchNorm parameters (i.e., batch statistics, affine transformation parameters) by visualizing their effectiveness in the performance and analyzed their connections with ICS. Afterward, we showed a different normalization scheme that fulfills all alternative explanations except reduction in ICS. Despite having all the alternative properties, we observed its poor performance, which nullifies the alternative claims rather signifies the importance of the ICS reduction. We performed comprehensive experiments on many variants of BatchNorm, finding that all of them similarly reduce ICS.

10. arXiv

    Adversarial attacks and batch normalization: a batch statistics perspective

    Awais, Muhammad, Shamshad, Fahad, and Bae, Sung-Ho

    IEEE Access 2023

    PDF
11. JAIHC

    Leveraging big data for politics: predicting general election of Pakistan using a novel rigged model

    Awais, Muhammad, Saeed UL, Hassan, and Ali, Ahmed

    Journal of Ambient Intelligence and Humanized Computing 2019

    HTML
12. ICCVw

    Adaptive Ptych: Leveraging Image Adaptive Generative Priors for Subsampled Fourier Ptychography

    Fahad, Shamshad, Asif, Hanif, Awais, Muhammad, Farwa, Abbas, and Ali, Ahmed

    International Conference on Computer Vision Workshops 2019

    Abs PDF

    Recently pretrained generative models have shown promising results for subsampled Fourier Ptychography (FP) in terms of quality of reconstruction for extremely low sampling rates. However, the representation capabilities of these pretrained generators do not capture the full distribution for complex classes of images, such as human faces or numbers, resulting in representation error. Moreover, recent studies have shown that these pretrained generative priors struggle at high-resolution in imaging inverse problems for reconstructing a faithful estimate of the true image, potentially due to mode collapse issue. To mitigate the issue of representation error of pretrained generative models for subsampled FP, we propose to make pretrained generator image adaptive by modifying it to better represent a single image (at test time) that is consistent with the subsampled FP measurements. Our experimental results demonstrate the superiority of the proposed approach over recent subsampled FP methods in terms of both quantitative metrics and visual quality

13. NIPSw

    Leveraging Deep Stein’s Unbiased Risk Estimator for Unsupervised X-ray Denoising

    Fahad, Shamshad, Awais, Muhammad, Muhammad, Asim, and Ali, Ahmed

    Neural Information Processing Systems Workshops 2018

    HTML PDF
14. ICTD

    Introducing Data mining for Predicting trends in School Education of Pakistan: Preliminary results and Future Directions

    Muhammad, Asim, Fahad, Shamshad, Awais, Muhammad, and Ali, Ahmed

    Conference on Information and Communication Technologies and Development 2017

    HTML PDF