Updates

  • (2025-09-18) The TransferBench paper has been accepted to NeurIPS D&B Track!

TransferBench

The TransferBench framework aims to systematically evaluate ensemble-based black-box transfer attacks in a fair and reproducible way. It defines a structured protocol across multiple controlled scenarios, addressing inconsistencies in surrogate model selection, query budgets, and victim model robustness.

  • We first define 16 different attack scenarios across two datasets, including combinations of homogeneous and heterogeneous surrogate model architectures, as well as robust and non-robust victim models.
  • Afterward, we evaluate each attack in terms of attack success rate and average queries per success, with consistent query budgets across all scenarios to ensure comparability.
  • Finally, we provide a complete open-source benchmarking suite with open-source code, allowing reproducible evaluation and community contribution.

Experimental coverage

2

Datasets

15

Attacks

11

Target Models

6

Scenarios

Leaderboard

For ImageNet, we evaluate transferability across 8 victim models starting from three different scenarios. In the Heterogeneous setting, we include: The Homeogeneous setting : Finally, the Robust+Homogeneous setting includes robust victim models :

ImageNet

Attack ASR (%) ▲ #Queries ▲

    CIFAR-10

    Attack ASR (%) ▲ #Queries ▲

      Authors


      Fabio Brau
      University of Cagliari
      Maura Pintor
      University of Cagliari
      Antonio Emanuele Cinà
      University of Genoa
      Raffaele Mura
      University of Cagliari
      Luca Scionis
      University of Cagliari
      Luca Oneto
      University of Cagliari
      Fabio Roli
      University of Genoa
      Battista Biggio
      University of Cagliari

      Acknowledgments


      TransferBench has been partially supported by the EU—NGEU National Sustainable Mobility Center (CN00000023), Italian Ministry of University and Research Decree n. 1033—17/06/2022 (Spoke 10); the project Sec4AI4Sec, under the EU’s Horizon Europe Research and Innovation Programme (grant agreement no. 101120393); the project ELSA, under the EU’s Horizon Europe Research and Innovation Programme (grant agreement no. 101070617); and projects SERICS (PE00000014) and FAIR (PE0000013) under the MUR NRRP funded by the EU—NGEU.

      +