bevformer

[1179902]

I am trying to reproduce the BEVFormer-based evaluation protocol reported in Table 1 and Section 4.2.1 of your paper, and I would like to confirm the exact setup used to compute mAP, NDS, and mAOE on synthetic nuScenes images.

My current understanding is the following: a BEVFormer detector pretrained on the real nuScenes train set is directly evaluated on the synthetic validation images, while the original nuScenes validation annotations are kept unchanged, and the final metrics are computed with the official nuScenes detection evaluation code.

I would be very grateful if you could help clarify the following points:

For the BEVFormer setting in the paper, was the input image resolution during evaluation 256×384?
When evaluating on synthetic images, did you only replace the image paths, or did you also update the camera intrinsics and/or related projection geometry to match the resized synthetic images?
Was the BEVFormer detector in Table 1 an official public checkpoint, or was it retrained under your paper’s setting?
Were the reported mAP, NDS, and mAOE values all obtained by directly running the official nuScenes detection evaluation on the synthetic validation set?
Was the synthetic validation set sample-wise aligned with the original nuScenes validation set, meaning that only the images were replaced while annotations, poses, CAN bus information, and other metadata remained unchanged?

In my own reproduction, the BEVFormer baseline on real images appears normal, but the performance drops significantly when I replace the validation images with synthetic ones. I would therefore like to verify whether my evaluation pipeline is consistent with the protocol used in your paper.

Thank you very much for your time and help.

[NuanBaobao]

Thank you for your interest in our work.

1. Yes, we retrained BEVFormer with an input resolution of 256×384. The results on the real validation set (256×384) can be referred to the Oracle column in Table 1.

2. Yes, the camera intrinsics need to be modified accordingly. Since the images are resized and cropped, both the focal length and the principal point will change, so the original intrinsics cannot be used directly.

3. The BEVFormer used for evaluation in the paper is a detector retrained at 256×384 resolution, rather than directly using the official public checkpoint.

4. Yes, the metrics in Table 1, including mAP, NDS, and mAOE, are all computed based on the predictions of our retrained 256×384 BEVFormer on the corresponding real/synthetic validation set (256×384), using the official nuScenes detection evaluation code.

5. Yes, the synthetic validation set is in one-to-one correspondence with the original nuScenes validation set samples. Since the images are resized/cropped from the original 900×1600 to 256×384, the corresponding camera intrinsics also need to be updated accordingly. In other words, apart from replacing the images and synchronously adjusting the camera intrinsics, the other information remains unchanged.

We hope these clarifications will help you reproduce our evaluation protocol more accurately.