This is our research code for CVPR 2022 paper: SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning.

We present SwinBERT, an end-to-end transformer-based model for video captioning. SwinBERT takes video frame patches directly as inputs, and outputs a natural language description. In this repository, we provide our research code for training and testing SwinBERT for video captioning.

• 05/05/2022: Init release

• Training and evaluation code
• Inference code
• Models and training logs
• TSV dataset annotations
• Tutorial for Frame-based TSV generation

• Model Card
• Requirements
• Download
• Launch Docker Container
• Quick Demo
• Evaluation
• Training
• Citation
• License
• Acknowledgments

• We release our best performing checkpoints for each dataset (corresponding to Table 1 in our paper). For clarity, we report performance on both validation and test splits below.

• We also report our results on private test splits, where the scores are obtained from VALUE Leaderboard Evaluation Server.

• We also release our 32-frame model below.

• Note: All results are based on single model. No CIDEr optimization used in our experiments.

We provide a Docker image for easier reproduction. Please install the following:

• nvidia driver (418+),
• Docker (19.03+),
• nvidia-container-toolkit.

We only support Linux with NVIDIA GPUs. We test on Ubuntu 18.04 and V100 cards. We use mixed-precision training hence GPUs with Tensor Cores are recommended. Our scripts require the user to have the docker group membership so that docker commands can be run without sudo.

1. Create folders that store pretrained models, datasets, and predictions.

   export REPO_DIR=$PWD
   mkdir -p $REPO_DIR/models  # pre-trained models
   mkdir -p $REPO_DIR/datasets  # datasets
   mkdir -p $REPO_DIR/predictions  # prediction outputs

2. Download pretrained models.

   Our pre-trained models can be downloaded with the following command.

   cd $REPO_DIR
   bash scripts/download_models.sh

   The script will download our models that are trained for VATEX, MSRVTT, MSVD, TVC and YouCook2, respectively. It will also download our training logs and output predictions.

   The resulting data structure should follow the hierarchy as below.

   ${REPO_DIR}  
   |-- models  
   |   |-- table1
   |   |   |-- vatex
   |   |   |   |-- best-checkpoint
   |   |   |   |   |-- model.bin
   |   |   |   |   |-- optmizer_state.bin
   |   |   |   |   |-- pred.*
   |   |   |   |-- tokenizer
   |   |   |   |   |-- added_tokens.json
   |   |   |   |   |-- special_tokens_map.json
   |   |   |   |   |-- vocab.txt
   |   |   |   |-- log
   |   |   |   |   |-- log.txt
   |   |   |   |   |-- args.json
   |   |   |-- msrvtt
   |   |   |-- msvd
   |   |   |-- tvc
   |   |   |-- youcook2
   |   |-- 32frm
   |   |   |-- vatex
   |   |   |   |-- best-checkpoint
   |   |   |   |   |-- model.bin
   |   |   |   |   |-- optmizer_state.bin
   |   |   |   |   |-- pred.*
   |   |   |   |-- tokenizer
   |   |   |   |   |-- added_tokens.json
   |   |   |   |   |-- special_tokens_map.json
   |   |   |   |   |-- vocab.txt
   |   |   |   |-- log
   |   |   |   |   |-- log.txt
   |   |   |   |   |-- args.json
   |   |   |-- msrvtt
   |   |   |-- msvd
   |   |   |-- tvc
   |   |   |-- youcook2
   |-- docs 
   |-- src
   |-- scripts 
   |-- README.md 
   |-- ... 
   |-- ... 
   

3. Download pretrained Video Swin Transformers.

   To run our code smoothly, please visit Video Swin Transformer to download pre-trained weights models.

   Download swin_base_patch244_window877_kinetics*_22k.pth, and place them under ${REPO_DIR}/models/video_swin_transformer directory. The data structure should follow the hierarchy below.

   ${REPO_DIR}  
   |-- models  
   |   |-- video_swin_transformer
   |    |   |-- swin_base_patch244_window877_kinetics600_22k.pth
   |    |   |-- swin_base_patch244_window877_kinetics400_22k.pth
   |   |-- table1
   |   |-- 32frm
   |-- docs 
   |-- src
   |-- scripts 
   |-- README.md 
   |-- ... 
   |-- ... 
   

4. Download prediction files that were evaluated on VALUE Leaderboard Evaluation Server

   The prediction files can be downloaded with the following command.

   cd $REPO_DIR
   bash scripts/download_value_preds.sh

   You could submit the prediction files to VALUE Leaderboard and reproduce our results.

5. Download datasets for training and evaluation

   In this project, we provide our pre-parsed annotation files in TSV format. To download the files, please use the following command.

   cd $REPO_DIR
   bash scripts/download_annotations.sh

   Following prior studies, we use the standard train/val/test splits for each dataset. Here, we just reorganize the data format in TSV files to better fit our codebase.

   Due to copyright issue, we could not release the raw videos. We suggest downloading the orignal raw videos from the official dataset websites. Please place the downloaded videos under raw_videos or videos of each dataset folder.

   The datasets directory structure should follow the below hierarchy.

   ${ROOT}  
   |-- datasets  
   |   |-- VATEX  
   |   |   |-- *.yaml 
   |   |   |-- *.tsv  
   |   |   |-- raw_videos  <<< please place the downloaded videos under this folder 
   |   |   |   |-- val_all
   |   |   |   |   |-- *.mp4
   |   |   |   |-- holdout_test
   |   |   |   |   |-- test
   |   |   |   |   |   |-- *.mp4
   |   |-- MSRVTT-v2  
   |   |   |-- *.yaml 
   |   |   |-- *.tsv  
   |   |   |-- videos <<< please place the downloaded videos under this folder 
   |   |   |   |-- *.mp4 
   |   |-- MSVD  
   |   |   |-- *.yaml 
   |   |   |-- *.tsv  
   |   |   |-- videos <<< please place the downloaded videos under this folder 
   |   |   |   |-- *.avi 
   |   |-- TVC  
   |   |   |-- *.yaml 
   |   |   |-- *.tsv  
   |   |   |-- videos <<< please place the downloaded videos under this folder 
   |   |   |   |-- bbt_new
   |   |   |   |-- castle
   |   |   |   |-- friends
   |   |   |   |-- grey
   |   |   |   |-- house
   |   |   |   |-- met 
   |   |-- YouCook2  
   |   |   |-- *.yaml 
   |   |   |-- *.tsv  
   |   |   |-- training <<< please place the downloaded training videos under this folder 
   |   |   |   |-- *.mp4 
   |   |   |-- validation <<< please place the downloaded validation videos under this folder 
   |   |   |   |-- *.mp4 
   |   |   |-- testing <<< please place the downloaded testing videos under this folder 
   |   |   |   |-- *.mp4 
   |-- docs
   |-- src
   |-- scripts
   |-- models 
   |-- README.md 
   |-- ... 
   |-- ... 
   
   

   We also provide example scripts to reproduce our annotation tsv files. You may find the examples below.

   ${ROOT}  
   |-- prepro  
   |   |-- tsv_preproc_vatex.py
   |   |-- tsv_preproc_msrvtt.py
   |   |-- tsv_preproc_msvd.py
   |   |-- tsv_preproc_tvc.py
   |   |-- tsv_preproc_youcook2.py
   |-- docs
   |-- src
   |-- scripts
   |-- README.md 
   |-- ... 
   |-- ... 
   
   

We provide a Docker image for easier reproduction. Please launch the docker container before running our codes.

export REPO_DIR=$PWD
DATASETS=$REPO_DIR'/datasets/'
MODELS=$REPO_DIR'/models/'
OUTPUT_DIR=$REPO_DIR'/output/'
source launch_container.sh $DATASETS $MODELS $OUTPUT_DIR

Our latest docker image linjieli222/videocap_torch1.7:fairscale supports the following mixed precision training

• Torch.amp (with limited GPU memory optimization, deprecated from this codebase)
• Nvidia Apex O2
• deepspeed (Best setting on VATEX, deepspeed fp16 with zero_opt_stage=1)
• fairscale

We provide a demo to run end-to-end inference on the test video.

Our inference code will take a video as input, and generate video caption.

# After launching the docker container 
EVAL_DIR='./models/table1/vatex/best-checkpoint/'
CHECKPOINT='./models/table1/vatex/best-checkpoint/model.bin'
VIDEO='./docs/G0mjFqytJt4_000152_000162.mp4'
CUDA_VISIBLE_DEVICES=0 python src/tasks/run_caption_VidSwinBert_inference.py \
       --resume_checkpoint $CHECKPOINT  \
       --eval_model_dir $EVAL_DIR \
       --test_video_fname $VIDEO \
       --do_lower_case \
       --do_test 

The prediction should look like

Prediction: a young boy is showing how to make a paper airplane.

We provide example scripts to evaluate pre-trained checkpoints

# Assume in the docker container 
EVAL_DIR='./models/table1/vatex/best-checkpoint/'
CUDA_VISIBLE_DEVICES=0 python src/tasks/run_caption_VidSwinBert.py \
       --val_yaml VATEX/public_test_128frames.yaml  \
       --do_eval true \
       --do_train false \
       --eval_model_dir $EVAL_DIR

Notes: Our dataloader supports two different modes:

• Online decoding: Extract video frames on-the-fly during experiments. It has less data prepro efforts.
• Offline decoding: Need to store all the extracted frames in a TSV file. But it usually run faster.

For online decoding, please use VATEX/public_test.yaml For offline decoding, please use VATEX/public_test_128frames.yaml

# Assume in the docker container 
EVAL_DIR='./models/table1/msrvtt/best-checkpoint/'
CUDA_VISIBLE_DEVICES=0 python src/tasks/run_caption_VidSwinBert.py \
       --val_yaml MSRVTT-v2/val_128frames.yaml  \
       --do_eval true \
       --do_train false \
       --eval_model_dir $EVAL_DIR

For online decoding, please use MSRVTT-v2/val.yaml For offline decoding, please use MSRVTT-v2/val_128frames.yaml

# Assume in the docker container 
EVAL_DIR='./models/table1/youcook2/best-checkpoint/'
CUDA_VISIBLE_DEVICES=0 python src/tasks/run_caption_VidSwinBert.py \
       --val_yaml YouCook2/testing_128frames.yaml  \
       --do_eval true \
       --do_train false \
       --eval_model_dir $EVAL_DIR

For online decoding, please use YouCook2/testing.yaml For offline decoding, please use YouCook2/testing_128frames.yaml

# Assume in the docker container 
EVAL_DIR='./models/table1/msvd/best-checkpoint/'
CUDA_VISIBLE_DEVICES=0 python src/tasks/run_caption_VidSwinBert.py \
       --val_yaml MSVD/val_32frames.yaml  \
       --do_eval true \
       --do_train false \
       --eval_model_dir $EVAL_DIR

For online decoding, please use MSVD/val.yaml For offline decoding, please use MSVD/val_32frames.yaml

# Assume in the docker container 
EVAL_DIR='./models/table1/tvc/best-checkpoint/'
CUDA_VISIBLE_DEVICES=0 python src/tasks/run_caption_VidSwinBert.py \
       --val_yaml TVC/val_128frames.yaml  \
       --do_eval true \
       --do_train false \
       --eval_model_dir $EVAL_DIR

For online decoding, please use TVC/val.yaml For offline decoding, please use TVC/val_128frames.yaml

We provide example scripts to train our model (with 32-frame inputs, soft sparse attention)

# Assume in the docker container 
python src/tasks/run_caption_VidSwinBert.py
        --config src/configs/VidSwinBert/vatex_8frm_default.json
        --train_yaml VATEX/train_32frames.yaml
        --val_yaml VATEX/public_test_32frames.yaml
        --per_gpu_train_batch_size 6
        --per_gpu_eval_batch_size 6
        --num_train_epochs 15
        --learning_rate 0.0003
        --max_num_frames 32
        --pretrained_2d 0
        --backbone_coef_lr 0.05
        --mask_prob 0.5
        --max_masked_token 45
        --zero_opt_stage 1
        --mixed_precision_method deepspeed
        --deepspeed_fp16
        --gradient_accumulation_steps 1
        --learn_mask_enabled
        --loss_sparse_w 0.5
        --output_dir ./output

# Assume in the docker container 
python src/tasks/run_caption_VidSwinBert.py
        --config src/configs/VidSwinBert/msrvtt_8frm_default.json
        --train_yaml MSRVTT-v2/train_32frames.yaml
        --val_yaml MSRVTT-v2/val_32frames.yaml
        --per_gpu_train_batch_size 6
        --per_gpu_eval_batch_size 6
        --num_train_epochs 15
        --learning_rate 0.0003
        --max_num_frames 32
        --pretrained_2d 0
        --backbone_coef_lr 0.05
        --mask_prob 0.5
        --max_masked_token 45
        --zero_opt_stage 1
        --mixed_precision_method deepspeed
        --deepspeed_fp16
        --gradient_accumulation_steps 4
        --learn_mask_enabled
        --loss_sparse_w 0.5
        --output_dir ./output

# Assume in the docker container 
python src/tasks/run_caption_VidSwinBert.py
        --config src/configs/VidSwinBert/youcook2_8frm_default.json
        --train_yaml YouCook2/training_128frames.yaml
        --val_yaml YouCook2/validation_128frames.yaml
        --per_gpu_train_batch_size 6
        --per_gpu_eval_batch_size 6
        --num_train_epochs 40
        --learning_rate 0.0003
        --max_num_frames 32
        --pretrained_2d 0
        --backbone_coef_lr 0.05
        --mask_prob 0.5
        --max_masked_token 45
        --zero_opt_stage 1
        --mixed_precision_method deepspeed
        --deepspeed_fp16
        --gradient_accumulation_steps 4
        --learn_mask_enabled
        --loss_sparse_w 0.5
        --output_dir ./output

# Assume in the docker container 
python src/tasks/run_caption_VidSwinBert.py
        --config src/configs/VidSwinBert/msvd_8frm_default.json
        --train_yaml MSVD/train_32frames.yaml
        --val_yaml MSVD/val_32frames.yaml
        --per_gpu_train_batch_size 6
        --per_gpu_eval_batch_size 6
        --num_train_epochs 15
        --learning_rate 0.0003
        --max_num_frames 32
        --pretrained_2d 0
        --backbone_coef_lr 0.05
        --mask_prob 0.5
        --max_masked_token 45
        --zero_opt_stage 1
        --mixed_precision_method deepspeed
        --deepspeed_fp16
        --gradient_accumulation_steps 1
        --learn_mask_enabled
        --loss_sparse_w 0.5
        --output_dir ./output

# Assume in the docker container 
python src/tasks/run_caption_VidSwinBert.py
        --config src/configs/VidSwinBert/tvc_8frm_default.json
        --train_yaml TVC/train_128frames.yaml
        --val_yaml TVC/val_128frames.yaml
        --per_gpu_train_batch_size 6
        --per_gpu_eval_batch_size 6
        --num_train_epochs 40
        --learning_rate 0.0003
        --max_num_frames 32
        --pretrained_2d 0
        --backbone_coef_lr 0.05
        --mask_prob 0.5
        --max_masked_token 45
        --zero_opt_stage 1
        --mixed_precision_method apex
        --amp_opt_level 2
        --gradient_accumulation_steps 1
        --learn_mask_enabled
        --loss_sparse_w 0.1
        --output_dir ./output

If you find our work useful in your research, please consider citing:

@inproceedings{lin2021end-to-end,
title={SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning},
author={Lin, Kevin and Li, Linjie and Lin, Chung-Ching and Ahmed, Faisal and Gan, Zhe and Liu, Zicheng and Lu, Yumao and Wang, Lijuan},
booktitle = {CVPR},
year = {2022},
}

Our research code is released under MIT license.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

We thank Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhengyuan Yang, Ehsan Azarnasab, Yue Cao, Lei Ji, Huaishao Luo and Ze Liu for their helpful discussions.

We also thank the anonymous reviewers for their constructive feedback.

Our code is built on top of open-source GitHub repositories. We thank all the authors who made their code public, which tremendously accelerates our project progress. If you find these works helpful, please consider citing them as well.

huggingface/transformers

jayleicn/ClipBERT

linjieli222/HERO

Microsoft/Oscar&VinVL

Microsoft/DeepSpeed

Nvidia/Apex

FAIR/SlowFast

FAIR/FairScale