README.md
3.6 KB · 93 lines · markdown Raw
1 ---
2 license: mit
3 pipeline_tag: video-classification
4 tags:
5 - video
6 library_name: transformers
7 datasets:
8 - HuggingFaceM4/something_something_v2
9 base_model:
10 - facebook/vjepa2-vitl-fpc64-256
11 ---
12
13 # V-JEPA 2
14
15 A frontier video understanding model developed by FAIR, Meta, which extends the pretraining objectives of [VJEPA](https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/), resulting in state-of-the-art video understanding capabilities, leveraging data and model sizes at scale.
16 The code is released [in this repository](https://github.com/facebookresearch/vjepa2).
17
18 <div style="background-color: rgba(251, 255, 120, 0.4); padding: 10px; color: black; border-radius: 5px; box-shadow: 0 4px 8px rgba(0,0,0,0.1);">
19 💡 This is V-JEPA 2 <a href="https://huggingface.co/facebook/vjepa2-vitl-fpc64-256">ViT-L 256</a> model with video classification head pretrained on <a href="https://paperswithcode.com/dataset/something-something-v2" style="color: black;">Something-Something-V2</a> dataset.
20 </div>
21 <br></br>
22
23 <img src="https://github.com/user-attachments/assets/914942d8-6a1e-409d-86ff-ff856b7346ab">&nbsp;
24
25 ## Installation
26
27 To run V-JEPA 2 model, ensure you have installed the latest transformers:
28
29 ```bash
30 pip install -U git+https://github.com/huggingface/transformers
31 ```
32
33 ## Video classification code snippet
34
35 ```python
36 import torch
37 import numpy as np
38
39 from torchcodec.decoders import VideoDecoder
40 from transformers import AutoVideoProcessor, AutoModelForVideoClassification
41
42 device = "cuda" if torch.cuda.is_available() else "cpu"
43
44 # Load model and video preprocessor
45 hf_repo = "facebook/vjepa2-vitl-fpc16-256-ssv2"
46
47 model = AutoModelForVideoClassification.from_pretrained(hf_repo).to(device)
48 processor = AutoVideoProcessor.from_pretrained(hf_repo)
49
50 # To load a video, sample the number of frames according to the model.
51 video_url = "https://huggingface.co/datasets/nateraw/kinetics-mini/resolve/main/val/bowling/-WH-lxmGJVY_000005_000015.mp4"
52 vr = VideoDecoder(video_url)
53 frame_idx = np.arange(0, model.config.frames_per_clip, 8) # you can define more complex sampling strategy
54 video = vr.get_frames_at(indices=frame_idx).data # frames x channels x height x width
55
56 # Preprocess and run inference
57 inputs = processor(video, return_tensors="pt").to(model.device)
58 with torch.no_grad():
59 outputs = model(**inputs)
60 logits = outputs.logits
61
62 print("Top 5 predicted class names:")
63 top5_indices = logits.topk(5).indices[0]
64 top5_probs = torch.softmax(logits, dim=-1).topk(5).values[0]
65 for idx, prob in zip(top5_indices, top5_probs):
66 text_label = model.config.id2label[idx.item()]
67 print(f" - {text_label}: {prob:.2f}")
68 ```
69 Output:
70 ```
71 Top 5 predicted class names:
72 - Stuffing [something] into [something]: 0.34
73 - Putting [something] into [something]: 0.25
74 - Putting [something] onto [something]: 0.04
75 - Spreading [something] onto [something]: 0.04
76 - Closing [something]: 0.03
77 ```
78
79 ## Citation
80
81 ```
82 @techreport{assran2025vjepa2,
83 title={V-JEPA~2: Self-Supervised Video Models Enable Understanding, Prediction and Planning},
84 author={Assran, Mahmoud and Bardes, Adrien and Fan, David and Garrido, Quentin and Howes, Russell and
85 Komeili, Mojtaba and Muckley, Matthew and Rizvi, Ammar and Roberts, Claire and Sinha, Koustuv and Zholus, Artem and
86 Arnaud, Sergio and Gejji, Abha and Martin, Ada and Robert Hogan, Francois and Dugas, Daniel and
87 Bojanowski, Piotr and Khalidov, Vasil and Labatut, Patrick and Massa, Francisco and Szafraniec, Marc and
88 Krishnakumar, Kapil and Li, Yong and Ma, Xiaodong and Chandar, Sarath and Meier, Franziska and LeCun, Yann and
89 Rabbat, Michael and Ballas, Nicolas},
90 institution={FAIR at Meta},
91 year={2025}
92 }
93 ```