Assignment 5
Points:
Task 1 is 50 points.
Task 2 is 50 points.
BasketTube - the AI Game Analyzer
Basketball games are complex, dynamic environments where players must make quick decisions based on their understanding of the game state. In this assignment, you will explore how to extract player actions from a basketball game such as the one shown below
You can use the following Colab notebook to complete this assignment: Basketball State Extraction
Task 1 - Analyzing Player Performance from Commentary
You should be able to respond to user queries in a chat interface that mimic Google’s responses to queries such as:
Q: “Analyze the player that scored the most in this game”
A: While the video does not provide a final leading scorer for the entire game, Stephen Curry is noted as having the most points among the individual player scores mentioned in the first half, reaching 12 points by (37:12).
Here’s an analysis of his performance in the first half:
Early Struggles and Defensive Coverage: The Lakers made it difficult for Curry early in the game (16:22). He took only three shots initially (15:17) as the Lakers forced him to get rid of the ball, often with two defenders when he came off a pick and roll (15:10-15:16).
First Field Goal: Curry scored his first field goal later in the first half (11:06).
Offensive Impact: By the end of the first half, he came “alive from three-point range” (42:22-42:25), hitting back-to-back three-pointers (37:09-37:12, 42:01-42:08), helping the Warriors extend their lead to 13 points by halftime (42:10).
Overall Role: He was described as the oldest scoring champion in the NBA in 23 years, showcasing his elite scoring ability (19:09-19:12). LeBron James even called him the MVP, stating he had the best season all year (19:23-19:29).
Underrated Aspects: Commentators highlighted his rebounding and finishing around the rim as underrated aspects of his game (18:54-19:06).
Note that this is based on the commentary provided in the video and may not reflect the reality (see task 2).
Task 2 - Verifying Player Actions from Video
You should be able to verify the actions of a player by analyzing the video footage itself. For example, if a user asks about a specific player’s performance, you should be able to provide evidence from the video to support your analysis.
Task 2.1 - Chunking the Video into a series of plays
In basketball, a play is a coordinated, pre-planned sequence of actions designed to create a scoring opportunity or a defensive stop. Plays can be called by the coach or initiated on the court by the point guard, and they rely on timing, spacing, and teamwork rather than improvisation.
There are offensively and defensively oriented plays but you dont have to classify them strictly. Here are some common types of plays:
- Offensive Plays
Offensive plays aim to generate an open shot or an advantageous matchup. They usually include screens, cuts, passes, and positioning.
Examples:
Pick and Roll: A player sets a screen for the ball handler and then rolls toward the basket for a pass.
Isolation (“Iso”): Spreading the floor to let one player attack 1-on-1.
Motion Offense: Continuous movement, passing, and cutting without set positions.
Triangle Offense: Three-player formation (corner, wing, post) that creates spacing and passing options.
- Defensive Plays
Defensive plays are organized systems for stopping the opponent.
Examples:
Man-to-Man Defense: Each defender guards a specific opposing player.
Zone Defense: Defenders guard areas of the court rather than individual players.
Full-Court Press: Applying pressure across the entire court to force turnovers.
Task 2.2 - Analyzing Specific Player Actions
For each play you can focus on the player of interest and recognize their actions. Examples of actions include:
- Shooting: Attempting to score by throwing the ball into the basket.
- Passing: Throwing the ball to a teammate.
- Dribbling: Bouncing the ball while moving around the court.
- Rebounding: Gaining possession of the ball after a missed shot.
- Defending: Preventing an opponent from scoring or advancing the ball.
- Assisting: Helping a teammate score by passing them the ball.
For each identified action, provide the starting timestamp, the action category and a birdseye view of the game as shown below. For example:
| Timestamp | Player | Action | Birds eye view |
|---|---|---|---|
| 12:34 | Stephen Curry | 3-Point Shot | |
| 15:20 | LeBron James | Assist |
You are free to select any Vision Language Model (VLM) or combination of models to help you analyze the video footage and extract the relevant player actions.