ActEV: Activities in Extended Video


The reporting of leaderboard will be delayed until 12/7/18

AD Leaderboard

Updated: 2018-11-15T07:52:30+00:00
RANK SUBMISSION_ID EVALUATION_NAME TRACK_NAME TEAM_NAME SYSTEM_NAME METRIC_NAME METRIC_VALUE
1 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00069_Team-Vision_20181009-155732-6403.sr-20181009-155732-6965 ActEV-2018 AD Team_Vision IBM_E2E mean-p_miss@0.15rfa 0.7087156226569999
2 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00083_Team-Vision_20181009-134800-1188.sr-20181009-134800-1731 ActEV-2018 AD Team_Vision IBM_E2E_ALL mean-p_miss@0.15rfa 0.718942449806
3 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00068_UCF_20181009-174811-7853.sr-20181009-174811-8151 ActEV-2018 AD UCF UCF mean-p_miss@0.15rfa 0.7329866448710001
4 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00074_BUPT-MCPRL_20181021-065538-7015.sr-20181021-065538-7426 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_ad mean-p_miss@0.15rfa 0.7485354298610001
5 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00077_BUPT-MCPRL_20181024-053432-9108.sr-20181024-053432-9511 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_aod2 mean-p_miss@0.15rfa 0.7507750715189999
6 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00080_BUPT-MCPRL_20181024-053231-6042.sr-20181024-053231-6330 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_ad3 mean-p_miss@0.15rfa 0.7507750715189999
7 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00076_BUPT-MCPRL_20181008-045311-3858.sr-20181008-045311-4151 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_ad2 mean-p_miss@0.15rfa 0.7507750715189999
8 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00075_BUPT-MCPRL_20181008-030636-8198.sr-20181008-030636-8583 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_aod mean-p_miss@0.15rfa 0.7555597605140001
9 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00097_Team-Vision_20181007-203040-8901.sr-20181007-203040-9533 ActEV-2018 AD Team_Vision IBM_E2E_ALL_NEW mean-p_miss@0.15rfa 0.7589621184540001
10 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00085_BUPT-MCPRL_20180928-002728-5146.sr-20180928-002728-5543 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_aod4 mean-p_miss@0.15rfa 0.759068532444
11 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00084_BUPT-MCPRL_20180928-002620-3845.sr-20180928-002620-4152 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_ad4 mean-p_miss@0.15rfa 0.759068532444
12 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00078_BUPT-MCPRL_20180921-234947-8187.sr-20180921-234947-8543 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_aod3 mean-p_miss@0.15rfa 0.759068532444
13 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00067_IBM-MIT-Purdue_20180823-154949-8470.sr-20180823-154949-8999 ActEV-2018 AD IBM-MIT-Purdue IBM mean-p_miss@0.15rfa 0.8396338981479999
14 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00102_INF_20181009-100009-2955.sr-20181009-100009-3440 ActEV-2018 AD INF localresnettrnval_all mean-p_miss@0.15rfa 0.843979455595
15 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00099_INF_20181009-095825-5765.sr-20181009-095825-6169 ActEV-2018 AD INF localresnettrnval mean-p_miss@0.15rfa 0.878536623884
16 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00087_VANT_20181001-041813-0184.sr-20181001-041813-0488 ActEV-2018 AD VANT ACT mean-p_miss@0.15rfa 0.8822791423299999
17 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00094_DIVA-TE-Baseline_20181005-161350-3844.sr-20181005-161350-4153 ActEV-2018 AD DIVA TE Baseline baselineACT_1_AD mean-p_miss@0.15rfa 0.8945666099160001
18 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00103_DIVA-TE-Baseline_20181009-130747-6384.sr-20181009-130747-6737 ActEV-2018 AD DIVA TE Baseline baselineACTMerged_1_AOD mean-p_miss@0.15rfa 0.905986406278
19 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00104_DIVA-TE-Baseline_20181009-130834-9585.sr-20181009-130835-0175 ActEV-2018 AD DIVA TE Baseline baselineACT_1_AOD mean-p_miss@0.15rfa 0.905986406278
20 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00065_NIST-TEST_20181029-144922-2766.sr-20181029-144922-3544 ActEV-2018 AD NIST-TEST Test-System mean-p_miss@0.15rfa 0.905986406278
21 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00091_DIVA-TE-Baseline_20181002-140233-6845.sr-20181002-140233-7141 ActEV-2018 AD DIVA TE Baseline baselineRC3D_1_AD mean-p_miss@0.15rfa 0.906375567631
22 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00093_NII-Hitachi-UIT_20181115-024032-6192.sr-20181115-024032-6592 ActEV-2018 AD NII_Hitachi_UIT baseline mean-p_miss@0.15rfa 0.919943970503
23 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00086_NII-Hitachi-UIT_20181115-025013-5968.sr-20181115-025013-6297 ActEV-2018 AD NII_Hitachi_UIT trail mean-p_miss@0.15rfa 0.919943970503
24 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00081_NII-Hitachi-UIT_20181113-232655-7826.sr-20181113-232655-8173 ActEV-2018 AD NII_Hitachi_UIT init mean-p_miss@0.15rfa 0.9213679349770001
25 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00071_UTS-CETC_20181005-073137-6294.sr-20181005-073137-6625 ActEV-2018 AD UTS-CETC window2 mean-p_miss@0.15rfa 0.924910657484
26 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00088_USF-Bulls_20181008-095726-6807.sr-20181008-095726-7157 ActEV-2018 AD USF Bulls usf1 mean-p_miss@0.15rfa 0.9344478809479999
27 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00095_USF-Bulls_20181008-005831-9550.sr-20181008-005831-9931 ActEV-2018 AD USF Bulls usf2 mean-p_miss@0.15rfa 0.9344478809479999
28 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00098_USF-Bulls_20181009-164339-7900.sr-20181009-164339-8194 ActEV-2018 AD USF Bulls usf-3 mean-p_miss@0.15rfa 0.9344478809479999
29 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00066_USF-Bulls_20181008-013009-7242.sr-20181008-013009-7545 ActEV-2018 AD USF Bulls usf0 mean-p_miss@0.15rfa 0.9344478809479999
30 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00072_UTS-CETC_20180907-024746-6012.sr-20180907-024746-6317 ActEV-2018 AD UTS-CETC base mean-p_miss@0.15rfa 0.951032165623

AOD Leaderboard

Updated: 2018-11-15T07:45:22+00:00
RANK SUBMISSION_ID EVALUATION_NAME TRACK_NAME TEAM_NAME SYSTEM_NAME METRIC_NAME METRIC_VALUE
1 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00070_Team-Vision_20181009-140238-5458.sr-20181009-140238-6040 ActEV-2018 AOD Team_Vision IBM_E2E mean-p_miss@0.15rfa 0.7520320713959999
2 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00083_Team-Vision_20181009-134800-1188.sr-20181009-134800-1731 ActEV-2018 AOD Team_Vision IBM_E2E_ALL mean-p_miss@0.15rfa 0.762919700625
3 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00097_Team-Vision_20181007-203040-8901.sr-20181007-203040-9533 ActEV-2018 AOD Team_Vision IBM_E2E_ALL_NEW mean-p_miss@0.15rfa 0.778533161446
4 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00077_BUPT-MCPRL_20181024-053432-9108.sr-20181024-053432-9511 ActEV-2018 AOD BUPT-MCPRL bupt_mcprl_aod2 mean-p_miss@0.15rfa 0.7859195998270001
5 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00075_BUPT-MCPRL_20181008-030636-8198.sr-20181008-030636-8583 ActEV-2018 AOD BUPT-MCPRL bupt_mcprl_aod mean-p_miss@0.15rfa 0.8487204831540001
6 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00085_BUPT-MCPRL_20180928-002754-1535.sr-20180928-002754-1965 ActEV-2018 AOD BUPT-MCPRL bupt_mcprl_aod4 mean-p_miss@0.15rfa 0.8522292550839999
7 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00079_BUPT-MCPRL_20180921-235304-7372.sr-20180921-235304-7867 ActEV-2018 AOD BUPT-MCPRL bupt_mcprl_aod3 mean-p_miss@0.15rfa 0.8522292550839999
8 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00073_UCF_20180917-135516-1904.sr-20180917-135516-2312 ActEV-2018 AOD UCF UCF mean-p_miss@0.15rfa 0.934219782411
9 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00093_NII-Hitachi-UIT_20181115-024032-6192.sr-20181115-024032-6592 ActEV-2018 AOD NII_Hitachi_UIT baseline mean-p_miss@0.15rfa 0.93871120108
10 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00104_DIVA-TE-Baseline_20181009-130834-9585.sr-20181009-130835-0175 ActEV-2018 AOD DIVA TE Baseline baselineACT_1_AOD mean-p_miss@0.15rfa 0.94080634278
11 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00065_NIST-TEST_20181029-144922-2766.sr-20181029-144922-3544 ActEV-2018 AOD NIST-TEST Test-System mean-p_miss@0.15rfa 0.94080634278
12 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00081_NII-Hitachi-UIT_20181110-003804-3118.sr-20181110-003804-3450 ActEV-2018 AOD NII_Hitachi_UIT init mean-p_miss@0.15rfa 0.942470599576
13 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00101_INF_20181009-154733-3763.sr-20181009-154733-4392 ActEV-2018 AOD INF localresnettrnval_all mean-p_miss@0.15rfa 0.9509898723899999
14 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00103_DIVA-TE-Baseline_20181009-130747-6384.sr-20181009-130747-6737 ActEV-2018 AOD DIVA TE Baseline baselineACTMerged_1_AOD mean-p_miss@0.15rfa 0.964845768267
15 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00100_INF_20181009-154712-8611.sr-20181009-154712-9045 ActEV-2018 AOD INF localresnettrnval mean-p_miss@0.15rfa 0.9712459955130001
Contact Us

For information on data, evaluation code, etc., please email: actev-nist@nist.gov

For ActEV evaluation discussion, please visit our Google Group: https://groups.google.com/a/list.nist.gov/forum/#!forum/trecvid.actev

Activity Examples

An ActEV activity is defined to be “one or more people performing a specified movement or interacting with an object or group of objects”. Activities are annotated by humans using a set of annotation guidelines that specify how to perform the annotation and the criteria to determine if the activity occurred. Each activity is formally defined by five elements:

  • Activity Name - A mnemonic handle for the activity
  • Activity Description - Textual description of the activity
  • Begin time rule definition - The specification of what determines the beginning time of the activity
  • End time rule definition - The specification of what determines the ending time of the activity
  • Required object type list - The list of objects systems are expected to identify for the activity. Note: this aspect of an activity not addressed by ActEV-PC.

For example:

Activity Name
Description and Example Chip Videos

Closing
  • Description: A person closing the door to a vehicle or facility.
  • Start: The event begins 1 s before the door starts to move.
  • End: The event ends after the door stops moving. People in cars who close the car door from within is a closing event if you can still see the person within the car. If the person is not visible once they are in the car, then the closing should not be annotated as an event.
  • Objects associated with the activity : Person; and Door or Vehicle


Vehicle_turning_left
  • Description: A vehicle turning left or right is determined from the POV of the driver of the vehicle. The vehicle may not stop for more than 10 s during the turn.
  • Start: Annotation begins 1 s before vehicle has noticeably changed direction.
  • End: Annotation ends 1 s after the vehicle is no longer changing direction and linear motion has resumed. Note: This event is determined after a reasonable interpretation of the video.
  • Objects associated with the activity : Vehicle


Loading
  • Description: An object moving from person to vehicle.
  • Start: The event begins 2 s before the cargo to be loaded is extended toward the vehicle (i.e., before a person’s posture changes from one of “carrying” to one of “loading”).
  • End: The event ends after the cargo is placed into the vehicle and the person-cargo contact is lost. In the event of occlusion, it ends when the loss of contact is visible.
  • Objects associated with the activity: Person; and Vehicle


The ActEV evalaution will continue to add, modify, and exclude activities each evalaution cycle. The table below provides a historical list of activities and the objects involved for each ActEV Evaluation. In this table, the expected object types are P={Person} and V={Construction_Vehicle, Vehicle}.

Activity Name/Objects Summer18 ActEV LeaderBoard Fall 18 ActEV Self Reported Winter 18 ActEV Prize Challenge (No Objects Required)
Closing (P, V) or (P) X X X
Closing_trunk (P, V) X X X
Entering (P, V) or (P) X X X
Exiting (P, V) or (P) X X X
Interacts (P, V) X
Loading (P, V) X X X
Open_Trunk (P, V) X X X
Opening (P, V) or (P) X X X
Transport_HeavyCarry (P, V) X X X
Unloading (P, V) X X X
Vehicle_turning_left (V) X X X
Vehicle_turning_right (V) X X X
Vehicle_u_turn (V) X X X
Pull (P) X X X
Riding (P) X X X
Talking (P) X X X
activity_carrying (P) X X X
specialized_talking_phone (P) X X X
specialized_texting_phone (P) X X X
Summary
ActEV is an series of evaluations to accelerate development of robust, multi-camera, automatic activity detection algorithms for forensic and real-time alerting applications. ActEV is an extension of the annual TRECVID Surveillance Event Detection (SED) evaluation where systems will also detect and track objects involved in the activities. Each evaluation will challenge systems with new data, system requirements, and/or new activities.

ActEV began with the Summer 2018 Blind and Leaderboard evaluations for 12 activities. The summer evaluation was followed by the ongoing Fall ActEV Self-Reported Evaluation which ends Dec 4, 2018 and included 18 activities. Beginning Dec 12, 2018, the Winter 2018, $50K ActEV Prize Challenge (ActEV-PC) kicks off with the opening of the Leaderboard Evaluation Phase.
What is Activity Detection in Extended Videos?
An ActEV activity is defined to be “one or more people performing a specified movement or interacting with an object or group of objects”. Activity detection technologies process extended video streams, such as those from a CCTV camera, and automatically detects all instances of the activity by: (1) identifying the type of activity, (2) producing a confidence score indicating the presence of instance, (3) temporally localizing the instance by indicating the begin and end times, and (4) optionally, detecting and tracking the objects (people, vehicles, objects) involved in the activity.

Click on the tabs above to see video examples, activity examples, and evaluation tasks
Who Can Participate ?
NIST invites all organizations, particularly universities and corporations, to participate in the ActEV evaluations. Participation is free. NIST does not provide funds to participants however NIST will administer several prize challenges related to ActEV.
How To Participate ?
Participation is easy: go to the Current Evaluations tab above for instructions.
Data
Each ActEV evaluation uses a new video data set, changes the evaluation tasks, or adds/changes activities. The data will be provided in MPEG-4 and AVI formatted files. See the individual evaluation pages for details.
Evaluation Metrics and Tools
The main scoring metrics will be based on detection, temporal localization, and spatio-temporal localization using evaluation measures that include the probability of missed detection and rate of false alarm. See details in the evaluation plans of each evaluation.

NIST maintains the ActEV Scoring Software on the Scoring software for the Activities in Extended Video (ActEV) evaluation GitHub repo.

News
4Dec
ActEV 1B Evaluation ends.

Fall 2018 ActEV Self-Reported Evaluation (Full Website)

December 4, 2018 (11 AM EST): Submissions period ends

December 21, 2018 : NIST reports the scores

Winter 2018 ActEV Prize Challenge (ActEV-PC) (Full Website)

December 12, 2018 - February 28, 2019: Phase 1 - ActEV-PC Open Leaderboard evaluation

March 1, 2019 - May 30, 2019: Phase 2 - ActEV-PC Independent evaluation
ActEV: Video Examples
Below you will find four example videos from our data sets. There are two example views each of indoor and outdoor.
Location View 1 View 2
Indoor
Outdoor
ActEV Evaluation Tasks
Activity detection has been researched for many years and remains an unsolved computer vision challenge that requires many capabilities beyond the current state of the art. The ActEV series supports several evaluation tasks each escalating the difficulty by requiring more specific information from the system. Presently, there are three evalaution tasks defined: 1) Activity Detection (AD), 2) Activity and Object Detection (AOD), and (3) Activity and Object Detection and Tracking (AODT). Each evaluation task is summarized below. For a full description of the evaluation tasks, read the Evaluation Plan for each specific evaluation.
Activity Detection (AD)

For the Activity Detection task, given a target activity, a system automatically detects and temporally localizes all instances of the activity. For a system-identified activity instance to be evaluated as correct, the type of activity must be correct and the temporal overlap must fall within a minimal requirement.
Activity and Object Detection (AOD)

For the Activity and Object Detection task, given a target activity, a system detects and temporally localizes all instances of the activity and spatially detects/localizes the people and/or objects associated with the target activity. For a system-identified instance to be scored as correct, it must meet the temporal overlap criteria for the AD task and in addition meet the spatial overlap of the identified objects during the activity instance.
Activity Object Detection and Tracking (AODT)

For the Activity Object Detection and Tracking task, given a target activity, a system detects and temporally localizes all instances of the activity, spatio-temporally detects/localizes the people and/or objects associated with the target activity, and properly assigns IDs the objects play in the activity. For a system-identified instance to be scored as correct, it must meet the temporal overlap criteria and spatio-temporal overlap of the objects for the AOD task and correctly assign the IDs to the objects as described in the activity definition.