TRECVID 2018 ActEV: Activities in Extended Video


AD Leaderboard

Updated: 2018-11-15T07:52:30+00:00
RANK SUBMISSION_ID EVALUATION_NAME TRACK_NAME TEAM_NAME SYSTEM_NAME METRIC_NAME METRIC_VALUE
1 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00069_Team-Vision_20181009-155732-6403.sr-20181009-155732-6965 ActEV-2018 AD Team_Vision IBM_E2E mean-p_miss@0.15rfa 0.7087156226569999
2 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00083_Team-Vision_20181009-134800-1188.sr-20181009-134800-1731 ActEV-2018 AD Team_Vision IBM_E2E_ALL mean-p_miss@0.15rfa 0.718942449806
3 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00068_UCF_20181009-174811-7853.sr-20181009-174811-8151 ActEV-2018 AD UCF UCF mean-p_miss@0.15rfa 0.7329866448710001
4 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00074_BUPT-MCPRL_20181021-065538-7015.sr-20181021-065538-7426 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_ad mean-p_miss@0.15rfa 0.7485354298610001
5 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00077_BUPT-MCPRL_20181024-053432-9108.sr-20181024-053432-9511 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_aod2 mean-p_miss@0.15rfa 0.7507750715189999
6 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00080_BUPT-MCPRL_20181024-053231-6042.sr-20181024-053231-6330 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_ad3 mean-p_miss@0.15rfa 0.7507750715189999
7 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00076_BUPT-MCPRL_20181008-045311-3858.sr-20181008-045311-4151 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_ad2 mean-p_miss@0.15rfa 0.7507750715189999
8 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00075_BUPT-MCPRL_20181008-030636-8198.sr-20181008-030636-8583 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_aod mean-p_miss@0.15rfa 0.7555597605140001
9 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00097_Team-Vision_20181007-203040-8901.sr-20181007-203040-9533 ActEV-2018 AD Team_Vision IBM_E2E_ALL_NEW mean-p_miss@0.15rfa 0.7589621184540001
10 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00085_BUPT-MCPRL_20180928-002728-5146.sr-20180928-002728-5543 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_aod4 mean-p_miss@0.15rfa 0.759068532444
11 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00084_BUPT-MCPRL_20180928-002620-3845.sr-20180928-002620-4152 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_ad4 mean-p_miss@0.15rfa 0.759068532444
12 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00078_BUPT-MCPRL_20180921-234947-8187.sr-20180921-234947-8543 ActEV-2018 AD BUPT-MCPRL bupt_mcprl_aod3 mean-p_miss@0.15rfa 0.759068532444
13 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00067_IBM-MIT-Purdue_20180823-154949-8470.sr-20180823-154949-8999 ActEV-2018 AD IBM-MIT-Purdue IBM mean-p_miss@0.15rfa 0.8396338981479999
14 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00102_INF_20181009-100009-2955.sr-20181009-100009-3440 ActEV-2018 AD INF localresnettrnval_all mean-p_miss@0.15rfa 0.843979455595
15 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00099_INF_20181009-095825-5765.sr-20181009-095825-6169 ActEV-2018 AD INF localresnettrnval mean-p_miss@0.15rfa 0.878536623884
16 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00087_VANT_20181001-041813-0184.sr-20181001-041813-0488 ActEV-2018 AD VANT ACT mean-p_miss@0.15rfa 0.8822791423299999
17 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00094_DIVA-TE-Baseline_20181005-161350-3844.sr-20181005-161350-4153 ActEV-2018 AD DIVA TE Baseline baselineACT_1_AD mean-p_miss@0.15rfa 0.8945666099160001
18 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00103_DIVA-TE-Baseline_20181009-130747-6384.sr-20181009-130747-6737 ActEV-2018 AD DIVA TE Baseline baselineACTMerged_1_AOD mean-p_miss@0.15rfa 0.905986406278
19 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00104_DIVA-TE-Baseline_20181009-130834-9585.sr-20181009-130835-0175 ActEV-2018 AD DIVA TE Baseline baselineACT_1_AOD mean-p_miss@0.15rfa 0.905986406278
20 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00065_NIST-TEST_20181029-144922-2766.sr-20181029-144922-3544 ActEV-2018 AD NIST-TEST Test-System mean-p_miss@0.15rfa 0.905986406278
21 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00091_DIVA-TE-Baseline_20181002-140233-6845.sr-20181002-140233-7141 ActEV-2018 AD DIVA TE Baseline baselineRC3D_1_AD mean-p_miss@0.15rfa 0.906375567631
22 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00093_NII-Hitachi-UIT_20181115-024032-6192.sr-20181115-024032-6592 ActEV-2018 AD NII_Hitachi_UIT baseline mean-p_miss@0.15rfa 0.919943970503
23 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00086_NII-Hitachi-UIT_20181115-025013-5968.sr-20181115-025013-6297 ActEV-2018 AD NII_Hitachi_UIT trail mean-p_miss@0.15rfa 0.919943970503
24 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00081_NII-Hitachi-UIT_20181113-232655-7826.sr-20181113-232655-8173 ActEV-2018 AD NII_Hitachi_UIT init mean-p_miss@0.15rfa 0.9213679349770001
25 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00071_UTS-CETC_20181005-073137-6294.sr-20181005-073137-6625 ActEV-2018 AD UTS-CETC window2 mean-p_miss@0.15rfa 0.924910657484
26 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00088_USF-Bulls_20181008-095726-6807.sr-20181008-095726-7157 ActEV-2018 AD USF Bulls usf1 mean-p_miss@0.15rfa 0.9344478809479999
27 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00095_USF-Bulls_20181008-005831-9550.sr-20181008-005831-9931 ActEV-2018 AD USF Bulls usf2 mean-p_miss@0.15rfa 0.9344478809479999
28 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00098_USF-Bulls_20181009-164339-7900.sr-20181009-164339-8194 ActEV-2018 AD USF Bulls usf-3 mean-p_miss@0.15rfa 0.9344478809479999
29 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00066_USF-Bulls_20181008-013009-7242.sr-20181008-013009-7545 ActEV-2018 AD USF Bulls usf0 mean-p_miss@0.15rfa 0.9344478809479999
30 ActEV-2018_AD_Phase-AD-ActEV-Leader_SYS-00072_UTS-CETC_20180907-024746-6012.sr-20180907-024746-6317 ActEV-2018 AD UTS-CETC base mean-p_miss@0.15rfa 0.951032165623

AOD Leaderboard

Updated: 2018-11-15T07:45:22+00:00
RANK SUBMISSION_ID EVALUATION_NAME TRACK_NAME TEAM_NAME SYSTEM_NAME METRIC_NAME METRIC_VALUE
1 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00070_Team-Vision_20181009-140238-5458.sr-20181009-140238-6040 ActEV-2018 AOD Team_Vision IBM_E2E mean-p_miss@0.15rfa 0.7520320713959999
2 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00083_Team-Vision_20181009-134800-1188.sr-20181009-134800-1731 ActEV-2018 AOD Team_Vision IBM_E2E_ALL mean-p_miss@0.15rfa 0.762919700625
3 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00097_Team-Vision_20181007-203040-8901.sr-20181007-203040-9533 ActEV-2018 AOD Team_Vision IBM_E2E_ALL_NEW mean-p_miss@0.15rfa 0.778533161446
4 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00077_BUPT-MCPRL_20181024-053432-9108.sr-20181024-053432-9511 ActEV-2018 AOD BUPT-MCPRL bupt_mcprl_aod2 mean-p_miss@0.15rfa 0.7859195998270001
5 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00075_BUPT-MCPRL_20181008-030636-8198.sr-20181008-030636-8583 ActEV-2018 AOD BUPT-MCPRL bupt_mcprl_aod mean-p_miss@0.15rfa 0.8487204831540001
6 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00085_BUPT-MCPRL_20180928-002754-1535.sr-20180928-002754-1965 ActEV-2018 AOD BUPT-MCPRL bupt_mcprl_aod4 mean-p_miss@0.15rfa 0.8522292550839999
7 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00079_BUPT-MCPRL_20180921-235304-7372.sr-20180921-235304-7867 ActEV-2018 AOD BUPT-MCPRL bupt_mcprl_aod3 mean-p_miss@0.15rfa 0.8522292550839999
8 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00073_UCF_20180917-135516-1904.sr-20180917-135516-2312 ActEV-2018 AOD UCF UCF mean-p_miss@0.15rfa 0.934219782411
9 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00093_NII-Hitachi-UIT_20181115-024032-6192.sr-20181115-024032-6592 ActEV-2018 AOD NII_Hitachi_UIT baseline mean-p_miss@0.15rfa 0.93871120108
10 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00104_DIVA-TE-Baseline_20181009-130834-9585.sr-20181009-130835-0175 ActEV-2018 AOD DIVA TE Baseline baselineACT_1_AOD mean-p_miss@0.15rfa 0.94080634278
11 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00065_NIST-TEST_20181029-144922-2766.sr-20181029-144922-3544 ActEV-2018 AOD NIST-TEST Test-System mean-p_miss@0.15rfa 0.94080634278
12 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00081_NII-Hitachi-UIT_20181110-003804-3118.sr-20181110-003804-3450 ActEV-2018 AOD NII_Hitachi_UIT init mean-p_miss@0.15rfa 0.942470599576
13 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00101_INF_20181009-154733-3763.sr-20181009-154733-4392 ActEV-2018 AOD INF localresnettrnval_all mean-p_miss@0.15rfa 0.9509898723899999
14 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00103_DIVA-TE-Baseline_20181009-130747-6384.sr-20181009-130747-6737 ActEV-2018 AOD DIVA TE Baseline baselineACTMerged_1_AOD mean-p_miss@0.15rfa 0.964845768267
15 ActEV-2018_AOD_Phase-AOD-ActEV-Leader_SYS-00100_INF_20181009-154712-8611.sr-20181009-154712-9045 ActEV-2018 AOD INF localresnettrnval mean-p_miss@0.15rfa 0.9712459955130001
Contact Us

For ActEV 1B Evaluation information (data, evaluation code, etc.) please email: actev-nist@nist.gov

For ActEV 1B Evaluation Discussion Please Visit our Google Group. https://groups.google.com/a/list.nist.gov/forum/#!forum/trecvid.actev

Activities and Tasks

List of Activities per Task


The table below provides a summary of the activity list and the required objects for the ActEV evaluations. The twelve target activities are used in the 1.A evaluation while nineteen target activities are used in both the ActEV Leaderboard and ActEV 1.B evaluations. The target objects for the ActEV evaluations (1.A, Leaderboard, and 1.B) are P={Person} and V={Construction_Vehicle, Vehicle}.


ActEV 1A Eval ActEV LeaderBoard Eval ActEV 1B Eval
Closing (P, V) or (P)
Closing_trunk (P, V)
Entering (P, V) or (P)
Exiting (P, V) or (P)
Loading (P, V)
Open_Trunk (P, V)
Opening (P, V) or (P)
Transport_HeavyCarry (P, V)
Unloading (P, V)
Vehicle_turning_left (V)
Vehicle_turning_right (V)
Vehicle_u_turn (V)










Closing (P, V) or (P)
Closing_trunk (P, V)
Entering (P, V) or (P)
Exiting (P, V) or (P)
Loading (P, V)
Open_Trunk (P, V)
Opening (P, V) or (P)
Transport_HeavyCarry (P, V)
Unloading (P, V)
Vehicle_turning_left (V)
Vehicle_turning_right (V)
Vehicle_u_turn (V)
Interacts (P)
Pull (P)
Riding (P)
Talking (P)
activity_carrying (P)
specialized_talking_phone (P)
specialized_texting_phone (P)



Closing (P, V) or (P)
Closing_trunk (P, V)
Entering (P, V) or (P)
Exiting (P, V) or (P)
Loading (P, V)
Open_Trunk (P, V)
Opening (P, V) or (P)
Transport_HeavyCarry (P, V)
Unloading (P, V)
Vehicle_turning_left (V)
Vehicle_turning_right (V)
Vehicle_u_turn (V)
Interacts (P)
Pull (P)
Riding (P)
Talking (P)
activity_carrying (P)
specialized_talking_phone (P)
specialized_texting_phone (P)



Different Tasks for the ActEV Evaluation
For the 1A Leaderboard evaluation, we will evaluate for the following two tasks: 1) Activity Detection (AD), and 2) Activity and Object Detection (AOD). Each task can be completed independently. For the TRECVID 2018 1.B ActEV evaluation we add the following task Activity and Object Detection and Tracking (AODT)
Activity Detection (AD)

For the Activity Detection task, given a target activity, a system automatically detects and temporally localizes all instances of the activity. For a system-identified activity instance to be evaluated as correct, the type of activity must be correct and the temporal overlap must fall within a minimal requirement as described in Evaluation Plan.
Activity and Object Detection (AOD)

For the Activity and Object Detection task, given a target activity, a system detects and temporally localizes all instances of the activity and spatially detects/localizes the people and/or objects associated with the target activity. For a system-identified instance to be scored as correct, it must meet the temporal overlap criteria for the AD task and in addition meet the spatial overlap of the identified objects during the activity instance as described in the Evaluation Plan.
Activity Object Detection and Tracking (AODT) (for the 1B evaluation)

For the Activity Object Detection and Tracking task, given a target activity, a system detects and temporally localizes all instances of the activity, spatio-temporally detects/localizes the people and/or objects associated with the target activity, and properly assigns IDs the objects play in the activity. For a system-identified instance to be scored as correct, it must meet the temporal overlap criteria and spatio-temporal overlap of the objects for the AOD task and correctly assign the IDs to the objects as prescribed in the activity definition as described in the Evaluation Plan.

TRECVID ActEV 2018 1B Evaluation

Summary
The ActEV evaluation seeks to evaluate robust automatic activity detection algorithms for a multi-camera streaming video environment. ActEV is an extension of the annual TRECVID Surveillance Event Detection (SED) evaluation by adding a large collection of multi-camera video data. Some of the applications of activity detection of interest are public safety and traffic monitoring and management. ActEV will address activity detection for both forensic applications and for real-time alerting. The ActEV evaluation was initially run as a blind evaluation and is currently open as a leaderboard evaluation. Participants may join the leaderboard evaluation independent of participation in the blind evaluation.
What is Activity Detection in Videos ?
By activity detection, we mean to detect visual events (people engaged in particular activities) in a large collection of streaming video data.
What
The ActEV evaluation is being conducted to assess the robustness of automatic activity detection for a multi-camera streaming video environment.
Who
NIST invites all organizations, particularly universities and corporations, to submit their results using their technologies to the ActEV evaluation server. The evaluation is open worldwide. Participation is free. NIST does not provide funds to participants.
How
To take part in the ActEV evaluation you need to email and register on the actev.nist.gov website. We will then send you information on how to download data and how to create an account on the ActEV scoring server (see website for details). Using a valid registered account, you will be able to upload your JSON format results (see website for: Evaluation Plan) to the ActEV scoring server and participate in the ActEV evaluation. The ActEV evaluation is currently running as a leaderboard evaluation.
Why
The primary driver of the evaluation is to support public safety and traffic monitoring and management by automatic activity detection in streaming video.
Data
The ActEV data used for this evaluation is an unreleased portion of the VIRAT dataset. The data will be provided in MPEG-4 format. (please send an email to receive information: Data download, Licensing and Participation Agreements, and Evaluation Plan).
Metrics
The main scoring metrics will be based on detection, temporal localization and spatio-temporal localization. We will use standard evaluation measures (e.g., probability of missed detection, rate of false alarm) (see: Evaluation Plan).
Evaluation Plan
Form
Please send an email to receive information on how to downloading the Video Dataset.
Task coordinator
ActEV NIST team (ActEV-nist@nist.gov )
News
12Nov
ActEV 1B Evaluation starts.
4Dec
ActEV 1B Evaluation ends.

TRECVID ActEV 2018 Leaderboard Evaluation Schedule

The blind ActEV evaluation is over. Currenty the Leaderboard evaluation is running

June 18, 2018: NIST start of ActEV Leaderboard Evaluation

November 15, 2018 11 AM EST : NIST end of ActEV Leaderboard Evaluation

November 15, 2018: NIST present the ActEV leaderboard evaluation results at TRECVID'18 and WACV workshop




Tentative DIVA 1.B Self-Reported Evaluation Schedule

August 27, 2018: NIST releases 1.B Evaluation Plan

August 27, 2018: Sample Training and Validation sets (same as the Leaderboard evaluation)

September 20, 2018: NIST releases Scoring Software 1.B (AO, AOD, AODT)

September 20, 2018: NIST posts JSON Data Format Specification for 1.B

October 15, 2018: NIST Dry Run for 1.B Evaluation begins on Scoring Server (1.B-Dryrun)

November 5, 2018: NIST Dry Run ends

November 12, 2018: NIST Evaluation Data Unlocked

November 12, 2018: NIST start of 1.B Evaluation Scoring Server

December 04, 2018, 11 AM EST: Performers submit AD, AOD, AODT, self-reported (1.B-Eval)

TRECVID 2018 ActEV: Activities in Extended Video
Data is no longer released for this evaluation. Please go to the "Current Evaluations" Tab on the Main ActEV Page.
Rules for Leaderboard Evaluation Schedule

TRECID ActEV 2018 You can submit maximum of 2 results per day and maximum of 50 results in total for AD and AOD.

The challenge participant can train their systems or tune parameters using any data complying with applicable laws and regulations. In the event that external limitations preclude sharing such data with others, participant are still permitted to use the data, but they must inform NIST that they are using such data, and provide appropriate detail regarding the type of data used and the limitations on distribution.

The challenge participant agree not to probe the test videos via manual/human means such as looking at the videos to produce the activity type and timing information from prior to the evaluation period to end of leaderboard evaluation.

All machine learning or statistical analysis algorithms must complete training, model selection, and tuning prior to running on the test data. This rule does not preclude online learning/adaptation during test data processing so long as the adaptation information is not reused for subsequent runs of the evaluation collection.

The only VIRAT data that may be used by the systems are the ActEV provided training and validation sets, associated annotations, and any derivatives of those sets (e.g. additional annotations on those videos). All other VIRAT data and associated annotations may not be used by any of the systems for the ActEV Leaderboard Evaluation.




If you have any question, please email to actev-nist@nist.gov