RANK | SUBMISSION_ID | EVALUATION_NAME | TRACK_NAME | TEAM_NAME | SYSTEM_NAME | W_PMISS@0.15RFA | PRIZE_ELIGIBLE |
---|---|---|---|---|---|---|---|
1 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00191_MUDSML_20190327-115431-4980.sr-20190327-115431-5488 | ActEV-2018 | AD | MUDSML | MUDSML_SECONDARY | 0.60473 | no |
2 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00224_MUDSML_20190401-061952-6662.sr-20190401-061952-7224 | ActEV-2018 | AD | MUDSML | MMVG-AlibabaAIC | 0.60473 | no |
3 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00188_MUDSML_20190412-103103-6528.sr-20190412-103103-7181 | ActEV-2018 | AD | MUDSML | MUDSML_MAIN | 0.64286 | no |
4 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00185_INF_20190413-223023-5926.sr-20190413-223023-6427 | ActEV-2018 | AD | INF | tst | 0.67599 | no |
5 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00216_BUPT-MCPRL_20190412-224234-5507.sr-20190412-224234-6508 | ActEV-2018 | AD | BUPT-MCPRL | MCPRL_PC | 0.69327 | |
6 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00221_BUPT-MCPRL_20190324-082801-6204.sr-20190324-082801-6550 | ActEV-2018 | AD | BUPT-MCPRL | MCPRL_C2 | 0.69371 | |
7 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00220_BUPT-MCPRL_20190403-100702-5648.sr-20190403-100702-6024 | ActEV-2018 | AD | BUPT-MCPRL | MCPRL_C1 | 0.69404 | |
8 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00214_BUPT-MCPRL_20190324-040537-0726.sr-20190324-040537-1127 | ActEV-2018 | AD | BUPT-MCPRL | MCPRL_BUPT | 0.69467 | |
9 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00261_INF_20190430-024948-8188.sr-20190430-024948-8860 | ActEV-2018 | AD | INF | clean | 0.70772 | no |
10 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00209_MUDSML_20190430-022449-0672.sr-20190430-022449-1455 | ActEV-2018 | AD | MUDSML | MUDSML_ADD | 0.71175 | no |
11 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00227_vireoJD-MM_20190424-103901-3123.sr-20190424-103901-3557 | ActEV-2018 | AD | vireoJD-MM | Pipeline-V3 | 0.73317 | |
12 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00218_UNSW-InsData-PC_20190321-152215-6408.sr-20190321-152215-7334 | ActEV-2018 | AD | UNSW_InsData_PC | UNSW_InsData | 0.74234 | |
13 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00237_vireoJD-MM_20190419-120757-3612.sr-20190419-120757-4173 | ActEV-2018 | AD | vireoJD-MM | Pipeline-V4 | 0.74516 | |
14 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00162_UCF_20190416-110758-3352.sr-20190416-110758-3742 | ActEV-2018 | AD | UCF | UCF-PC | 0.74997 | no |
15 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00201_UMD_20190207-125352-3189.sr-20190207-125352-3536 | ActEV-2018 | AD | UMD | UMD | 0.75031 | no |
16 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00189_IBM-MIT-Purdue_20190312-153824-9633.sr-20190312-153825-0992 | ActEV-2018 | AD | IBM-MIT-Purdue | k1 | 0.75658 | no |
17 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00189_STARK_20181221-134529-1523.sr-20181221-134529-2194 | ActEV-2018 | AD | STARK | k1 | 0.75780 | no |
18 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00198_STR-DIVA-Team_20190320-180832-6691.sr-20190320-180832-7046 | ActEV-2018 | AD | STR-DIVA Team | STR | 0.76192 | no |
19 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00229_STR-DIVA-Team_20190324-160109-5400.sr-20190324-160109-5739 | ActEV-2018 | AD | STR-DIVA Team | STR2 | 0.76192 | no |
20 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00204_UMD_20190221-211220-5543.sr-20190221-211220-5868 | ActEV-2018 | AD | UMD | UMD3 | 0.76432 | no |
21 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00202_UMD_20190321-132838-6826.sr-20190321-132838-7142 | ActEV-2018 | AD | UMD | UMD2 | 0.76441 | no |
22 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00223_vireoJD-MM_20190329-055409-2177.sr-20190329-055409-2581 | ActEV-2018 | AD | vireoJD-MM | Pipeline-V2 | 0.76539 | |
23 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00196_vireoJD-MM_20190321-155052-3583.sr-20190321-155052-3896 | ActEV-2018 | AD | vireoJD-MM | Pipeline-V1 | 0.76826 | |
24 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00186_INF_20190201-135322-9654.sr-20190201-135323-0281 | ActEV-2018 | AD | INF | tst2 | 0.77348 | no |
25 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00199_JHUDIVATeam_20190322-195555-4851.sr-20190322-195555-5913 | ActEV-2018 | AD | JHUDIVATeam | Simulation_Synthesis | 0.79336 | no |
26 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00168_JHUDIVATeam_20190201-010657-0332.sr-20190201-010657-1986 | ActEV-2018 | AD | JHUDIVATeam | Structured_Genralization_AD | 0.79420 | no |
27 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00228_NtechLab_20190329-052339-1867.sr-20190329-052339-2168 | ActEV-2018 | AD | NtechLab | Combined s1 and s2 | 0.79968 | |
28 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00161_SRI_20190206-141136-4690.sr-20190206-141136-7075 | ActEV-2018 | AD | SRI | sri_pc_test1 | 0.80460 | no |
29 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00171_SRI_20190204-075310-0761.sr-20190204-075310-5138 | ActEV-2018 | AD | SRI | TardisVS | 0.80518 | no |
30 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00200_NtechLab_20190321-092010-7359.sr-20190321-092010-7736 | ActEV-2018 | AD | NtechLab | Detect&Track | 0.80620 | |
31 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00197_SRI_20190208-175020-5929.sr-20190208-175021-0000 | ActEV-2018 | AD | SRI | TardisVP_pruning | 0.81229 | no |
32 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00203_NtechLab_20190321-161831-5586.sr-20190321-161831-5894 | ActEV-2018 | AD | NtechLab | ActionRCNN | 0.82827 | |
33 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00170_SRI_20181217-094621-1223.sr-20181217-094621-1972 | ActEV-2018 | AD | SRI | TardisVP | 0.83609 | no |
34 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00217_Shandong-Normal-University_20190314-084457-6128.sr-20190314-084457-6690 | ActEV-2018 | AD | Shandong Normal University | sdnu-resnet | 0.85823 | |
35 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00169_JHUDIVATeam_20181214-181827-2859.sr-20181214-181827-3922 | ActEV-2018 | AD | JHUDIVATeam | CLI_Indepedent_Eval | 0.87234 | no |
36 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00219_USF-Bulls_20190326-112622-4715.sr-20190326-112622-5039 | ActEV-2018 | AD | USF Bulls | PC2 | 0.88561 | |
37 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00193_USF-Bulls_20190129-003000-8660.sr-20190129-003000-9033 | ActEV-2018 | AD | USF Bulls | PC1 | 0.88816 | |
38 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00166_DIVA-TE-Baseline_20181213-083956-4174.sr-20181213-083956-4501 | ActEV-2018 | AD | DIVA TE Baseline | baselineACT_1_AD | 0.90723 | no |
39 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00133_DIVA-TE-Baseline_20181212-084002-2087.sr-20181212-084002-2555 | ActEV-2018 | AD | DIVA TE Baseline | baselineRC3D_1_AD | 0.91297 | no |
40 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00225_IVP_20190321-095107-2888.sr-20190321-095107-3306 | ActEV-2018 | AD | IVP | IVP-baseline_1_AD | 0.93736 | |
41 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00226_IVP_20190321-101007-2212.sr-20190321-101007-2509 | ActEV-2018 | AD | IVP | IVP_2_AD | 0.93736 | |
42 | ActEV-2018_AD_Prize-Challenge-Leader_SYS-00215_XXR_20190314-004302-2142.sr-20190314-004302-2628 | ActEV-2018 | AD | XXR | XXR | 0.97173 |
The top 6 Phase 1 ActEV Challenge participants delivered their algorithms in a form compatible with the ActEV Command Line Interface (ActEV CLI) to NIST for Phase 2 testing. The command line interface implementation that you provided formalizes the entire process of evaluating a system including: (1) downloading and installing your software via a single URL, (2) verifying the delivery works AND produces output that is “consistent” with output you produce, and (3) enabling a user to process a large collection of video in a fault-tolerant, parallelizable manner.
To complete this task you needed the following things described in detail below:
NIST installed your system from a fresh Ubuntu 16.04 cloud image available from https://cloud-images.ubuntu.com/releases/16.04/release/ on the following hardware.
List of Activities |
---|
Closing Closing_trunk Entering Exiting Loading Open_Trunk Opening Transport_HeavyCarry Unloading Vehicle_turning_left Vehicle_turning_right Vehicle_u_turn Pull Riding Talking activity_carrying specialized_talking_phone specialized_texting_phone |
Given a target activity, a system automatically detects and temporally localizes all instances of the activity. For a system-identified activity instance to be evaluated as correct, the type of activity must be correct and the temporal overlap must fall within a minimal requirement as described in the Evaluation Plan
ActEV-PC Open Leaderboard Evaluation (Phase 1)
For the ActEV-PC Open Leaderboard evaluation, challenge participants ran their activity detection software on their compute hardware and submit system output defined by the Evaluation Plan to the NIST ActEV Scoring Server. This phase served as a qualifying stage where the top 6 participants proceeded to phase 2.
ActEV-PC Independent Evaluation (Phase 2)
For the ActEV-PC Independent evaluation, invited challenge participants submited their runnable activity detection software to NIST using the Evaluation Commandline Interface Submission Instructions. NIST then evaluated system performance on sequestered data using NIST hardware.
See the complete eligibility requirements in Evaluation and Rules tab.
Phase 1 - ActEV-PC Open Leaderboard evaluation:
Phase 2 - ActEV-PC Independent evaluation:
Each entrant retains full ownership and title in and to their submission and expressly reserve all intellectual property rights not expressly granted under the challenge agreement. By participating in the challenge, each entrant hereby irrevocably grants to the Office of the Director of National Intelligence (ODNI ), of which IARPA is a component, and NIST a limited, non-exclusive, royalty-free, worldwide license and right to reproduce, publically perform, publically display, and use the submission for internal ODNI and NIST business and to the extent necessary to administer the challenge. By submitting an Entry, you grant a non-exclusive right and license to ODNI and NIST to use your name, likeness, biographical information, image, any other personal data submitted with your Entry and the contents in your Entry (including any created works, such as YouTube® videos, but not including any App software submitted with or as part of your Entry), in connection with the Competition. You also agree that this license is perpetual and irrevocable.
You agree that nothing in this Notice grants you a right or license to use any names or logos of ODNI, IARPA, or NIST or any other intellectual property or proprietary rights of ODNI, IARPA, and NIST. You grant to ODNI and NIST the right to include your company or institution name and logo (if your Entry is from a company or institution) as a Participant on the Event Web site and in materials from IARPA announcing winners of or Participants in the Competition. Other than these uses or as otherwise set forth herein, you are not granting ODNI or NIST any rights to your trademarks.
ODNI and NIST would retain and use the system for a period of no longer than two years after submission or when the Participant ceases to participate in ActEV evaluations, whichever is later.
Any data or documentation that qualifies as business proprietary information, as defined by the Freedom of Information Act (5 USC Section 552), and is properly marked as such would be treated as confidential and would only be used for the purposes of the ActEV test.