ActEV Sequestered Data Leaderboard


ActEV Sequestered Data Leaderboard (SDL) Evaluation

Updates
  • March 01, 2020: The ActEV 2020 SDL opens with the expanded MEVA Test3 dataset
  • March 01, 2020: ActEV 2020 SDL is a guest task under ActivityNet CVPR'20 ActivityNet workshop
  • May 17, 2020 May 10, 2020 at 12:00 noon EST: New Deadline for CLI submissions to be included in ActEV SDL ActivityNet rankings. Top two submissions to be announced on June 1, 2020.
  • June 14, 2020: CVPR'20 ActivityNet workshop ActEV SDL guest task presentations
  • April 28th, 2020: CLI submissions deadline extended to May 17th, 2020 at 12:00 noon EST
Summary
The ActEV (Activities in Extended Video) Sequestered Data Leaderboard is an ongoing ranking of software systems that watch lengthy videos and detect activities of interest. Anyone can submit their system to NIST, which will then run the system on sequestered data, score the results and post the score to the leaderboard.
The sequestered data is from the MEVA dataset, which contains hours of videos, including indoor and outdoor scenes, night and day, crowds and individuals, and videos are from both EO (Electro-Optical) and IR (Infrared) sensors. Hours can go by with no activities, but then multiple activities happen simultaneously. The data is multi-camera, in that multiple cameras may be pointed at the same scene at the same time. There are also two separate leaderboards for EO and IR videos.
What
Build and submit a software system to watch videos and detect if and when an activity of interest occurs. System runtime must be 1 times the video length on the designated evaluation hardware.
Who
Everyone. Anyone who registers can submit to the evaluation server.
How
Register here and then follow the instructions on the Algorithm Submission tab above. Systems must follow a NIST-defined command line interface and automatically run on NIST’s servers, both of which are described in the instructions.
Evaluation Task
Detect if and when an activity occurs. Given a target activity type and a set of videos, submitted systems must automatically detect all instances of the activity in the videos. While different activities instances have different durations, a submitted system will be considered to have detected an activity if it correctly identifies at least 1 second of the activity.
Data
The data is from the Multiview Extended Video with Activities (MEVA) dataset and the videos are from both EO (Electro-Optical) and IR (Infrared) sensors. The data used for SDL evaluation from March 01, 2020 is the the expanded MEVA Test3 dataset. The public MEVA dataset includes hundreds of hours of data from the same cameras at the same facility, which can be used for training. You can download the public MEVA dataset for free at mevadata.org; and more info about the datasets is on the data tab. We also provide annotations for 20 hours of MEVA data, and instructions on how to make and share activity annotations are at mevadata.org.
Metrics
Submitted activity detection systems must give a confidence score for each activity they detect. Detected activities are then thresholded based on this confidence score. Varying the threshold makes a trade-off between being sensitive enough to identify true activity instances (low threshold) vs. not making false alarms when no activity is present (high threshold). Submitted systems are scored on both of these, measured by Probability of Missed Detection (Pmiss) and Time-based False Alarm (TFA). Pmiss is the portion of activities where the system did not detect the activity for at least 1 second. TFA is the portion of time that the system detected an activity when in fact there was none. Submitted systems are scored for Pmiss and TFA at multiple thresholds, creating a detect error tradeoff (DET) curve. The leaderboard ranking of a system is based on a summary of its DET curve: the area under the DET Curve across the TFA range between 0% to 20% divided by 0.2 to normalize the value to [0:1]. Lower numbers are better, as they reflect fewer errors. See details in the SDL evaluation plan or check out the ActEV Scoring Software GitHub repo.
Evaluation Plan
Task coordinator
ActEV NIST team (ActEV-nist@nist.gov)
News
01March
ActEV 2020 SDL starts with expanded MEVA Test3 dataset
17May
New Deadline to submit for ActivityNet task

ActEV SDL Evaluation Schedule
March 01, 2020: ActEV 2020 SDL opens with MEVA Test3.

May 17 May 10 , 2020 at 12 noon EST : New Deadline for CLI submissions to be included in ActEV guest task under CVPR'20 ActivityNet workshop.

June 01, 2020 : We will invite the top two teams on the ActEV 2020 SDL leaderboard to give ActEV guest task oral presentations at the CVPR'20 ActivityNet workshop based on the CLI submission deadline (May 17th May 10th , 2020).

June 14, 2020: CVPR'20 ActivityNet workshop ActEV SDL guest task presentations.

Leaderboard Remains Open: Leaderboard will continue to remain open to allow participants to show continued progress on this challenging problem.
ActEV SDL Dataset

Multiview Extended Video with Activities (MEVA)

The ActEV SDL evaluation is based on the Multiview Extended Video with Activities (MEVA) dataset (mevadata.org) collected at the Muscatatuck Urban Training Center with a team of over 100 actors performing in various scenarios. The data was built by the Intelligence Advanced Research Projects Activity (IARPA) Deep Intermodal Video Analytics (DIVA) program to support activity detection in multi-camera environments for both DIVA performers and the broader research community.
The MEVA dataset has two parts: the public training and development data and sequestered evaluation data used only by NIST to test systems. The data is accompanied by activity annotations.

Public Training and Development Data
The Multiview Extended Video with Activities (MEVA) dataset website mevadata.org is to share the public MEVA video dataset and annotations. The size of the public MEVA dataset is 333 hours of ground-camera and UAV video. The size of the provided annotated video dataset is 28 hours. ActEV participants are encouraged to annotate the MEVA KF1 dataset for the 37 activities as described at mevadata.org.


The MEVA data GIT repo is the data distribution mechanism for MEVA Related annotations and documentation. The repo presently consists of schemas for the activity annotations https://gitlab.kitware.com/meva/meva-data-repo.

The ActEV data GIT repo, is the data distribution mechanism for the ActEV evaluation. The repo presently consists of a collection of corpora and partition definition files to be used for the evaluations https://gitlab.kitware.com/actev/actev-data-repo.

Sequestered Evaluation Data
As of March 2020, NIST is using a 140-hour collection of annotated MEVA data for sequestered data evaluations. The data set consists of both EO and IR cameras, public cameras (examples of which are in the public data set). The leaderboard presents results on the full 140-hour collection reporting separately by EO and IR data. Developers receive additional scores by activity for the EO_subset1 and the IR_subset1. Both subsets consist of data from public cameras.
ActEV SDL Leaderboard

Update June 3, 2020: Reported scores for all submissions on the ActEV2020 leaderboards were revised using the ActEV_SDL_V2 scoring protocol which slightly decreased all error rates.

SDL20-scoring-EO

Updated: 2020-07-12 01:33:59 -0400
RANK SCORING_SUBMISSION_ID SCORING REQUEST NAME TEAM NAME SUBMISSION ID SUBMISSION DATE SYSTEM NAME SYSTEM ID SCORING PROTOCOL PARTIAL AUDC* TIME LIMITED PARTIAL AUDC* RELATIVE PROCESSING TIME DETECTED ACTIVITY TYPES† PROCESSED FILES‡ MEAN-P_MISS@0.04TFA TIME LIMITED MEAN-P_MISS@0.04TFA
1 19372 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200626-225528-9522.sr-20200626-225529-2904 UCF submission_description/19025|19025 2020-06-19 UCF-P 270 ActEV_SDL_V2 0.36126 0.722 100% 100% 0.42337
2 18744 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00326_UMD_20200611-111630-1181.sr-20200611-111630-4277 UMD submission_description/18450|18450 2020-06-07 UMD+UCF 326 ActEV_SDL_V2 0.36655 0.49867 1.486 94% 100% 0.42493 0.53357
3 17057 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00298_INF_20200516-013223-2420.sr-20200526-131614-0410 INF submission_description/16309|16309 2020-05-08 speed1x 298 ActEV_SDL_V2 0.38699 0.498 97% 100% 0.46352
4 16601 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00328_VUS_20200511-132633-4864.sr-20200515-123841-5202 VUS submission_description/16259|16259 2020-05-08 VUS-V1 328 ActEV_SDL_V2 0.40599 0.49453 1.344 100% 100% 0.46931 0.53313
5 20364 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200711-232330-7159.sr-20200711-232331-0498 IBM-MIT-Purdue submission_description/20207|20207 2020-07-10 Purdue 312 ActEV_SDL_V2 0.41002 0.093 100% 100% 0.53802
6 16200 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200507-153946-0409.sr-20200519-235031-1300 Team_Vision submission_description/16060|16060 2020-05-05 STARK 282 ActEV_SDL_V2 0.52993 0.53006 1.003 100% 100% 0.62047 0.62012
7 17477 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00337_vireoJD-MM_20200520-200345-0804.sr-20200526-131616-1250 vireoJD-MM submission_description/17093|17093 2020-05-16 vireo3 337 ActEV_SDL_V2 0.53876 0.149 100% 96% 0.67389
8 17913 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00338_BUPT-MCPRL_20200529-112230-6942.sr-20200529-112231-8566 BUPT-MCPRL submission_description/17732|17732 2020-05-26 bupt-mcprl 338 ActEV_SDL_V2 0.61532 0.969 100% 100% 0.63203
9 17965 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00335_CIS-JHU_20200529-142402-1419.sr-20200529-142403-2822 CIS_JHU submission_description/17471|17471 2020-05-20 jhu_stmpgnn 335 ActEV_SDL_V2 0.62911 0.77784 4.520 94% 69% 0.70393 0.79736
10 18862 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00276_DIVA-TE-Baseline_20200612-131408-4506.sr-20200612-131408-7185 DIVA TE Baseline submission_description/18234|18234 2020-06-04 RC3D 276 ActEV_SDL_V2 0.93925 0.94669 1.327 89% 100% 0.95265 0.95662
11 19360 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200626-094820-3881.sr-20200626-094820-6286 NIST-TEST submission_description/19117|19117 2020-06-23 NIST Test 263 ActEV_SDL_V2 1.00000 0.030 0% 100% 1.00000

SDL20-scoring-EO

Updated: 2020-07-12 01:33:58 -0400

SDL20-scoring-EO

Updated: 2020-07-12 01:33:58 -0400
RANK SCORING_SUBMISSION_ID SCORING REQUEST NAME TEAM NAME SUBMISSION ID SUBMISSION DATE SYSTEM NAME SYSTEM ID SCORING PROTOCOL PARTIAL AUDC* TIME LIMITED PARTIAL AUDC* RELATIVE PROCESSING TIME DETECTED ACTIVITY TYPES† PROCESSED FILES‡ MEAN-P_MISS@0.04TFA TIME LIMITED MEAN-P_MISS@0.04TFA
1 19372 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200626-225528-9522.sr-20200626-225529-2904 UCF submission_description/19025|19025 2020-06-19 UCF-P 270 ActEV_SDL_V2 0.36126 0.722 100% 100% 0.42337
2 17885 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200528-094551-9941.sr-20200528-094553-2266 UCF submission_description/17152|17152 2020-05-17 UCF-P 270 ActEV_SDL_V2 0.36452 0.684 100% 100% 0.42292
3 18744 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00326_UMD_20200611-111630-1181.sr-20200611-111630-4277 UMD submission_description/18450|18450 2020-06-07 UMD+UCF 326 ActEV_SDL_V2 0.36655 0.49867 1.486 94% 100% 0.42493 0.53357
4 19876 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00326_UMD_20200707-171737-4910.sr-20200707-171737-8499 UMD submission_description/19562|19562 2020-06-30 UMD+UCF 326 ActEV_SDL_V2 0.36941 0.51425 1.582 94% 100% 0.43018 0.54987
5 19469 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200629-055542-1886.sr-20200629-055542-4982 UCF submission_description/19365|19365 2020-06-26 UCF-P 270 ActEV_SDL_V2 0.36969 0.688 100% 100% 0.42401
6 17151 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200516-215958-2007.sr-20200519-235025-5233 UCF submission_description/16743|16743 2020-05-12 UCF-P 270 ActEV_SDL_V2 0.37281 0.581 94% 100% 0.42796
7 17057 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00298_INF_20200516-013223-2420.sr-20200526-131614-0410 INF submission_description/16309|16309 2020-05-08 speed1x 298 ActEV_SDL_V2 0.38699 0.498 97% 100% 0.46352
8 16601 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00328_VUS_20200511-132633-4864.sr-20200515-123841-5202 VUS submission_description/16259|16259 2020-05-08 VUS-V1 328 ActEV_SDL_V2 0.40599 0.49453 1.344 100% 100% 0.46931 0.53313
9 16051 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00288_INF_20200505-093715-5641.sr-20200519-235025-8703 INF submission_description/15846|15846 2020-05-01 meva_inf2 288 ActEV_SDL_V2 0.40897 0.927 97% 100% 0.48110
10 20364 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200711-232330-7159.sr-20200711-232331-0498 IBM-MIT-Purdue submission_description/20207|20207 2020-07-10 Purdue 312 ActEV_SDL_V2 0.41002 0.093 100% 100% 0.53802
11 15414 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00307_INF_20200421-115525-8081.sr-20200519-235026-1773 INF submission_description/15132|15132 2020-04-13 INF_MEVA_IOD 307 ActEV_SDL_V2 0.41143 0.848 97% 98% 0.48364
12 16256 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00326_UMD_20200508-110531-4667.sr-20200519-235026-4616 UMD submission_description/15747|15747 2020-04-30 UMD+UCF 326 ActEV_SDL_V2 0.41450 0.51127 1.253 100% 100% 0.50159 0.57103
13 17326 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00334_VUS_20200519-030816-7908.sr-20200526-131615-6056 VUS submission_description/16908|16908 2020-05-14 VUS-V1-FAST 334 ActEV_SDL_V2 0.41493 0.829 100% 100% 0.48191
14 17473 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00326_UMD_20200520-155344-4251.sr-20200526-131615-1214 UMD submission_description/16984|16984 2020-05-15 UMD+UCF 326 ActEV_SDL_V2 0.41646 0.50800 1.226 97% 100% 0.49014 0.56142
15 14577 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00283_INF_20200317-124405-8174.sr-20200519-235026-8006 INF submission_description/14480|14480 2020-03-16 INF_MEVA1 283 ActEV_SDL_V2 0.42496 0.527 97% 98% 0.49599
16 19647 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00288_INF_20200702-000615-6292.sr-20200702-000616-3752 INF submission_description/19451|19451 2020-06-28 meva_inf2 288 ActEV_SDL_V2 0.43139 0.610 100% 100% 0.54264
17 14857 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200319-153022-2982.sr-20200519-235027-0971 UCF submission_description/14760|14760 2020-03-18 UCF-P 270 ActEV_SDL_V2 0.43526 0.349 100% 100% 0.53063
18 14963 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200320-041131-0946.sr-20200519-235027-3712 UCF submission_description/14858|14858 2020-03-19 UCF-P 270 ActEV_SDL_V2 0.43555 0.346 100% 100% 0.52764
19 20361 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200711-122139-6464.sr-20200711-122140-0051 IBM-MIT-Purdue submission_description/20206|20206 2020-07-10 Purdue 312 ActEV_SDL_V2 0.43758 0.093 100% 100% 0.56295
20 19897 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200707-213937-2081.sr-20200707-213937-5374 UMD submission_description/19565|19565 2020-06-30 UMD 281 ActEV_SDL_V2 0.43796 0.45792 1.055 97% 100% 0.52779 0.54450
21 15986 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200504-164235-0111.sr-20200519-235027-6683 UCF submission_description/15544|15544 2020-04-27 UCF-P 270 ActEV_SDL_V2 0.43888 0.356 97% 100% 0.50250
22 16558 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200510-225902-1029.sr-20200519-235027-9601 UCF submission_description/16212|16212 2020-05-07 UCF-P 270 ActEV_SDL_V2 0.43937 0.317 94% 100% 0.51352
23 18849 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200612-022023-7687.sr-20200612-022024-0711 UMD submission_description/18557|18557 2020-06-09 UMD 281 ActEV_SDL_V2 0.44296 0.44346 1.002 100% 100% 0.52413 0.52440
24 17599 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00290_INF_20200523-181607-2999.sr-20200605-094432-1913 INF submission_description/17472|17472 2020-05-20 set3 290 ActEV_SDL_V2 0.46181 0.915 100% 100% 0.57696
25 16257 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200508-110705-2778.sr-20200519-235028-2659 UCF submission_description/15949|15949 2020-05-03 UCF-P 270 ActEV_SDL_V2 0.46326 0.480 100% 100% 0.56739
26 18749 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200611-125827-6623.sr-20200611-125828-0367 UMD submission_description/18150|18150 2020-06-01 UMD 281 ActEV_SDL_V2 0.46455 0.884 100% 100% 0.57492
27 14856 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200319-133322-4135.sr-20200519-235029-1164 UMD submission_description/13739|13739 2020-03-11 UMD 281 ActEV_SDL_V2 0.46562 0.684 97% 100% 0.55473
28 15300 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200416-142124-7336.sr-20200519-235029-4101 UMD submission_description/13021|13021 2020-03-05 UMD 263 ActEV_SDL_V2 0.46563 0.822 97% 100% 0.55441
29 14483 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00298_INF_20200316-112137-9583.sr-20200519-235028-5577 INF submission_description/13770|13770 2020-03-11 speed1x 298 ActEV_SDL_V2 0.46565 0.613 97% 100% 0.54743
30 14202 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00298_INF_20200313-132531-6559.sr-20200519-235028-8388 INF submission_description/13315|13315 2020-03-10 speed1x 298 ActEV_SDL_V2 0.46566 0.615 97% 100% 0.54743
31 14819 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200319-091913-4675.sr-20200519-235029-6961 UMD submission_description/14296|14296 2020-03-13 UMD 281 ActEV_SDL_V2 0.46662 0.703 100% 100% 0.55902
32 15026 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200320-131358-4621.sr-20200519-235029-9885 UCF submission_description/13949|13949 2020-03-11 UCF-P 270 ActEV_SDL_V2 0.46704 0.333 97% 100% 0.53785
33 14669 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200318-053708-4851.sr-20200519-235030-2750 UCF submission_description/14578|14578 2020-03-17 UCF-P 270 ActEV_SDL_V2 0.46704 0.296 97% 100% 0.53785
34 16557 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00290_INF_20200510-225720-2108.sr-20200519-235030-5862 INF submission_description/16142|16142 2020-05-06 set3 290 ActEV_SDL_V2 0.47294 0.559 97% 100% 0.53440
35 19586 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200701-042026-6851.sr-20200701-042026-9738 UCF submission_description/19448|19448 2020-06-28 UCF-P 270 ActEV_SDL_V2 0.47688 0.515 100% 100% 0.51324
36 17346 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200519-111655-5218.sr-20200526-131613-0088 UCF submission_description/16386|16386 2020-05-09 UCF-P 270 ActEV_SDL_V2 0.48113 0.305 94% 100% 0.57069
37 18071 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00307_INF_20200530-125344-8660.sr-20200530-125346-3964 INF submission_description/17726|17726 2020-05-26 INF_MEVA_IOD 307 ActEV_SDL_V2 0.48144 0.824 97% 97% 0.57248
38 17744 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200526-170134-8475.sr-20200526-170136-0564 UMD submission_description/17521|17521 2020-05-22 UMD 281 ActEV_SDL_V2 0.48305 0.905 97% 100% 0.57807
39 19367 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200626-155016-8498.sr-20200626-155017-1824 IBM-MIT-Purdue submission_description/19119|19119 2020-06-24 Purdue 312 ActEV_SDL_V2 0.48664 0.090 100% 100% 0.62999
40 18555 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00288_INF_20200609-113657-6904.sr-20200609-113658-4373 INF submission_description/18247|18247 2020-06-05 meva_inf2 288 ActEV_SDL_V2 0.49394 0.991 100% 100% 0.61605
41 18229 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00283_INF_20200603-135148-1770.sr-20200603-135149-1523 INF submission_description/18151|18151 2020-06-02 INF_MEVA1 283 ActEV_SDL_V2 0.49492 0.950 100% 98% 0.61675
42 20297 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200711-022734-9206.sr-20200711-022735-2782 IBM-MIT-Purdue submission_description/20129|20129 2020-07-10 Purdue 312 ActEV_SDL_V2 0.49857 0.063 100% 100% 0.63895
43 18235 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200605-095636-6760.sr-20200605-095637-0432 IBM-MIT-Purdue submission_description/18073|18073 2020-05-30 Purdue 312 ActEV_SDL_V2 0.50541 0.366 102% 100% 0.65305
44 17319 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200519-012627-9123.sr-20200519-235030-8540 Team_Vision submission_description/16275|16275 2020-05-08 STARK 282 ActEV_SDL_V2 0.52993 1.002 100% 100% 0.62047
45 16200 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200507-153946-0409.sr-20200519-235031-1300 Team_Vision submission_description/16060|16060 2020-05-05 STARK 282 ActEV_SDL_V2 0.52993 0.53006 1.003 100% 100% 0.62047 0.62012
46 17090 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200516-094544-8275.sr-20200519-235031-3894 IBM-MIT-Purdue submission_description/16902|16902 2020-05-14 Purdue 312 ActEV_SDL_V2 0.53528 0.080 97% 100% 0.66782
47 17477 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00337_vireoJD-MM_20200520-200345-0804.sr-20200526-131616-1250 vireoJD-MM submission_description/17093|17093 2020-05-16 vireo3 337 ActEV_SDL_V2 0.53876 0.149 100% 96% 0.67389
48 16135 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00329_vireoJD-MM_20200506-100639-0175.sr-20200519-235031-6711 vireoJD-MM submission_description/15735|15735 2020-04-30 Vireo 329 ActEV_SDL_V2 0.54107 0.160 100% 96% 0.67380
49 17008 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00331_vireoJD-MM_20200515-163029-9498.sr-20200519-235031-9723 vireoJD-MM submission_description/16445|16445 2020-05-09 vireo2 331 ActEV_SDL_V2 0.55665 0.163 100% 96% 0.68832
50 17387 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200519-233350-8538.sr-20200526-131614-5743 IBM-MIT-Purdue submission_description/16909|16909 2020-05-14 Purdue 312 ActEV_SDL_V2 0.56720 0.080 97% 100% 0.69927
51 19993 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200708-194028-8463.sr-20200708-194029-1870 Team_Vision submission_description/19871|19871 2020-07-07 STARK 282 ActEV_SDL_V2 0.57102 0.993 97% 100% 0.65497
52 18233 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200604-113128-5818.sr-20200604-113129-9201 Team_Vision submission_description/16705|16705 2020-05-12 STARK 282 ActEV_SDL_V2 0.61351 0.63166 1.096 100% 100% 0.70557 0.71890
53 17913 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00338_BUPT-MCPRL_20200529-112230-6942.sr-20200529-112231-8566 BUPT-MCPRL submission_description/17732|17732 2020-05-26 bupt-mcprl 338 ActEV_SDL_V2 0.61532 0.969 100% 100% 0.63203
54 17965 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00335_CIS-JHU_20200529-142402-1419.sr-20200529-142403-2822 CIS_JHU submission_description/17471|17471 2020-05-20 jhu_stmpgnn 335 ActEV_SDL_V2 0.62911 0.77784 4.520 94% 69% 0.70393 0.79736
55 19251 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200624-152809-7118.sr-20200624-152810-0495 Team_Vision submission_description/18959|18959 2020-06-18 STARK 282 ActEV_SDL_V2 0.62962 0.62956 1.005 100% 100% 0.71889 0.71876
56 15224 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200415-114050-1365.sr-20200519-235032-2671 IBM-MIT-Purdue submission_description/15129|15129 2020-04-10 Purdue 312 ActEV_SDL_V2 0.64260 0.148 100% 100% 0.72913
57 14009 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200312-162922-5840.sr-20200519-235032-5355 Team_Vision submission_description/13117|13117 2020-03-08 STARK 282 ActEV_SDL_V2 0.69971 0.685 97% 100% 0.76985
58 15027 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200320-133708-7523.sr-20200519-235032-8099 Team_Vision submission_description/13533|13533 2020-03-10 STARK 282 ActEV_SDL_V2 0.70162 0.681 97% 98% 0.77027
59 15413 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200421-105807-5524.sr-20200519-235033-1130 IBM-MIT-Purdue submission_description/15320|15320 2020-04-18 Purdue 312 ActEV_SDL_V2 0.74163 0.047 100% 100% 0.85468
60 16614 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200511-173821-3000.sr-20200519-235033-4110 IBM-MIT-Purdue submission_description/15562|15562 2020-04-28 Purdue 312 ActEV_SDL_V2 0.74383 0.047 100% 100% 0.85239
61 20127 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200710-095000-2256.sr-20200710-095000-5705 Team_Vision submission_description/19750|19750 2020-07-06 STARK 282 ActEV_SDL_V2 0.81637 0.918 97% 100% 0.84071
62 18862 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00276_DIVA-TE-Baseline_20200612-131408-4506.sr-20200612-131408-7185 DIVA TE Baseline submission_description/18234|18234 2020-06-04 RC3D 276 ActEV_SDL_V2 0.93925 0.94669 1.327 89% 100% 0.95265 0.95662
63 18868 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00341_DIVA-TE-Baseline_20200614-225857-0186.sr-20200614-225857-2696 DIVA TE Baseline submission_description/18498|18498 2020-06-08 RC3D-kw-annotations 341 ActEV_SDL_V2 0.96674 0.97091 1.236 75% 100% 0.97908 0.98109
64 19032 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00307_INF_20200623-035629-8422.sr-20200623-035630-2041 INF submission_description/18968|18968 2020-06-18 INF_MEVA_IOD 307 ActEV_SDL_V2 0.97072 0.523 97% 3% 0.97095
65 19360 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200626-094820-3881.sr-20200626-094820-6286 NIST-TEST submission_description/19117|19117 2020-06-23 NIST Test 263 ActEV_SDL_V2 1.00000 0.030 0% 100% 1.00000
66 18855 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200612-101533-8300.sr-20200612-101534-1008 NIST-TEST submission_description/18753|18753 2020-06-11 NIST Test 263 ActEV_SDL_V2 1.00000 0.031 0% 100% 1.00000
67 18859 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200612-105746-5020.sr-20200612-105746-7748 NIST-TEST submission_description/18238|18238 2020-06-05 NIST Test 263 ActEV_SDL_V2 1.00000 0.031 0% 100% 1.00000
68 18863 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200612-174502-8142.sr-20200612-174503-0813 NIST-TEST submission_description/18565|18565 2020-06-09 NIST Test 263 ActEV_SDL_V2 1.00000 0.031 0% 100% 1.00000

SDL20-scoring-EO

Updated: 2020-07-12 01:33:58 -0400

SDL19-scoring-EO

Updated: 2020-06-24 08:44:02 -0400
RANK TEAM NAME SUBMISSION ID SUBMISSION DATE SYSTEM NAME PARTIAL AUDC* TIME LIMITED PARTIAL AUDC* RELATIVE PROCESSING TIME DETECTED ACTIVITY TYPES† PROCESSED FILES‡ MEAN-P_MISS@0.04TFA TIME LIMITED MEAN-P_MISS@0.04TFA
1 UMD 11045 2020-01-24 UMD+UCF 0.41585 0.65517 1.154 100% 100% 0.49360 0.67902
2 UCF 10640 2020-01-21 UCF-P 0.43810 0.362 100% 100% 0.52356
3 UCF 12650 2020-02-25 UCF-P 0.44047 0.364 97% 100% 0.54081
4 UCF 9105 2020-01-15 UCF-P 0.44056 0.424 97% 100% 0.52513
5 UCF 12222 2020-02-11 UCF-P 0.44198 0.395 100% 100% 0.53514
6 UCF 8328 2020-01-10 UCF-P 0.44360 0.403 100% 100% 0.52972
7 UCF 10821 2020-01-22 UCF-P 0.44473 0.407 100% 100% 0.54228
8 UCF 11849 2020-02-02 UCF-P 0.45165 0.378 100% 100% 0.51752
9 UMD 5595 2019-12-06 UMD 0.47596 0.725 97% 100% 0.54479
10 UMD 12468 2020-02-18 UMD 0.47611 0.849 97% 100% 0.55945
11 UMD 8222 2020-01-10 UMD 0.47668 0.68909 1.260 100% 100% 0.54492 0.71689
12 UMD 8198 2020-01-09 UMD 0.48037 0.68601 1.134 97% 100% 0.54825 0.71585
13 UCF 7728 2020-01-04 UCF-P 0.48290 0.438 94% 100% 0.51235
14 UMD 7992 2020-01-08 UMD 0.48333 0.68298 1.098 97% 100% 0.55028 0.71410
15 INF 11911 2020-02-06 set3 0.48978 0.646 97% 100% 0.55928
16 UMD 10000 2020-01-19 UMD 0.49061 0.803 100% 100% 0.56996
17 UMD 7472 2020-01-02 UMD 0.49689 0.61793 1.042 97% 93% 0.55515 0.65471
18 UCF 7235 2019-12-30 UCF-P 0.51676 0.347 97% 100% 0.55164
19 UCF 6499 2019-12-18 UCF-P 0.52068 0.333 97% 100% 0.57885
20 UMD 4809 2019-11-15 UMD 0.53144 0.965 100% 98% 0.60011
21 UMD 3666 2019-10-10 UMD 0.53147 0.970 100% 98% 0.60101
22 UMD 6285 2019-12-12 UMD 0.53459 0.723 100% 98% 0.59596
23 INF 12068 2020-02-09 speed1x 0.53654 0.642 97% 94% 0.59701
24 INF 11850 2020-02-03 meva_inf2 0.59044 0.960 94% 97% 0.64252
25 UCF 5851 2019-12-09 UCF-P 0.60431 0.293 97% 100% 0.64157
26 Edge-Intelligence 8003 2020-01-08 Edge-Intelligence 0.62842 0.939 97% 100% 0.75488
27 IBM-MIT-Purdue 6513 2019-12-18 Purdue 0.64182 0.272 100% 100% 0.73321
28 IBM-MIT-Purdue 11346 2020-01-26 Purdue 0.64182 0.128 100% 100% 0.73320
29 IBM-MIT-Purdue 11345 2020-01-26 Purdue 0.64182 0.125 100% 100% 0.73320
30 Edge-Intelligence 7845 2020-01-06 Edge_Intelligence 0.64737 0.887 94% 100% 0.76268
31 UMD 2074 2019-08-16 UMD 0.65325 0.744 97% 98% 0.73860
32 IBM-MIT-Purdue 6815 2019-12-20 Purdue 0.65351 0.315 100% 100% 0.73730
33 IBM-MIT-Purdue 5004 2019-11-25 Purdue 0.65503 0.124 100% 98% 0.73745
34 IBM-MIT-Purdue 5005 2019-11-25 Purdue 0.65913 0.124 100% 98% 0.73971
35 IBM-MIT-Purdue 6501 2019-12-18 Purdue 0.66055 0.442 100% 100% 0.74490
36 IBM-MIT-Purdue 4660 2019-11-11 Purdue 0.66383 0.087 100% 98% 0.74381
37 IBM-MIT-Purdue 11343 2020-01-26 Purdue 0.66394 0.038 100% 98% 0.74397
38 IBM-MIT-Purdue 11344 2020-01-26 Purdue 0.66394 0.035 100% 98% 0.74397
39 IBM-MIT-Purdue 11268 2020-01-26 Purdue 0.66496 0.124 100% 100% 0.74147
40 IBM-MIT-Purdue 6303 2019-12-12 Purdue 0.66900 0.096 100% 100% 0.75098
41 IBM-MIT-Purdue 6547 2019-12-18 Purdue 0.67114 0.099 100% 100% 0.75100
42 UMD 1907 2019-08-13 UMD 0.67831 0.727 0.74917
43 IBM-MIT-Purdue 4999 2019-11-25 Purdue 0.67912 0.098 100% 98% 0.76349
44 INF 11670 2020-01-28 INF_MEVA1 0.67983 0.928 100% 100% 0.78032
45 IBM-MIT-Purdue 5001 2019-11-25 Purdue 0.68547 0.099 100% 98% 0.76606
46 IBM-MIT-Purdue 11188 2020-01-25 Purdue 0.69501 0.108 100% 98% 0.75741
47 UCF 5596 2019-12-06 UCF-P 0.69848 0.304 86% 100% 0.73165
48 IBM-MIT-Purdue 6443 2019-12-17 Purdue 0.70117 0.82690 1.072 100% 100% 0.77819 0.87029
49 IBM-MIT-Purdue 4876 2019-11-19 Purdue 0.70382 0.088 97% 98% 0.76211
50 Team_Vision 6865 2019-12-20 STARK 0.71736 0.793 97% 100% 0.77673
51 Team_Vision 4434 2019-11-05 STARK 0.71911 0.774 97% 100% 0.77884
52 UCF 5439 2019-12-04 UCF-P 0.72830 0.304 86% 100% 0.75572
53 IBM-MIT-Purdue 8154 2020-01-09 Purdue 0.74090 0.380 100% 100% 0.82467
54 IBM-MIT-Purdue 9978 2020-01-18 Purdue 0.74091 0.145 100% 100% 0.82468
55 UCF 5077 2019-11-27 UCF-P 0.74243 0.312 86% 100% 0.77197
56 IBM-MIT-Purdue 4228 2019-10-31 Purdue 0.75410 0.093 97% 98% 0.82355
57 IBM-MIT-Purdue 4520 2019-11-08 Purdue 0.75509 0.087 100% 98% 0.82359
58 IBM-MIT-Purdue 10037 2020-01-19 Purdue 0.79701 0.151 86% 100% 0.84131
59 IBM-MIT-Purdue 4179 2019-10-30 Purdue 0.81573 0.093 97% 98% 0.86015
60 IBM-MIT-Purdue 10041 2020-01-19 Purdue 0.82020 0.143 100% 100% 0.88273
61 IBM-MIT-Purdue 3830 2019-10-21 Purdue 0.82025 0.246 100% 97% 0.86425
62 Team_Vision 4071 2019-10-29 STARK 0.82121 0.757 75% 100% 0.85712
63 Team_Vision 8506 2020-01-12 STARK 0.82442 0.739 94% 58% 0.85486
64 Team_Vision 3557 2019-10-08 STARK 0.83722 0.723 78% 100% 0.88187
65 IBM-MIT-Purdue 10038 2020-01-19 Purdue 0.84098 0.144 81% 100% 0.87514
66 Team_Vision 3382 2019-09-23 STARK 0.84258 0.726 81% 100% 0.88996
67 UCF 3472 2019-09-26 UCF-P 0.84269 0.292 100% 100% 0.89750
68 INF 8367 2020-01-11 set3 0.84990 0.88917 1.332 89% 85% 0.89547 0.92076
69 IBM-MIT-Purdue 8290 2020-01-10 Purdue 0.85131 0.301 100% 100% 0.88623
70 INF 7212 2019-12-29 INF_MEVA1 0.85864 0.922 89% 100% 0.87950
71 INF 6967 2019-12-23 set3 0.86506 0.911 89% 100% 0.88947
72 IBM-MIT-Purdue 10973 2020-01-24 Purdue 0.86613 0.151 72% 100% 0.88674
73 IBM-MIT-Purdue 10972 2020-01-24 Purdue 0.86613 0.152 72% 100% 0.88674
74 IBM-MIT-Purdue 10896 2020-01-23 Purdue 0.86614 0.143 72% 100% 0.88673
75 IBM-MIT-Purdue 9866 2020-01-17 Purdue 0.86887 0.142 72% 100% 0.88733
76 UCF 3104 2019-09-10 UCF-P 0.87068 0.291 97% 100% 0.91260
77 INF 6965 2019-12-23 INF_MEVA1 0.87216 0.906 89% 100% 0.89303
78 IBM-MIT-Purdue 10040 2020-01-19 Purdue 0.87422 0.147 70% 100% 0.89421
79 Team_Vision 2694 2019-08-27 STARK 0.87562 0.91641 1.795 86% 80% 0.90833 0.92705
80 IBM-MIT-Purdue 9873 2020-01-17 Purdue 0.87606 0.139 70% 100% 0.89165
81 UCF 4510 2019-11-06 UCF-P 0.87657 0.293 97% 100% 0.91490
82 INF 7152 2019-12-26 speed1x 0.88292 0.989 89% 86% 0.89853
83 Team_Vision 12208 2020-02-11 STARK 0.89021 0.783 94% 36% 0.89837
84 UCF 2014 2019-08-15 UCF-P 0.89321 0.236 91% 81% 0.92557
85 Team_Vision 12319 2020-02-12 STARK 0.89465 0.841 97% 26% 0.89748
86 UCF 4060 2019-10-25 UCF-P 0.89623 0.273 97% 100% 0.92032
87 Edge-Intelligence 7427 2020-01-02 Edge_Intelligence 0.92170 0.736 13% 96% 0.95539
88 INF 3276 2019-09-13 INF_MEVA_IOD 0.93009 0.588 94% 91% 0.94479
89 INF 8273 2020-01-10 INF_MEVA1 0.93512 0.95304 1.054 59% 100% 0.93680 0.95338
90 INF 2928 2019-08-29 INF_MEVA1 0.97344 0.788 89% 98% 0.98175
91 IBM-MIT-Purdue 8278 2020-01-10 Purdue 0.97569 0.361 29% 100% 0.98028
92 INF 7151 2019-12-26 meva_inf2 0.97762 0.931 83% 100% 0.97655
93 IBM-MIT-Purdue 8287 2020-01-10 Purdue 0.97778 0.361 24% 100% 0.98142
94 IBM-MIT-Purdue 8284 2020-01-10 Purdue 0.97778 0.374 24% 100% 0.98142
95 IBM-MIT-Purdue 8288 2020-01-10 Purdue 0.97778 0.359 24% 100% 0.98142
96 IBM-MIT-Purdue 8286 2020-01-10 Purdue 0.97778 0.357 24% 100% 0.98142
97 IBM-MIT-Purdue 8289 2020-01-10 Purdue 0.97778 0.370 24% 100% 0.98142
98 INF 4959 2019-11-21 meva_inf2 0.98157 0.98830 1.703 83% 89% 0.98886 0.99250
99 DIVA TE Baseline 1786 2019-08-02 RC3D 0.98216 0.959 29% 100% 0.98722
100 INF 3163 2019-09-12 speed1x 0.98315 0.860 78% 94% 0.98496
101 Team_Vision 8299 2020-01-10 STARK 0.99165 0.061 83% 86% 0.99143
102 Team_Vision 8295 2020-01-10 STARK 0.99211 0.060 72% 96% 0.99190
103 DIVA TE Baseline 7911 2020-01-07 RC3D 0.99255 0.99552 2.791 27% 75% 0.99302 0.99521
104 DIVA TE Baseline 7928 2020-01-07 RC3D-WHEEL 0.99608 0.99608 2.715 27% 37% 0.99580 0.99580
105 INF 2682 2019-08-27 INF_MEVA1 1.00000 0.000 0% 100% 1.00000
106 Team_Vision 8313 2020-01-10 STARK 1.00000 0.049 0% 79% 1.00000
107 IBM-MIT-Purdue 6056 2019-12-11 Purdue 1.00000 0.037 0% 100% 1.00000
108 IBM-MIT-Purdue 6055 2019-12-11 Purdue 1.00000 0.037 0% 98% 1.00000
109 IBM-MIT-Purdue 6000 2019-12-10 Purdue 1.00000 0.037 0% 98% 1.00000
110 Jay Chou 5904 2019-12-10 Jack 1.00000 0.179 0% 0% 1.00000
111 IBM-MIT-Purdue 5818 2019-12-09 Purdue 1.00000 0.037 0% 98% 1.00000
112 WeiLai 5817 2019-12-09 Tryyitry 1.00000 0.177 0% 0% 1.00000
113 WeiLai 5651 2019-12-07 Tryyitry 1.00000 0.178 0% 0% 1.00000
114 WeiLai 5576 2019-12-05 Tryyitry 1.00000 0.178 0% 0% 1.00000
115 IBM-MIT-Purdue 5506 2019-12-04 Purdue 1.00000 0.039 0% 98% 1.00000
116 IBM-MIT-Purdue 11642 2020-01-27 Purdue 1.00000 0.016 0% 100% 1.00000
117 WeiLai 5452 2019-12-04 Tryyitry 1.00000 0.178 0% 0% 1.00000
118 IBM-MIT-Purdue 5384 2019-12-02 Purdue 1.00000 0.039 0% 98% 1.00000
119 INF 4513 2019-11-08 INF_MEVA1 1.00000 0.039 0% 98% 1.00000

SDL19-scoring-EO

Updated: 2020-06-24 08:38:25 -0400

SDL19-scoring-IR

Updated: 2020-07-12 01:33:49 -0400
RANK SCORING_SUBMISSION_ID SCORING REQUEST NAME TEAM NAME SUBMISSION ID SUBMISSION DATE SYSTEM NAME SYSTEM ID SCORING PROTOCOL PARTIAL AUDC* TIME LIMITED PARTIAL AUDC* RELATIVE PROCESSING TIME DETECTED ACTIVITY TYPES† PROCESSED FILES‡ MEAN-P_MISS@0.04TFA TIME LIMITED MEAN-P_MISS@0.04TFA

SDL19-scoring-IR

Updated: 2020-07-12 01:33:49 -0400