Skip to main content

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

ActEV Sequestered Data Leaderboard (SDL) 2020/2019


ActEV Sequestered Data Leaderboard (SDL) Evaluation
Updates
  • March 01, 2020: The ActEV 2020 SDL opens with the expanded MEVA Test3 dataset
  • March 01, 2020: ActEV 2020 SDL is a guest task under ActivityNet CVPR'20 ActivityNet workshop
  • May 17, 2020 May 10, 2020 at 12:00 noon EST: New Deadline for CLI submissions to be included in ActEV SDL ActivityNet rankings. Top two submissions to be announced on June 1, 2020.
  • June 14, 2020: CVPR'20 ActivityNet workshop ActEV SDL guest task presentations
  • April 28th, 2020: CLI submissions deadline extended to May 17th, 2020 at 12:00 noon EST
Summary
The ActEV (Activities in Extended Video) Sequestered Data Leaderboard is an ongoing ranking of software systems that watch lengthy videos and detect activities of interest. Anyone can submit their system to NIST, which will then run the system on sequestered data, score the results and post the score to the leaderboard.
The sequestered data is from the MEVA dataset, which contains hours of videos, including indoor and outdoor scenes, night and day, crowds and individuals, and videos are from both EO (Electro-Optical) and IR (Infrared) sensors. Hours can go by with no activities, but then multiple activities happen simultaneously. The data is multi-camera, in that multiple cameras may be pointed at the same scene at the same time. There are also two separate leaderboards for EO and IR videos.
What
Build and submit a software system to watch videos and detect if and when an activity of interest occurs. System runtime must be 1 times the video length on the designated evaluation hardware.
Who
Everyone. Anyone who registers can submit to the evaluation server.
How
Register here and then follow the instructions on the Algorithm Submission tab above. Systems must follow a NIST-defined command line interface and automatically run on NIST’s servers, both of which are described in the instructions.
Evaluation Task
Detect if and when an activity occurs. Given a target activity type and a set of videos, submitted systems must automatically detect all instances of the activity in the videos. While different activities instances have different durations, a submitted system will be considered to have detected an activity if it correctly identifies at least 1 second of the activity.
Data
The data is from the Multiview Extended Video with Activities (MEVA) dataset and the videos are from both EO (Electro-Optical) and IR (Infrared) sensors. The data used for SDL evaluation from March 01, 2020 is the the expanded MEVA Test3 dataset. The public MEVA dataset includes hundreds of hours of data from the same cameras at the same facility, which can be used for training. You can download the public MEVA dataset for free at mevadata.org; and more info about the datasets is on the data tab. We also provide annotations for 20 hours of MEVA data, and instructions on how to make and share activity annotations are at mevadata.org.
Metrics
Submitted activity detection systems must give a confidence score for each activity they detect. Detected activities are then thresholded based on this confidence score. Varying the threshold makes a trade-off between being sensitive enough to identify true activity instances (low threshold) vs. not making false alarms when no activity is present (high threshold). Submitted systems are scored on both of these, measured by Probability of Missed Detection (Pmiss) and Time-based False Alarm (TFA). Pmiss is the portion of activities where the system did not detect the activity for at least 1 second. TFA is the portion of time that the system detected an activity when in fact there was none. Submitted systems are scored for Pmiss and TFA at multiple thresholds, creating a detect error tradeoff (DET) curve. The leaderboard ranking of a system is based on a summary of its DET curve: the area under the DET Curve across the TFA range between 0% to 20% divided by 0.2 to normalize the value to [0:1]. Lower numbers are better, as they reflect fewer errors. See details in the SDL evaluation plan or check out the ActEV Scoring Software GitHub repo.
Evaluation Plan
SDL Evaluation Plan
Task coordinator
ActEV NIST team (ActEV-nist@nist.gov)
News
01March
ActEV 2020 SDL starts with expanded MEVA Test3 dataset
17May
New Deadline to submit for ActivityNet task
ActEV SDL Evaluation Schedule
March 01, 2020: ActEV 2020 SDL opens with MEVA Test3.

May 17 May 10 , 2020 at 12 noon EST : New Deadline for CLI submissions to be included in ActEV guest task under CVPR'20 ActivityNet workshop.

June 01, 2020 : We will invite the top two teams on the ActEV 2020 SDL leaderboard to give ActEV guest task oral presentations at the CVPR'20 ActivityNet workshop based on the CLI submission deadline (May 17th May 10th , 2020).

June 14, 2020: CVPR'20 ActivityNet workshop ActEV SDL guest task presentations.

Leaderboard Remains Open: Leaderboard will continue to remain open to allow participants to show continued progress on this challenging problem.
ActEV SDL Dataset

Multiview Extended Video with Activities (MEVA)

The ActEV SDL evaluation is based on the Multiview Extended Video with Activities (MEVA) dataset (mevadata.org) collected at the Muscatatuck Urban Training Center with a team of over 100 actors performing in various scenarios. The data was built by the Intelligence Advanced Research Projects Activity (IARPA) Deep Intermodal Video Analytics (DIVA) program to support activity detection in multi-camera environments for both DIVA performers and the broader research community.
The MEVA dataset has two parts: the public training and development data and sequestered evaluation data used only by NIST to test systems. The data is accompanied by activity annotations.

Public Training and Development Data
The Multiview Extended Video with Activities (MEVA) dataset website mevadata.org is to share the public MEVA video dataset and annotations. The size of the public MEVA dataset is 333 hours of ground-camera and UAV video. The size of the provided annotated video dataset is 28 hours. ActEV participants are encouraged to annotate the MEVA KF1 dataset for the 37 activities as described at mevadata.org.


The MEVA data GIT repo is the data distribution mechanism for MEVA Related annotations and documentation. The repo presently consists of schemas for the activity annotations https://gitlab.kitware.com/meva/meva-data-repo.

The ActEV data GIT repo, is the data distribution mechanism for the ActEV evaluation. The repo presently consists of a collection of corpora and partition definition files to be used for the evaluations https://gitlab.kitware.com/actev/actev-data-repo.

Sequestered Evaluation Data
As of March 2020, NIST is using a 140-hour collection of annotated MEVA data for sequestered data evaluations. The data set consists of both EO and IR cameras, public cameras (examples of which are in the public data set). The leaderboard presents results on the full 140-hour collection reporting separately by EO and IR data. Developers receive additional scores by activity for the EO_subset1 and the IR_subset1. Both subsets consist of data from public cameras.
ActEV SDL Leaderboard

Update June 3, 2020: Reported scores for all submissions on the ActEV2020 leaderboards were revised using the ActEV_SDL_V2 scoring protocol which slightly decreased all error rates.

SDL20-scoring-EO

Updated: 2020-12-09 14:46:36 -0500
RANK SCORING_SUBMISSION_ID SCORING REQUEST NAME TEAM NAME SUBMISSION ID SUBMISSION DATE SYSTEM NAME SYSTEM ID SCORING PROTOCOL PARTIAL AUDC* TIME LIMITED PARTIAL AUDC* RELATIVE PROCESSING TIME DETECTED ACTIVITY TYPES† PROCESSED FILES‡ MEAN-P_MISS@0.04TFA TIME LIMITED MEAN-P_MISS@0.04TFA
1 21907 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00290_INF_20200802-111248-0247.sr-20200802-111248-6519 INF submission_description/21756|21756 2020-07-27 set3 290 ActEV_SDL_V2 0.35042 0.570 100% 100% 0.43284
2 22213 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00346_TeamVision2_20200917-045128-0151.sr-20200917-045129-0454 TeamVision2 submission_description/22212|22212 2020-09-14 dummy_test 346 ActEV_SDL_V2 0.35362 0.590 100% 100% 0.46523
3 21562 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00345_IBM-MIT-Purdue_20200723-223132-7371.sr-20200723-223133-0801 IBM-MIT-Purdue submission_description/20923|20923 2020-07-16 IBM-Purdue 345 ActEV_SDL_V2 0.35548 0.941 100% 100% 0.46655
4 19372 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200626-225528-9522.sr-20200626-225529-2904 UCF submission_description/19025|19025 2020-06-19 UCF-P 270 ActEV_SDL_V2 0.36126 0.722 100% 100% 0.42337
5 22080 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00326_UMD_20200813-111436-3293.sr-20200813-111443-6136 UMD submission_description/18450|18450 2020-06-07 UMD+UCF 326 ActEV_SDL_V2 0.36655 0.49867 1.486 94% 100% 0.42493 0.53357
6 16601 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00328_VUS_20200511-132633-4864.sr-20200515-001742-2385 VUS submission_description/16259|16259 2020-05-08 VUS-V1 328 ActEV_SDL_V2 0.40599 ** 1.344 100% 100% 0.46931 **
7 22204 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00349_UMCMU_20200908-213841-1038.sr-20200908-213842-3874 UMCMU submission_description/22202|22202 2020-09-05 UMCMU-T 349 ActEV_SDL_V2 0.43836 0.64138 2.779 100% 100% 0.53971 0.68352
8 15300 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200416-142124-7336.sr-20200519-235029-4101 NIST-TEST submission_description/13021|13021 2020-03-05 NIST Test 263 ActEV_SDL_V2 0.46563 0.822 97% 100% 0.55441
9 17319 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200519-012627-9123.sr-20200519-235030-8540 Team_Vision submission_description/16275|16275 2020-05-08 STARK 282 ActEV_SDL_V2 0.52993 1.002 100% 100% 0.62047
10 17477 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00337_vireoJD-MM_20200520-200345-0804.sr-20200526-131616-1250 vireoJD-MM submission_description/17093|17093 2020-05-16 vireo3 337 ActEV_SDL_V2 0.53876 0.149 100% 96% 0.67389
11 17913 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00338_BUPT-MCPRL_20200529-112230-6942.sr-20200529-112231-8566 BUPT-MCPRL submission_description/17732|17732 2020-05-26 bupt-mcprl 338 ActEV_SDL_V2 0.61532 0.969 100% 100% 0.63203
12 22210 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00335_CIS-JHU_20200910-111927-6409.sr-20200910-111939-5650 CIS_JHU submission_description/17471|17471 2020-05-20 jhu_stmpgnn 335 ActEV_SDL_V2 0.62911 0.77784 4.520 94% 69% 0.70393 0.79736
13 22200 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00276_DIVA-TE-Baseline_20200903-202020-0140.sr-20200903-202032-7226 DIVA TE Baseline submission_description/18234|18234 2020-06-04 RC3D 276 ActEV_SDL_V2 0.93925 0.94669 1.327 89% 100% 0.95265 0.95662
Download CSV

SDL20-scoring-EO

Updated:

SDL20-scoring-EO

Updated: 2020-12-09 14:46:25 -0500
RANK SCORING_SUBMISSION_ID SCORING REQUEST NAME TEAM NAME SUBMISSION ID SUBMISSION DATE SYSTEM NAME SYSTEM ID SCORING PROTOCOL PARTIAL AUDC* TIME LIMITED PARTIAL AUDC* RELATIVE PROCESSING TIME DETECTED ACTIVITY TYPES† PROCESSED FILES‡ MEAN-P_MISS@0.04TFA TIME LIMITED MEAN-P_MISS@0.04TFA
1 21907 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00290_INF_20200802-111248-0247.sr-20200802-111248-6519 INF submission_description/21756|21756 2020-07-27 set3 290 ActEV_SDL_V2 0.35042 0.570 100% 100% 0.43284
2 22213 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00346_TeamVision2_20200917-045128-0151.sr-20200917-045129-0454 TeamVision2 submission_description/22212|22212 2020-09-14 dummy_test 346 ActEV_SDL_V2 0.35362 0.590 100% 100% 0.46523
3 21562 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00345_IBM-MIT-Purdue_20200723-223132-7371.sr-20200723-223133-0801 IBM-MIT-Purdue submission_description/20923|20923 2020-07-16 IBM-Purdue 345 ActEV_SDL_V2 0.35548 0.941 100% 100% 0.46655
4 22092 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00346_TeamVision2_20200818-120721-8627.sr-20200818-120722-9978 TeamVision2 submission_description/22087|22087 2020-08-17 dummy_test 346 ActEV_SDL_V2 0.35861 0.165 100% 100% 0.46920
5 21720 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00345_IBM-MIT-Purdue_20200727-100139-3414.sr-20200727-100139-7088 IBM-MIT-Purdue submission_description/21563|21563 2020-07-24 IBM-Purdue 345 ActEV_SDL_V2 0.35903 0.164 100% 100% 0.46970
6 21558 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00345_IBM-MIT-Purdue_20200721-224447-0979.sr-20200723-115701-6484 IBM-MIT-Purdue submission_description/20925|20925 2020-07-16 IBM-Purdue 345 ActEV_SDL_V2 0.36026 0.933 100% 100% 0.47363
7 19372 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200626-225528-9522.sr-20200626-225529-2904 UCF submission_description/19025|19025 2020-06-19 UCF-P 270 ActEV_SDL_V2 0.36126 0.722 100% 100% 0.42337
8 17885 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200528-094551-9941.sr-20200528-094553-2266 UCF submission_description/17152|17152 2020-05-17 UCF-P 270 ActEV_SDL_V2 0.36452 0.684 100% 100% 0.42292
9 22080 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00326_UMD_20200813-111436-3293.sr-20200813-111443-6136 UMD submission_description/18450|18450 2020-06-07 UMD+UCF 326 ActEV_SDL_V2 0.36655 0.49867 1.486 94% 100% 0.42493 0.53357
10 22186 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00326_UMD_20200903-134945-6697.sr-20200903-135004-3443 UMD submission_description/19562|19562 2020-06-30 UMD+UCF 326 ActEV_SDL_V2 0.36941 0.51425 1.582 94% 100% 0.43018 0.54987
11 19469 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200629-055542-1886.sr-20200629-055542-4982 UCF submission_description/19365|19365 2020-06-26 UCF-P 270 ActEV_SDL_V2 0.36969 0.688 100% 100% 0.42401
12 17151 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200516-215958-2007.sr-20200519-235025-5233 UCF submission_description/16743|16743 2020-05-12 UCF-P 270 ActEV_SDL_V2 0.37281 0.581 94% 100% 0.42796
13 21246 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00345_IBM-MIT-Purdue_20200719-014924-2963.sr-20200719-014924-6450 IBM-MIT-Purdue submission_description/20951|20951 2020-07-16 IBM-Purdue 345 ActEV_SDL_V2 0.37300 0.939 100% 100% 0.49878
14 22066 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00345_IBM-MIT-Purdue_20200807-122924-7589.sr-20200807-122938-3167 IBM-MIT-Purdue submission_description/21386|21386 2020-07-20 IBM-Purdue 345 ActEV_SDL_V2 0.37434 0.134 100% 100% 0.49792
15 21560 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00345_IBM-MIT-Purdue_20200723-065908-6567.sr-20200723-115906-1667 IBM-MIT-Purdue submission_description/21401|21401 2020-07-20 IBM-Purdue 345 ActEV_SDL_V2 0.37434 0.165 100% 100% 0.49790
16 21363 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00345_IBM-MIT-Purdue_20200720-100421-3618.sr-20200720-100421-7590 IBM-MIT-Purdue submission_description/20924|20924 2020-07-16 IBM-Purdue 345 ActEV_SDL_V2 0.38039 0.945 100% 100% 0.50065
17 17057 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00298_INF_20200516-013223-2420.sr-20200526-131614-0410 INF submission_description/16309|16309 2020-05-08 speed1x 298 ActEV_SDL_V2 0.38699 0.498 97% 100% 0.46352
18 20991 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00283_INF_20200717-020850-9110.sr-20200717-020851-7129 INF submission_description/20674|20674 2020-07-14 INF_MEVA1 283 ActEV_SDL_V2 0.39638 0.569 100% 100% 0.50412
19 20788 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200715-135050-7837.sr-20200715-135051-1502 IBM-MIT-Purdue submission_description/20639|20639 2020-07-14 Purdue 312 ActEV_SDL_V2 0.40001 0.446 100% 100% 0.52175
20 21214 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200718-161337-9332.sr-20200718-161338-3634 IBM-MIT-Purdue submission_description/20790|20790 2020-07-15 Purdue 312 ActEV_SDL_V2 0.40279 0.105 100% 100% 0.53051
21 16601 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00328_VUS_20200511-132633-4864.sr-20200515-001742-2385 VUS submission_description/16259|16259 2020-05-08 VUS-V1 328 ActEV_SDL_V2 0.40599 ** 1.344 100% 100% 0.46931 **
22 20522 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200713-020327-1299.sr-20200713-020327-5401 IBM-MIT-Purdue submission_description/20365|20365 2020-07-12 Purdue 312 ActEV_SDL_V2 0.40889 0.094 100% 100% 0.53662
23 16051 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00288_INF_20200505-093715-5641.sr-20200519-235025-8703 INF submission_description/15846|15846 2020-05-01 meva_inf2 288 ActEV_SDL_V2 0.40897 0.927 97% 100% 0.48110
24 20535 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200713-140322-3163.sr-20200713-140322-6513 IBM-MIT-Purdue submission_description/20390|20390 2020-07-12 Purdue 312 ActEV_SDL_V2 0.40987 0.094 100% 100% 0.53775
25 20364 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200711-232330-7159.sr-20200711-232331-0498 IBM-MIT-Purdue submission_description/20207|20207 2020-07-10 Purdue 312 ActEV_SDL_V2 0.41002 0.093 100% 100% 0.53802
26 15414 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00307_INF_20200421-115525-8081.sr-20200519-235026-1773 INF submission_description/15132|15132 2020-04-13 INF_MEVA_IOD 307 ActEV_SDL_V2 0.41143 0.848 97% 98% 0.48364
27 22206 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00326_UMD_20200910-111409-6010.sr-20200910-111427-9528 UMD submission_description/15747|15747 2020-04-30 UMD+UCF 326 ActEV_SDL_V2 0.41450 0.51127 1.253 100% 100% 0.50159 0.57103
28 17326 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00334_VUS_20200519-030816-7908.sr-20200526-131615-6056 VUS submission_description/16908|16908 2020-05-14 VUS-V1-FAST 334 ActEV_SDL_V2 0.41493 0.829 100% 100% 0.48191
29 22207 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00326_UMD_20200910-111512-8994.sr-20200910-111526-9232 UMD submission_description/16984|16984 2020-05-15 UMD+UCF 326 ActEV_SDL_V2 0.41646 0.50800 1.226 97% 100% 0.49014 0.56142
30 22190 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200903-143511-7879.sr-20200903-143525-5231 UMD submission_description/21909|21909 2020-08-03 UMD 281 ActEV_SDL_V2 0.41865 0.43988 1.057 97% 100% 0.49523 0.51203
31 14577 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00283_INF_20200317-124405-8174.sr-20200519-235026-8006 INF submission_description/14480|14480 2020-03-16 INF_MEVA1 283 ActEV_SDL_V2 0.42496 0.527 97% 98% 0.49599
32 19647 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00288_INF_20200702-000615-6292.sr-20200702-000616-3752 INF submission_description/19451|19451 2020-06-28 meva_inf2 288 ActEV_SDL_V2 0.43139 0.610 100% 100% 0.54264
33 14857 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200319-153022-2982.sr-20200519-235027-0971 UCF submission_description/14760|14760 2020-03-18 UCF-P 270 ActEV_SDL_V2 0.43526 0.349 100% 100% 0.53063
34 14963 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200320-041131-0946.sr-20200519-235027-3712 UCF submission_description/14858|14858 2020-03-19 UCF-P 270 ActEV_SDL_V2 0.43555 0.346 100% 100% 0.52764
35 20361 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200711-122139-6464.sr-20200711-122140-0051 IBM-MIT-Purdue submission_description/20206|20206 2020-07-10 Purdue 312 ActEV_SDL_V2 0.43758 0.093 100% 100% 0.56295
36 22191 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200903-144008-5507.sr-20200903-144024-9839 UMD submission_description/19565|19565 2020-06-30 UMD 281 ActEV_SDL_V2 0.43796 0.45792 1.055 97% 100% 0.52779 0.54450
37 22204 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00349_UMCMU_20200908-213841-1038.sr-20200908-213842-3874 UMCMU submission_description/22202|22202 2020-09-05 UMCMU-T 349 ActEV_SDL_V2 0.43836 0.64138 2.779 100% 100% 0.53971 0.68352
38 15986 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200504-164235-0111.sr-20200519-235027-6683 UCF submission_description/15544|15544 2020-04-27 UCF-P 270 ActEV_SDL_V2 0.43888 0.356 97% 100% 0.50250
39 16558 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200510-225902-1029.sr-20200519-235027-9601 UCF submission_description/16212|16212 2020-05-07 UCF-P 270 ActEV_SDL_V2 0.43937 0.317 94% 100% 0.51352
40 22192 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200903-144542-8180.sr-20200903-144554-4687 UMD submission_description/18557|18557 2020-06-09 UMD 281 ActEV_SDL_V2 0.44296 0.44346 1.002 100% 100% 0.52413 0.52440
41 22166 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00298_INF_20200824-120056-4943.sr-20200824-120111-4317 INF submission_description/22076|22076 2020-08-11 speed1x 298 ActEV_SDL_V2 0.44860 0.759 100% 100% 0.55369
42 22160 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00349_UMCMU_20200822-025139-4835.sr-20200822-025140-6360 UMCMU submission_description/22100|22100 2020-08-20 UMCMU-T 349 ActEV_SDL_V2 0.44974 0.48978 1.142 100% 100% 0.55711 0.58248
43 22164 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00307_INF_20200824-114239-5597.sr-20200824-114253-9849 INF submission_description/22079|22079 2020-08-12 INF_MEVA_IOD 307 ActEV_SDL_V2 0.45250 0.531 100% 98% 0.52495
44 22158 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00283_INF_20200821-140731-1281.sr-20200821-140759-0942 INF submission_description/22084|22084 2020-08-16 INF_MEVA1 283 ActEV_SDL_V2 0.45250 0.540 100% 98% 0.52526
45 17599 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00290_INF_20200523-181607-2999.sr-20200605-094432-1913 INF submission_description/17472|17472 2020-05-20 set3 290 ActEV_SDL_V2 0.46181 0.915 100% 100% 0.57696
46 16257 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200508-110705-2778.sr-20200519-235028-2659 UCF submission_description/15949|15949 2020-05-03 UCF-P 270 ActEV_SDL_V2 0.46326 0.480 100% 100% 0.56739
47 18749 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200611-125827-6623.sr-20200611-125828-0367 UMD submission_description/18150|18150 2020-06-01 UMD 281 ActEV_SDL_V2 0.46455 0.884 100% 100% 0.57492
48 14856 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200319-133322-4135.sr-20200519-235029-1164 UMD submission_description/13739|13739 2020-03-11 UMD 281 ActEV_SDL_V2 0.46562 0.684 97% 100% 0.55473
49 15300 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200416-142124-7336.sr-20200519-235029-4101 NIST-TEST submission_description/13021|13021 2020-03-05 NIST Test 263 ActEV_SDL_V2 0.46563 0.822 97% 100% 0.55441
50 14483 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00298_INF_20200316-112137-9583.sr-20200519-235028-5577 INF submission_description/13770|13770 2020-03-11 speed1x 298 ActEV_SDL_V2 0.46565 0.613 97% 100% 0.54743
51 14202 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00298_INF_20200313-132531-6559.sr-20200519-235028-8388 INF submission_description/13315|13315 2020-03-10 speed1x 298 ActEV_SDL_V2 0.46566 0.615 97% 100% 0.54743
52 14819 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200319-091913-4675.sr-20200519-235029-6961 UMD submission_description/14296|14296 2020-03-13 UMD 281 ActEV_SDL_V2 0.46662 0.703 100% 100% 0.55902
53 15026 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200320-131358-4621.sr-20200519-235029-9885 UCF submission_description/13949|13949 2020-03-11 UCF-P 270 ActEV_SDL_V2 0.46704 0.333 97% 100% 0.53785
54 14669 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200318-053708-4851.sr-20200519-235030-2750 UCF submission_description/14578|14578 2020-03-17 UCF-P 270 ActEV_SDL_V2 0.46704 0.296 97% 100% 0.53785
55 16557 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00290_INF_20200510-225720-2108.sr-20200519-235030-5862 INF submission_description/16142|16142 2020-05-06 set3 290 ActEV_SDL_V2 0.47294 0.559 97% 100% 0.53440
56 22093 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00351_UMCMU_20200818-155914-5142.sr-20200818-155917-4156 UMCMU submission_description/22088|22088 2020-08-17 UMCMU-SF 351 ActEV_SDL_V2 0.47326 0.50639 1.096 100% 100% 0.58134 0.60281
57 19586 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200701-042026-6851.sr-20200701-042026-9738 UCF submission_description/19448|19448 2020-06-28 UCF-P 270 ActEV_SDL_V2 0.47688 0.515 100% 100% 0.51324
58 22081 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00349_UMCMU_20200813-201514-5379.sr-20200813-201515-6990 UMCMU submission_description/22072|22072 2020-08-11 UMCMU-T 349 ActEV_SDL_V2 0.47947 0.50746 1.084 100% 100% 0.58884 0.60606
59 17346 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20200519-111655-5218.sr-20200526-131613-0088 UCF submission_description/16386|16386 2020-05-09 UCF-P 270 ActEV_SDL_V2 0.48113 0.305 94% 100% 0.57069
60 18071 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00307_INF_20200530-125344-8660.sr-20200530-125346-3964 INF submission_description/17726|17726 2020-05-26 INF_MEVA_IOD 307 ActEV_SDL_V2 0.48144 0.824 97% 97% 0.57248
61 22157 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00351_UMCMU_20200821-120111-4361.sr-20200821-120116-5860 UMCMU submission_description/22095|22095 2020-08-19 UMCMU-SF 351 ActEV_SDL_V2 0.48268 0.51874 1.108 100% 100% 0.59095 0.61479
62 17744 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20200526-170134-8475.sr-20200526-170136-0564 UMD submission_description/17521|17521 2020-05-22 UMD 281 ActEV_SDL_V2 0.48305 0.905 97% 100% 0.57807
63 19367 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200626-155016-8498.sr-20200626-155017-1824 IBM-MIT-Purdue submission_description/19119|19119 2020-06-24 Purdue 312 ActEV_SDL_V2 0.48664 0.090 100% 100% 0.62999
64 22096 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00288_INF_20200819-093352-6433.sr-20200819-093353-7810 INF submission_description/22090|22090 2020-08-18 meva_inf2 288 ActEV_SDL_V2 0.48932 0.486 100% 100% 0.55322
65 18555 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00288_INF_20200609-113657-6904.sr-20200609-113658-4373 INF submission_description/18247|18247 2020-06-05 meva_inf2 288 ActEV_SDL_V2 0.49394 0.991 100% 100% 0.61605
66 18229 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00283_INF_20200603-135148-1770.sr-20200603-135149-1523 INF submission_description/18151|18151 2020-06-02 INF_MEVA1 283 ActEV_SDL_V2 0.49492 0.950 100% 98% 0.61675
67 20297 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200711-022734-9206.sr-20200711-022735-2782 IBM-MIT-Purdue submission_description/20129|20129 2020-07-10 Purdue 312 ActEV_SDL_V2 0.49857 0.063 100% 100% 0.63895
68 22101 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00283_INF_20200820-033210-9803.sr-20200820-033212-8662 INF submission_description/22097|22097 2020-08-19 INF_MEVA1 283 ActEV_SDL_V2 0.50539 0.461 100% 100% 0.56714
69 18235 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200605-095636-6760.sr-20200605-095637-0432 IBM-MIT-Purdue submission_description/18073|18073 2020-05-30 Purdue 312 ActEV_SDL_V2 0.50541 0.366 102% 100% 0.65305
70 17319 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200519-012627-9123.sr-20200519-235030-8540 Team_Vision submission_description/16275|16275 2020-05-08 STARK 282 ActEV_SDL_V2 0.52993 1.002 100% 100% 0.62047
71 22208 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200910-111621-4009.sr-20200910-111632-2445 Team_Vision submission_description/16060|16060 2020-05-05 STARK 282 ActEV_SDL_V2 0.52993 0.53006 1.003 100% 100% 0.62047 0.62012
72 17090 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200516-094544-8275.sr-20200519-235031-3894 IBM-MIT-Purdue submission_description/16902|16902 2020-05-14 Purdue 312 ActEV_SDL_V2 0.53528 0.080 97% 100% 0.66782
73 22174 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00290_INF_20200826-171114-9686.sr-20200826-171116-0653 INF submission_description/22170|22170 2020-08-25 set3 290 ActEV_SDL_V2 0.53531 0.518 100% 100% 0.59422
74 17477 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00337_vireoJD-MM_20200520-200345-0804.sr-20200526-131616-1250 vireoJD-MM submission_description/17093|17093 2020-05-16 vireo3 337 ActEV_SDL_V2 0.53876 0.149 100% 96% 0.67389
75 22194 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200903-162052-0365.sr-20200903-162102-4232 Team_Vision submission_description/20960|20960 2020-07-16 STARK 282 ActEV_SDL_V2 0.54007 0.54326 1.021 100% 100% 0.63513 0.63803
76 22173 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00349_UMCMU_20200826-133244-4242.sr-20200826-133245-6193 UMCMU submission_description/22165|22165 2020-08-24 UMCMU-T 349 ActEV_SDL_V2 0.54015 0.973 100% 100% 0.61852
77 16135 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00329_vireoJD-MM_20200506-100639-0175.sr-20200519-235031-6711 vireoJD-MM submission_description/15735|15735 2020-04-30 Vireo 329 ActEV_SDL_V2 0.54107 0.160 100% 96% 0.67380
78 22195 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200903-170336-8983.sr-20200903-170348-6197 Team_Vision submission_description/20532|20532 2020-07-13 STARK 282 ActEV_SDL_V2 0.54918 0.55134 1.014 100% 100% 0.64655 0.64749
79 22161 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00288_INF_20200822-124810-1592.sr-20200822-124811-3286 INF submission_description/22159|22159 2020-08-22 meva_inf2 288 ActEV_SDL_V2 0.54966 0.520 100% 100% 0.60651
80 17008 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00331_vireoJD-MM_20200515-163029-9498.sr-20200519-235031-9723 vireoJD-MM submission_description/16445|16445 2020-05-09 vireo2 331 ActEV_SDL_V2 0.55665 0.163 100% 96% 0.68832
81 17387 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200519-233350-8538.sr-20200526-131614-5743 IBM-MIT-Purdue submission_description/16909|16909 2020-05-14 Purdue 312 ActEV_SDL_V2 0.56720 0.080 97% 100% 0.69927
82 19993 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200708-194028-8463.sr-20200708-194029-1870 Team_Vision submission_description/19871|19871 2020-07-07 STARK 282 ActEV_SDL_V2 0.57102 0.993 97% 100% 0.65497
83 22209 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200910-111813-0426.sr-20200910-111827-9542 Team_Vision submission_description/16705|16705 2020-05-12 STARK 282 ActEV_SDL_V2 0.61351 0.63166 1.096 100% 100% 0.70557 0.71890
84 17913 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00338_BUPT-MCPRL_20200529-112230-6942.sr-20200529-112231-8566 BUPT-MCPRL submission_description/17732|17732 2020-05-26 bupt-mcprl 338 ActEV_SDL_V2 0.61532 0.969 100% 100% 0.63203
85 22210 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00335_CIS-JHU_20200910-111927-6409.sr-20200910-111939-5650 CIS_JHU submission_description/17471|17471 2020-05-20 jhu_stmpgnn 335 ActEV_SDL_V2 0.62911 0.77784 4.520 94% 69% 0.70393 0.79736
86 22198 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200903-200540-0013.sr-20200903-200553-6396 Team_Vision submission_description/18959|18959 2020-06-18 STARK 282 ActEV_SDL_V2 0.62962 0.62956 1.005 100% 100% 0.71889 0.71876
87 15224 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200415-114050-1365.sr-20200519-235032-2671 IBM-MIT-Purdue submission_description/15129|15129 2020-04-10 Purdue 312 ActEV_SDL_V2 0.64260 0.148 100% 100% 0.72913
88 22175 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00307_INF_20200827-105401-5621.sr-20200827-105403-0050 INF submission_description/22167|22167 2020-08-24 INF_MEVA_IOD 307 ActEV_SDL_V2 0.68832 0.551 100% 100% 0.71984
89 14009 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200312-162922-5840.sr-20200519-235032-5355 Team_Vision submission_description/13117|13117 2020-03-08 STARK 282 ActEV_SDL_V2 0.69971 0.685 97% 100% 0.76985
90 15027 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200320-133708-7523.sr-20200519-235032-8099 Team_Vision submission_description/13533|13533 2020-03-10 STARK 282 ActEV_SDL_V2 0.70162 0.681 97% 98% 0.77027
91 15413 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200421-105807-5524.sr-20200519-235033-1130 IBM-MIT-Purdue submission_description/15320|15320 2020-04-18 Purdue 312 ActEV_SDL_V2 0.74163 0.047 100% 100% 0.85468
92 16614 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00312_IBM-MIT-Purdue_20200511-173821-3000.sr-20200519-235033-4110 IBM-MIT-Purdue submission_description/15562|15562 2020-04-28 Purdue 312 ActEV_SDL_V2 0.74383 0.047 100% 100% 0.85239
93 20127 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20200710-095000-2256.sr-20200710-095000-5705 Team_Vision submission_description/19750|19750 2020-07-06 STARK 282 ActEV_SDL_V2 0.81637 0.918 97% 100% 0.84071
94 22185 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00349_UMCMU_20200902-115359-4665.sr-20200902-115400-5492 UMCMU submission_description/22181|22181 2020-08-31 UMCMU-T 349 ActEV_SDL_V2 0.82685 0.85776 1.262 43% 100% 0.85385 0.87633
95 22171 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00283_INF_20200826-035642-4842.sr-20200826-035643-6712 INF submission_description/22169|22169 2020-08-25 INF_MEVA1 283 ActEV_SDL_V2 0.85228 0.076 100% 100% 0.85927
96 22200 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00276_DIVA-TE-Baseline_20200903-202020-0140.sr-20200903-202032-7226 DIVA TE Baseline submission_description/18234|18234 2020-06-04 RC3D 276 ActEV_SDL_V2 0.93925 0.94669 1.327 89% 100% 0.95265 0.95662
97 22199 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00341_DIVA-TE-Baseline_20200903-201905-7262.sr-20200903-201917-7742 DIVA TE Baseline submission_description/18498|18498 2020-06-08 RC3D-kw-annotations 341 ActEV_SDL_V2 0.96674 0.97091 1.236 75% 100% 0.97908 0.98109
98 19032 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00307_INF_20200623-035629-8422.sr-20200623-035630-2041 INF submission_description/18968|18968 2020-06-18 INF_MEVA_IOD 307 ActEV_SDL_V2 0.97072 0.523 97% 3% 0.97095
99 18855 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200612-101533-8300.sr-20200612-101534-1008 NIST-TEST submission_description/18753|18753 2020-06-11 NIST Test 263 ActEV_SDL_V2 1.00000 0.031 0% 100% 1.00000
100 22153 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200820-163504-5674.sr-20200820-163505-5395 NIST-TEST submission_description/22067|22067 2020-08-10 NIST Test 263 ActEV_SDL_V2 1.00000 0.096 0% 100% 1.00000
101 22182 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200831-172129-2294.sr-20200831-172130-1681 NIST-TEST submission_description/22180|22180 2020-08-31 NIST Test 263 ActEV_SDL_V2 1.00000 0.029 0% 100% 1.00000
102 18863 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200612-174502-8142.sr-20200612-174503-0813 NIST-TEST submission_description/18565|18565 2020-06-09 NIST Test 263 ActEV_SDL_V2 1.00000 0.031 0% 100% 1.00000
103 22184 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200901-234020-6634.sr-20200901-234021-9031 NIST-TEST submission_description/22183|22183 NIST Test 263 ActEV_SDL_V2 1.00000 0.030 0% 100% 1.00000
104 22155 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200821-104029-2935.sr-20200821-104030-2742 NIST-TEST submission_description/22154|22154 2020-08-20 NIST Test 263 ActEV_SDL_V2 1.00000 0.030 0% 100% 1.00000
105 18859 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200612-105746-5020.sr-20200612-105746-7748 NIST-TEST submission_description/18238|18238 2020-06-05 NIST Test 263 ActEV_SDL_V2 1.00000 0.031 0% 100% 1.00000
106 19360 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20200626-094820-3881.sr-20200626-094820-6286 NIST-TEST submission_description/19117|19117 2020-06-23 NIST Test 263 ActEV_SDL_V2 1.00000 0.030 0% 100% 1.00000
Download CSV

SDL20-scoring-EO

Updated:

SDL19-scoring-EO

Updated: 2020-06-24 08:44:02 -0400
RANK TEAM NAME SUBMISSION ID SUBMISSION DATE SYSTEM NAME PARTIAL AUDC* TIME LIMITED PARTIAL AUDC* RELATIVE PROCESSING TIME DETECTED ACTIVITY TYPES† PROCESSED FILES‡ MEAN-P_MISS@0.04TFA TIME LIMITED MEAN-P_MISS@0.04TFA
1 UMD 11045 2020-01-24 UMD+UCF 0.41585 0.65517 1.154 100% 100% 0.49360 0.67902
2 UCF 10640 2020-01-21 UCF-P 0.43810 0.362 100% 100% 0.52356
3 UCF 12650 2020-02-25 UCF-P 0.44047 0.364 97% 100% 0.54081
4 UCF 9105 2020-01-15 UCF-P 0.44056 0.424 97% 100% 0.52513
5 UCF 12222 2020-02-11 UCF-P 0.44198 0.395 100% 100% 0.53514
6 UCF 8328 2020-01-10 UCF-P 0.44360 0.403 100% 100% 0.52972
7 UCF 10821 2020-01-22 UCF-P 0.44473 0.407 100% 100% 0.54228
8 UCF 11849 2020-02-02 UCF-P 0.45165 0.378 100% 100% 0.51752
9 UMD 5595 2019-12-06 UMD 0.47596 0.725 97% 100% 0.54479
10 UMD 12468 2020-02-18 UMD 0.47611 0.849 97% 100% 0.55945
11 UMD 8222 2020-01-10 UMD 0.47668 0.68909 1.260 100% 100% 0.54492 0.71689
12 UMD 8198 2020-01-09 UMD 0.48037 0.68601 1.134 97% 100% 0.54825 0.71585
13 UCF 7728 2020-01-04 UCF-P 0.48290 0.438 94% 100% 0.51235
14 UMD 7992 2020-01-08 UMD 0.48333 0.68298 1.098 97% 100% 0.55028 0.71410
15 INF 11911 2020-02-06 set3 0.48978 0.646 97% 100% 0.55928
16 UMD 10000 2020-01-19 UMD 0.49061 0.803 100% 100% 0.56996
17 UMD 7472 2020-01-02 UMD 0.49689 0.61793 1.042 97% 93% 0.55515 0.65471
18 UCF 7235 2019-12-30 UCF-P 0.51676 0.347 97% 100% 0.55164
19 UCF 6499 2019-12-18 UCF-P 0.52068 0.333 97% 100% 0.57885
20 UMD 4809 2019-11-15 UMD 0.53144 0.965 100% 98% 0.60011
21 UMD 3666 2019-10-10 UMD 0.53147 0.970 100% 98% 0.60101
22 UMD 6285 2019-12-12 UMD 0.53459 0.723 100% 98% 0.59596
23 INF 12068 2020-02-09 speed1x 0.53654 0.642 97% 94% 0.59701
24 INF 11850 2020-02-03 meva_inf2 0.59044 0.960 94% 97% 0.64252
25 UCF 5851 2019-12-09 UCF-P 0.60431 0.293 97% 100% 0.64157
26 Edge-Intelligence 8003 2020-01-08 Edge-Intelligence 0.62842 0.939 97% 100% 0.75488
27 IBM-MIT-Purdue 6513 2019-12-18 Purdue 0.64182 0.272 100% 100% 0.73321
28 IBM-MIT-Purdue 11346 2020-01-26 Purdue 0.64182 0.128 100% 100% 0.73320
29 IBM-MIT-Purdue 11345 2020-01-26 Purdue 0.64182 0.125 100% 100% 0.73320
30 Edge-Intelligence 7845 2020-01-06 Edge_Intelligence 0.64737 0.887 94% 100% 0.76268
31 UMD 2074 2019-08-16 UMD 0.65325 0.744 97% 98% 0.73860
32 IBM-MIT-Purdue 6815 2019-12-20 Purdue 0.65351 0.315 100% 100% 0.73730
33 IBM-MIT-Purdue 5004 2019-11-25 Purdue 0.65503 0.124 100% 98% 0.73745
34 IBM-MIT-Purdue 5005 2019-11-25 Purdue 0.65913 0.124 100% 98% 0.73971
35 IBM-MIT-Purdue 6501 2019-12-18 Purdue 0.66055 0.442 100% 100% 0.74490
36 IBM-MIT-Purdue 4660 2019-11-11 Purdue 0.66383 0.087 100% 98% 0.74381
37 IBM-MIT-Purdue 11343 2020-01-26 Purdue 0.66394 0.038 100% 98% 0.74397
38 IBM-MIT-Purdue 11344 2020-01-26 Purdue 0.66394 0.035 100% 98% 0.74397
39 IBM-MIT-Purdue 11268 2020-01-26 Purdue 0.66496 0.124 100% 100% 0.74147
40 IBM-MIT-Purdue 6303 2019-12-12 Purdue 0.66900 0.096 100% 100% 0.75098
41 IBM-MIT-Purdue 6547 2019-12-18 Purdue 0.67114 0.099 100% 100% 0.75100
42 UMD 1907 2019-08-13 UMD 0.67831 0.727 0.74917
43 IBM-MIT-Purdue 4999 2019-11-25 Purdue 0.67912 0.098 100% 98% 0.76349
44 INF 11670 2020-01-28 INF_MEVA1 0.67983 0.928 100% 100% 0.78032
45 IBM-MIT-Purdue 5001 2019-11-25 Purdue 0.68547 0.099 100% 98% 0.76606
46 IBM-MIT-Purdue 11188 2020-01-25 Purdue 0.69501 0.108 100% 98% 0.75741
47 UCF 5596 2019-12-06 UCF-P 0.69848 0.304 86% 100% 0.73165
48 IBM-MIT-Purdue 6443 2019-12-17 Purdue 0.70117 0.82690 1.072 100% 100% 0.77819 0.87029
49 IBM-MIT-Purdue 4876 2019-11-19 Purdue 0.70382 0.088 97% 98% 0.76211
50 Team_Vision 6865 2019-12-20 STARK 0.71736 0.793 97% 100% 0.77673
51 Team_Vision 4434 2019-11-05 STARK 0.71911 0.774 97% 100% 0.77884
52 UCF 5439 2019-12-04 UCF-P 0.72830 0.304 86% 100% 0.75572
53 IBM-MIT-Purdue 8154 2020-01-09 Purdue 0.74090 0.380 100% 100% 0.82467
54 IBM-MIT-Purdue 9978 2020-01-18 Purdue 0.74091 0.145 100% 100% 0.82468
55 UCF 5077 2019-11-27 UCF-P 0.74243 0.312 86% 100% 0.77197
56 IBM-MIT-Purdue 4228 2019-10-31 Purdue 0.75410 0.093 97% 98% 0.82355
57 IBM-MIT-Purdue 4520 2019-11-08 Purdue 0.75509 0.087 100% 98% 0.82359
58 IBM-MIT-Purdue 10037 2020-01-19 Purdue 0.79701 0.151 86% 100% 0.84131
59 IBM-MIT-Purdue 4179 2019-10-30 Purdue 0.81573 0.093 97% 98% 0.86015
60 IBM-MIT-Purdue 10041 2020-01-19 Purdue 0.82020 0.143 100% 100% 0.88273
61 IBM-MIT-Purdue 3830 2019-10-21 Purdue 0.82025 0.246 100% 97% 0.86425
62 Team_Vision 4071 2019-10-29 STARK 0.82121 0.757 75% 100% 0.85712
63 Team_Vision 8506 2020-01-12 STARK 0.82442 0.739 94% 58% 0.85486
64 Team_Vision 3557 2019-10-08 STARK 0.83722 0.723 78% 100% 0.88187
65 IBM-MIT-Purdue 10038 2020-01-19 Purdue 0.84098 0.144 81% 100% 0.87514
66 Team_Vision 3382 2019-09-23 STARK 0.84258 0.726 81% 100% 0.88996
67 UCF 3472 2019-09-26 UCF-P 0.84269 0.292 100% 100% 0.89750
68 INF 8367 2020-01-11 set3 0.84990 0.88917 1.332 89% 85% 0.89547 0.92076
69 IBM-MIT-Purdue 8290 2020-01-10 Purdue 0.85131 0.301 100% 100% 0.88623
70 INF 7212 2019-12-29 INF_MEVA1 0.85864 0.922 89% 100% 0.87950
71 INF 6967 2019-12-23 set3 0.86506 0.911 89% 100% 0.88947
72 IBM-MIT-Purdue 10973 2020-01-24 Purdue 0.86613 0.151 72% 100% 0.88674
73 IBM-MIT-Purdue 10972 2020-01-24 Purdue 0.86613 0.152 72% 100% 0.88674
74 IBM-MIT-Purdue 10896 2020-01-23 Purdue 0.86614 0.143 72% 100% 0.88673
75 IBM-MIT-Purdue 9866 2020-01-17 Purdue 0.86887 0.142 72% 100% 0.88733
76 UCF 3104 2019-09-10 UCF-P 0.87068 0.291 97% 100% 0.91260
77 INF 6965 2019-12-23 INF_MEVA1 0.87216 0.906 89% 100% 0.89303
78 IBM-MIT-Purdue 10040 2020-01-19 Purdue 0.87422 0.147 70% 100% 0.89421
79 Team_Vision 2694 2019-08-27 STARK 0.87562 0.91641 1.795 86% 80% 0.90833 0.92705
80 IBM-MIT-Purdue 9873 2020-01-17 Purdue 0.87606 0.139 70% 100% 0.89165
81 UCF 4510 2019-11-06 UCF-P 0.87657 0.293 97% 100% 0.91490
82 INF 7152 2019-12-26 speed1x 0.88292 0.989 89% 86% 0.89853
83 Team_Vision 12208 2020-02-11 STARK 0.89021 0.783 94% 36% 0.89837
84 UCF 2014 2019-08-15 UCF-P 0.89321 0.236 91% 81% 0.92557
85 Team_Vision 12319 2020-02-12 STARK 0.89465 0.841 97% 26% 0.89748
86 UCF 4060 2019-10-25 UCF-P 0.89623 0.273 97% 100% 0.92032
87 Edge-Intelligence 7427 2020-01-02 Edge_Intelligence 0.92170 0.736 13% 96% 0.95539
88 INF 3276 2019-09-13 INF_MEVA_IOD 0.93009 0.588 94% 91% 0.94479
89 INF 8273 2020-01-10 INF_MEVA1 0.93512 0.95304 1.054 59% 100% 0.93680 0.95338
90 INF 2928 2019-08-29 INF_MEVA1 0.97344 0.788 89% 98% 0.98175
91 IBM-MIT-Purdue 8278 2020-01-10 Purdue 0.97569 0.361 29% 100% 0.98028
92 INF 7151 2019-12-26 meva_inf2 0.97762 0.931 83% 100% 0.97655
93 IBM-MIT-Purdue 8287 2020-01-10 Purdue 0.97778 0.361 24% 100% 0.98142
94 IBM-MIT-Purdue 8284 2020-01-10 Purdue 0.97778 0.374 24% 100% 0.98142
95 IBM-MIT-Purdue 8288 2020-01-10 Purdue 0.97778 0.359 24% 100% 0.98142
96 IBM-MIT-Purdue 8286 2020-01-10 Purdue 0.97778 0.357 24% 100% 0.98142
97 IBM-MIT-Purdue 8289 2020-01-10 Purdue 0.97778 0.370 24% 100% 0.98142
98 INF 4959 2019-11-21 meva_inf2 0.98157 0.98830 1.703 83% 89% 0.98886 0.99250
99 DIVA TE Baseline 1786 2019-08-02 RC3D 0.98216 0.959 29% 100% 0.98722
100 INF 3163 2019-09-12 speed1x 0.98315 0.860 78% 94% 0.98496
101 Team_Vision 8299 2020-01-10 STARK 0.99165 0.061 83% 86% 0.99143
102 Team_Vision 8295 2020-01-10 STARK 0.99211 0.060 72% 96% 0.99190
103 DIVA TE Baseline 7911 2020-01-07 RC3D 0.99255 0.99552 2.791 27% 75% 0.99302 0.99521
104 DIVA TE Baseline 7928 2020-01-07 RC3D-WHEEL 0.99608 0.99608 2.715 27% 37% 0.99580 0.99580
105 INF 2682 2019-08-27 INF_MEVA1 1.00000 0.000 0% 100% 1.00000
106 Team_Vision 8313 2020-01-10 STARK 1.00000 0.049 0% 79% 1.00000
107 IBM-MIT-Purdue 6056 2019-12-11 Purdue 1.00000 0.037 0% 100% 1.00000
108 IBM-MIT-Purdue 6055 2019-12-11 Purdue 1.00000 0.037 0% 98% 1.00000
109 IBM-MIT-Purdue 6000 2019-12-10 Purdue 1.00000 0.037 0% 98% 1.00000
110 Jay Chou 5904 2019-12-10 Jack 1.00000 0.179 0% 0% 1.00000
111 IBM-MIT-Purdue 5818 2019-12-09 Purdue 1.00000 0.037 0% 98% 1.00000
112 WeiLai 5817 2019-12-09 Tryyitry 1.00000 0.177 0% 0% 1.00000
113 WeiLai 5651 2019-12-07 Tryyitry 1.00000 0.178 0% 0% 1.00000
114 WeiLai 5576 2019-12-05 Tryyitry 1.00000 0.178 0% 0% 1.00000
115 IBM-MIT-Purdue 5506 2019-12-04 Purdue 1.00000 0.039 0% 98% 1.00000
116 IBM-MIT-Purdue 11642 2020-01-27 Purdue 1.00000 0.016 0% 100% 1.00000
117 WeiLai 5452 2019-12-04 Tryyitry 1.00000 0.178 0% 0% 1.00000
118 IBM-MIT-Purdue 5384 2019-12-02 Purdue 1.00000 0.039 0% 98% 1.00000
119 INF 4513 2019-11-08 INF_MEVA1 1.00000 0.039 0% 98% 1.00000
Download CSV

SDL19-scoring-EO

Updated:

SDL19-scoring-IR

Updated: 2023-09-18 16:41:11 -0400
RANK SCORING REQUEST NAME TEAM NAME SUBMISSION ID SYSTEM NAME SYSTEM ID PARTIAL AUDC* RELATIVE PROCESSING TIME DETECTED ACTIVITY TYPES† PROCESSED FILES‡ MEAN-P_MISS@0.04TFA
1 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20190903-171132-2439.sr-20190904-151050-8165 UMD 1907 UMD 281 0.82140 0.727 97% 94% 0.87521
2 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00281_UMD_20190905-120934-4541.sr-20190905-120934-7789 UMD 2074 UMD 281 0.84254 0.744 97% 94% 0.88553
3 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20190913-094214-5138.sr-20190913-094214-7316 UCF 3104 UCF-P 270 0.96630 0.291 78% 100% 0.97125
4 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00263_NIST-TEST_20190814-164723-7643.sr-20190814-164723-9641 NIST-TEST 1790 NIST Test 263 0.99139 0.930 27% 100% 0.99280
5 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00276_DIVA-TE-Baseline_20190919-133858-1097.sr-20190919-133858-3388 DIVA TE Baseline 1786 RC3D 276 0.99139 0.959 27% 100% 0.99280
6 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20190930-141448-4379.sr-20190930-141451-2761 Team_Vision 3382 STARK 282 0.99268 0.726 45% 100% 0.99345
7 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00270_UCF_20190905-151917-3849.sr-20190905-151917-6400 UCF 2014 UCF-P 270 0.99346 0.236 56% 100% 0.99547
8 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20190905-154602-1817.sr-20190905-154603-7118 Team_Vision 2048 STARK 282 0.99682 1.132 51% 100% 0.99673
9 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00282_Team-Vision_20190905-165243-1769.sr-20190905-165244-9598 Team_Vision 2694 STARK 282 0.99804 1.795 48% 94% 0.99869
10 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00283_INF_20190903-152553-6173.sr-20190903-152553-9672 INF 2928 INF_MEVA1 283 1.00000 0.788 54% 100% 1.00000
11 ActEV-2018_AD_ActEV19-SDL-Scoring_SYS-00283_INF_20190828-173003-1496.sr-20190828-173003-3976 INF 2682 INF_MEVA1 283 1.00000 0.000 0% 100% 1.00000
Download CSV

SDL19-scoring-IR

Updated:

**The specified submissions exceeded the 1X realtime proccessing speed however Time-limited nAUDCs were not computed.

*Partial AUDC is the area under the DET curve between a Time-based False Alarm rate of 0 and 0.2. Value of a perfect system is 0.

†The fraction of activity types with detected instances in the processed data set.

‡The fraction of videos processed and scored.

Activities for the ActEV Sequestered Data Leaderboard

We have updated the activity names for the SDL. We intend for these names to be used for the duration of the DIVA program. The list below shows the "ActEV 2020 SDL Activity Name" and "ActEV 2019 SDL Activity Name" of the 37 activities to be detected for the ActEV SDL evaluation. Detailed activity definitions are in the ActEV Annotation Definitions for MEVA Data document. Note that current scoring for the SDL is based exclusively on activity detection, and object detection is not considered. The link to the activity-name-mapping.csv .

ActEV SDL Activities


ActEV 2020 SDL Activity Name ActEV 2019 SDL Activity Name (DEPRECATED)
person_abandons_package abandon_package
person_closes_facility_door person_closes_facility_door
person_closes_trunk Closing_Trunk
person_closes_vehicle_door person_closes_vehicle_door
person_embraces_person person_person_embrace
person_enters_scene_through_structure person_enters_through_structure
person_enters_vehicle person_enters_vehicle
person_exits_scene_through_structure person_exits_through_structure
person_exits_vehicle person_exits_vehicle
hand_interacts_with_person hand_interaction
person_carries_heavy_object Transport_HeavyCarry
person_interacts_with_laptop person_laptop_interaction
person_loads_vehicle person_loads_vehicle
person_transfers_object object_transfer
person_opens_facility_door person_opens_facility_door
person_opens_trunk Open_Trunk
person_opens_vehicle_door person_opens_vehicle_door
person_talks_to_person Talking
person_picks_up_object person_picks_up_object
person_purchases person_purchasing
person_reads_document person_reading_document
person_rides_bicycle Riding
person_puts_down_object person_sets_down_object
person_sits_down person_sitting_down
person_stands_up person_standing_up
person_talks_on_phone specialized_talking_phone
person_texts_on_phone specialized_texting_phone
person_steals_object theft
person_unloads_vehicle Unloading
vehicle_drops_off_person vehicle_drops_off_person
vehicle_picks_up_person vehicle_picks_up_person
vehicle_reverses vehicle_reversing
vehicle_starts vehicle_starting
vehicle_stops vehicle_stopping
vehicle_turns_left vehicle_turning_left
vehicle_turns_right vehicle_turning_right
vehicle_makes_u_turn vehicle_u_turn
Task for the ActEV Sequestered Data Leaderboard

In the SDL evaluation, there is one Activity Detection (AD) task for detecting and temporally localizing activities.


Activity Detection (AD)
For the Activity Detection task, given a target activity, a system automatically detects and temporally localizes all instances of the activity. For a system-identified activity instance to be evaluated as correct, the type of activity must be correct and temporally overlap the true activity for at least one second. Additional details may be found in the SDL Evaluation Plan.
Algorithm Delivery for the SDL participants

System delivery to the leaderboard must be in a form compatible with the ActEV Command Line Interface (ActEV CLI) and submitted to NIST for testing. The command line interface implementation that you will provide formalizes the entire process of evaluating a system, by providing the evaluation team a means to: (1) download and install your software via a single URL, (2) verify that the delivery works AND produces output that is “consistent” with output you produce, and (3) process a large collection of video in a fault-tolerant, parallelizable manner.

To complete this task you will need the following items described in detail below:

  1. FAQ - Validation Phase Processing
  2. CLI Description
  3. The Abstract CLI Git Repository
  4. The CLI Implementation Primer
  5. The Validation Data Set
  6. Example CLI-Compliant Implementation
  7. NIST Hardware and Initial Operating System Description
  8. SDL Submission Processing Pipeline

1. FAQ - Validation Phase Processing

The ActEV SDL - Validation Phase Processing

FAQ

2. CLI Description

The ActEV CLI description

3. The Abstract CLI GIT Repository

The Abstract CLI GIT repository. The repo contains documentation.

4. The CLI Implementation Primer

There are 6 steps to adapt your code to the CLI. The ActEV Evaluation CLI Programming Primer describes the steps to clone the Abstract CLI and begin adapting the code to your implementation.

5. The Validation Data Set

As mentioned above, the CLI is used to verify the downloaded software is correctly installed and produces the same output at NIST as you produce locally. In order to do so, we have provided a small validation data set (ActEV-Eval-CLI-Validation-Set3) as part of the ActEV SDL Dataset that will be processed both at your site and at NIST. Please use this data in Step 3 of the Primer described above.

6. Example CLI-Compliant Implementation

The links below provide two example CLI implementations for the leaderboard baseline algorithm:

7. NIST Independent Evaluation Infrastructure Specification

NIST will begin installing your system from a fresh Ubuntu 18.04 cloud image available from https://cloud-images.ubuntu.com/releases/18.04/release/ on the following hardware.

  • Chassis: Asus ESC4000 G4S
  • CPU: 2 x Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz (12 cores/CPU)
  • Motherboard: Asus Intel® C621 PCH chipset
  • HDD/SSD: 2x 1.92GB Intel SSD DC S4500
  • RAM: 12x 16GB DDR4-2400 ECC RDIMM
  • GPU: Four PNY RTX2080Ti blower style
  • OS: Ubuntu 18.04
  • Storage Volume- 1TB (variable)
  • Supplied object store (read only) for source video

8. SDL Submission Processing Pipeline

There are three stages for the submission processing pipeline. They are:

  • Validation: during this stage we run through each of the ActEV CLI commands to install your system, run the system on the validation set, compare the produced output to the validation set, and finally, take a snapshot of your system to re-use during execution.
  • Execution: during this stage, we use the snapshot to process the sequestered data. Presently, we can divide the sequestered data into 1-hour sub-parts of the data set or process the whole dataset. Each sub-part is nominally 12, 5-minute files. We are only processing MEVA data through your system. Presently, we have a runtime limit per part and if a part fails to be processed, we retry it once.
  • Scoring: After all the parts have been processed, the outputs are merged, and scored.

Datasets

Framework

The DIVA Framework is a software framework designed to provide an architecture and a set of software modules which will facilitate the development of activity recognition analytics. The Framework is developed as a fully open source project on GitHub. The following links will help you get started with the framework: The DIVA Framework is based on KWIVER, an open source framework designed for building complex computer vision systems. The following links will help you learn more about KWIVER:
  • KWIVER Github Repository This is the main KWIVER site, all development of the framework happens here.
  • KWIVER Issue Tracker Submit any bug reports or feature requests for the KWIVER here. If there's any question about whether your issues belongs in the KWIVER or DIVA framework issues tracker, submit to the DIVA tracker and we'll sort it out..
  • KWIVER Main Documentation Page The source for the KWIVER documentation is maintained in the Github repository using Sphinx. A built version is maintained on ReadTheDocs at this link. A good place to get started in the documentation, after reading the Introduction are the Arrows and Sprokit sections, both of which are used by the KWIVER framework.
The framework based R-C3D baseline algorithm implementation with CLI; see for details.

Baseline Algorithms

KITWARE has adapted two "baseline" activity recognition algorithms to work within the DIVA Framework:

Visualization Tools

Annotation Tools

Contact Us

For ActEV Evaluation information (data, evaluation code, etc.) please email: actev-nist@nist.gov

For ActEV Evaluation Discussion Please Visit our Google Group.