Activity Extended Video (ActEV) challenge

ActEV'21 Sequestered Data Leaderboard (SDL) Evaluation

Updates

Sep 21, 2020: ActEV'21 SDL UF with Known Activities opens
Sep 21, 2020: ActEV'21 SDL UF is a competition under WACV'21 HADCV workshop
Sep 25, 2020: ActEV'21 SDL KF with Known Activities opens
Nov 02, 2020: ActEV'21 SDL UF with Surprise Activities are available for NIST-internal debugging
Nov 20, 2020: ActEV'21 SDL UF with Surprise Activities are available for automation debugging
Dec 04, 2020: ActEV'21 SDL UF with Surprise Activities opens

Summary

The Activities in Extended Video (ActEV) series of evaluations is designed to accelerate development of robust, multi-camera, automatic activity detection systems in known/unknown facilities for forensic and real-time alerting applications.

The ActEV 2021 Sequestered Data Leaderboard (ActEV’21 SDL) includes two separate evaluations of real-time or faster activity detection systems:

A Known Facility (KF) evaluation with Electro-Optical (EO)/Infrared (IR) modalities
An Unknown Facility (UF) evaluation with Known/Surprise activities

Systems submitted to the SDL are expected to process video in real-time or faster. Scores for systems operating slower than real-time will be computed from the output generated before the real-time clock expires per the Metrics section below.

What

Participants are invited to submit their runnable activity detection software using an ActEV Command Line Interface (CLI) submission. There are two leaderboards (KF and UF) with sub-leaderboards as follows:

UF: Unknown Facilities Leaderboards
- UF Known Activities on EO
- UF Surprise Activities on EO
KF: Known Facilities Leaderboards
- KF Known Activities on EO
- KF Known Activities on IR

Participants may submit systems that address only a single sub-leaderboard task as described in the evaluation plan below. NIST will evaluate system performance on sequestered data using NIST hardware and results will be posted to a public leaderboard.

Who

Everyone. Anyone who registers can submit to the evaluation server.

How

Systems must follow a NIST-defined CLI and automatically run on NIST’s servers, both of which are described in the instructions.

Evaluation Task

In the ActEV'21 SDL evaluation, there is one Activity Detection (AD) task which is defined as: given a target activity, a system automatically detects and temporally localizes all instances of the activity. For a system-identified activity instance to be evaluated as correct, the type of activity must be correct, and the temporal overlap must fall within a minimal requirement.

Data

The ActEV'21 SDL evaluation is based on the KF and UF datasets. The KF and UF data were collected and annotated for the Intelligence Advanced Research Projects Activity (IARPA) Deep Intermodal Video Analytics (DIVA) program.

You can download the public MEVA dataset for free at mevadata.org. We also provide annotations for 27 hours of MEVA data, and instructions on how to make and share activity annotations are at mevadata.org.

Metrics

Submitted systems are measured by Probability of Missed Detection (Pmiss) and Time-based False Alarm (TFA). Pmiss is the portion of activities where the system did not detect the activity for at least 1 second. TFA is the portion of time that the system detected an activity when in fact there was none. Submitted systems are scored for Pmiss and TFA at multiple thresholds (based on confidence scores produced by the systems), creating a detection error tradeoff (DET).

The ActEV ’21 SDL Leaderboard will report scores on how systems would perform in real-time. If an SDL system takes longer than real-time to process test videos, NIST will score the system as if execution stopped where the runtime exceeded real time. The leaderboard ranking of a system is based on the scores as follows:

Time limited partial AUDC for the public leaderboard ranking
Time limited mean Pmiss at TFA = 0.02 for IARPA DIVA program ranking

Please see details in the ActEV'21 SDL evaluation plan below or check out the ActEV Scoring Software GitHub repo.

Evaluation Plan

ActEV'21 SDL Evaluation Plan

Contact

For ActEV Evaluation information, please email: actev-nist@nist.gov

For ActEV Evaluation Discussion, please visit our Google Group.

News

21Sep

ActEV'21 SDL UF (with KA) starts

25Sep

ActEV'21 SDL KF starts

27Nov

Deadline to submit system for WACV'21 HADCV workshop

04Dec

ActEV'21 SDL UF (with SA) starts

ActEV'21 SDL Evaluation Schedule

Sep 21, 2020: ActEV'21 SDL UF (with Known Activities) leaderboard opens

Sep 25, 2020: ActEV'21 SDL KF with EO (Electro-Optical) and IR (Infrared) leaderboards open

Nov 27, 2020 : Deadline for UF CLI submissions to be included in WACV'21 HADCV workshop.

Dec 20, 2020 : Top two teams on the ActEV'21 SDL UF (with Known Activities) leaderboard invited to give oral presentations at the WACV'21 HADCV workshop based on the CLI submission deadline (Nov 27, 2020).

Jan 05, 2021: WACV'21 HADCV workshop

Leaderboard Remains Open: Leaderboard will continue to remain open to allow participants to show continued progress on the challenging problem.

ActEV SDL Dataset

The ActEV SDL evaluation is based on the Known Facility (KF) and Unknown Facility (UF) datasets. The KF data was collected at the Muscatatuck Urban Training Center (MUTC) with a team of over 100 actors performing in various scenarios. The KF dataset has two parts: (1) the public training and development data and (2) sequestered evaluation data used by NIST to test systems. The UF data is a sequestered evaluation data which was collected at different places and includes both known and surprise activities.

The KF and UF data were collected and annotated for the Intelligence Advanced Research Projects Activity (IARPA) Deep Intermodal Video Analytics (DIVA) program. A primary goal of the DIVA program is to support activity detection in multi-camera environments for both DIVA performers and the broader research community.

Training and Development Data

The public KF dataset has been released as the Multiview Extended Video with Activities (MEVA) dataset. As of December 2019, 328 hours of ground-camera data and 4.2 hours of Unmanned Arial Vehicle video have been released. 28 hours of the ground camera video have been annotated by the same team that has annotated the ActEV test set. Additional annotations have been performed by the public and are also available in the annotation repository.

Sequestered Data

The KF test set is a 140-hour collection of videos which consists of both EO and IR camera modalities, public cameras (video from the cameras and associated metadata are in the public training set) and non-public cameras (video is not provided on metadata.org and camera parameters are only provided to the systems at test time). The KF leaderboard presents results on the full 140-hour collection reporting separately for EO and IR data. Developers receive additional scores by activity for the EO_Set1 and the IR_Set1. The EO-Set1 and IR-Set1 are subsets of the entire test sets. For example, EO-Set1 is a random 50% of the EO data from public cameras and likewise for IR-Set1.

The UF test set has a large collection of videos exclusively in the EO spectrum (as of September 9, 2020). The UF leaderboard presents results separately for known activity and surprise activity types.

Data Download

You can download the public MEVA video and annotations dataset for free from the mevadata.org website (http://mevadata.org/)

Then complete these steps:

Get an up-to-date copy of the ActEV Data Repo via GIT. You'll need to either clone the repo (the first time you access it) or update a previously downloaded repo with 'git pull'.
- Clone: git clone https://gitlab.kitware.com/actev/actev-data-repo.git
- Update: cd "Your_Directory_For_actev-data-repo"; git pull
Get an up-to-date copy of the MEVA Data Repo via GIT. You'll need to either clone the repo (the first time you access it) or update a previously downloaded repo with 'git pull'.
- Clone: git clone https://gitlab.kitware.com/meva/meva-data-repo
- Update: cd "Your_Directory_For_meva-data-repo"; git pull

ActEV'21 SDL Leaderboard

Update Dec. 4, 2020: Reported scores for the UF Surprise Activity systems are preliminary scores. NIST is continuing to review the scoring protocol and reported Surprise Activity Scores.