---

Nov 27, 2020 : Deadline for UF CLI submissions to be included in WACV'21 HADCV workshop.

The ActEV SDL evaluation is based on the Known Facility (KF) and Unknown Facility (UF) datasets. The KF data was collected at the Muscatatuck Urban Training Center (MUTC) with a team of over 100 actors performing in various scenarios. The KF dataset has two parts: (1) the public training and development data and (2) sequestered evaluation data used by NIST to test systems. The UF data is a sequestered evaluation data which was collected at different places and includes both known and surprise activities.

The KF and UF data were collected and annotated for the Intelligence Advanced Research Projects Activity (IARPA) Deep Intermodal Video Analytics (DIVA) program. A primary goal of the DIVA program is to support activity detection in multi-camera environments for both DIVA performers and the broader research community.

The public KF dataset has been released as the Multiview Extended Video with Activities (MEVA) dataset. As of December 2019, 328 hours of ground-camera data and 4.2 hours of Unmanned Arial Vehicle video have been released. 28 hours of the ground camera video have been annotated by the same team that has annotated the ActEV test set. Additional annotations have been performed by the public and are also available in the annotation repository.

The KF test set is a 140-hour collection of videos which consists of both EO and IR camera modalities, public cameras (video from the cameras and associated metadata are in the public training set) and non-public cameras (video is not provided on metadata.org and camera parameters are only provided to the systems at test time). The KF leaderboard presents results on the full 140-hour collection reporting separately for EO and IR data. Developers receive additional scores by activity for the EO_Set1 and the IR_Set1. The EO-Set1 and IR-Set1 are subsets of the entire test sets. For example, EO-Set1 is a random 50% of the EO data from public cameras and likewise for IR-Set1.

The UF test set has a large collection of videos exclusively in the EO spectrum (as of September 9, 2020). The UF leaderboard presents results separately for known activity and surprise activity types.

You can download the public MEVA video and annotations dataset for free from the mevadata.org website (http://mevadata.org/)

Then complete these steps:

• Get an up-to-date copy of the ActEV Data Repo via GIT. You'll need to either clone the repo (the first time you access it) or update a previously downloaded repo with 'git pull'.
  • Clone: git clone https://gitlab.kitware.com/actev/actev-data-repo.git
  • Update: cd "Your_Directory_For_actev-data-repo"; git pull
• Get an up-to-date copy of the MEVA Data Repo via GIT. You'll need to either clone the repo (the first time you access it) or update a previously downloaded repo with 'git pull'.
  • Clone: git clone https://gitlab.kitware.com/meva/meva-data-repo
  • Update: cd "Your_Directory_For_meva-data-repo"; git pull

Update June 3, 2020: Reported scores for all submissions on the ActEV2020 leaderboards were revised using the ActEV_SDL_V2 scoring protocol which slightly decreased all error rates.

In the ActEV'21 SDL evaluation, there is one Activity Detection (AD) task for detecting and localizing activities; given a target activity, a system automatically detects and temporally localizes all instances of the activity. For a system-identified activity instance to be evaluated as correct, the type of activity must be correct, and the temporal overlap must fall within a minimal requirement. The minimum temporal overlap with a single reference instance in the evaluation is 1 second. If the reference instance duration is less than 1 second, 50% of the reference duration is required as the minimum temporal overlap. The AD task applies to both Known facilities (KF) and Unknown facilities (UF) as well as both known and surprise activities.

Systems will be tested on detecting Known Activities in both Electro-Optical (EO) and Infrared (IR) camera modalities, where each modality is scored individually on separate sub-leaderboards. Both the facility and the activities are known to the developers, Facility data is available at https://mevadata.org including a site map with approximate camera locations and sample FOVs, camera models, a 3D site model, and additional metadata and site information. Sample representative video from the known facility is also provided, with over 20 hours of video annotated for leaderboard activities. All available metadata and site information may be used during system development.

Systems will be tested on both Known and Surprise activities in EO video. Unlike the KF testing, no facility information will be provided to developers. A subset of the facility-defining metadata and the Surprise Activity definitions (textual definitions and at least one chip video exemplars with localization annotations) will be provided at test time. Thus, systems must be able to build detectors on the fly. The Known Activities are the same those used for KF testing. Systems will be provided a maximum of 10 hours to train SA detectors while executing on the NIST hardware.

For the Known Activities (KA) tests, developers are provided a list of activities in advance for use during system development (e.g., training) for the system to automatically detect and localize all instances of the activities.

Detailed activity definitions are in the ActEV Annotation Definitions for MEVA Data document.

For the Surprise Activities (SA) tests, the pre-built system is provided a set of activities with training materials (text description and at least one exemplar video chip) during system test time (in execution) to automatically develop detection and localization models. Then the system must automatically detect and localize all instances of the activities. To facilitate activity training at test time, systems will be provided a maximum of 10 hours to train for SAs while executing on the NIST hardware.

Please click Surprise Activities to see how to make SDL submission for SA-capable algorithms.

System delivery to the leaderboard must be in a form compatible with the ActEV Command Line Interface (ActEV CLI) and submitted to NIST for testing. The command line interface implementation that you will provide formalizes the entire process of evaluating a system, by providing the evaluation team a means to: (1) download and install your software via a single URL, (2) verify that the delivery works AND produces output that is “consistent” with output you produce, and (3) process a large collection of video in a fault-tolerant, parallelizable manner.

To complete this task you will need the following items described in detail below:

1. FAQ - Validation Phase Processing
2. CLI Description
3. The Abstract CLI Git Repository
4. The CLI Implementation Primer
5. The Validation Data Set
6. Example CLI-Compliant Implementation
7. NIST Hardware and Initial Operating System Description
8. SDL Submission Processing Pipeline

1. FAQ - Validation Phase Processing

The ActEV SDL - Validation Phase Processing

FAQ

2. CLI Description

The ActEV CLI description

3. The Abstract CLI GIT Repository

The Abstract CLI GIT repository. The repo contains documentation.

4. The CLI Implementation Primer

There are 6 steps to adapt your code to the CLI. The ActEV Evaluation CLI Programming Primer describes the steps to clone the Abstract CLI and begin adapting the code to your implementation.

5. The Validation Data Set

As mentioned above, the CLI is used to verify the downloaded software is correctly installed and produces the same output at NIST as you produce locally. In order to do so, we have provided a small validation data set (ActEV-Eval-CLI-Validation-Set3) as part of the ActEV SDL Dataset that will be processed both at your site and at NIST. Please use this data in Step 3 of the Primer described above.

6. Example CLI-Compliant Implementation

The link below provides CLI implementations for the leaderboard baseline algorithm for Known Activities only:

• Framework based RC3D baseline system with CLI implementation see for details.

7. NIST Independent Evaluation Infrastructure Specification

Hardware specifications for ActEV’21 SDL evaluations are as follows.

1080 Hardware for Known Facility Evaluation:

• 32 VCPU 2.2Ghz (16 hyperthreaded)
• 128GB memory
• Root Disk: 40GB
• Ephemeral Disk: 256GB
• GPU: 4x - 1080Ti
• OS: Ubuntu 18.04

2080 Hardware for Unknown Facility Evaluation:

• 32 VCPU 2.2Ghz (16 hyperthreaded)
• 128GB memory
• Root Disk: 128GB
• Ephemeral Disk: 256GB
• GPU: 4x - 2080Ti
• OS: Ubuntu 18.04

8. SDL Submission Processing Pipeline

There are three stages for the submission processing pipeline. They are:

• Validation: during this stage NIST runs through each of the ActEV CLI commands to install your system, runs the system on the validation set, compares the produced output to the validation set, and finally, takes a snapshot of your system to re-use during execution.
• Execution: during this stage, we use the snapshot to process the sequestered data. Presently, we can divide the sequestered data into 2-hour sub-parts of the data set or process the whole dataset. Each sub-part is nominally 24, 5-minute files. We are only processing ActEV'21 SDL data through your system. Presently, we have a time limit per part and if a part fails to be processed, we retry it once.
• Scoring: After all the parts have been processed, the outputs are merged, and scored.

---

The content below is a jumping off point to begin developing an ActEV CLI (Command Line Interface) compliant system capable of training on Surprise Activities (SA) and being able to test the system’s capabilities for the surprise activities detection on sequestered data. This document describes the steps to obtain ActEV-provided data resources and information sources to make sure you have all the critical information.

• Step 1: Obtain the ActEV-provided data resources needed to develop an SA-capable system by updating to actev-data-repo. The commands to update the repo are as follows and these steps assume you have already completed the setup steps in the top-level "README.md":
  • cd actev-data-repo
  • git pull
  • python3 scripts/actev-corpora-maint.py --operation download --corpus MEVA


• Step 2: Review the ActEV-provided data resources. There are three directories within actev-data-repo relevant to SA:
  • partitions/ActEV-Eval-CLI-Validation-Set7: This is a new validation data set for CLI testing and must be used for Surprise Activity Enabled systems. This is a larger data set (5 hours) than other validation sets so that SA can be reasonably tested both during development and during validation on SDL.
  • partitions/MEVA-Public-SimUF-KA-SimSA5-20201023: This is a revision of the MEVA-Public-20200288 data partion to support internal development of a Surprise Activity-enabled system. The data partition uses the MEVA Public data found on mevadata.org and the meva-data-repo. Consult the README.md for details.
  • corpora/MEVA/SimSA: This directory contains derived data sources for the Surprise Activities. This directory is the source for the two partitions above and demonstrates how Kitware-produced MEVA annotations are transformed to be simulated Surprise Activities.


• Step 3: Familiarize yourself with the Surprise Activity (SA) evaluation task by reviewing the evaluation plan in particular Sections 1 (Overview) and 4.3 (Activity Definitions and Annotations). Also note that SA-capable systems will be simultaneously evaluated both surprise and known activities.


• Step 4: Familiarize yourself with activity definition materials that the system will process to build detection models. For each activity, a system will be provided two components; a textual description and a set of exemplar instances. The textual description includes a natural language description of the activity, common examples, and rules to determine the start and end of an activity instance. An exemplar instance consists of a video-chip clip and a KWIVER Packet Format (KPF) annotation file outlining the people AND objects involved in the activity instance. The ActEV-provided data resources include “Simulated Surprise Activities” (SimSA) to help developers understand the data resources. The table below shows an example of the SimSA “SimSA5_person_closes_trunk” that was derived from the MEVA known activity “person_closes_trunk”.
  • Text Description
  • Exemplar 001: video chip clip for a single instance


• Step 5: Familiarize yourself with the ActEV Evaluation JSON files that are used as input to the system by reading the ActEV Evaluation JSON Formats Document found in (https://gitlab.kitware.com/meva/meva-data-repo/-/tree/master/documents/nist-json-for-actev). In particular, review the structure and content of the “activity-index” JSON found in Section 1.2 (Activity Index). The activity index enumerates all the activity-defining materials that are provided to a CLI-compliant system during activity training. For example, see actev-data-repo/partitions/ActEV..Set7/activity-index.json.


• Step 6: Familiarize yourself with the modifications to the ActEV Abstract CLI to support SA found in Algorithm Submission. In particular, note the new ‘actev-train-system’ command that takes as input an activity-index.json and a directory of video. The command is responsible for building the models to detect the activities and storing the models in the CLI directory for subsequent uses. When the actev-train-system command is called to train/configure the system to detect the surprise activities, the step must run within 10 hours on the NIST SDL server.


• Step 7: Develop a CLI-compliant system for submission
  • Follow the procedure to update the CLI to the latest version as described in the ActEV Evaluation CLI Programming Primer.
  • Implement the ‘actev-train-system’ command. Keep in mind, models need to be accessible for subsequent CLI calls and models produced during training should be written to the same/similar location as existing models.
  • Use the required validation set “ActEV-Eval-CLI-Validation-Set7” and include both Known Activity and Surprise Activity detections.
  • Make your submission to the SDL.

Datasets

• ActEV-supported data sets
  • Multiview Extended Video with Activities (MEVA)
  • VIRAT
  • Data access from mevadata.org: Accessing and using MEVA and MEVA Download Instructions (This includes, 4.6 hr UAV video collected, 3D model of the facility)
  • Data access : See actev-data-repo. Access credentials provided during signup
• Kinetics
• AVA
• Moments-in-Time
• ActivityNet
• NVIDIA's CityFlow dataset
• Live Datasets for Visual AI (Visym Collector)
  • People in Public - 175k: 184,379 video clips of the ActEV activity classes for training in the unknown facility use-case.

Framework

The DIVA Framework is a software framework designed to provide an architecture and a set of software modules which will facilitate the development of activity recognition analytics. The Framework is developed as a fully open source project on GitHub. The following links will help you get started with the framework:

• DIVA Framework Github Repository This is the main DIVA Framework site, all development of the framework happens here.
• DIVA Framework Issue Tracker Submit any bug reports or feature requests for the framework here.
• DIVA Framework Main Documentation PageThe source code for the framework documentation is maintained in the Github repository using Sphinx. A built version is maintained on ReadTheDocs at this link. A good place to get started in the documentation, after reading the Introduction is the UseCase section which will walk you though a number of typical use cases with the framework.

The DIVA Framework is based on KWIVER, an open source framework designed for building complex computer vision systems. The following links will help you learn more about KWIVER:

• KWIVER Github Repository This is the main KWIVER site, all development of the framework happens here.
• KWIVER Issue Tracker Submit any bug reports or feature requests for the KWIVER here. If there's any question about whether your issues belongs in the KWIVER or DIVA framework issues tracker, submit to the DIVA tracker and we'll sort it out..
• KWIVER Main Documentation Page The source for the KWIVER documentation is maintained in the Github repository using Sphinx. A built version is maintained on ReadTheDocs at this link. A good place to get started in the documentation, after reading the Introduction are the Arrows and Sprokit sections, both of which are used by the KWIVER framework.

The framework based R-C3D baseline algorithm implementation with CLI found in the link.

Baseline Algorithms

KITWARE has adapted two "baseline" activity recognition algorithms to work within the DIVA Framework:

• R-C3D
• ACT

Visualization Tools

• Visym Python Tools for Computer Vision and Machine Learning

Annotation Tools

• Kitware annotation tool (the tool natively supports the DIVA format)
• The VGG Image Annotator
• Scalabel (used for annotation of Berkeley DeepDrive project)
• VATIC - Video Annotation Tool
• BeaverDam
• VoTT: Visual Object Tagging Tool
• Computer Vision Annotation Tool (CVAT)
• Efficient Annotation of Segmentation Datasets with Polygon-RNN++