ActEV Sequestered Data Leaderboard

System delivery to the leaderboard must be in a form compatible with the ActEV Command Line Interface (ActEV CLI) and submitted to NIST for testing. The command line interface implementation that you will provide formalizes the entire process of evaluating a system, by providing the evaluation team a means to: (1) download and install your software via a single URL, (2) verify that the delivery works AND produces output that is “consistent” with output you produce, and (3) process a large collection of video in a fault-tolerant, parallelizable manner.

To complete this task you will need the following items described in detail below:

1. FAQ - Validation Phase Processing
2. CLI Description
3. The Abstract CLI Git Repository
4. The CLI Implementation Primer
5. The Validation Data Set
6. Example CLI-Compliant Implementation
7. NIST Hardware and Initial Operating System Description

FAQ - Validation Phase Processing

CLI Description

The Abstract CLI GIT Repository

The CLI Implementation Primer

The Validation Data Set

Example CLI-Compliant Implementation

• RC3D baseline system with CLI implementation by Alexandre Boyer

NIST Independent Evaluation Infrastructure Specification

NIST will begin installing your system from a fresh Ubuntu 18.04 cloud image available from https://cloud-images.ubuntu.com/releases/18.04/release/ on the following hardware.

• Chassis: Asus ESC4000 G4S
• CPU: 2 x Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz (12 cores/CPU)
• Motherboard: Asus Intel® C621 PCH chipset
• HDD/SSD: 2x 1.92GB Intel SSD DC S4500
• RAM: 12x 16GB DDR4-2400 ECC RDIMM
• GPU: Four PNY RTX2080Ti blower style
• OS: Ubuntu 18.04
• Storage Volume- 1TB (variable)
• Supplied object store (read only) for source video

For ActEV Evaluation information (data, evaluation code, etc.) please email: actev-nist@nist.gov

For ActEV Evaluation Discussion Please Visit our Google Group. https://groups.google.com/a/list.nist.gov/forum/#!forum/trecvid.actev

The list below specifies the activities to be detected for the ActEV SDL evaluation. Detailed activity definitions are in the ActEV Annotation Definitions for MEVA Data document. Note that current scoring for the SDL is based exclusively on activity detection, and object detection is not considered.

Datasets

• ActEV-supported data sets
  • Multiview Extended Video with Activities (MEVA)
  • VIRAT
  • Data access from mevadata.org: Accessing and using MEVA and MEVA Download Instructions (This includes, 4.6 hr UAV video collected, 3D model of the facility)
  • Data access : See actev-data-repo. Access credentials provided during signup
• Kinetics
• AVA
• Moments-in-Time
• ActivityNet
• NVIDIA's CityFlow dataset

Framework

• DIVA Framework Github Repository This is the main DIVA Framework site, all development of the framework happens here.
• We have also used the framework to implement submissions to the SDL (in this case, the R-C3D baseline algorithm); see for details.
• DIVA Framework Issue Tracker Submit any bug reports or feature requests for the framework here.
• DIVA Framework Main Documentation PageThe source for the framework documentation is maintained in the Github repository using Sphinx. A built version is maintained on ReadTheDocs at this link. A good place to get started in the documentation, after reading the Introduction is the UseCase section which will walk you though a number of typical use cases with the framework.

• KWIVER Github Repository This is the main KWIVER site, all development of the framework happens here.
• KWIVER Issue Tracker Submit any bug reports or feature requests for the KWIVER here. If there's any question about whether your issues belongs in the KWIVER or DIVA framework issues tracker, submit to the DIVA tracker and we'll sort it out..
• KWIVER Main Documentation PageThe source for the KWIVER documentation is maintained in the Github repository using Sphinx. A built version is maintained on ReadTheDocs at this link. A good place to get started in the documentation, after reading the Introduction are the Arrows and Sprokit sections, both of which are used by the KWIVER framework.

Baseline Algorithms

• R-C3D
• ACT

Annotation Tools

• Kitware annotation tool (the tool natively supports the DIVA format)
• The VGG Image Annotator
• Scalabel (used for annotation of Berkeley DeepDrive project)
• VATIC - Video Annotation Tool
• BeaverDam
• VoTT: Visual Object Tagging Tool
• Computer Vision Annotation Tool (CVAT)
• Efficient Annotation of Segmentation Datasets with Polygon-RNN++