Development Plan

Last Update Dec 19, 2018

The primary purpose of this target release is to stabilize and complete development of NWB:N 2.0beta. This release will also include a number of changes to the format and APIs that have not made it into the beta release. Requests for new features for this release has been frozen. Additional new features may be included in future NWB:N 2.x releases.

Timeline

• NWB:N 2.0:
  • Jan. 18, 2019: Full release of NWB:N 2.O (final) and PyNWB and MatNWB
  • Dec. 23, 2018: NWB:N 2.0 schema finalized
  • SfN 2018 (~Nov.5 2018) complete all changes that break compatibility with previous versions of NWB:N.
• Other:
  • May, 2019: Rename and release pynwb.form as a separate library. Tentative library name Hiearchical Data Modeling Framework (HDMF)

ToDo for the release

• PyNWB
  • Close (or as approbritate update) open issues for the NWB:N 2.0 Full milestone. One critical issues is #715, #747 to fix the validator.
• MatNWB
  • Sync MatNWB to the latest schema
• nwb-docutils
  • Make PyPi release of the latest verison
• Documentation:
  • Test that the schema docs build and are organized correctly on ReadTheDocs
  • Update builds on ReadTheDocs for all docs
  • Change links on https://neurodatawithoutborders.github.io/ to point to the stable rather than the latest build of the docs.
• nwb-schema:
  • Sync schema with PyNWB
  • Finalize release notes
  • Close (or as approbritate update) open issues for the NWB:N 2.0 Full milestone
• Publication:
  • Post manuscript describing the advances in the NWB:N 2.0 schema online
  • Post news release with announcement of the release

Last Update Dec 19, 2018

The primary purpose of this target release is to stabilize and complete development of NWB:N 2.0beta. This release will also include a number of changes to the format and APIs that have not made it into the beta release. Requests for new features for this release has been frozen. Additional new features may be included in future NWB:N 2.x releases.

Timeline

SfN 2018 (~Nov.5 2018) complete all changes that break compatibility with previous versions of NWB:N.

Open issues for this release

• PyNWB – NWB:N 2.0 issues
• NWB:N Schema – NWB:N 2.0 issues
• MatNWB – All Open Issues

PyNWB: Python Reference API

• PyNWB Interfaces
  • Adapt to schema changes described below
  • Expand and fix bugs in PyNWB interfaces #512, #45, #148, #306, #327, #350, #412, #433, #445, #447, #459
  • I/O bug fixes and enhancements #32, #117, #146
  • Expand legacy read support to more sets of Allen data #275, #392, [#384][pi385]
• Extensions support
  • Support saving and using schema cached in NWB:N HDF5 files as part of the /general/specifications #497, #522,
  • Test and fix bugs in autogenerated extension classes #520, #48, #289
  • Expand online validation of new specs during creation #51,pr#525
• Testing, Continuous Integration, Deployment:
  • Add integration test cases to ensure broad coverage of integration tests #112, #406
  • Add further unit tests to ensure broad coverage
  • Expand/ensure proper function of CI #152, #270, #274, #277, #399, #444, #446, #461
  • Add tests for CLI tools #208
  • Define application test cases, i.e., sets of tests that implement select application test cases (e.g, convert of lab data to NWB:N).
• Documentation:
  • Add more tutorials and examples #505,#206, #268, #309
  • Fix bugs and clarify the documentation #517, #514, #263,#291, #324, #335, #433, #488
  • Create central resource for NWB:N documents (see https://neurodatawithoutborders.github.io/) :white_check_mark:
• Branding and Templates #281, #425

NWB:N Format Schema

• Reorganize /general to ease adding and managing custom, lab-specific metadata via extensions (#19)
• Restructure the storage of Epochs data (see #119
• Restructure the storage of ROI-related data (see #118, #124, #60 )
• Improve/add support for trial data (see #152)
• Add support for dynamic data tables
• Add support for efficient storage of vector data
• Restructure unit data (consolidate UnitTimes, ClusterWaveforms, and Clustering)
• Use region and object reference for explicit referencing of data when possible
• Make processing interfaces (Clustering, FeatureExtraction) that describe a time series inherit from TimeSeries (see #112)
• Fix bugs and clarify fields (see #12, #59, #136, #139, #143, #4, #7, #53, #54 )
• Various minor restructuring/clarification of metadata (see #145, #148, )
• Complete release notes describing the changes to the format, specification language etc. (see #161, #133, #122, #115, #102, #6, #28, #39, )

NWB:N Specification Language

• No new features are planned for the specification language itself beyond what was changed/added in NWB:N 2.0 beta. :white_check_mark:

NWB:N Documentation Utilties

• Move documentation tools from nwb-schema to nwb-docutils) to make them easier to access and reuse. :white_check_mark:
• Add tests for documentation tools

MatNWB: Matlab API

• Ensure complete coverage and support for read and write of NWB:N HDF5 files
• Ensure consistency between files generated by the Matlab and Python APIs
• Add support for compound data types
• Add support for the above-described changes to the specification language and schema
• Add support for lazy data load
• Add further documentation

This is a first public beta release of the new version NWB:N. The intent of this beta release is to enable early adopters to start exploring the new format and software. As development is ongoing, there may still be bugs and further changes to the APIs as well as the format are still planed between this beta and the first full release of NWB:N2.0. As such backward compatibility cannot be promised at this point yet.

New features for this release:

• PyNWB: A new advanced python API for the new version of NWB:N 2.0. PyNWB defines a new modular software architecture and API to enable users/developers to efficiently interact with the NWB:N data format, format files, and specifications. The new software architecture decouples the various aspects of NWB:N (i.e. the specification language, format specification, storage, and API) to allow each to be used and maintained more independently. PyNWB also provides and API for interacting with NWB:N format specifications and provides mechanisms to easily create and use format extensions.
  • Available online: https://github.com/NeurodataWithoutBorders/pynwb
  • Documentation: PyNWB : Python Reference API Docs
  • Supported Languages :
    • Python >2.7.x
    • Python >3.6.x
  • Core features:
    • Write NWB:N HDF5 files
    • Read NWB:N files (base round-trip read/write)
    • Validate NWB:N files
    • Create new format extensions
    • Read/Write data using extensions
    • Read legacy NWB:N1.0x files from the Allen Institute for Brain Science
  • Main contributors for this release :
    • Andrew Tritt, Oliver Ruebel, David Camp, Kris Bouchard (LBNL) : Lead architects and development team
    • Nicholas Cain (Allen Institute for Brain Science) and Andrew Tritt (LBNL) : Port to Python 2 and legacy read for Allen data
    • Bug fixes, feedback and suggestions have also been contributed by numerous users and developers from the NWB:N community. For details see https://github.com/NeurodataWithoutBorders/pynwb/graphs/contributors
• PyNWB testing, continuous integration, deployment We have developed an extensive set of unit tests for PyNWB. We have also created base infrastructure for integration tests to test end-to-end read/write of NWB:N files and data objects. While coverage of the tests is already quite good (>80%) coverage of the integration test suite is expected to be incomplete. We have also setup infrastructure for continuous integration to execute all tests on a broad range of platforms as well as defined software deployment paths (e.g., via PIP, CONDA)
  • Available online: For an overview of the various software health checks and tests being perfomed regulalary on PyNWB, see https://github.com/NeurodataWithoutBorders/pynwb
  • Continuous integration :
    • CircleCI (Linux, py27, py36)
    • Travis (MacOS, py27, py36)
    • AppVeyor (Windows, py27, py36)
    • Codecov.io
    • PEP8Speaks
  • Deployment:
    • PIP (PyPI)
    • Conda (CondaCloud)
  • Main contributors for this release
    • Andrew Tritt (LBNL) : Lead for design and implementation of the Python test suites
    • Jean-Christophe Fillion-Robin, Doruk Ozturk, Chris Kotfila, Michael Grauer, Will Schroeder (Kitware) : Lead development team for creation of continuous integration, testing, and deployment mechanisms. For details see https://github.com/NeurodataWithoutBorders/pynwb/graphs/contributors
• Specification Language We have simplified and extended the specification language used to describe the NWB:N format to ease readability, interpretability and expressiveness as well as added support for compound data types and references among others.
  • Documentation: Specification Language Docs. The documentation also includes the release notes describing the changes in NWB:N 2 beta.
  • Main contributors for this release :
    • Andrew Tritt, Oliver Ruebel, Kris Bouchard (LBNL) : Development leads
    • Changes to the specification language have also been reviewed and new changes have been proposed at a community hackathon at Janelia Farms HHMI in August 2017. Many users have also contributed via GitHub issues.
• Data Standard Specification Changes to the format have focused mainly on improving the structure and usability of NWB:N rather than adding new features to the format itself. Updates to the NWB:N format include among others: 1) extensions to clarify and extend the concept of NWBContainer (previously Interface), 2) avoid implicit links, 3) improve consistency and ease-of-use, 4) refactor of /general/extracellular_ephys and ElectrodeGroup to use data tables via compound data types.
  • Available online: nwb-schema is available on GitHub at https://github.com/NeurodataWithoutBorders/nwb-schema
  • Documentation: Data Standard Schema Docs.The documentation also includes the release notes describing the changes in NWB:N 2 beta.
  • Main contributors for this release :
    • Andrew Tritt, Oliver Ruebel (LBNL) : Implementation and documentation leads
    • Changes to the format specification have also been reviewed and new changes have been proposed at a community hackathon at Janelia Farms HHMI in August 2017. Many users have also contributed via GitHub issues.
• Documentation We have created dedicated online documentations for the various aspects of NWB:N. While the documents are quite extensive, completeness and consistency of the documents is not guaranteed for the beta release and examples, tutorials and release notes (i.e., changes), may be a bit out of date given pace of development.
  • Available online:
    • Project Overview
    • PyNWB : Python Reference API
    • Data Standard Schema
    • Specification Language
    • Data Storage Specification: HDF5
  • Format Documentation Tools To ensure consistency between the NWB:N format specification and documentation we have developed a set of tools that generate Sphinx RST documents from the YAML specification sources. The tools are available in Python 2/3 as part of the NWB:N Documentation Utilities.
  • Main contributors for this release :
    • Oliver Ruebel and Andrew Tritt (LBNL) : Documentation leads
    • Several of the documents have been ported from NWB:N 1.x, which were originally created by Jeff Teeters (UCB) et al.
    • Other teams have also contributed bug fixes
• MatNWB MatNWB is a Matlab API for NWB:N. MatNWB generates Matlab classes for representing NWB:N neurodata_type directly from the YAML specification.
  • Core features:
    • Write NWB:N HDF5 files
    • Read NWB:N HDF5 files
    • Read/Write data for custom extensions
  • Main contributors for this release:
    • Nathan Clack and Lawrence Niu (Vidriotech) : Software design and development lead
    • Karel Svoboda (Janelia Farms HHMI): Project Lead
    • Andrew Tritt and Oliver Ruebel : Review and discussion
    • See also https://github.com/NeurodataWithoutBorders/matnwb/graphs/contributors