skip to Main Content

The development plan is work in progress.

NWB 2.x refers to the next major feature release of NWB beyond the main release of NWB:N 2. On GitHub we are using the milestones

  1. NWB 2.x to organize issues we are targeting for upcoming releases as part of 2.x
  2. Future to organize issues targets for future releases that are beyond the scope of our current development plan.

In the table below, we summarize our upcoming, high-level milestones:

Community Outreach and Engagement 

Our community outreach and engagement activities will continue to be a major focus to enhance adoption and dissemination of NWB. In the following we focus on known key events for user/developer training and engagement. More events may be added.

Target  Milestone and Deliverable
Jan. 2020 International Brain Initiative (IBI) meeting, Tokyo, Japan
Feb. 3-6, 2020 Developer hackathon at the Allen Institute for Brain Science aimed at increasing support for ontologies, intracellular electrophysiology, and data management and visualization tools with NWB (also part of Aim 1)
Feb. 27, 2020 NWB Tutorial at Cosyne 2020
May 6-9, 2020 NWB User Days and Developer Days, at HHMI Janelia  (also part of Aim 1)
June 2020 NIH BRAIN Initiative PI Meeting
Aug. 2020 NWB Workshop at INCF Neuroinformatics 2020
Sept. 2020 Joint OpenSourceBrain and NWB Hackathon (tentative, pending available support)
Oct. 2020 Society for Neuroscience (SfN 2020)
May 2021 NWB User Days and Developer Days (tentative, pending available funds)
Aim 1: To develop and maintain an accessible and sustainable open source software ecosystem for NWB  Efforts in this aim will primarily focus on continued software support and maintenance and continued engagement with the developer community to enhance integration of NWB software with data analysis tools.
Target  Milestone and Deliverable
April 2020 Prepare and submit journal submission of manuscript on the NWB 2.x schema (preprint at https://www.biorxiv.org/content/10.1101/523035v1)
Dec. 2019 – May 2020 Respond to user issues and implement bug fixes for PyNWB, HDMF, NWB Schema etc.
Dec. 2019 – May 2020 Continue efforts to improve documentation, tutorials, testing, continuous integration, and deployment of NWB software
Aim 2: To design methods for integration of controlled vocabulary, provenance, modeling of data relationships and external data management systems.
Aim 2 – Task 2.1: Design and integrate standardized metadata models with associated controlled vocabularies and ontologies with NWB. Efforts in extending data models and schemas to represent anatomy, spatial coordinates, genetic tools, stimulus, task and behavior with focus in supporting four major use cases: visual behavior, 3D grasping, maze task and movement planning.
Target  Milestone and Deliverable
Feb. 2020 Lead Allen Hackathon projects on stimulus and task representations in use cases
April 2020 Completed site visits to use case labs to collect metadata from 4 focus areas
May 2020 Release draft of version 1.0  metadata models at Janelia Hackathon and collect feedback
July 2020 Develop mechanisms for integration of controlled vocabularies and ontologies with NWB:N
Oct. 2020 Release draft of version 2.0 metadata models to release at SfN
Feb. 2021 Cosyne workshop on ontology and provenance
May 2021 Evaluation and refinement of the proposed ontology models with our chosen use cases
Aim 2 – Task 2.2: Develop methods for integration of provenance, data relationships and data management systems with NWB:N.   Efforts in this aim will focus on three key areas: 1) modeling of structural relationships between data, primarily via the concept of coordinate scales, 2) support for data referencing via foreign fields to enhance integration of NWB with data archives and management, and 3) development of a prototype provenance model for NWB. In addition, we will continue our ongoing efforts on integration of Zarr and engagement with DataJoint and DANDI.
Target  Milestone and Deliverable
Dec. 2019 – April 2020 Support coordinate scales for modeling of structural data relationships
Feb. 2020 Design data standard for defining structural relationships (in the form of coordinate scales)
May 2020 Implementation and release of coordinate scales in NWB:N, including enhancement of the current NWB:N standard schema to use the new feature to describe relationships explicitly
Dec. 2019 – July 2020 Demonstrate the use of Zarr as alternate storage backend
April 2020 Release Zarr Directory Store as Python-only storage backend for NWB via HDMF/PyNWB
July 2020 Evaluate the use of Zarr for NWB use cases
April 2020 – Feb. 2021 Support Foreign Fields for integration with data archives and data management
April 2020 Design storage standard for foreign fields
June 2020 Develop REST API standard for foreign fields
Aug. 2020 Develop reference client for the foreign field REST API
Feb. 2021 Evaluation, refinement, and release of foreign fields with NWB
Feb. 2020 – May 2021 Develop extension for integration of data provenance with NWB
Aug. 2020 Release provenance graphs for use cases in Aim 2, Task 2.1 at INCF neuroinformatics meeting
Oct. 2020 Design draft provenance standard for describing Activities
Dec. 2020 Design draft provenance standard for describing Agents
March 2021 Develop prototype extension for integration of the provenance model with NWB
May 2021 Evaluation and review of the proposed provenance extension and model
Aim 3: To develop methods and tools for facilitating community adoption, extension, and curation of NWB:N and integration of new use cases. With the release of the NDX Catalog in Oct. 2019, our focus in this aim will shift towards enhancement and refinement of the software and methods developed as part of the NDX catalog, as well as evaluation of our NWB extensions strategies on real use cases (e.g., ICEphys).
Target Milestone and Deliverable
May 2020 Enhance continuous integration and automated testing for the NDX catalog
June 2020 Prepare and submit manuscript describing the NDX catalog to a relevant journal (e.g., Frontiers in Neuroinformatics or Nature Scientific Data)
July 2020 Complete community review and release of new intracellular electrophysiology metadata models currently in development at [E3.12]. This milestone is part of our milestone to develop example extensions and engage with the community to evaluate the NDX Catalog and developed processes.
Aug. 2020 Implement advanced extension search and tracking of older extension versions in the NDX Catalog UI. Enhance tools for installation, discussion, and review of extensions.

Last Update Dec 19, 2018

The primary purpose of this target release is to stabilize and complete development of NWB:N 2.0beta. This release will also include a number of changes to the format and APIs that have not made it into the beta release. Requests for new features for this release has been frozen. Additional new features may be included in future NWB:N 2.x releases.

Timeline

  • NWB:N 2.0:
    • Jan. 18, 2019: Full release of NWB:N 2.O (final) and PyNWB and MatNWB
    • Dec. 23, 2018: NWB:N 2.0 schema finalized
    • SfN 2018 (~Nov.5 2018) complete all changes that break compatibility with previous versions of NWB:N.
  • Other:
    • May, 2019: Rename and release pynwb.form as a separate library. Tentative library name Hiearchical Data Modeling Framework (HDMF)

ToDo for the release

  • PyNWB
  • MatNWB
    • Sync MatNWB to the latest schema
  • nwb-docutils
    • Make PyPi release of the latest verison
  • Documentation:
    • Test that the schema docs build and are organized correctly on ReadTheDocs
    • Update builds on ReadTheDocs for all docs
    • Change links on https://neurodatawithoutborders.github.io/ to point to the stable rather than the latest build of the docs.
  • nwb-schema:
    • Sync schema with PyNWB
    • Finalize release notes
    • Close (or as approbritate update) open issues for the NWB:N 2.0 Full milestone
  • Publication:
    • Post manuscript describing the advances in the NWB:N 2.0 schema online
    • Post news release with announcement of the release

Last Update Dec 19, 2018

The primary purpose of this target release is to stabilize and complete development of NWB:N 2.0beta. This release will also include a number of changes to the format and APIs that have not made it into the beta release. Requests for new features for this release has been frozen. Additional new features may be included in future NWB:N 2.x releases.

Timeline

SfN 2018 (~Nov.5 2018) complete all changes that break compatibility with previous versions of NWB:N.

Open issues for this release

PyNWB: Python Reference API

  • PyNWB Interfaces
  • Extensions support
    • Support saving and using schema cached in NWB:N HDF5 files as part of the /general/specifications #497#522,
    • Test and fix bugs in autogenerated extension classes #520#48#289
    • Expand online validation of new specs during creation #51,pr#525
  • Testing, Continuous Integration, Deployment:
    • Add integration test cases to ensure broad coverage of integration tests #112#406
    • Add further unit tests to ensure broad coverage
    • Expand/ensure proper function of CI #152#270#274#277#399#444#446#461
    • Add tests for CLI tools #208
    • Define application test cases, i.e., sets of tests that implement select application test cases (e.g, convert of lab data to NWB:N).
  • Documentation:
  • Branding and Templates #281#425

NWB:N Format Schema

  • Reorganize /general to ease adding and managing custom, lab-specific metadata via extensions (#19)
  • Restructure the storage of Epochs data (see #119
  • Restructure the storage of ROI-related data (see #118#124#60 )
  • Improve/add support for trial data (see #152)
  • Add support for dynamic data tables
  • Add support for efficient storage of vector data
  • Restructure unit data (consolidate UnitTimes, ClusterWaveforms, and Clustering)
  • Use region and object reference for explicit referencing of data when possible
  • Make processing interfaces (Clustering, FeatureExtraction) that describe a time series inherit from TimeSeries (see #112)
  • Fix bugs and clarify fields (see #12#59#136#139#143#4#7#53#54 )
  • Various minor restructuring/clarification of metadata (see #145#148, )
  • Complete release notes describing the changes to the format, specification language etc. (see #161#133#122#115#102#6#28#39, )

NWB:N Specification Language

  • No new features are planned for the specification language itself beyond what was changed/added in NWB:N 2.0 beta. :white_check_mark:

NWB:N Documentation Utilties

  • Move documentation tools from nwb-schema to nwb-docutils) to make them easier to access and reuse. :white_check_mark:
  • Add tests for documentation tools

MatNWB: Matlab API

  • Ensure complete coverage and support for read and write of NWB:N HDF5 files
  • Ensure consistency between files generated by the Matlab and Python APIs
  • Add support for compound data types
  • Add support for the above-described changes to the specification language and schema
  • Add support for lazy data load
  • Add further documentation

This is a first public beta release of the new version NWB:N. The intent of this beta release is to enable early adopters to start exploring the new format and software. As development is ongoing, there may still be bugs and further changes to the APIs as well as the format are still planed between this beta and the first full release of NWB:N2.0. As such backward compatibility cannot be promised at this point yet.

New features for this release:

  • PyNWB: A new advanced python API for the new version of NWB:N 2.0. PyNWB defines a new modular software architecture and API to enable users/developers to efficiently interact with the NWB:N data format, format files, and specifications. The new software architecture decouples the various aspects of NWB:N (i.e. the specification language, format specification, storage, and API) to allow each to be used and maintained more independently. PyNWB also provides and API for interacting with NWB:N format specifications and provides mechanisms to easily create and use format extensions.
    • Available online: https://github.com/NeurodataWithoutBorders/pynwb
    • Documentation: PyNWB : Python Reference API Docs
    • Supported Languages :
      • Python >2.7.x
      • Python >3.6.x
    • Core features:
      • Write NWB:N HDF5 files
      • Read NWB:N files (base round-trip read/write)
      • Validate NWB:N files
      • Create new format extensions
      • Read/Write data using extensions
      • Read legacy NWB:N1.0x files from the Allen Institute for Brain Science
    • Main contributors for this release :
      • Andrew Tritt, Oliver Ruebel, David Camp, Kris Bouchard (LBNL) : Lead architects and development team
      • Nicholas Cain (Allen Institute for Brain Science) and Andrew Tritt (LBNL) : Port to Python 2 and legacy read for Allen data
      • Bug fixes, feedback and suggestions have also been contributed by numerous users and developers from the NWB:N community. For details see https://github.com/NeurodataWithoutBorders/pynwb/graphs/contributors
  • PyNWB testing, continuous integration, deployment We have developed an extensive set of unit tests for PyNWB. We have also created base infrastructure for integration tests to test end-to-end read/write of NWB:N files and data objects. While coverage of the tests is already quite good (>80%) coverage of the integration test suite is expected to be incomplete. We have also setup infrastructure for continuous integration to execute all tests on a broad range of platforms as well as defined software deployment paths (e.g., via PIP, CONDA)
    • Available online: For an overview of the various software health checks and tests being perfomed regulalary on PyNWB, see https://github.com/NeurodataWithoutBorders/pynwb
    • Continuous integration :
      • CircleCI (Linux, py27, py36)
      • Travis (MacOS, py27, py36)
      • AppVeyor (Windows, py27, py36)
      • Codecov.io
      • PEP8Speaks
    • Deployment:
      • PIP (PyPI)
      • Conda (CondaCloud)
    • Main contributors for this release
      • Andrew Tritt (LBNL) : Lead for design and implementation of the Python test suites
      • Jean-Christophe Fillion-Robin, Doruk Ozturk, Chris Kotfila, Michael Grauer, Will Schroeder (Kitware) : Lead development team for creation of continuous integration, testing, and deployment mechanisms. For details see https://github.com/NeurodataWithoutBorders/pynwb/graphs/contributors
  • Specification Language We have simplified and extended the specification language used to describe the NWB:N format to ease readability, interpretability and expressiveness as well as added support for compound data types and references among others.
    • Documentation: Specification Language Docs. The documentation also includes the release notes describing the changes in NWB:N 2 beta.
    • Main contributors for this release :
      • Andrew Tritt, Oliver Ruebel, Kris Bouchard (LBNL) : Development leads
      • Changes to the specification language have also been reviewed and new changes have been proposed at a community hackathon at Janelia Farms HHMI in August 2017. Many users have also contributed via GitHub issues.
  • Data Standard Specification Changes to the format have focused mainly on improving the structure and usability of NWB:N rather than adding new features to the format itself. Updates to the NWB:N format include among others: 1) extensions to clarify and extend the concept of NWBContainer (previously Interface), 2) avoid implicit links, 3) improve consistency and ease-of-use, 4) refactor of /general/extracellular_ephys and ElectrodeGroup to use data tables via compound data types.
    • Available online: nwb-schema is available on GitHub at https://github.com/NeurodataWithoutBorders/nwb-schema
    • Documentation: Data Standard Schema Docs.The documentation also includes the release notes describing the changes in NWB:N 2 beta.
    • Main contributors for this release :
      • Andrew Tritt, Oliver Ruebel (LBNL) : Implementation and documentation leads
      • Changes to the format specification have also been reviewed and new changes have been proposed at a community hackathon at Janelia Farms HHMI in August 2017. Many users have also contributed via GitHub issues.
  • Documentation We have created dedicated online documentations for the various aspects of NWB:N. While the documents are quite extensive, completeness and consistency of the documents is not guaranteed for the beta release and examples, tutorials and release notes (i.e., changes), may be a bit out of date given pace of development.
  • MatNWB MatNWB is a Matlab API for NWB:N. MatNWB generates Matlab classes for representing NWB:N neurodata_type directly from the YAML specification.
    • Core features:
      • Write NWB:N HDF5 files
      • Read NWB:N HDF5 files
      • Read/Write data for custom extensions
    • Main contributors for this release:
Back To Top