Changes
Version 3.0.1
Date: Jan 05, 2021
This version is mostly a bug fix release that allows the python packages of Connectome Mapper 3 to be available on PyPI. It incorporates Pull Request #132 which includes the following changes.
Bug fix
Rename the project name in
setup.py
andsetup_pypi.py
from"cmp"
to"connectomemapper"
. Such a"cmp"
project name was already existing on PyPI, that caused continuous integration on CircleCI to fail during the lastv3.0.0
release, while uploading the python packages of CMP3 to PyPI.
Code refactoring
Make
cmp.bidsappmanager.gui.py
more lightweight by splitting the classes defined there in different files. (See Issue #129 for more discussion details)Split the
create_workflow()
method of theRegistrationStage
into thecreate_ants_workflow()
,create_flirt_workflow()
, andcreate_bbregister_workflow()
. (See Issue #95 for more discussion details)
Code style
Correct a number of code style issues with class names
Contributors
Please check the main pull request 132 page for more details.
Version 3.0.0
Date: Dec 24, 2021
This version corresponds to the first official release of Connectome Mapper 3 (CMP3). It incorporates Pull Request #88 (>450 commits) which includes the following changes.
Updates
traits has been updated from
6.0.0
to6.2.0
.traitsui has been updated from
6.1.3
to7.0.0
.pybids has been updated from
0.10.2
to0.14.0
.nipype has been updated to
1.5.1
to1.7.0
.dipy has been updated from
1.1.0
to1.3.0
.obspy has been updated from
1.2.1
to1.2.2
.
New features
CMP3 can take custom segmentation (brain, white-matter, gray-matter and CSF masks, Freesurfer’s aparcaseg - used for ACT for PFT) and parcellation files as long as they comply to BIDS Derivatives specifications, by providing the label value for the different entity in the filename. This has led to the creation of the new module
cmtklib.bids.io
, which provides different classes to represent the diversity of custom input BIDS-formatted files. (PR #88)CMP3 generates generic label-index mapping
tsv
files along with the parcellation files, in accordance to BIDS derivatives. This has led to the creation of theCreateBIDSStandardParcellationLabelIndexMappingFile
andCreateCMPParcellationNodeDescriptionFilesFromBIDSFile
interfaces, which allows us to create the BIDS label-index mapping file from the parcellation node description files employed by CMP3 (that includes_FreeSurferColorLUT.txt
and_dseg.graphml
), and vice versa.CMP3 provide python wrappers to the Docker and Singularity container images (
connectomemapper3_docker
andconnectomemapper3_singularity
) that will generate and execute the appropriate command to run the BIDS App. (PR #109,
Major changes
Lausanne2018 parcellation has completely replaced the old Lausanne2008 parcellation. In brief, the new parcellation was introduced to provide (1) symmetry of labels between hemispheres, and (2) a more optimal generation of the volumetric parcellation images, that now are generated at once from
annot
files. This fixes the issue of overwritten labels encountered by in the process of creating the Lausanne2008 parcellation. Any code and data related to Lausanne2008 has been removed. If one still wish to use this old parcellation scheme, one should use CMP3 (v3.0.0-RC4
).
Output updates
Directories for the derivatives produced by cmp (
cmp
,freesurfer
,nipype
) were renamed tocmp-
,freesurfer-
, andnipype-
to comply with BIDS 1.4.0+. (PR #3 (fork))
Code refactoring
Creation in
AnatomicalPipeline
,DiffusionPipeline
,fMRIPipeline
ofcreate_datagrabber_node()
andcreate_datasinker_node()
methods to reduce the code increate_workflow()
.The
run(command)
function ofcmp.bidsappmanager.core
has been moved tocmtklib.process
, which is used by the python wrappers incmp.cli
.
Pipeline Improvements
Better handle of existing Freesurfer outputs. In this case, CMP3 does not re-create the
mri/orig/001.mgz
and connect the reconall interface anymore.Creation of 5TT, gray-matter / white-matter interface, and partial volume maps images are performed in the preprocessing stage of the diffusion pipeline only if necessary
Code Style
Clean code and remove a number of commented lines that are now obsolete. Code related to the connection of nodes in the Nipype
Workflow
adopts a specific format and are protected from being reformatted by BLACK with the# fmt: off
and# fmt: on
tags.
Documentation
Add instructions to use custom segmentation and parcellation files as inputs.
Add description in contributing page of format for code related to the connection of the nodes in a Nipype
Workflow
.Add instructions to use the python wrappers for running the BIDS App. (PR #115)
Add notification about the removal of the old Lausanne2008 parcellation, and remove any other mentions in the documentation.
Software container
Define multiple build stages in Dockerfile, which can be run in parallel at build with BUILDKIT. (PR #88)
Software development life cycle
Update the list of outputs of circleci tests with the new names of directories produced by cmp in
output_dir/
.Following major changes in the pricing plans of CircleCI but also to improve its readability,
circleci/config.yml
has been dramatically refactored, including: * Use BUILDKIT in docker build to take advantage of the multi-stage build * Reordering and modularization of the tests:tests 01-02 (Docker): anatomical pipeline for each parcellation scheme
tests 03-06 (Docker): diffusion pipeline for dipy/mrtrix deterministic/probabilistic tractography
tests 07-08 (Docker): fMRI pipeline for FLIRT and BBRegistration registrations
test 09 (Singularity): anatomical pipeline for Lausanne2018 scheme
Creation of commands for steps that are shared between jobs to reduce code duplication
(PR #88)
Contributors
Please check the main pull request 88 page for more details.
Version 3.0.0-RC4
Date: March 07, 2021
This version corresponds to the fourth and final release candidate of Connectome Mapper 3 (CMP3). It incorporates the relatively large Pull Request #74 (~270 commits) which includes the following changes such that it marks the end of the release candidate phase.
New features
CMP3 pipeline configuration files adopt
JSON
as new format. (PR #76)CMP3 is compatible with PyPI for installation. (PR #78)
BIDS convention naming of data derived from parcellation atlas adopt now the new BIDS entity
atlas-<atlas_label>
to distinguish data derived from different parcellation atlases. The use of the entitydesc-<scale_label>
to distinguish between parcellation scale has been replaced by the use of the entityres-<scale_label>
. (PR #79)
Updates
Content of
dataset_description.json
for each derivatives folder has been updated to conform to BIDS version 1.4.0. (PR #79)
Code refactoring
Major refactoring of the
cmtklib.config
module with the addition and replacement of a number of new methods to handleJSON
configuration files. (See full diff on GitHub) Configuration files in the oldINI
format can be converted automatically with the help of the two new methodscheck_configuration_format()
andconvert_config_ini_2_json()
to detect if configuration files are in theINI
format and to make the conversion. (PR #76)Major changes to make
cmp
andcmpbidsappmanager
compatible with the Python Package Index (pip
) for package distribution and installation. This includes the merge ofsetup.py
andsetup_gui.py
, which have been merged into onesetup.py
and a major refactoring to makepip
happy, as well as the creation of a newcmp.cli
module, migration tocmp.cli
module and refactoring of the scriptsconnectomemapper3
,showmatrix_gpickle
, andcmpbidsappmanager
with correction of code style issues and addition of missing docstrings. (PR #78)
Improvements
Clean parameters to be saved in configuration files with the new API. (PR #74)
Clean output printed by the cmpbidsappmanager Graphical User Interface. (PR #74)
Add in
cmtklib.config
the three new functionsprint_error
,print_blue
, andprint_warning
to use different colors to differentiate general info (default color), error (red), command or action (blue), and highlight or warning (yellow). (PR #74)Clean code and remove a number of commented lines that are now obsolete. (PR #74, PR #79)
Documentation
Review usage and add a note regarding the adoption of the new
JSON
format for configuration files. (PR #76)Update tutorial on using CMP3 and Datalad for collaboration. (PR #77)
Update installation instruction of
cmpbidsappmanager
usingpip install .
. (PR #78)Update list of outputs following the new BIDS derivatives naming convention introduced. (PR #79)
Bug fixes
Correct attributes related to the diffusion imaging model type
multishell
. (PR #74)Review code in
cmtklib/connectome.py
for saving functional connectome files in GRAPHML format. (PR #74)
Software Updates
Update version of datalad and dependencies (PR #77):
datalad[full]==0.13.0
todatalad[full]==0.14.0
.datalad-container==0.3.1
todatalad-container==1.1.2
.datalad_neuroimaging==0.2.0
todatalad-neuroimaging==0.3.1
.git-annex=8.20200617
togit-annex=8.20210127
.datalad-revolution
was removed.
Software development life cycle
Improve code coverage by calling the methods
check_stages_execution()
andfill_stages_outputs()
on each pipeline when executed with coverage. (PR #75)Improve code coverage by saving in test-01 structural connectome files in MAT and GRAPHML format. (PR #74)
Improve code coverage by saving in test-07 functional connectome files in GRAPHML format. (PR #74)
Update the list of outputs for all tests. (PR #74)
Add
test-python-install
job that test the build and installation ofcmp
andcmpbidsappmanager
packages compatible withpip
. (PR #78)
Please check the main pull request 74 page for more details.
Version 3.0.0-RC3
Date: February 05, 2021
This version corresponds to the third release candidate of Connectome Mapper 3. In particular, it integrates Pull Request #62 which includes:
Updates
MRtrix3 has been updated from
3.0_RC3_latest
to3.0.2
.Numpy has been updated from
1.18.5
to1.19.2
.Nipype has been updated to
1.5.0
to1.5.1
.Dipy has been updated from
1.0.0
to1.3.0
.CVXPY has been updated from
1.1.5
to1.1.7
.
Documentation
Update outdated screenshots for GUI documentation page at readthedocs reported at CMTK user-group.
Correction of multiple typos.
Bug fixes
Update code for Dipy tracking with DTI model following major changes in Dipy 1.0 (Fix reported issue #54).
Update to Dipy 1.3.0 has removed the deprecated warnings related to CVXPY when using MAP_MRI (#63)
Do not set anymore
OMP_NUM_THREADS
at execution due to allocation errors raised when using numpy function dot in Dipy.
Software development life cycle
Add
Test 08
that runs anatomical and fMRI pipelines with: Lausanne2018 parcellation, FSL FLIRT co-registration, all nuisance regression, linear detrending and scrubbingAdd
Test 09
that runs anatomical and dMRI pipelines with: Lausanne2018 parcellation, FSL FLIRT, Dipy SHORE, MRtrix SD_Stream tracking, MRtrix SIFT tractogram filteringRemove
deploy_singularity_latest
from the workflow for the sake of space on Sylabs.io.
Please check the main pull request 62 page for more details.
Version 3.0.0-RC2-patch1
Date: February 4, 2021
This version fixes bugs in the second release candidate of Connectome Mapper 3 (v3.0.0-RC2). In particular, it includes:
Bug fixes
Fix the error to save connectome in GraphML format reported in #65 and (Pull Request #66).
Software development life cycle
Remove publication of the Singularity image to sylabs.io when the master branch is updated for the sake of space (11GB limit)
Commits
CI: remove publication of latest tag image on sylabs.io for space (2 days ago) - commit c765f79
Merge pull request #66 from connectomicslab/v3.0.0-RC2-hotfix1 (3 days ago) - commit 0a2603e
FIX: update g2.node to g2.nodes when saving connectomes as graphml (fix #65) (6 days ago) - commit d629eef
FIX: enabled/disabled gray-out button “Run BIDS App” with Qt Style sheet [skip ci] (3 weeks ago) - commit 10e78d9
MAINT: removed commented lines in cmpbidsappmanager/gui.py [skip ci] (3 weeks ago) - commit 4cc11e7
FIX: check availability of modalities in the BIDS App manager window [skip ci] (3 weeks ago) - commit 80fbee2
MAINT: update copyright year [skip ci] (3 weeks ago) - commit f7d0ffb
CI: delete previous container with latest TAG on sylabs.io [skip ci] (4 weeks ago) - commit 15c9b18
DOC: update tag to latest in runonhpc.rst [skip ci] (4 weeks ago) - commit 3165bcc
CI: comment lines related to version for singularity push (4 weeks ago) - commit 3952d46
Version 3.0.0-RC2
Date: December 24, 2020
This version corresponds to the second release candidate of Connectome Mapper 3. In particular, it integrates Pull Request #45 which includes:
New feature
Add a tracker to support us seeking for new funding. User is still free to opt-out and disable it with the new option flag
--notrack
.Add options suggested by Theaud G et al. (2020) to better control factors having impacts on reproducibility. It includes:
Set the number of ITK threads used by ANTs for registration (option flag
--ants_number_of_threads
).Set the seed of the random number generator used by ANTs for registration (option flag
--ants_random_seed
).Set the seed of the random number generator used by MRtrix for tractography seeding and track propagation (option flag
--mrtrix_random_seed
).
Full support of Singularity (see Software development life cycle).
Code refactoring
A number of classes describing interfaces to
fsl
andmrtrix3
have been moved fromcmtklib/interfaces/util.py
tocmtklib/interfaces/fsl.py
andcmtklib/interfaces/mrtrix3.py
.Capitalize the first letter of a number of class names.
Lowercase a number of variable names in
cmtklib/parcellation.py
.
Graphical User Interface
Improve display of qpushbuttons with images in the GUI (PR #52).
Make the window to control BIDS App execution scrollable.
Allow to specify a custom output directory.
Tune new options in the window to control BIDS App multi-threading (OpenMP and ANTs) and random number generators (ANTs and MRtrix).
Documentation
Full code documentation with numpydoc-style docstrings.
API documentation page at readthedocs.
Bug fixes
Fix the error reported in #17 if it is still occuring.
Review statements for creating contents of BIDS App entrypoint scripts to fix issue with Singularity converted images reported in #47.
Install
dc
package inside the BIDS App to fix the issue with FSL BET reported in #50.Install
libopenblas
package inside the BIDS App to fix the issue with FSL EDDY_OPENMP reported in #49.
Software development life cycle
Add a new job
test_docker_fmri
that test the fMRI pipeline.Add
build_singularity
,test_singularity_parcellation
,deploy_singularity_latest
, anddeploy_singularity_release
jobs to build, test and deploy the Singularity image in CircleCI (PR #56).
Please check the main pull request 45 page for more details.
Version 3.0.0-RC1
Date: August 03, 2020
This version corresponds to the first release candidate of Connectome Mapper 3. In particular, it integrates Pull Request #40 where the last major changes prior to its official release have been made, which includes in particular:
Migration to Python 3
Fixes automatically with
2to3
and manually a number of Python 2 statements invalid in python 3 including the print() functionCorrect automatically PEP8 code style issues with autopep8
Correct manually a number of code stly issues reported by Codacy (bandits/pylints/flake8)
Major dependency upgrades including:
dipy 0.15 -> 1.0
and related code changes incmtklib/interfaces/dipy
(Check here for more details about Dipy 1.0 changes)
Warning
Interface for tractography based on Dipy DTI model and EuDX tractography, which has been drastically changed in Dipy 1.0, has not been updated yet, It will be part of the next release candidate.
nipype 1.1.8 -> 1.5.0
pybids 0.9.5 -> 0.10.2
pydicom 1.4.2 -> 2.0.0
networkX 2.2 -> 2.4
statsmodels 0.9.0 -> 0.11.1
obspy 1.1.1 -> 1.2.1
traits 5.1 -> 6.0.0
traitsui 6.0.0 -> 6.1.3
numpy 1.15.4 -> 1.18.5
matplotlib 1.1.8 -> 1.5.0
fsleyes 0.27.3 -> 0.33.0
mne 0.17.1 -> 0.20.7
sphinx 1.8.5 -> 3.1.1
sphinx_rtd_theme 0.4.3 -> 0.5.0
recommonmark 0.5.0 -> 0.6.0
New feature
Option to run Freesurfer recon-all in parallel and to specify the number of threads used by not only Freesurfer but also all softwares relying on OpenMP for multi-threading. This can be achieved by running the BIDS App with the new option flag
--number_of_threads
.
Changes in BIDS derivatives
Renamed connectivity graph files to better conform to the BIDS extension proposal on connectivity data schema. They are now saved by default in a TSV file as a list of edges.
Code refactoring
Functions to save and load pipeline configuration files have been moved to
cmtklib/config.py
Bug fixes
Major changes in how inspection of stage/pipeline outputs with the graphical user interface (cmpbidsappmanager) which was not working anymore after migration to Python3
Fixes to compute the structural connectivity matrices following migration to python 3
Fixes to computes ROI volumetry for Lausanne2008 and NativeFreesurfer parcellation schemes
Add missing renaming of the ROI volumetry file for the NativeFreesurfer parcellation scheme following BIDS
Create the mask used for computing peaks from the Dipy CSD model when performing Particle Filtering Tractography (development still on-going)
Add missing renaming of Dipy tensor-related maps (AD, RD, MD) following BIDS
Remove all references to use Custom segmentation / parcellation / diffusion FOD image / tractogram, inherited from CMP2 but not anymore functional following the adoption of BIDS standard inside CMP3.
Software development life cycle
Use Codacy to support code reviews and monitor code quality over time.
Use coveragepy in CircleCI during regression tests of the BIDS app and create code coverage reports published on our Codacy project page.
- Add new regression tests in CircleCI to improve code coverage:
Test 01: Lausanne2018 (full) parcellation + Dipy SHORE + Mrtrix3 SD_STREAM tractography
Test 02: Lausanne2018 (full) parcellation + Dipy SHORE + Mrtrix3 ACT iFOV2 tractography
Test 03: Lausanne2018 (full) parcellation + Dipy SHORE + Dipy deterministic tractography
Test 04: Lausanne2018 (full) parcellation + Dipy SHORE + Dipy Particle Filtering tractography
Test 05: Native Freesurfer (Desikan-Killiany) parcellation
Test 06: Lausanne2008 parcellation (as implemented in CMP2)
Moved pipeline configurations for regression tests in CircleCI from
config/
to.circle/tests/configuration_files
Moved lists of expected regression test outputs in CircleCI from
.circle/
to.circle/tests/expected_outputs
Please check the pull request 40 page for more details.
Version 3.0.0-beta-RC2
Date: June 02, 2020
This version integrates Pull Request #33 which corresponds to the last beta release that still relies on Python 2.7. It includes in particular:
Upgrade
Uses
fsleyes
instead offslview
(now deprecated), which now included in the conda environment of the GUI (py27cmp-gui
).
New feature
Computes of ROI volumetry stored in
<output_dir>/sub-<label>(/ses<label>)/anat
folder, recognized by their_stats.tsv
file name suffix.
Improved replicability
Sets the
MATRIX_RNG_SEED
environment variable (used by MRtrix) and seed for the numpy random number generator (numpy.random.seed()
)
Bug fixes
Fixes the output inspector window of the cmpbidsappmanager (GUI) that fails to find existing outputs, after adoption of /bids_dir and /output_dir in the bidsapp docker image.
Fixes the way to get the list of networkx edge attributes in
inspect_outputs()
ofConnectomeStage
for the output inspector window of the cmpbidsappmanager (GUI)Added missing package dependencies (
fury
andvtk
) that fixes dipy_CSD execution error when trying to import module actor from dipy.viz to save the results in a pngFixes a number of unresolved references identified by pycharm code inspection tool
Code refactoring
Interfaces for fMRI processing were moved to
cmtklib/functionalMRI.py
.Interface for fMRI connectome creation (
rsfmri_conmat
) moved tocmtklib/connectome.py
Please check the pull request 33 page for change details.
Version 3.0.0-beta-RC1
Date: March 26, 2020
This version integrates Pull Request #28 which includes in summary:
A major revision of continuous integration testing and deployment with CircleCI which closes Issue 14 integrates an in-house dataset published and available on Zenodo @ https://doi.org/10.5281/zenodo.3708962.
Multiple bug fixes and enhancements incl. close Issue 30 , update mrtrix3 to RC3 version, bids-app run command generated by the GUI, location of the configuration and log files to be more BIDS compliant.
Change in tagging beta version which otherwise might not be meaningfull in accordance with the release date (especially when the expected date is delayed due to unexpected errors that might take longer to be fixed than expected).
Please check the pull request 28 page for a full list of changes.
Version 3.0.0-beta-20200227
Date: February 27, 2020
This version addresses multiple issues to make successful conversion and run of the CMP3 BIDS App on HPC (Clusters) using Singularity.
Revised the build of the master and BIDS App images:
Install locales and set
$LC_ALL
and$LANG
to make freesurfer hippocampal subfields and brainstem segmentation (matlab-based) modules working when run in the converted SIngularity imageBIDS input and output directories inside the BIDS App container are no longer the
/tmp
and/tmp/derivatives
folders but/bids_dir
and/output_dir
. .. warning:: this might affect the use of Datalad container (To be confirmed.)Fix the branch of mrtrix3 to check out
Updated metadata
Fix the configuration of CircleCI to not use Docker layer cache feature anymore as this feature is not included anymore in the free plan for open source projects.
Improved documentation where the latest version should be dynamically generated everywhere it should appear.
Version 3.0.0-beta-20200206
Date: February 06, 2020
Implementation of an in-house Nipype interface to AFNI 3DBandPass which can handle to check output as ..++orig.BRIK or as ..tlrc.BRIK (The later can occur with HCP preprocessed fmri data)
Version 3.0.0-beta-20200124
Date: January 24, 2020
Updated multi-scale parcellation with a new symmetric version:
The right hemisphere labels were projected in the left hemisphere to create a symmetric version of the multiscale cortical parcellation proposed by Cammoun2012.
For scale 1, the boundaries of the projected regions over the left hemisphere were matched to the boundaries of the original parcellation for the left hemisphere.
This transformation was applied for the rest of the scales.
Updated documentation with list of changes