Connectome Mapper 3 (UNDER DEVELOPMENT)

This neuroimaging processing pipeline software is developed by the Connectomics Lab at the University Hospital of Lausanne (CHUV) for use within the SNF Sinergia Project 170873, as well as for open-source software distribution.

Warning

THIS SOFTWARE IS FOR RESEARCH PURPOSES ONLY AND SHALL NOT BE USED FOR ANY CLINICAL USE. THIS SOFTWARE HAS NOT BEEN REVIEWED OR APPROVED BY THE FOOD AND DRUG ADMINISTRATION OR EQUIVALENT AUTHORITY, AND IS FOR NON-CLINICAL, IRB-APPROVED RESEARCH USE ONLY. IN NO EVENT SHALL DATA OR IMAGES GENERATED THROUGH THE USE OF THE SOFTWARE BE USED IN THE PROVISION OF PATIENT CARE.

About

Connectome Mapper 3, part of the Connectome Mapping Toolkit (CMTK), implements full anatomical, diffusion and resting-state MRI processing pipelines, from raw Diffusion / T1 / T2 / BOLD data to multi-resolution connection matrices.

The Connectome Mapper 3 pipelines uses a combination of tools from well-known software packages, including FSL, FreeSurfer, ANTs, MRtrix3, Dipy and AFNI. These pipelines were designed to provide the best software implementation for each state of processing, and will be updated as newer and better neuroimaging software become available.

This tool allows you to easily do the following:

• Take T1 / Diffusion / resting-state MRI data from raw to multi-resolution connection matrices.
• Implement tools from different software packages.
• Achieve optimal data processing quality by using the best tools available
• Automate and parallelize processing steps, which provides a significant speed-up from typical linear, manual processing.

Reproducibility and replicatibility is achieved through the distribution of a BIDSApp, a software container image which provide a frozen environment where versions of all external softwares and libraries are fixed.

Installation Instructions for Users

Warning

This software is for research purposes only and shall not be used for any clinical use. This software has not been reviewed or approved by the Food and Drug Administration or equivalent authority, and is for non-clinical, IRB-approved Research Use Only. In no event shall data or images generated through the use of the Software be used in the provision of patient care.

The Connectome Mapper 3 is composed of a Docker image, namely the Connectome Mapper 3 BIDS App, and a Python Graphical User Interface, namely the Connectome Mapper BIDS App Manager.

• Installation instructions for the Connectome mapper 3 BIDS App are found in Installation.
• Installation instructions for the Connectome mapper 3 BIDS App Manager are found in Installation.

Make sure that you have installed the following prerequisites.

The Connectome Mapper 3 BIDSApp

Prerequisites

• Installed Docker Engine corresponding to your system:

  • For Ubuntu 14.04/16.04/18.04, follow the instructions from the web page:

    $ firefox https://docs.docker.com/install/linux/docker-ce/ubuntu/
    

  • For Mac OSX (>=10.10.3), get the .dmg installer from the web page:

    $ firefox https://store.docker.com/editions/community/docker-ce-desktop-mac
    

  • For Windows (>=10), get the installer from the web page:

    $ firefox https://store.docker.com/editions/community/docker-ce-desktop-windows
    

Note

Connectome Mapper 3 BIDSApp has been tested only on Ubuntu and MacOSX. For Windows users, it might be required to make few patches in the Dockerfile.

• Docker managed as a non-root user

  • Open a terminal

  • Create the docker group:

    $ sudo groupadd docker
    

  • Add the current user to the docker group:

    $ sudo usermod -G docker -a $USER
    

  • Reboot

    After reboot, test if docker is managed as non-root:

    $ docker run hello-world
    

Installation

Installation of the Connectome Mapper 3 has been facilitated through the distribution of a BIDSApp relying on the Docker software container technology.

• Open a terminal

• Get the latest release (v3.0.0-beta-20200227) of the BIDS App:

  $ docker pull sebastientourbier/connectomemapper-bidsapp:v3.0.0-beta-20200227
  

• To display all docker images available:

  $ docker images
  

You should see the docker image “connectomemapper-bidsapp” with tag “v3.0.0-beta-20200227” is now available.

• You are ready to use the Connectome Mapper 3 BIDS App from the terminal. See its commandline usage.

The Connectome Mapper 3 BIDSApp Manager (GUI)

Prerequisites

• Installed miniconda2 (Python 2.7) from the web page:

  $ firefox https://conda.io/miniconda.html
  

  Download the Python 2.7 installer corresponding to your 32/64bits MacOSX/Linux/Win system.

Installation

The installation of the Connectome Mapper 3 BIDS App Manager (CMPBIDSAPPManager) consists of a clone of the source code repository, the creation of conda environment with all python dependencies installed, and eventually the installation of the CMPBIDSAPPManager itself, as follows:

• Open a terminal

• Go to the folder in which you would like to clone the source code repository:

  $ cd <INSTALLATION DIRECTORY>
  

• Clone the source code repository:

  $ git clone https://github.com/connectomicslab/connectomemapper3.git connectomemapper3
  

• Create a branch and checkout the code corresponding to this version release:

  $ git fetch
  $ git checkout tags/v3.0.0-beta-20200227 -b v3.0.0-beta-20200227
  

• Create a miniconda2 environment where all python dependencies will be installed, this by using the spec list “conda_packages_list.txt” provided by the repository:

  $ conda env create -f connectomemapper3/environment.yml
  

• Activate the conda environment:

  $ source activate py27cmp-gui
  

  or:

  $ conda activate py27cmp-gui
  

• Install the Connectome Mapper BIDS App Manager from the Bash Shell using following commands:

  (py27cmp-gui)$ cd connectomemapper3/
  (py27cmp-gui)$ python setup_gui.py install
  

• You are ready to use the Connectome Mapper 3 BIDS App Manager. See the dedicated user guide.

Help/Questions

If you run into any problems or have any questions, you can post to the CMTK-users group. Code bugs can be reported by creating a “New Issue” on the source code repository.

Graphical User Interface

Introduction

Connectome Mapper 3 comes with a Graphical User Interface, the Connectome Mapper BIDS App manager, designed to facilitate the configuration of all pipeline stages, the configuration of the BIDS App run and its execution, and the inspection of the different stage outputs with appropriate viewers.

Main window of the Connectome Mapper BIDS App Manager

Start the Graphical User Interface

In a terminal, enter to following:

$ source activate py27cmp-gui

or:

$ conda activate py27cmp-gui

Please see Section Installation for more details about installation.

After activation of the conda environment, start the graphical user interface called Connectome Mapper 3 BIDS App Manager

$ cmpbidsappmanager

Load a BIDS dataset

The Connectome Mapper 3 BIDS App Manager allows you to:

• load a BIDS dataset stored locally

You only have to select the root directory of your valid BIDS dataset (see note below)

• create a new datalad/BIDS dataset locally from an existing local or remote datalad/BIDS dataset (This is a feature under development)

Select the mode “Install a Datalad/BIDS dataset”.

If ssh connection is used, make sure to enable the “install via ssh” and to provide all connection details (IP address / Remote host name, remote user, remote password)

Note

The input dataset MUST be a valid BIDS structured dataset and must include at least one T1w or MPRAGE structural image. We highly recommend that you validate your dataset with the free, online BIDS Validator.

Pipeline stage configuration

Start the Configurator Window

• From the main window, click on the left button to start the Configurator Window.

• The window of the Connectome Mapper BIDS App Configurator will appear, which will assist you note only in configuring the pipeline stages (each pipeline has a tab panel), but also in creating appropriate configuration files which could be used outside the Graphical User Interface.

Configurator Window of the Connectome Mapper

The outputs depend on the chosen parameters.

Anatomical pipeline stages

Panel for configuration of anatomical pipeline stages

Segmentation

Performs tissue segmentation using Freesurfer or custom segmentation.

Freesurfer

• Freesurfer args: used to specify Freesurfer processing options
• Use existing freesurfer data: Check this box if you have already Freesurfer output data available

Custom segmentation

Warning

Not fully tested. Development and testing in progress.

• White matter mask: select the file containing your white matter binary mask

Parcellation

Generates the Native Freesurfer or Lausanne2008/Lausanne2018 parcellation from Freesurfer data, or takes a custom parcellation atlas.

Parcellation scheme

• NativeFreesurfer:

  

  Atlas composed of 83 regions from the Freesurfer aparc+aseg file

• Lausanne2008:

  

  Multi-resolution atlas

• Lausanne2018:

  

  Lausanne 2008 atlas extended with 7 thalamic nuclei, 12 hippocampal subfields, and 4 brainstem sub-structure per hemisphere

• Custom:

  Warning

  Not fully tested. Development and testing in progress.

  

  Custom atlas. Specify the atlas name, the number of regions, the nifti file and a corresponding graphml file. The Graphml file must contain at least a “dn_correspondence_id” field for each node. This field should contain the region’s label in the nifti file.

Diffusion pipeline stages

Panel for configuration of diffusion pipeline stages

Preprocessing

Preprocessing includes denoising, bias field correction, motion and eddy current correction for diffusion data.

Denoising

Remove noise from diffusion images using (1) MRtrix3 MP-PCA method or (2) Dipy Non-Local Mean (NLM) denoising with Gaussian or Rician noise models

Bias field correction

Remove intensity inhomogeneities due to the magnetic resonnace bias field using (1) MRtrix3 N4 bias field correction or (2) the bias field correction provided by FSL FAST.

Motion correction

Aligns diffusion volumes to the b0 volume using FSL’s MCFLIRT.

Note

For hemi-sphere DSI aquisitions, warning outputs will be displayed in the console when processing empty volumes.

Eddy current correction

Corrects for eddy current distortions using FSL’s Eddy correct tool.

Resampling

Resample morphological and diffusion data to F0 x F1 x F2 mm^3

Registration

Registration mode

• FSL (Linear):

  

  Perform linear registration from T1 to diffusion b0 using FSL’s flirt.

• BBregister (FS):

  

  Perform linear registration using Freesurfer BBregister tool.

• Non-linear (ANTS):

  

  Perform symmetric diffeomorphic SyN registration from T1 to b0

Diffusion reconstruction and tractography

Perform diffusion reconstruction and local deterministic or probabilistic tractography based on several tools. ROI dilation is required to map brain connections when the tracking only operates in the white matter.

Diffusion stage configuration window

Reconstruction tool

Dipy: perform SHORE, tensor, CSD and MAP-MRI reconstruction.

• SHORE:

  

  SHORE performed only on DSI data

• Tensor:

  

  Tensor performed only on DTI data

• CSD:

  

  CSD performed on DTI and multi-shell data

• MAP_MRI:

  

  MAP-MRI performed only on multi-shell data

MRtrix: perform CSD reconstruction.

• CSD:

  

  CSD performed on DTI and multi-shell data

Tractography tool

Dipy: perform deterministic and probabilistic fiber tracking as well as particle filtering tractography.

• Deterministic tractography:

  

  Deterministic tractography (SD_STREAM) performed on single tensor or CSD reconstruction

• Probabilistic tractography:

  

  Probabilistic tractography (iFOD2) performed on SHORE or CSD reconstruction

• Probabilistic particle filtering tractography (PFT):

  

  Probabilistic PFT tracking performed on SHORE or CSD reconstruction. Seeding from the gray matter / white matter interface is possible.

Note

We noticed a shift of the center of tractograms obtained by dipy. As a result, tractograms visualized in TrackVis are not commonly centered despite the fact that the tractogram and the ROIs are properly aligned.

MRtrix: perform deterministic and probabilistic fiber tracking as well as anatomically-constrained tractography. ROI dilation is required to map brain connections when the tracking only operates in the white matter.

• Deterministic tractography:

  

  Deterministic tractography (SD_STREAM) performed on single tensor or CSD reconstruction

• Deterministic anatomically-constrained tractography (ACT):

  

  Deterministic ACT tracking performed on single tensor or CSD reconstruction. Seeding from the gray matter / white matter interface is possible. Backtrack option is not available in deterministic tracking.

• Probabilistic tractography:

  

  Probabilistic tractography (iFOD2) performed on SHORE or CSD reconstruction

• Probabilistic anatomically-constrained tractography (ACT):

  

  Probabilistic ACT tracking performed on SHORE or CSD reconstruction. Seeding from the gray matter / white matter interface is possible.

Connectome

Compute fiber length connectivity matrices. If DTI data is processed, FA additional map is computed. In case of DSI, additional maps include GFA and RTOP. In case of MAP-MRI, additional maps are RTPP, RTOP, …

Output types

Select in which formats the connectivity matrices should be saved.

FMRI pipeline stages

Panel for configuration of fMRI pipeline stages

Preprocessing

Preprocessing refers to processing steps prior to registration. It includes discarding volumes, despiking, slice timing correction and motion correction for fMRI (BOLD) data.

Discard n volummes

Discard n volumes from further analysis

Despiking

Perform despiking of the BOLD signal using AFNI.

Slice timing and Repetition time

Perform slice timing correction using FSL’s slicetimer.

Motion correction

Align BOLD volumes to the mean BOLD volume using FSL’s MCFLIRT.

Registration

Registration mode

• FSL (Linear):

  

  Perform linear registration from T1 to mean BOLD using FSL’s flirt.

• BBregister (FS)

  

  Perform linear registration using Freesurfer BBregister tool from T1 to mean BOLD via T2.

  Warning

  development in progress

fMRI processing

Performs detrending, nuisance regression, bandpass filteringdiffusion reconstruction and local deterministic or probabilistic tractography based on several tools. ROI dilation is required to map brain connections when the tracking only operates in the white matter.

Detrending

Detrending of BOLD signal using: 1. linear trend removal algorithm provided by the scipy library 2. quadratic trend removal algorithm provided by the obspy library

Nuisance regression

A number of options for removing nuisance signals is provided. They consist of: 1. Global signal regression 2. CSF regression 3. WM regression 4. Motion parameters regression

Bandpass filtering

Perform bandpass filtering of the time-series using FSL’s slicetimer

Connectome

Computes ROI-averaged time-series and the correlation connectivity matrices.

Output types

Select in which formats the connectivity matrices should be saved.

Save the configuration files

You can save the pipeline stage configuration files in two different way:

1. You can save all configuration files at once by clicking on the Save All Pipeline Configuration Files. This will save automatically the configuration file of the anatomical / diffusion / fMRI pipeline to <bids_dataset>/code/ref_anatomical_config.ini / <bids_dataset>/code/ref_diffusion_config.ini / <bids_dataset>/code/ref_fMRI_config.ini respectively.
2. You can save individually each of the pipeline configuration files and edit its filename in the File menu (File -> Save anatomical/diffusion/fMRI configuration file as…)

Nipype

The Connectome Mapper processing relies on nipype. For each stage, a processing folder is created in $Base_directory/derivatives/nipype/sub-<participant_label>/<pipeline_name>/<stage_name>.

All intermediate steps for the processing are saved in the corresponding stage folders.

Run the BIDS App

Start the Connectome Mapper BIDS App GUI

• From the main window, click on the middle button to start the Connectome Mapper BIDS App GUI.

• The window of the Connectome Mapper BIDS App GUI will appear, which will help you in setting up and launching the BIDS App run.

Window of the Connectome Mapper BIDS App GUI

Run configuration

• Select the subject labels to be processed

  

• Check/Uncheck the pipelines to be performed

  

• Specify your Freesurfer license

  

  Note

  Your freesurfer license will be copied to your dataset directory as <bids_dataset>/code/license.txt which will be mounted inside the BIDS App container image.

• When the run is set up, you can click on the Check settings button.

  

• If the setup is complete and valid, this will enable the Run BIDS App button.

  

You are ready to launch the BIDS App run!

Launch the BIDS App run

• Click on the Run BIDS App button to launch the BIDS App run

  

• You can see the complete docker run command generated by the Connectome Mapper BIDS App GUI from the terminal output such as in this example

  Start BIDS App
  > Copy FreeSurfer license (BIDS App Manager)
  ... src : /usr/local/freesurfer/license.txt
  ... dst : /media/localadmin/HagmannHDD/Seb/ds-testLausanne2008SHOREPFT/code/license.txt
  > Datalad available: True
  *... Docker cmd 2 : ['docker', 'run', '-it', '--rm', '-v', '/media/localadmin/HagmannHDD/Seb/ds-testLausanne2008SHOREPFT:/tmp', '-u', '1000:1000', 'sebastientourbier/connectomemapper-bidsapp:3.0.0-beta-singularity', '/tmp', '/tmp/derivatives', 'participant', '--participant_label', 'A001', '--anat_pipeline_config', '/tmp/code/ref_anatomical_config.ini', '--dwi_pipeline_config', '/tmp/code/ref_diffusion_config.ini']*
  > BIDS dataset: /tmp
  > Subjects to analyze : ['A001']
  > Copy FreeSurfer license (BIDS App)
  > Sessions to analyze : ['ses-20150203160809']
  > Process subject sub-A001 session ses-20150203160809
  WARNING: rewriting config file /tmp/derivatives/sub-A001_ses-20150203160809_anatomical_config.ini
  ... Anatomical config created : /tmp/derivatives/sub-A001_ses-20150203160809_anatomical_config.ini
  WARNING: rewriting config file /tmp/derivatives/sub-A001_ses-20150203160809_diffusion_config.ini
  ... Diffusion config created : /tmp/derivatives/sub-A001_ses-20150203160809_diffusion_config.ini
  ... Running pipelines :
          - Anatomical MRI (segmentation and parcellation)
          - Diffusion MRI (structural connectivity matrices)
  ... cmd : connectomemapper3 /tmp /tmp/derivatives sub-A001 ses-20150203160809 /tmp/derivatives/sub-A001_ses-20150203160809_anatomical_config.ini True /tmp/derivatives/sub-A001_ses-20150203160809_diffusion_config.ini True
  

  Note

  Also, this can be helpful in you wish to design your own batch scripts to call the BIDS App with the correct syntax.

Check progress

For each subject, the execution output of the pipelines are redirected to a log file, written as <bids_dataset/derivatives>/cmp/sub-<subject_label>_log.txt. Execution progress can be checked by the means of these log files.

Check stages outputs

Start the Inspector Window

• From the main window, click on the right button to start the Inspector Window.

• The window of the Connectome Mapper BIDS App Inspector will appear, which will assist you in inspecting outputs of the different pipeline stages (each pipeline has a tab panel).

Anatomical pipeline stages

• Click on the stage you wish to check the output(s):

  

  Panel for configuration of anatomical pipeline stages

Segmentation

• Select the desired output from the list and click on view:

  

Segmentation results

Surfaces extracted using Freesurfer.

T1 segmented using Freesurfer.

Parcellation

• Select the desired output from the list and click on view:

  

Parcellation results

Cortical and subcortical parcellation are shown with Freeview.

Diffusion pipeline stages

• Click on the stage you wish to check the output(s):

  

  Panel for configuration of diffusion pipeline stages

Preprocessing

• Select the desired output from the list and click on view:

  

Registration

• Select the desired output from the list and click on view:

  

Registration results

Registration of T1 to Diffusion space (b0). T1 in copper overlayed to the b0 image.

Diffusion reconstruction and tractography

• Select the desired output from the list and click on view:

  

Tractography results

DSI Tractography results are displayed with TrackVis.

Connectome

• Select the desired output from the list and click on view:

  

Generated connection matrix

Displayed using a:

1. matrix layout with pyplot

2. circular layout with pyplot and MNE

FMRI pipeline stages

• Click on the stage you wish to check the output(s):

  

  Panel for configuration of fMRI pipeline stages

Preprocessing

• Select the desired output from the list and click on view:

  

Registration

• Select the desired output from the list and click on view:

  

fMRI processing

• Select the desired output from the list and click on view:

  

ROI averaged time-series

Connectome

• Select the desired output from the list and click on view:

  

Generated connection matrix

Displayed using a:

1. matrix layout with pyplot

2. circular layout with pyplot and MNE

Commandline Usage

Introduction

The Connectome Mapper 3 BIDS App takes as principal input the path of the dataset that is to be processed. The input dataset is required to be in valid BIDS format, and it must include at least one T1w or MPRAGE structural image. We highly recommend that you validate your dataset with the free, online BIDS Validator.

Commandline Arguments

The command to run Connectome Mapper 3 follow the BIDS-Apps definition with additional options for loading pipeline configuration files.

Entrypoint script of the BIDS-App Connectome Mapper version v3.0.0-beta-20200227

usage: connectomemapper3 [-h]
                         [--participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]
                         [--session_label SESSION_LABEL [SESSION_LABEL ...]]
                         [--anat_pipeline_config ANAT_PIPELINE_CONFIG]
                         [--dwi_pipeline_config DWI_PIPELINE_CONFIG]
                         [--func_pipeline_config FUNC_PIPELINE_CONFIG]
                         [--number_of_participants_processed_in_parallel NUMBER_OF_PARTICIPANTS_PROCESSED_IN_PARALLEL]
                         [--fs_license FS_LICENSE] [-v]
                         bids_dir output_dir {participant,group}

Positional Arguments

bids_dir	The directory with the input dataset formatted according to the BIDS standard.
output_dir	The directory where the output files should be stored. If you are running group level analysis this folder should be prepopulated with the results of theparticipant level analysis.
analysis_level	Possible choices: participant, group Level of the analysis that will be performed. Multiple participant level analyses can be run independently (in parallel) using the same output_dir.

Named Arguments

--participant_label
	The label(s) of the participant(s) that should be analyzed. The label corresponds to sub-<participant_label> from the BIDS spec (so it does not include “sub-“). If this parameter is not provided all subjects should be analyzed. Multiple participants can be specified with a space separated list.
--session_label
	The label(s) of the session that should be analyzed. The label corresponds to ses-<session_label> from the BIDS spec (so it does not include “ses-“). If this parameter is not provided all sessions should be analyzed. Multiple sessions can be specified with a space separated list.
--anat_pipeline_config
	Configuration .txt file for processing stages of the anatomical MRI processing pipeline
--dwi_pipeline_config
	Configuration .txt file for processing stages of the diffusion MRI processing pipeline
--func_pipeline_config
	Configuration .txt file for processing stages of the fMRI processing pipeline
--number_of_participants_processed_in_parallel
	The number of subjects to be processed in parallel (One core used by default).
--fs_license	Freesurfer license.txt
-v, --version	show program’s version number and exit

Participant Level Analysis

To run the docker image in participant level mode (for one participant):

$ docker run -t --rm -u $(id -u):$(id -g) \
        -v /home/localadmin/data/ds001:/bids_dir \
        -v /media/localadmin/data/ds001/derivatives:/output_dir \
        (-v /usr/local/freesurfer/license.txt:/bids_dir/code/license.txt \)
        sebastientourbier/connectomemapper3:|release \\|
        /bids_dir /output_dir participant --participant_label 01 \(--session_label 01 \)
            --anat_pipeline_config /bids_dir/code/ref_anatomical_config.ini \)
        (--dwi_pipeline_config /bids_dir/code/ref_diffusion_config.ini \)
        (--func_pipeline_config /bids_dir/code/ref_fMRI_config.ini \)
        (--number_of_participants_processed_in_parallel 1)

Note

The local directory of the input BIDS dataset (here: /home/localadmin/data/ds001) and the output directory (here: /media/localadmin/data/ds001/derivatives) used to process have to be mapped to the folders /bids_dir and /output_dir respectively using the -v docker run option.

Important

The user is requested to use its own Freesurfer license (available here). CMP expects by default to find a copy of the FreeSurfer license.txt in the code/ folder of the BIDS directory. However, one can also mount with the -v docker run option a freesurfer license.txt, which can be located anywhere on its computer (as in the example above, i.e. /usr/local/freesurfer/license.txt) to the code/ folder of the BIDS directory inside the docker container (i.e. /bids_dir/code/license.txt).

Note

At least a configuration file describing the processing stages of the anatomical pipeline should be provided. Diffusion and/or Functional MRI pipeline are performed only if a configuration file is set. The generation of such configuration files, the execution of the BIDS App docker image and output inpection are facilitated through the use of the Connectome Mapper GUI, i.e. cmpbidsappmanager (see dedicated documentation page)

Debugging

Logs are outputted into <output dir>/cmp/sub-<participant_label>/sub-<participant_label>_log-cmpbidsapp.txt.

Support and communication

The documentation of this project is found here: https://connectome-mapper-3.readthedocs.io/en/latest/.

All bugs, concerns and enhancement requests for this software can be submitted here: https://gitlab.com/connectomicslab/connectomemapper3/issues.

If you run into any problems or have any questions, you can post to the CMTK-users group.

Not running on a local machine? - Data transfer

If you intend to run connectomemapper3 on a remote system, you will need to make your data available within that system first. Comprehensive solutions such as Datalad will handle data transfers with the appropriate settings and commands. Datalad also performs version control over your data.

Adopting Datalad for collaboration

Datalad is a powerful tool for the versioning and sharing raw and processed data as well as for the tracking of data provenance (i.e. the recording on how data was processed). This page was created with the intention to share with the user how we adopted the use of datalad datasets with the connectome mapper in in our lab at the time of creation of this document (2019 Jan 8). For more details and tutorials on Datalad,please check the recent Datalad Handbook

Warning

This was tested on Ubuntu 16.04 with Datalad 0.11.3 and its extensions datalad-container 0.3.1, datalad_neuroimaging 0.2.0 and datalad_revolution 0.6.0. This example might not work with their latest versions as they are under intensive developement and a number of new versions with minor and major changes have been released in the meantime.

Move original BIDS dataset to server

rsync -P -v -avz -e 'ssh' --exclude 'derivatives' --exclude 'code' --exclude '.datalad' --exclude '.git' --exclude '.gitattributes' /media/localadmin/HagmannHDD/Seb/ds-newtest2/* tourbier@<SERVER_IP_ADDRESS>:/home/tourbier/Data/ds-newtest2

Datalad setup and dataset creation on Server (accessible via ssh)

Connect to server

ssh tourbier@<SERVER_IP_ADDRESS>

Install git-annex and liblzma-dev (datalad dependencies), Datalad and its extensions

sudo apt-get install git-annex liblzma-dev
pip install datalad[all]==0.11.3
pip install datalad-container==0.3.1
pip install datalad_neuroimaging==0.2.0
pip install datalad_revolution==0.6.0

Note

Tested using git-annex version 7.20190219-gad7c11b

Go to source dataset directory, create a Datalad dataset and save all

cd /home/tourbier/Data/ds-newtest2
datalad rev-create -f -D "Original test dataset on lab server"
datalad rev-save -m 'Source (Origin) BIDS dataset' --version-tag origin

Report on the state of dataset content

datalad rev-status --recursive

Processing using the Connectome Mapper BIDS App on a local workstation

Dataset installation

datalad install -s ssh://tourbier@<SERVER_IP_ADDRESS>:/home/tourbier/Data/ds-newtest2  \
/home/localadmin/Data/ds-newtest2

cd /home/localadmin/Data/ds-newtest2

Get T1w and Diffusion images to be processed, written in a bash script for reproducibility

datalad get -J 4 sub-*/ses-*/anat/sub-*_T1w.nii.gz
datalad get -J 4 sub-*/ses-*/dwi/sub-*_dwi.nii.gz
datalad get -J 4 sub-*/ses-*/dwi/sub-*_dwi.bvec
datalad get -J 4 sub-*/ses-*/dwi/sub-*_dwi.bval

Write datalad get commands to get_required_files_for_analysis.sh:

mkdir code
echo "datalad get -J 4 sub-*/ses-*/anat/sub-*_T1w.nii.gz" > code/get_required_files_for_analysis.sh
echo "datalad get -J 4 sub-*/ses-*/dwi/sub-*_dwi.nii.gz" >> code/get_required_files_for_analysis.sh
echo "datalad get -J 4 sub-*/ses-*/dwi/sub-*_dwi.bvec" >> code/get_required_files_for_analysis.sh
echo "datalad get -J 4 sub-*/ses-*/dwi/sub-*_dwi.bval" >> code/get_required_files_for_analysis.sh

Add all content in the code/ directory directly to git:

datalad add --to-git code

Add the container image of the connectome mapper to the dataset

datalad containers-add connectomemapper-bidsapp-|release| \
--url dhub://sebastientourbier/connectomemapper-bidsapp:|release| \
--update

Save the state of the dataset prior to analysis

datalad rev-save -m "Seb's test dataset on local \
workstation ready for analysis with connectomemapper-bidsapp:|release|" \
--version-tag ready4analysis-<date>-<time>

Run Connectome Mapper on all subjects

datalad containers-run --container-name connectomemapper-bidsapp-|release| \
'/tmp' '/tmp/derivatives' participant \
--anat_pipeline_config '/tmp/code/ref_anatomical_config.ini' \
--dwi_pipeline_config '/tmp/code/ref_diffusion_config.ini' \

Save the state

datalad rev-save -m "Seb's test dataset on local \
workstation processed by connectomemapper-bidsapp:|release|, {Date/Time}" \
--version-tag processed-<date>-<time>

Report on the state of dataset content

datalad rev-status --recursive

With DataLad with don’t have to keep those inputs around – without losing the ability to reproduce an analysis.

Let’s uninstall them – checking the size on disk before and after:

datalad uninstall sub-*/*

Local collaboration with Bob for Electrical Source Imaging

Processed dataset installation on Bob’s workstation

datalad install -s (ssh://)localadmin@HOS51827:/home/localadmin/Data/ds-newtest2  \
/home/bob/Data/ds-newtest2

cd /home/bob/Data/ds-newtest2

Get connectome mapper output files (Brain Segmentation and Multi-scale Parcellation) used by Bob in his analysis

datalad get -J 4 derivatives/cmp/sub-*/ses-*/anat/sub-*_mask.nii.gz
datalad get -J 4 derivatives/cmp/sub-*/ses-*/anat/sub-*_class-*_dseg.nii.gz
datalad get -J 4 derivatives/cmp/sub-*/ses-*/anat/sub-*_scale*_atlas.nii.gz

Write datalad get commands to get_required_files_for_analysis_by_bob.sh for reproducibility:

echo "datalad get -J 4 derivatives/cmp/sub-*/ses-*/anat/sub-*_mask.nii.gz" > code/get_required_files_for_analysis_by_bob.sh
echo "datalad get -J 4 derivatives/cmp/sub-*/ses-*/anat/sub-*_class-*_dseg.nii.gz" >> code/get_required_files_for_analysis_by_bob.sh
echo "datalad get -J 4 derivatives/cmp/sub-*/ses-*/anat/sub-*_scale*_atlas.nii.gz" >> code/get_required_files_for_analysis_by_bob.sh

Add all content in the code/ directory directly to git:

datalad add --to-git code

Update derivatives

cd /home/bob/Data/ds-newtest2
mkdir derivatives/cartool ...

Save the state

datalad rev-save -m "Bob's test dataset on local \
workstation processed by cartool:|release|, {Date/Time}" \
--version-tag processed-<date>-<time>

Report on the state of dataset content

datalad rev-status --recursive

With DataLad with don’t have to keep those inputs around – without losing the ability to reproduce an analysis.

Let’s uninstall them – checking the size on disk before and after:

datalad uninstall sub-*/*
datalad uninstall derivatives/cmp/*
datalad uninstall derivatives/freesurfer/*
datalad uninstall derivatives/nipype/*

• Created by Sebastien Tourbier - 2019 Jan 8

Running on a cluster (HPC)

Before the Connectome Mapper 3 BIDS App can be run on a cluster, it first needs to be saved to an Singularity-compatible image file (Please check the Singularity documentation website for more details).

Conversion to a Singularity image

Let’s say we want to store Singularity-compatible image file in ~/Softwares/singularity/ :

$ singularity build ~/Softwares/singularity/cmp-v3.0.0-beta-20200227.simg docker://sebastientourbier/connectomemapper-bidsapp:v3.0.0-beta-20200227

This command will directly download the latest version release of the Docker image from the DockerHub and convert it to a Singularity image.

Running the singularity image

The following example shows how to call from the terminal the Singularity image of the CMP3 BIDS App to perform both anatomical and diffusion pipelines for sub-01, sub-02 and sub-03 of a BIDS dataset whose root directory is located at ${localDir}:

$ singularity run --bind ${localDir}:/bids_dir --bind ${localDir}/derivatives:/output_dir \
~/Softwares/singularity/cmp-v3.0.0-beta-20200227.simg \
/bids_dir /output_dir participant --participant_label 01 02 03 \
--anat_pipeline_config /bids_dir/code/ref_anatomical_config.ini \
--dwi_pipeline_config /bids_dir/code/ref_diffusion_config.ini \
--fs_license /bids_dir/code/license.txt \
--number_of_participants_processed_in_parallel 3

Useful singularity commands

• Display a container’s metadata:

  $ singularity inspect ~/Softwares/singularity/cmp-v3.0.0-beta-20200227.simg
  

• Clean cache:

  $ singularity cache clean
  

Created by Sebastien Tourbier - 2020 Mar 04

BSD 3-Clause License

Copyright (C) 2009-2020, Ecole Polytechnique Fédérale de Lausanne (EPFL) and Hospital Center and University of Lausanne (UNIL-CHUV), Switzerland All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Neither the name of the Ecole Polytechnique Fédérale de Lausanne (EPFL) and Hospital Center and University of Lausanne (UNIL-CHUV) nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL “Ecole Polytechnique Fédérale de Lausanne (EPFL) and Hospital Center and University of Lausanne (UNIL-CHUV)” BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Warning

THIS SOFTWARE IS FOR RESEARCH PURPOSES ONLY AND SHALL NOT BE USED FOR ANY CLINICAL USE. THIS SOFTWARE HAS NOT BEEN REVIEWED OR APPROVED BY THE FOOD AND DRUG ADMINISTRATION OR EQUIVALENT AUTHORITY, AND IS FOR NON-CLINICAL, IRB-APPROVED RESEARCH USE ONLY. IN NO EVENT SHALL DATA OR IMAGES GENERATED THROUGH THE USE OF THE SOFTWARE BE USED IN THE PROVISION OF PATIENT CARE.

Changes

Version 3.0.0-beta-20200227

Date: February 27, 2020

This version addresses multiple issues to make successful conversion and run of the CMP3 BIDS App on HPC (Clusters) using Singularity.

• Revised the build of the master and BIDS App images:

  • Install locales and set $LC_ALL and $LANG to make freesurfer hippocampal subfields and brainstem segmentation (matlab-based) modules working when run in the converted SIngularity image
  • BIDS input and output directories inside the BIDS App container are no longer the /tmp and /tmp/derivatives folders but /bids_dir and /output_dir.

  Warning

  this might affect the use of Datalad container (To be confirmed.)

  • Fix the branch of mrtrix3 to check out
  • Updated metadata

• Fix the configuration of CircleCI to not use Docker layer cache feature anymore as this feature is not included anymore in the free plan for open source projects.

• Improved documentation where the latest version should be dynamically generated everywhere it should appear.

Version 3.0.0-beta-20200206

Date: February 06, 2020

• Implementation of an in-house Nipype interface to AFNI 3DBandPass which can handle to check output as ..++orig.BRIK or as ..tlrc.BRIK (The later can occur with HCP preprocessed fmri data)

Version 3.0.0-beta-20200124

Date: January 24, 2020

• Updated multi-scale parcellation with a new symmetric version:

  1. The right hemisphere labels were projected in the left hemisphere to create a symmetric version of the multiscale cortical parcellation proposed by Cammoun2012.
  2. For scale 1, the boundaries of the projected regions over the left hemisphere were matched to the boundaries of the original parcellation for the left hemisphere.
  3. This transformation was applied for the rest of the scales.

• Updated documentation with list of changes

Citing

If your are using the Connectome Mapper 3 in your work, please acknowledge this software and its dependencies.

To help you to do so, we recommend you to use, modify to your needs, and include in your work the following text:

Results included in this manuscript come from the Connectome Mapper 3 version latest [ref1], a processing pipeline, written in Python which uses Nipype [ref2] [ref3]. It is encapsulated in a BIDS app [ref4] based on Docker [ref5] and Singularity [ref6] container technologies. Resampling to isotropic resolution, Desikan-Killiany brain parcellation [ref7], brainstem parcellation [ref8], and hippocampal subfields segmentation [ref9] were performed using FreeSurfer 6.0.1. Final parcellations were created by performing cortical brain parcellation on at 5 different scales [ref10], probabilistic atlas-based segmentation of the thalamic nuclei [ref11],and combination of all segmented structures, using in-house CMTK tools and the antsRegistrationSyNQuick tool of ANTS v2.2.0 [ref12].

[ref1]

Tourbier S, Aleman-Gomez Y, Griffa A, Bach Cuadra M, Hagmann P (2019, October 8). connectomicslab/connectomemapper3: Connectome Mapper v3.0.0-beta-20200227 (Version v3.0.0-beta-20200227). Zenodo. http://doi.org/10.5281/zenodo.3475969.

[ref2]

Gorgolewski K, Burns CD, Madison C, Clark D, Halchenko YO, Waskom ML, Ghosh SS (2011). Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python. Front Neuroinform, vol. 5, no. 13. doi:10.3389/fninf.2011.00013.

[ref3]

Gorgolewski KJ, Esteban O, Ellis DG, Notter MP, Ziegler E, Johnson H, Hamalainen C, Yvernault B, Burns C, Manhães-Savio A, Jarecka D, Markiewicz CJ, Salo T, Clark D, Waskom M, Wong J, Modat M, Dewey BE, Clark MG, Dayan M, Loney F, Madison C, Gramfort A, Keshavan A, Berleant S, Pinsard B, Goncalves M, Clark D, Cipollini B, Varoquaux G, Wassermann D, Rokem A, Halchenko YO, Forbes J, Moloney B, Malone IB, Hanke M, Mordom D, Buchanan C, Pauli WM, Huntenburg JM, Horea C, Schwartz Y, Tungaraza R, Iqbal S, Kleesiek J, Sikka S, Frohlich C, Kent J, Perez-Guevara M, Watanabe A, Welch D, Cumba C, Ginsburg D, Eshaghi A, Kastman E, Bougacha S, Blair R, Acland B, Gillman A, Schaefer A, Nichols BN, Giavasis S, Erickson D, Correa C, Ghayoor A, Küttner R, Haselgrove C, Zhou D, Craddock RC, Haehn D, Lampe L, Millman J, Lai J, Renfro M, Liu S, Stadler J, Glatard T, Kahn AE, Kong X-Z, Triplett W, Park A, McDermottroe C, Hallquist M, Poldrack R, Perkins LN, Noel M, Gerhard S, Salvatore J, Mertz F, Broderick W, Inati S, Hinds O, Brett M, Durnez J, Tambini A, Rothmei S, Andberg SK, Cooper G, Marina A, Mattfeld A, Urchs S, Sharp P, Matsubara K, Geisler D, Cheung B, Floren A, Nickson T, Pannetier N, Weinstein A, Dubois M, Arias J, Tarbert C, Schlamp K, Jordan K, Liem F, Saase V, Harms R, Khanuja R, Podranski K, Flandin G, Papadopoulos Orfanos D, Schwabacher I, McNamee D, Falkiewicz M, Pellman J, Linkersdörfer J, Varada J, Pérez-García F, Davison A, Shachnev D, Ghosh S (2017). Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in Python. doi:10.5281/zenodo.581704.

[ref4]

Gorgolewski KJ, Alfaro-Almagro F, Auer T, Bellec P, Capota M, Chakravarty MM, Churchill NW, Cohen AL, Craddock RC, Devenyi GA, Eklund A, Esteban O, Flandin G, Ghosh SS, Guntupalli JS, Jenkinson M, Keshavan A, Kiar G, Liem F, Raamana PR, Raffelt D, Steele CJ, Quirion P, Smith RE, Strother SC, Varoquaux G, Wang Y, Yarkoni T, Poldrack RA (2017). BIDS apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods. PLOS Computational Biology, vol.13, no. 3, e1005209. doi:10.1371/journal.pcbi.1005209.

[ref5]

Merkel D (2014). Docker: lightweight Linux containers for consistent development and deployment. Linux Journal, vol. 2014, no. 239. https://dl.acm.org/citation.cfm?id=2600239.2600241

[ref6]

Kurtzer GM, Sochat V, Bauer MW (2017). Singularity: Scientific containers for mobility of compute. PLoS ONE, vol. 12, no. 5, e0177459. doi: 10.1371/journal.pone.0177459

[ref7]

Desikan RS, Ségonne F, Fischl B, Quinn BT, Dickerson BC, Blacker D, Buckner RL, Dale AM, Maguire RP, Hyman BT, Albert MS, Killiany RJ. An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. NeuroImage, vol. 31, no. 3, pp. 968-980. doi:10.1016/j.neuroimage.2006.01.021.

[ref8]

Iglesias JE, Van Leemput K, Bhatt P, Casillas C, Dutt S, Schuff N, Truran-Sacrey D, Boxer A, Fischl B (2015). Bayesian segmentation of brainstem structures in MRI. Neuroimage, vol. 113, pp. 184-195. doi: 10.1016/j.neuroimage.2015.02.065.

[ref9]

Iglesias JE, Augustinack JC, Nguyen K, Player CM, Player A, Wright M, Roy N, Frosch MP, McKee AC, Wald LL, Fischl B, Van Leemput K (2015). A computational atlas of the hippocampal formation using ex vivo, ultra-high resolution MRI: Application to adaptive segmentation of in vivo MRI. Neuroimage, vol. 115, July, pp. 117-137. doi: 10.1016/j.neuroimage.2015.04.042.

[ref10]

Cammoun L, Gigandet X, Meskaldji D, Thiran JP, Sporns O, Do KQ, Maeder P, Meuli RA, Hagmann P (2012). Mapping the human connectome at multiple scales with diffusion spectrum MRI. Journal of neuroscience methods, vol. 203, no. 2, pp. 386-397. doi: 10.1016/j.jneumeth.2011.09.031.

[ref11]

Najdenovska E, Alemán-Gómez Y, Battistella G, Descoteaux M, Hagmann P, Jacquemont S, Maeder P, Thiran JP, Fornari E, Bach Cuadra M (2018). In-vivo probabilistic atlas of human thalamic nuclei based on diffusion- weighted magnetic resonance imaging. Scientific Data, vol. 5, no. 180270. doi: 10.1038/sdata.2018.270

[ref12]

Avants BB, Epstein CL, Grossman M, Gee JC (2008). Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical Image Analysis, vol. 12, no. 1, pp. 26–41. doi:10.1016/j.media.2007.06.004.

Contributing to Connectome Mapper

Authors:Sebastien Tourbier, Adrien Birbaumer Version:Revision: 2 Copyright:Copyright (C) 2017-2019, Brain Communication Pathways Sinergia Consortium, Switzerland. This software is distributed under the open-source license Modified BSD.

Contents

• Contributing to Connectome Mapper
  • Philosophy
  • Enhancements

Philosophy

The development philosophy for this new version of the Connectome Mapper is to:

I. Keep the code of the processing as much as possible outside of the actual main Connectome Mapper code, through the use and extension of existing Nipype interfaces and an external library (dubbed cmtklib).

2. Separation between the code of the graphical interface and the actual main Connectomer Mapper code, achieved through inheritance of the actual main stages and pipelines.
3. Enhance portability by freezing the computing environment with all software dependencies installed, through the adoption of the BIDS App framework relying on light software container technologies.
4. Enhance interoperability by working with datasets structured following the Brain Imaging Data Structure structured dataset.

Further development, typically additions of other tools and configuration options should go in this direction.

Enhancements

Possible enhancements are probably to be included in the following list:

1. Adding of a configuration option to an existing stage
2. Adding a new interface to cmtklib
3. Adding of a new stage
4. Adding of a new pipeline

The adding of newer configuration options to existing stages should be self- understandable. If the addition is large enough to be considered a “sub-module” of an existing stage, see the Diffusion stage example.

Adding a new stage implies the addition of the stage folder to the cmp/stages and cmp/bidsappmanager/stages directory and according modification of the parent pipeline along with insertion of a new image in cmp/bidsappmanager/stages. Copy-paste of existing stage (such as segmentation stage) is recommended.

Adding a new pipeline implies the creation of a new pipeline script and folder in the cmp/pipelines and cmp/bidsappmanager/pipelines directories Again copy-pasting an existing pipeline is the better idea here. Modification of cmp/project.py and cmp/bidsappmanager/project.py file is also needed.

Funding

Work supported by the SNF Sinergia Grant 170873 (http://p3.snf.ch/Project-170873).

License information

This software is distributed under the open-source license Modified BSD. See license for more details.

All trademarks referenced herein are property of their respective holders.

Aknowledgment

If your are using the Connectome Mapper 3 in your work, please acknowledge this software and its dependencies. See Citing for more details.

Eager to contribute?

See Contributing to Connectome Mapper for more details.