Message Passing Interface for ROOT

Mpi Logo


What is mpi?

An Interface Specification:
M P I = Message Passing Interface
MPI is a specification for the developers and users of message passing libraries. By itself, it is NOT a library - but rather the specification of what such a library should be.
Simply stated, the goal of the Message Passing Interface is to provide a widely used standard for writing message passing programs. The interface attempts to be, practical, portable, efficient and flexible.

More information in Message Passing Interface (MPI)


ROOT Mpi is a message passing interface for ROOT based in MPI standard that uses libraries with the implementation like OpenMPI, Mpich, MVAPICH etc..
ROOT Mpi lets you to communicate objects that can be serialized by ROOT like the ROOT and c++ standard classes, also is a modern design that uses the new features from c++ and the ROOT interpreter Cling.
A old prototype was written with Cint but is was deprecated because Cint have so meny limitations respect to cling to use templates and features like RTTI(Run-time type information).

In this new design we provide a set of classes that lets to write parallel code easily without dealing with low level code of standard MPI.


ROOT Mpi is currently under development and we have a prototype to do a proof of concept.
The code can be found at


It can be compiled with OpenMPI or Mpich libraries, to enable it with for compilation in ROOT just run

cmake -Dmpi=ON ..
Plugin execution pending approval

This plugin was recently added or modified. Until an editor of the site validates the parameters, execution will not be possible.

rootmpi (command line tool)

Normally the mpi binaries are execute with the command mpirun, but in ROOTMpi you can
execute ROOT macros without compile it like is usual to run ROOT code.
rootmpi(executable) is a command line tool that helps you run binary code, root macros or compiling source code to generate binary code.
Internally to run a macro it uses an TRint class object to create a ROOT interpreter and call mpirun, to compile it uses gSystem, to call mpic++, and to run a binary executable directly it calls mpirun.

Usage for Macro: rootmpi (mpirun options) (root/cling options) (macro file.C )
Usage for Binary Executable: rootmpi -R (mpirun options) (Binary Executable)
Usage to Compile Code: rootmpi -C (mpic++ options) (Souce1.cxx Source2.cpp ..)
--help-mpic++ show mpi options for compilation
--help-mpirun show mpi options for execution
Options Cint/ROOT:
-b : run in batch mode without graphics
-n : do not execute logon and logoff macros as specified in .rootrc
-q : exit after processing command line macro files
-l : do not show splash screen
-x : exit on exception
dir : if dir is a valid directory cd to it before executing
-memstat : run with memory usage monitoring

Basic prototype examples

In this examples you can see the basic object communication between processes and
an application to TMVA.

Basics in communication

Basic example for p2p communication seding/recieving a ROOT object, a std C++ object and a simple double type.

using namespace ROOT::Mpi;
void p2p()
  TEnvironment env;          //environment to start communication system
  TCommunicator comm;   //Communicator to send/recv messages
  //data to send/recv
  std::map<std::string,std::string> mymap; //std oebjct
  TMatrixD mymat(2,2);                     //ROOT object
  Double_t a;                              //default datatype

	mymat[0][0] = 0.1;
	mymat[0][1] = 0.2;
	mymat[1][0] = 0.3;
	mymat[1][1] = 0.4;
        std::cout<<"Sending scalar = "<<a<<std::endl;
	    std::cout<<"Sending map = "<<mymap["key"]<<std::endl;
        std::cout<<"Sending mat = ";
        std::cout<<"Recieved scalar = "<<a<<std::endl;
        std::cout<<"Received map = "<<mymap["key"]<<std::endl;
        std::cout<<"Received mat = ";

To run the code just save it in a file p2p.C and run it using the command rootmpi that is a tool provided (see rootmpi (command line tool)). The rootmpi executable is a tool that call the cling interpreter with mpirun and provide facilities to execute the macros.

rootmpi -np 2 p2p.C

The output is something like


You can find a broadcast example in $ROOTSYS/tutorials/mpi/

Basic TMVA example

This example is to perform classification with four different methods using TMVA,
the DataLoader object is shared using broadcast and every classficiation method is executed in a defferent process.

#include "TMVA/Factory.h"
#include "TMVA/DataLoader.h"
#include "TMVA/Tools.h"

using namespace ROOT::Mpi;
using namespace TMVA;

void classification()

   TEnvironment env;
   TCommunicator comm;
   auto rank = comm.GetRank();
   if (comm.GetSize() != 4) {
      Error("classification", "Please run wih 4 processors.");

   auto outputFile = TFile::Open(Form("TMVA%d.root", rank), "RECREATE");

   Factory factory("TMVAClassification", outputFile,

   DataLoader dataloader("dataset");

   if (comm.IsMainProcess()) {
      if (gSystem->AccessPathName("./tmva_class_example.root"))    // file does not exist in local directory
         gSystem->Exec("curl -O http://root.cern.ch/files/tmva_class_example.root");

      TFile *input = TFile::Open("./tmva_class_example.root");

      TTree *signalTree     = (TTree *)input->Get("TreeS");
      TTree *background     = (TTree *)input->Get("TreeB");

      dataloader.AddVariable("myvar1 := var1+var2", 'F');
      dataloader.AddVariable("myvar2 := var1-var2", "Expression 2", "", 'F');
      dataloader.AddVariable("var3",                "Variable 3", "units", 'F');
      dataloader.AddVariable("var4",                "Variable 4", "units", 'F');

      dataloader.AddSpectator("spec1 := var1*2",  "Spectator 1", "units", 'F');
      dataloader.AddSpectator("spec2 := var1*3",  "Spectator 2", "units", 'F');

      // global event weights per tree (see below for setting event-wise weights)
      Double_t signalWeight     = 1.0;
      Double_t backgroundWeight = 1.0;

      // You can add an arbitrary number of signal or background trees
      dataloader.AddSignalTree(signalTree,     signalWeight);
      dataloader.AddBackgroundTree(background, backgroundWeight);


      dataloader.PrepareTrainingAndTestTree("", "",                                            "nTrain_Signal=4000:nTrain_Background=4000:SplitMode=Random:NormMode=NumEvents:!V");

   comm.Bcast(dataloader, comm.GetMainProcess());

   if (rank == 0) {
      factory.BookMethod(&dataloader, TMVA::Types::kMLP, "MLP","H:!V:NeuronType=tanh:VarTransform=N:NCycles=600:HiddenLayers=N+5:TestRate=5:!UseRegulator");
   if (rank == 1) {
      factory.BookMethod(&dataloader, TMVA::Types::kBDT, "BDT","!H:!V:NTrees=850:MinNodeSize=2.5%:MaxDepth=3:BoostType=AdaBoost:AdaBoostBeta=0.5");
   if (rank == 2) {
      factory.BookMethod(&dataloader, TMVA::Types::kSVM, "SVM", "Gamma=0.25:Tol=0.001:VarTransform=Norm");
   if (rank == 3) {
      factory.BookMethod(&dataloader, TMVA::Types::kKNN, "KNN","H:nkNN=20:ScaleFrac=0.8:SigmaFact=1.0:Kernel=Gaus:UseKernel=F:UseWeight=T:!Trim");

To run the code just save it in a file classification.C and run it using the command rootmpi that is a tool provided (see rootmpi (command line tool)).

rootmpi -np 4 classification.C

Here we have some times trying to classify the data with different amount of processors. This can be useful for cross validation, resampling, hyper parameter search etc..



requires OpenMpi >= 1.8 and Mpich >= 1.5 that supports Ibcast


Related Sites