95 Parallel Computing (NM72)

Introduction to NONMEM 7.6.0

Author

Robert J. Bauer

Published

June 1, 2025

1 General Concepts of Parallel Computing

If you have a run that takes a long time to estimate, you may submit it for parallel computing. This is the process of splitting the objective function evaluations of individual subjects among a set of computers or CPUs, to speed up analysis of a particular run. Only estimations ($EST) and covariance assessments ($COV) are parallel processed.

From our tests, we have found that the optimal number of processes needed depends on the problem. On one extreme, if the problem contains many subjects, and each subject takes a long time to evaluate because of a large number of differential equations, and/or a large number of dose events, so that one subject takes a minute to evaluate on each function evaluation, then as many cores as there are subjects would still be efficient. Our parallelization algorithm does not split up the problem beyond one subject per process. On the other hand, if the problem takes just 0.01 second to evaluate all subjects for a function evaluation, then it may not be worth using parallel processing. For each function call, the manager process packages a subset of subjects and sends the data to a worker process, then the worker process returns its results to the manager, and the manager summarizes the information from all of the workers. For the next function call, the procedure begins again.

The length of time to perform one subject’s evaluation in a function call varies with the estimation method as well. In importance sampling, there is one function call per iteration, and if you have high ISAMPLE, then it can take some time to evaluate each subject. Such a problem is very efficiently parallelized. On the other hand, BAYES analysis performs only one sample per subject per function call, so it may perform a function evaluation very quickly on a single process, and parallelization may not improve computation time.

NONMEM can parallelize across computers as well as to individual cores on those computers. However, depending on your intranet connection between computers, the process will be a little slower across computers than among cores on the manager computer alone. Eight to 16 cores per computer with about 2 GB RAM per core should be sufficient for almost any problem in NONMEM. Alternatively, 0.4 GB per core is more than enough for many NONMEM problems. If there is insufficient RAM, many operating systems utilize virtual memory (usually mapped to hard drives), but this may slow down execution.

The manager process is the user’s process that runs the nmfe75 script, reads the control stream file, executes NMTRAN, and runs the main NONMEM process. The worker process is NONMEM in worker mode, not taking any input from the user, only from the manager NONMEM process.

If the manager process is on one computer and the worker process is on a second computer, then a network communication must be possible between these computers, and the manager computer must be able to have access to a network drive and directory that is mapped to a drive and directory that is locally accessible by the worker directory. It is possible for this directory to also be accessible from the worker computer as a network drive, but this can slow down the data transfer. If the manager process and the worker process are on the same computer, but are simply running on different cores, then they can communicate on an agreed upon directory on a local drive. Both manager and worker must have read and write privileges.

To obtain the greatest efficiency in parallel computing, make sure the LIM values to buffers 1, 3, 4, 13, and 15 are set to the largest needed for ensuring the buffers can be loaded all into memory, and no file reading and writing is required. See the section I.11 Changing the Size of NONMEM Buffers on how to do this.

2 File Passing Interface (FPI) Method

Two information passing methods between manager and worker processes are available, file passing interface (FPI), and message passing interface (MPI). The FPI method requires no additional software installation other than what is normally required to run a single process NONMEM run (that is, it needs only NONMEM plus compiler). All transfer of information between a manager NONMEM process and its worker processes is done by writing files to a directory throughout the analysis.

3 Message Passing Interface (MPI) method

The message passing interface (MPI) allows exchange of data much more rapidly than the FPI. MPI requires installation of free but ubiquitous use third party software, and we recommend you set this up for your cluster. Fortunately, MPI is free and available for most platforms and Fortran compilers. The MPI’s speed is particularly notable over FPI when FOCE, Laplace, SAEM and BAYES are done. For ITS and IMP/IMPMAP, the speed difference is less noticeable. There is some initial file copying required between manager and worker directories (or computers), but after the initial loading of the NONMEM processes, all information transfer is via the message passing interface without requiring file transfer.

4 The PARAFILE

Parallel computing with NONMEM 7.2.0 uses a “parallel file” (or parafile) that controls the parallelization process implemented by NONMEM, and is written by the user. The NONMEM installed ..\run directory has sample pnm files that can be used as a template. The name of the parallel file may be given at the command line as:

Nmfe75 myexample.ctl myexample.res -parafile=myparallel.pnm

(quotes of some kind may be needed for Windows, otherwise the parameters are improperly parsed). This parallel file will remain in effect throughout the control-stream file, to be used in all $EST methods.

If no -parafile switch was given, then the default name parallel.pnm is assumed. The reserved default name of parallel.pnm should not be used, as it is only for the worker process. Make sure no file called parallel.pnm exists in your manager’s run directory.

The PARAFILE option may be alternatively set to the keywords ON or OFF. If a PARAFILE parameter is set to OFF in a $EST command, then parallelization does not occur for that $EST command. If a subsequent PARAFILE is set to ON, the parallelization occurs using the most recent PARAFILE file specification. If -parafile=off is given at the command line, then no parallelization is done for the entire control stream, regardless of PARAFILE options within the control stream file.

The format of the parallel file is best shown by this example, which is heavily commented to describe the meanings of the records and options available. This parafile example is set up for FPI method on Windows:

$GENERAL
NODES=2 PARSE_TYPE=3 PARSE_NUM=200 TIMEOUTI=60 TIMEOUT=10 PARAPRINT=0
TRANSFER_TYPE=0
; NODES=number of nodes (that is process, whether cores or computers)
; SINGLE node: NODES=1
; MULTI node (node means process, whether cores or computers): NODES>1
; WORKER node: NODES=0
;
; parse_num=number of subjects to give to each node
; parse_type=0, give each node parse_num subjects
; parse_type=1, evenly distribute numbers of subjects among available nodes
; parse_type=2, load balance among nodes
; parse_type=3, assign subjects to nodes based on idranges
; parse_type=4, load balance among nodes, taking into account loading time.
; This setting of parse_type will assess ideal number of nodes.
; If loading time too costly, will eventually revert to single CPU mode.
;
; timeouti=seconds to wait for node to start. if not started in time,
; deassign node, and give its load to next worker, until next iteration
; timeout=minutes to wait for node to complete. As of nm75, only a warning
; is given if node takes longer than timeout minutes to return
; paraprint=1 print to console the parallel computing process. Can be
; modified at run-time with ctrl-B toggle.
; Regardless of paraprint setting, <control_stream>.log always records
; parallelization progress.
;
; transfer_type=0 for file transfer, unloading and reloading workers with
; each estimation
; transfer_type=1 for mpi
; transfer type=2 for file transfer, maintaining a single loaded process
; throughout the run.

;THE EXCLUDE/INCLUDE may be used to selectively use certain nodes,
; out of a large list.
; $EXCLUDE 5-7 ; exclude nodes 5-7
; or
;$EXCLUDE ALL
;$INCLUDE 1,4-6

$NAMES ; Give a label to each node for convenience
1:MANAGER
2:WORKER1
3:WORKER2
4:WORKER3

$COMMANDS ;each node gets a command line, used to launch the node session.
; Command lines must be on one line for each process. The following commands
; are for FPI method on Windows.

; First node is manager, so it does not get a command line when using FPI
1:NONE
;
; load on a core of the same computer as manager:
; For psexec, notice that the worker directories are named
; as the worker sees them, not as the manager sees them. Very important
; distinction for remote worker computers.
; -w refers to working directory for particular process
2:psexec -d -w worker1\ cmd.exe /C nonmem.exe
; load on a core of the same computer as manager:
3:psexec -d -w worker2\ cmd.exe /C nonmem.exe
; load on a core of a different computer than manager:
4:psexec \\any_computer -d -w c:\share\worker3 cmd.exe /C nonmem.exe

$DIRECTORIES ; Names of directories as a manager sees them.
1:NONE ; FIRST DIRECTORY IS THE COMMON DIRECTORY. Make it NONE if no
; common directory is to be used. This is the best option.
2:worker1\ ; NEXT SET ARE THE WORKER directories.
3:worker2\
4:w:\share\worker3\ ; This directory is on a different computer from manager

$IDRANGES ; USED IF PARSE_TYPE=3
1:1,50
2:51,100

You may load the problem as follows:

nmfe75 mycontrol.ctl mycontrol.res -parafile=fpiwini8.pnm

Strictly speaking, drive letter mapping on the manager side is not necessary. One could refer to the network drive as \\any_computer\share\worker3\ instead of w:\share\worker3 in the pnm file.

The most versatile PARSE_TYPE selections are 2 and 4. If you select PARSE_TYPE=0, make sure that PARSE_NUM>=(no. of subjects)/(no. of nodes), otherwise the problem may not run properly. If you select PARES_TYPE=3, make sure all subjects are accounted for in the $IDRANGES listings.

By default, at the beginning of each step (estimation, covariance, wres diagnostics, etc.), the individuals are evenly distributed among the nodes. As of nm742, an option called PARSE_PRESERVE may be set to a non-zero value to have the most efficient ID range settings that were determined from the previous step be used for the first iteration after the next step. For example, if the previous step was estimation, then the ID distributions among the nodes of the last iteration of estimation will be used as the start of covariance assessment. The supposition is that the load distribution that was assessed in estimation should be suitable for covariance assessment. Set the bit for the appropriate previous step:

If Previous estimation was estimation, use its last ID load distribution: set bit 0
If Previous estimation was chain method, use its last ID load distribution: set bit 1
If Previous estimation was covariance, use its last ID load distribution: set bit 2
If Previous estimation was wres assessment, use its last ID load distribution: set bit 3
If Previous estimation was final eta assessment, use its last ID load distribution: set bit 4
If Previous estimation was simulation, use its last ID load distribution: set bit 5
If Previous estimation was nonparametric, use its last ID load distribution: set bit 6

The most sensible is to set bits 0, 2, 3, and 4, as these algorithms tend to provide the most accurate load distribution assessments, suitable for launching the next step with the same initial load distribution (PARSE_PRESERVE=29). After several iterations, each step empirically adjusts its load distribution anyway, regardless of PARSE_PRESERVE setting.

The $NAMES record is optional. If left out, or if a name is not defined for a process, the default name is MANAGER for position 1, WORKER1 for position 2, WORKER2 for position 3, etc.

The structure of the COMMANDS lines for launching the worker nodes is completely dependent on your computing and parallel distribution environment, and the syntax requirements of the launching program. The psexec.exe program (located in the ..\run directory of the NONMEM folder) is available for Windows to launch a program on the same computer (as with the first 2 worker nodes), or on a remote computer (last worker node). An alternative launching program may be used. The -w option in psexec specifies the working directory (as the worker identifies it) from which the NONMEM programs is to be launched.

The index numbers that begin an item in a list (1:, 2:, etc), are optional. If present, it refers to node 1 (manager), node 2, node 3, etc. If not present, the item number is determined by the order in which the item was listed. It is best to use them for greater clarity.

In $DIRECTORIES, the directory names must follow syntax rules of the particular operating system. The $DIRECTORIES record is optional. If left out, or if a directory name is not given for a process. Then the default values are NONE for common directory (position 1), worker1 for the first worker (position 2), worker2 for the second worker (position 3), etc. These are interpreted as sub-directories to the present run directory.

There is no need to create the worker directories ahead of time (although its parent directory, whether local or network, must exist), or be concerned with populating them with the appropriate files, including the nonmem executable. NONMEM will take care of this automatically. For example, while w:\share needs to exist before the run, as it was the share directory that needed to be set up, w:\share\worker3 did not have to exist before the NONMEM run. Make sure that the managers and workers have appropriate read/write access to these directories, and proper privileges to load on remote computers.

The $COV statement also allows a PARAFILE setting, to turn on or off parallel computing for the $COV step for classical NONMEM methods, or changing the parallelization profile. The $TABLE statement also allows a PARAFILE setting, to turn on or off parallel computing for calculation of weighted residuals, or changing the parallelization profile (nm74).

The $SIML and $NONP also have PARAFILE settings to turn on or off parallel computing for their calculations.

Sometimes the parallelization log files can become very large during the $EST and $COV steps. Each of these records have parafprint options to control the print intervals. For example.

$EST METHOD= BAYES parafprint=100

will cause only every 100th iteration to be printed.

Or, you can control the print iterations globally with the -parafprint option at the command line:

nmfe75 mycontrol.ctl myconotrl.res -parafile=mpiwini8.pnm -parafprint=100

Examples of PARAFILE files are given in NONMEM’s ..\run directory as a list of *.pnm files. Examples are shown in the next sections as well. The files fpiwini8.pnm, fpilinux8.pnm, mpilinux8.pnm, and fpilinux8.pnm are particularly versatile, in that they are useful for multiple cores on a single computer, and are designed to be used in any run directory.

5 Substitution Variables in the parafile

Substitution variables provide flexibility in the use of the parafile. Certain substitution variables are reserved words as follows, which can be passed as arguments to the worker nonmem executable (although typically this is not necessary to do so). That is, they are placed at the end of a $COMMANDS process command line, coming after nonmem.exe, as arguments to nonmem.exe, as needed:

<control_stream>: substitute the control stream file name given at the command line of the nmfe75 script.

<licfile>: substitute the entire -licfile option, including its value, provided by the nmfe75 script. For example, -licfile=c:\mynonmem\license\nmlicense.lic is substituted into <licfile>.

<background>: substitute -background switch, if given by user on the nmfe75 command line.

<parafile>: substitute -parafile option, such as -parafile=myparallel.pnm, given at nmfe75 command line. Never use the <parafile> switch on a worker process.

Substitution variables need not be used just as arguments to the nonmem executables that are loaded. In some cases, they are needed in other parts of the command line of the process launch, or in the directory listing of $DIRECTORIES. In such cases, it is not desired to substitute the entire -option=value string, but just the value portion. Where the value of the option itself is to be substituted, use <<option>>. For example, suppose the nmexec option is used to specify an alternative nonmem executable name. In such cases, you would specify <<nmexec>> in place of the usual nonmem.exe:

3:psexec -d -w worker2\ cmd.exe /C <<nmexec>> <control_stream>

This principle of using <> versus <<>> applies to the other substitution parameters as well.

You may also define your own substitution parameters to be used in the pnm file, as long as the substitution variable begins with a [ or <. For example, you may enter at the command line of nmfe75 the following variable [wd] for a worker directory definition:

Nmfe75 mycontrol.ctl mycontrol.res -parafile=mypara.pnm [wd]=c:\myworker

and your pnm file may contain the following loading $COMMANDS:

2:psexec -d -w [wd]\q1 cmd.exe /C nonmem.exe
3:psexec -d -w [wd]\q2 cmd.exe /C nonmem.exe

and $DIRECTORIES

2:[wd]\q1
3:[wd]\q2

For user defined variables, the value of the variable is substituted into the placeholder, rather than the entire [var]=value. Then c:\myworker will be substituted in place of [wd], in the $COMMANDS and $DIRECTORIES entries. Add as many substitution variables as you need to create a generalized pnm file.

To make the user substitution process even more flexible, default values for these variables may be defined, in case the user does not specify a value for it on the command line. For example, in ..\run\fpiwini8.pnm, There is a section called $DEFAULTS (or $DEFAULT), where a default value for [nodes] is given:

$DEFAULTS
[nodes]=8

, and in $GENERAL, [nodes] is used as the number of nodes:

$GENERAL
; [nodes] is a User defined variable
NODES=[nodes] PARSE_TYPE=2 PARSE_NUM=50 TIMEOUTI=500 TIMEOUT=2000 PARAPRINT=0
TRANSFER_TYPE=0

Make sure that $DEFAULTS is placed at the head of the file, so the default variable substitution value is available to the parafile interpreter by the time it needs to use it in the rest of the parafile.

In addition, if a file called defaults.pnm exists in the run directory, it may list alternative defaults that override those in the parafile, such as:

$DEFAULTS
[nodes]=2

The defaults.pnm file is expected to have only entries for $DEFAULTS, and no other parafile records. The order of override is:

Command line on nmfe75 script overrides
defaults.pnm, which overrides
defaults defined in parafile.

The advantage to this ordering is that a generic parafile file can be created for most environments. A user may then override defaults specified in this generic parafile with his own in defaults.pnm, that may be more suitable to his environment. Finally, a user can temporarily override his own defaults by giving an alternative value as an nmfe75 script command option. For example, the *8.pnm files listed in the NONMEM ..\run directory serve as generic parafiles that can be run for up to 8 nodes on a multi-core single computer system. Also in the NONMEM ..\run directory there is an example defaults.pnm file that has [nodes]=2 defined as a default. If this file were placed in the user’s run directory, and the user used fpiliwini8.pnm as a parafile:

nmfe75 mycontrol.ctl mresults.res -parafile=fpiwini8.pnm

then the number of nodes would be that given in defaults.pnm, nodes=2. The user may override this by specifying an alternative number of nodes on the command line:

nmfe75 mycontrol.ctl mresults.res -parafile=fpiwini8.pnm [nodes]=4

in which case the first 4 nodes (or node numbers 1, 2, 3, 4) listed in $COMMANDS and $DIRECTORIES would be executed.

To also make distinct commands easy to write when launching many processes, number list substitution can also be performed. For example,

$GENERAL
NODES=8 PARSE_TYPE=4 PARSE_NUM=200 TIMEOUTI=600 TIMEOUT=1000 PARAPRINT=0
TRANSFER_TYPE=1

$NAMES ;Give a name to each node, which is displayed
1:MANAGER
2-8:WORKER{10-17}

$COMMANDS ;each node gets a command line, used to launch the node session
; %cd% refers to current directory
; Beyond the first position, a ; will not be interpreted as a comment for
; commands
1:mpiexec -wdir "%cd%" -hosts 1 localhost 1 nonmem.exe %*
2-8:-wdir "%cd%\wk{#-1}" -hosts 1 localhost 1 nonmem.exe

$DIRECTORIES
1:NONE ; FIRST DIRECTORY IS THE COMMON DIRECTORY
2-8:wk{#-1} ; NEXT SET ARE THE WORKER directories

In the above example, the name of processes 2 through 8 are given as:

2-8:WORKER{10-16}

In this case, each number represented in the list within the braces {} is expanded and matched with the process number, so this line is equivalent to:

2:WORKER10
3:WORKER11
4:WORKER12
5:WORKER13
6:WORKER14
7:WORKER15
8:WORKER16

Make sure that the number of items represented in the number list in the braces is at least as many as the number list before the colon. Another example:

2,4,7:WORKER{1-3}

Expands to

2:WORKER1
4:WORKER2
7:WORKER3

Another method is to use the expression {#offset}, which directly substitutes the process number listed before the colon into the place at the braces, with an offset added to it. So,

2-8:-wdir "%cd%\wk{#-1}" -hosts 1 localhost 1 nonmem.exe

Expands to

2:-wdir "%cd%\wk1"   -hosts 1 localhost 1 nonmem.exe
3:-wdir "%cd%\wk2"   -hosts 1 localhost 1 nonmem.exe
4:-wdir "%cd%\wk3"   -hosts 1 localhost 1 nonmem.exe
5:-wdir "%cd%\wk4"   -hosts 1 localhost 1 nonmem.exe
6:-wdir "%cd%\wk5"   -hosts 1 localhost 1 nonmem.exe
7:-wdir "%cd%\wk6"   -hosts 1 localhost 1 nonmem.exe
8:-wdir "%cd%\wk7"   -hosts 1 localhost 1 nonmem.exe

Similarly,

2,4,7:-wdir "%cd%\wk{#+11}" -hosts 1 localhost 1 nonmem.exe

Expands to:

2:-wdir "%cd%\wk13"  -hosts 1 localhost 1 nonmem.exe
4:-wdir "%cd%\wk15"  -hosts 1 localhost 1 nonmem.exe
7:-wdir "%cd%\wk18"  -hosts 1 localhost 1 nonmem.exe

6 Easy to Use Parafiles

For easy use, there are a series of pnm files in the ..\run directory that can take any number of cores on a single computer. These are fpiwini8.pnm, mpiwini8.pnm, fpilinux8.pnm, and mpilinux8.pnm (for MAC OSX, use the *linux8.pnm files), located in the NONMEM ..\run directory. The 8 refers to the default number of nodes (processes) being 8, if it is not specified on the command line, or in a defaults.pnm file. An example of its use is as follows:

Nmfe75 foce_parallel.ctl foce_parallel.res -parafile=mpiwini8.pnm [nodes]=4

The example control stream file foce_parallel.ctl is in the ..\examples directory.

7 Windows

Setting up a network drive on Windows for multiple Computers:

Both FPI and MPI methods require the user to set up network drives to pass files between manager and worker computers. If you are running your multiple process on multiple cores of just a single computer, then you may skip this section.

From the worker computer, select a directory (or create a directory) which you would like to have shared with the manager computer. Suppose it is called c:\share. On windows XP, open “my computers”, or right click on Start ->Explore, go to directory tree, right click on c:\share, select properties, then select Sharing, and click on share this folder. On other Windows systems, there may be a different menu path to follow. A suggested share name will be given. You may keep this as is, or change to a name you prefer. Click on Permissions, for user Everyone select Full control, click on apply. Consult your IT representative if you are not able to obtain privileges.

From the manager computer, right click on the my computer icon and select map network drive. Select an available drive letter, which for this example will be w. Then enter \\, the computer name of the remote computer, or its IP address. This is followed by a \ and a share name of an accessible directory. For this example, the computer name is any_computer, and the share name of the directory is share, so enter

\\any_computer\share

Thus, from the manager side, drive w: will be associated with \\any_computer\share, which is in fact c:\share as seen by the worker computer. You may be asked to enter username and password.

7.1 Psexec

Setting up FPI on Windows:

A versatile loading program called psexec.exe (freeware, from www.sysinternals.com), supplied with the NONMEM installation in the ..\run directory, can be used, that allows one to load processes locally or on other computers. You may choose alternative loading programs. Copy psexec.exe from the NONMEM’s ..\run directory to your managers run directory. From a DOS console window, type

Psexec

to see the parameters options for this launching program.

To test that your manager computer can load the NONMEM program on the worker computer (if different from manager), copy a computername.exe from NONMEM’s .\run directory (we shall assume it is named NONMEM7.2.0) to the network mapped directory that is local to the worker.

Copy \nonmem7.2.0\run\computername.exe w:\share

Then type from the manager console window:

Psexec \\any_computer c:\share\computername.exe

(remember, these are just example names of computers and network share directories. Your particular environment will be different). The computer name of the worker computer should be displayed. You may be required to enter a user name and password. If this is the case, you should make sure that your user account and password on your manager computer is the same as on the worker computer, so that user name and password is not requested. Otherwise, when you run the NONMEM program, the run will be continually interrupted for this information.

During the parallelization process, NONMEM sends a copy of its program (nonmem.exe on Windows, nonmem on Linux) to the worker processes’s directory, and then loads it there. Therefore, the worker computers must typically be of the same operating system (although not necessarily same version) as the manager computer (but see below to get around this). The worker computer does not have to have Intel or gfortran installed.

For a quick test on a single multi-core computer, try the following. Copy foce_parallel.ctl and example1.csv from the NONMEM ..\examples directory, fpiwini8.pnm from the NONMEM ..\run directory, and psexec.exe from the NONMEM ..\run directory, into your standard run directory. Then, execute the following from your standard run directory:

Nmfe75 foce_parallel.ctl foce_parallel.res -parafile=fpiwini8.pnm [nodes]=4

where the values of [nodes] should be no greater than the number of cores available on your computer.

A parafile example set up for FPI method on Windows is as follows (set TRANSFER_TYPE=0):

$GENERAL
NODES=2 PARSE_TYPE=3 PARSE_NUM=200 TIMEOUTI=60 TIMEOUT=10 PARAPRINT=0
TRANSFER_TYPE=0
; NODES=number of nodes (that is process, whether cores or computers)
; SINGLE node: NODES=1
; MULTI node (node means process, whether cores or computers): NODES>1
; WORKER node: NODES=0
;
; parse_num=number of subjects to give to each node
; parse_type=0, give each node parse_num subjects
; parse_type=1, evenly distribute numbers of subjects among available nodes
; parse_type=2, load balance among nodes
; parse_type=3, assign subjects to nodes based on idranges
; parse_type=4, load balance among nodes, taking into account loading time.
; This setting of parse_type will assess ideal number of nodes.
; If loading time too costly, will eventually revert to single CPU mode.
;
; timeouti=seconds to wait for node to start. if not started in time,
; deassign node, and give its load to next worker, until next iteration
; timeout=minutes to wait for node to complete. As of nm75, only a warning
; is given if node takes longer than timeout minutes to return
; paraprint=1 print to console the parallel computing process. Can be
; modified at run-time with ctrl-B toggle.
; Regardless of paraprint setting, <control_stream>.log always records
; parallelization progress.
;
; transfer_type=0 for file transfer, unloading and reloading workers with
; each estimation
; transfer_type=1 for mpi
; transfer type=2 for file transfer, maintaining a single loaded process
; throughout the run.

;THE EXCLUDE/INCLUDE may be used to selectively use certain nodes,
; out of a large list.
; $EXCLUDE 5-7 ; exclude nodes 5-7
; or
;$EXCLUDE ALL
;$INCLUDE 1,4-6

$NAMES ; Give a label to each node for convenience
1:MANAGER
2:WORKER1
3:WORKER2
4:WORKER3

$COMMANDS ;each node gets a command line, used to launch the node session.
; Command lines must be on one line for each process. The following commands
; are for FPI method on Windows.

; First node is manager, so it does not get a command line when using FPI
1:NONE
;
; load on a core of the same computer as manager: Note that worker does not
; really need a control stream file, but something must be there as a place
; holder. Also, for psexec, notice that the worker directories are named
; as the worker sees them, not as the manager sees them. Very important
; distinction for remote worker computers.
; -wdir refers to working directory for particular process
; do not user %cd% with psexec. Just user relative directory notation
2:psexec -d -w worker1 cmd.exe /C nonmem.exe
; load on a core of the same computer as manager:
3:psexec -d -w worker2\ cmd.exe /C nonmem.exe
; load on a core of a different computer than manager:
4:psexec \\any_computer -d -w c:\share\worker3 cmd.exe /C nonmem.exe

$DIRECTORIES ; Names of directories as a manager sees them.
1:NONE ; FIRST DIRECTORY IS THE COMMON DIRECTORY. Make it NONE if no
; common directory is to be used. This is the best option.
2:worker1\ ; NEXT SET ARE THE WORKER directories.
3:worker2\
4:w:\share\worker3\ ; This directory is on a different computer from manager

$IDRANGES ; USED IF PARSE_TYPE=3
1:1,50
2:51,100

After an estimation step is performed, the worker processes exit. For the next estimation step that follows (if there is one), the manager will reload the worker processes.

For the FPI method with TRANSFER_TYPE=0, a PARAFILE file name may be given specific to a $EST command:

$EST METHOD=IMP INTERACTION NITER=20 PARAFILE=myparallel_imp.pnm
$EST METHOD=1 INTERACTION PARAFILE=myparallel_foce.pnm

If no parallel file is given for an estimation method, it takes the PARAFILE name of the previous $EST command. If no PARAFILE option was given for the first $EST method, then it takes the value given in the command line switch -parafile. If no -parafile switch was given, then the default name parallel.pnm is assumed. If parallel.pnm file does not exist, then NONMEM runs on a single CPU.

If you want worker processes to remain resident until all estimations and problems listed in the control stream file are completed, then select TRANSFER_TYPE=2. In these cases, new PARAFILE settings at $EST steps within the control stream file will be ignored, except for PARAFILE=ON or PARAFILE=OFF.

7.2 Installing MPI on Windows

Go to the web site

https://www.mpich.org/downloads/

and select the suitable Windows version, with extension .msi. Or, select the mpich2-1.2.1p1-win-ia32.msi file listed in https://nonmem.iconplc.com/mpich2/ . Install the full version on the manager computer by double clicking on the .msi file, or running it from START->run. Follow the instructions in section 7 of mpich2-1.2.1-windevguide.pdf, and verify that the MPI system is working. Copy the program mpiexec.exe from the bin directory of the MPICH2 directory, to your manager NONMEM run directory.

NONMEM comes with the MPI library files (they are located in ..\mpi\MPI_WINI for Intel Fortran and ..\mpi\MPI_WING for gfortran). For communication across computers, make sure you also have a network file allocated, as described above. If the MPI library files do not match the version which you downloaded, or there are linking difficulties when you run nmfe75.bat, then copy the appropriate .lib file from the MPICH2 installed directory mpich2\lib to ..\mpi\MPI_WINI directory. Keep in mind that we have supplied 32 bit versions of libraries. Environments with 64 bit processing may require libraries from the mpich2 web site.

The MPI Windows installation guide (section 9) may offer other ways to supply user name and password via the program mpiexec. For example, from the manager computer

mpiexec -register
Enter name
Enter password.

In addition, the MPI system needs certain executable files available on the worker computer. A minimal installation on the worker computer can be implemented by copying smpd.exe (found in the bin directory of you manager’s MPICH2 directory) to the worker computer, and executing

Smpd.exe -install

See section 9 of the MPI Windows installation guide about the full use of smpd.exe.

Also, the MPI system needs certain dll library files placed in each worker processor’s directory of the worker computer, or in the windows\system32 directory (more generally, in %systemroot%\system32):

Fmpich2.dll (intel) or fmpich2g.dll (gfortran)
Mpich2.dll
Mpich2mpi.dll

The dll files are located in the manager’s %systemroot%\system32 directory.

Next, make sure the library file in the NONMEM system is the same as that of your MPICH2 system. For example, for Intel fortran:

cd \nm750\mpi\mpi_wini
cp fmpich2.lib fmpich2_orig.lib
cp "\program files\MPICH2\lib\fmpich2.lib"

For gfortran:

cd \nm750\mpi\mpi_wing
cp libfmpich2g.a libfmpich2g_orig.a
cp "\program files\MPICH2\lib\libfmpich2g.a"

In addition, the file PNM_MPI.f90 may need to be newly compiled with the appropriate mpi.mod file present. Sometimes there will be an appropriate gfortran or intel fortran based mpi.mod file in the appropriate ..\lib directory of the MPI installation, and you just copy it over:

gfortran:

cd \nm75g65\mpi_mpi_wing\
copy "c:\program files (x86)\MPICH2\lib\mpi.mod"
copy mpi.mod ..\..\resource

intel fortran:

cd \nm75g64\mpi_mpi_wini\
copy "c:\program files (x86)\MPICH2\lib\mpi.mod"
copy mpi.mod ..\..\resource

Or you may have to build it from a file called mpi.h or mpi.f, probably located in the ..\include folder of the MPI installation.

gfortran:

gfortran -c -fno-range-check mpi.f90
copy mpi.mod ..\..\resource
gfortran -c -O3 -ffast-math -m64 -mpc64 -I../../resource -J../../resource PNM_MPI.f90

(note you may want to use whatever compile options you used for the main NONMEM installation).

Intel fortran:

ifort /c mpi.f90
copy mpi.mod ..\..\resource
ifort /c PNM_MPI.f90 /I:..\..\resource

(note you may want to use whatever compile options you used for the main NONMEM installation).

Other MPI software are available, such as OpenMpi, and Microsoft MPI for Windows environment. In fact, MPICH may no longer work for Windows 10 and above. Instructions and materials for Microsoft MPI installation are located at https://nonmem.iconplc.com/msmpi

As an added precaution that your NONMEM installation will use the correct MPI setup (and compiler), you may set up a bash script, called nmloc, in which you establish the following environment variables, and place nmloc.bat in the top NONMEM installation directory. See the end of section I.9 Invoking NONMEM regarding the nmloc system.

Once you have an MPI system set up, for a quick test on a single multi-core computer, try the following. Copy foce_parallel.ctl and example1.csv from the NONMEM ..\examples directory, mpiwini8.pnm from the NONMEM ..\run directory, and mpiexec.exe from the NONMEM ..\run directory, into your standard run directory. Then, execute the following from your standard run directory:

Nmfe75 foce_parallel.ctl foce_parallel.res -parafile=mpiwini8.pnm [nodes]=4

where the values of [nodes] should be no greater than the number of cores available on your computer.

For instructional purposes, a typical structure of a PARAFILE is listed below that would be used for NONMEM on Windows using MPI (note the setting of TRANSFER_TYPE=1):

$GENERAL
NODES=2 PARSE_TYPE=3 PARSE_NUM=200 TIMEOUTI=60 TIMEOUT=10 PARAPRINT=0
TRANSFER_TYPE=1 COMPUTERS=2

; NODES=number of nodes (that is process, whether cores or computers)
; SINGLE node: NODES=1
; MULTI node (node means process, whether cores or computers): NODES>1
; WORKER node: NODES=0
;
; parse_num=number of subjects to give to each node
; parse_type=0, give each node parse_num subjects
; parse_type=1, evenly distribute numbers of subjects among available nodes
; parse_type=2, load balance among nodes
; parse_type=3, assign subjects to nodes based on idranges
; parse_type=4, load balance among nodes, taking into account loading time.
; This setting of parse_type will assess ideal number of nodes.
; If loading time too costly, will eventually revert to single CPU mode.
;
; timeouti=seconds to wait for node to start. if not started in time,
; deassign node, and give its load to next worker, until next iteration
; timeout=minutes to wait for node to complete. As of nm75, only a warning
; is given if node takes longer than timeout minutes to return
; paraprint=1 print to console the parallel computing process. Can be
; modified at run-time with ctrl-B toggle.
; Regardless of paraprint setting, <control_stream>.log always records
; parallelization progress.
;
; transfer_type=0 for file transfer, unloading and reloading workers with
; each estimation
; transfer_type=1 for mpi
; transfer type=2 for file transfer, maintaining a single loaded process
; throughout the run.

;THE EXCLUDE/INCLUDE may be used to selectively use certain nodes,
; out of a large list.

 $EXCLUDE 5-7 ; exclude nodes 5-7
; or
;$EXCLUDE ALL
;$INCLUDE 1,4-6

$NAMES ; Give a name to each node, which is displayed
1:MANAGER
2:WORKER1
3:WORKER2

$COMMANDS ;each node gets a command line, used to launch the node session
; The first one launches the manager's NONMEM.
; -wdir refers to working directory for particular process
; %* mean to transfer all options from command line to
; manager process's nonmem.exe
1:mpiexec -wdir "%cd%" -hosts 1 localhost 1 -noprompt nonmem.exe %*
; the next one launches a worker process on the manager's computer
; the worker only needs certain of the parameters from the command line.
2:-wdir "%cd%"\worker1 -hosts 1 localhost 1 -noprompt nonmem.exe
;
; This launches a worker process on a separate computer.
3:-wdir c:\share\worker3 -n 1 -host any_worker -noprompt c:\share\worker3\nonmem.exe

$DIRECTORIES
1:NONE ; FIRST DIRECTORY IS THE COMMON DIRECTORY
2:worker1\ ; NEXT SET ARE THE WORKER directories
3:w:\share\worker3\

$IDRANGES ; USED IF PARSE_TYPE=3
1:1,50
2:51,100

If Previous estimation was estimation, use its last ID load distribution: set bit 0
If Previous estimation was chain method, use its last ID load distribution: set bit 1
If Previous estimation was covariance, use its last ID load distribution: set bit 2
If Previous estimation was wres assessment, use its last ID load distribution: set bit 3
If Previous estimation was final eta assessment, use its last ID load distribution: set bit 4
If Previous estimation was simulation, use its last ID load distribution: set bit 5
If Previous estimation was nonparametric, use its last ID load distribution: set bit 6

An additional setting in $GENERAL is introduced, called COMPUTERS. By default COMPUTERS is equal to 1. However, if you are running MPI method on Windows, and you have at least one of the worker processes on another computer, and your LIM values are not maximized, so that some file buffers are being used, then you may need to set COMPUTERS=2. If you obtain a read/write error on FILE10, or other FILEXX error, then set COMPUTERS=2.

Unlike FPI, the MPI system can only use the starting parallel.pnm file specified at the command line, and it may not be easily switched later in the control stream. All processes remain resident throughout the entire job, although it will honor requests of parafile=off or parafile=on at individual $EST records, which allows you to have control of which estimation method will use parallel processing.

In the FPI method, the manager NONMEM process has total control of loading followed by implementing all the workers, and is in fact loaded before the pnm file is interpreted and acted upon. With MPI, the mpi system has control, and the manager NONMEM program is just the first of a set of processes. The mpi system is first loaded using a DOS batch file called nmmpi.bat (constructed by the nmfe75 script by a call to nonmem_mpi), and with commands constructed from the $COMMANDS entries in the pnm file. The mpi program loads all the processes, including the manager. Therefore the manager’s $COMMANDS entry has to have all of the parameters passed to it that was entered at the nmfe75 command line by the user, as shown in the example above, by using %*.

For the Windows version of MPI, sometimes you have to specify the full file path of the nonmem.exe program when launching on a remote computer.

8 Linux

Setting up share directory, and ssh on a Linux System

The ssh system and share directory used to pass files between worker and manager must be set up for FPI and MPI methods, if the worker computer differs from the manager computer. The following instructions serve only as a guide as to how to set up the ssh system. You may need to vary some of the commands to suit your environment. Consult your Linux user manual as well.

The network files system (NFS) is used for the manager computer to access a network drive that points to a worker computer’s local drive. Consider the following example.

From the worker computer, create a share directory, such as:

mkdir /home/myself/share

Next, use your editor, and sudo privilege, to modify the /etc/hosts file,

sudo gedit /etc/hosts

And map IP address to computer names:

127.0.0.1 localhost
192.168.1.3 my_manager
192.168.1.2 any_computer

Then save and exit. Use your editor to edit /etc/exports:

sudo gedit /etc/exports

Add the following line:

/home/myself/share 192.168.1.0/24(rw,sync)

Which allows IP addresses 192.168.1.0 through 192.168.1.255 to access this share directory.

Then exit.

sudo exportfs -a

Stop and restart NFS system (this is for Ubuntu: the command may differ on your computer)

sudo /etc/init.d/nfs-kernel-server Stop
sudo /etc/init.d/nfs-kernel-server restart

Go to the manager computer, and also place computer names to IP address mapping in /etc/hosts:

127.0.0.1 localhost
192.168.1.3 my_manager
192.168.1.2 any_computer

Then, create a mount drive for the remote directory:

mkdir /mnt/share
sudo gedit /etc/fstab

Enter the mount drive entry for the remote directory:

any_computer:/home/myself/share /mnt/share nfs rw,sync 0 0

and exit the editor. Then,

sudo mount /mnt/share

Test by copying a file from the manager to the worker:

cp myfile /mnt/share

Next, the ssh component must be set up. Check that you have ssh installed on both manager and worker computers: From the manager, run the standard Linux date program on the worker computer:

ssh -n any_computer date
enter password

If the date is returned from the worker computer, you have ssh connection. You might have to enter user account name:

ssh -n my_account@any_computer date

For ssh to work in parallel computing, you need to set up ssh so it does not always ask for your password. From the manager computer:

ssh-keygen -t dsa

Respond yes to writing to ~/.ssh, and enter in a passphrase. Copy id_dsa.pub from the manager to the worker computer (possibly via the share drive you had set up):

cp ~./ssh/id_dsa.pub /mnt/share

Then concatenate this manager created id_dsa.pub to the authorized_keys file on the worker computer:

cd $HOME
chmod +w .ssh/authorized_keys
touch .ssh/authorized_keys
cat id_dsa.pub >> .ssh/authorized_keys
chmod 400 .ssh/authorized_keys

From the manager computer, repeat the command

ssh -n any_computer date

it should ask you for the pass-phrase, then give you the date.

Do it again:

ssh -n any_computer date

the pass phrase should not be requested this time, nor should a password be requested, and a date from the worker computer should return.

During the parallelization process, NONMEM sends a copy of its program to the worker computer, and then loads it there. Therefore, the worker computers must be of the same operating system (although not necessarily same version) as the manager computer. For Intel fortran, the worker computer does not have to have Intel Fortran installed. For gfortran, -static option for the FPI is used in the nmfe75 script, which makes gfortran portable to the worker computer without requiring the gfortran share library (libgfortran.so.3). If for some reason you needed to remove the -static option, then gfortran requires its share library available for the worker process, and in the path designated by the manager’s LD_LIBRARY_PATH setting, such as:

LD_LIBRARY_PATH="$HOME/gcc-trunk/lib:$HOME/libgf:$LD_LIBRARY_PATH"
Export LD_LIBRARY_PATH

where $HOME/gcc-trunk/lib is the library path for the manager’s gfortran, and $HOME/libgf is the path on the worker computer containing at least the file libgfortran.so.3. You may place these lines in the .bashrc file. Therefore, if upon loading NONMEM on the worker computer, a message is displayed indicating that certain share files are missing, etc., then you may need to either install gfortran, or selectively make the share file available.

8.1 Setting up FPI on Linux

For a quick test on a single multi-core computer, try the following. Copy foce_parallel.ctl and example1.csv from the NONMEM ..\examples directory, fpilinux8.pnm from the NONMEM ..\run directory, and beolaunch.sh from the NONMEM ..\run directory, into your standard run directory. Then, execute the following from your standard run directory:

Nmfe75 foce_parallel.ctl foce_parallel.res -parafile=fpilinux8.pnm [nodes]=4

where the values of [nodes] should be no greater than the number of cores available on your computer.

For instructional purposes, here is an example pnm file for FPI on Linux systems (note TRANSFER_TYPE=0):

$GENERAL
NODES=3 PARSE_TYPE=2 PARSE_NUM=50 TIMEOUTI=300 TIMEOUT=20 PARAPRINT=0
TRANSFER_TYPE=0

; NODES=number of nodes (that is process, whether cores or computers)
; SINGLE node: NODES=1
; MULTI node (node means process, whether cores or computers): NODES>1
; WORKER node: NODES=0
;
; parse_num=number of subjects to give to each node
; parse_type=0, give each node parse_num subjects
; parse_type=1, evenly distribute numbers of subjects among available nodes
; parse_type=2, load balance among nodes
; parse_type=3, assign subjects to nodes based on idranges
; parse_type=4, load balance among nodes, taking into account loading time.
; This setting of parse_type will assess ideal number of nodes.
; If loading time too costly, will eventually revert to single CPU mode.
;
; timeouti=seconds to wait for node to start. if not started in time,
; deassign node, and give its load to next worker, until next iteration
; timeout=minutes to wait for node to complete. As of nm75, only a warning
; is given if node takes longer than timeout minutes to return
; paraprint=1 print to console the parallel computing process. Can be
; modified at run-time with ctrl-B toggle.
; Regardless of paraprint setting, <control_stream>.log always records
; parallelization progress.
;
; transfer_type=0 for file transfer, unloading and reloading workers with
; each estimation
; transfer_type=1 for mpi
; transfer type=2 for file transfer, maintaining a single loaded process
; throughout the run.

;THE EXCLUDE/INCLUDE may be used to selectively use certain nodes,
; out of a large list.

 $EXCLUDE 5-7 ; exclude nodes 5-7
; or
;$EXCLUDE ALL
;$INCLUDE 1,4-6

$NAMES ; Give a label to each node for convenience
1:MANAGER
2:WORKER1
3:WORKER2

$COMMANDS ;each node gets a command line, used to launch the node session
; Command lines must be on one line for each process.
; command not needed for node 1, manager
1:NONE
;
; following is a launch on a core of the manager computer. Beolaunch.sh is a
; simple script available from the NONMEM ../run directory
2:./beolaunch.sh wrk_ftif/ ./nonmem >worker1.out
;
; following is a launch on a remote worker computer
3:ssh -n any_computer cd /home/myself/share/worker1';'./nonmem >worker1.out &

$DIRECTORIES
1:NONE ; FIRST DIRECTORY IS THE COMMON DIRECTORY
2:wrk_ftif/ ; NEXT SET ARE THE WORKER directories.
3:/mnt/share/worker1/

$CONTROL
;MTOUCH=1 for manager to "touch" the worker directory to get
; up-to-date information
;WTOUCH=1 for worker to "touch" its directory;
;MSLEEP=milliseconds for manager to wait between writing its content files
; to the remote worker directory
;WSLEEP=milliseconds for worker to wait between writing its content files
; to the worker directory
3: MTOUCH=1 WSLEEP=5 WTOUCH=0 MSLEEP=0

$IDRANGES ; USED IF PARSE_TYPE=3
1:1,50
2:51,100

There is an additional record introduced here, called $CONTROL. When working between computers on Linux with FPI, some network file systems (such as NFS on Unix) may require that the manager ‘touch’ the remote worker directory for that directory to show the up-to-date file information to the manager. Also, the process may need a period of waiting time before the signal file is created. Hence the need for the $CONTROL statements.

After an estimation step is performed, the worker processes exit. For the next estimation step that follows (if there is one), the manager will reload the worker processes. If you want worker processes to remain resident until all estimations and problems listed in the control stream file are completed, then select TRANSFER_TYPE=2.

Running Parallel Processes in a Mixed Platform Environment.

Suppose the manager process may be a new Linux operating system with a GLIBC that is new, while a worker computer may be Linux with an older operating system with an old GLIBC. This typically is not an easy environment to set up, but if you wish to do so, it means that you would need to create the nonmem executable on the Linux machine ahead of time, name it nonmem2, or some other name, so it is not copied over with the nonmem executable of the manager process, and use that nonmem2 on the worker $COMMANDS line:

2:./beolaunch.sh wrk_ftif/ ./nonmem2 >worker1.out

One would do something similar if the manager were a Windows process, and the worker were a Linux process, for example, but it is up to the user to find a means of launching a remote Linux process. The psexec launcher only works between Windows computers.

8.2 Installing MPI on Linux

If you are communicating across computers, make sure you set up a share drive and the ssh system as described earlier. Go to the web site

http://phase.hpcc.jp/mirrors/mpi/mpich2/

and select the appropriate *.tar.gz file. Or, select the mpich2_1.2.1.1.orig.tar.gz file in the MPI directory given in the NONMEM installation disk. On the manager computer, unpack the tar.gz file:

tar xfz mpich2_1.2.1.orig.tar.gz

Follow the instructions in section 2.2 of mpich2-1.2.1-installguide.pdf, and verify that the MPI system is working. NONMEM comes with the MPI library files (they are located in ..\mpi\mpi_lini for Intel Fortran and ..\mpi\mpi_ling for gfortran). For communication across computers, make sure you also have a network file allocated, just as with the FPI method. If the MPI library files do not match the version which you downloaded, or there are linking difficulties when you run nmfe75, then copy the appropriate *.a file from the MPICH2 installed directory mpich2\lib to the ..\mpi\mpi_lini directory. Keep in mind that we have supplied 32 bit versions of libraries. Environments with 64 bit processing may require libraries from the mpich2 web site.

For easy access of the mpi utility programs, you should expand the $PATH to include the path to the bin directory of the MPICH2 system, if it is not there already. You can insert the following line in the manager’s $HOME/.bashrc file, for example:

export PATH=$HOME/MPICH2_LINUX/mpich2-install/bin:$PATH

During the parallelization process, NONMEM sends a copy of its program (in nonmem.exe on Windows, nonmem on Linux) to the worker computer, and then loads it there. Therefore, the worker computers must be of the same operating system (although not necessarily same version) as the manager computer. For Intel fortran, the worker computer does not have to have Intel Fortran installed. For gfortran, -static option for the MPI method cannot be used in the nmfe75 script, as it prevents the MPI components from being properly linked. Thus the gfortran version of NONMEM with MPI requires its share library (libgfortran.so.3) available for the worker process, and in the path designated by the manager’s LD_LIBRARY_PATH setting:

LD_LIBRARY_PATH="$HOME/gcc-trunk/lib:$HOME/libgf:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH

In addition, the MPI system needs certain executable files available on the worker computer. These are (obtained from the bin directory of the MPICH2 system):

mpdlib.py
mpdman.py
mpd.py

Place these files in a directory on the worker computer that has the same path as MPICH2 is installed in the manager’s computer. For example, if the manager’s MPICH2 bin path is $HOME/MPICH2_LINUX/mpich2-install/bin, then this should be where the worker computer’s *.py files are.

Upon booting up, before executing your first NONMEM run, load up the mpi system:

mpdboot -n <number_of_computers> -f mpd.hosts

as instructed in the install guide. The mpd.hosts file contains a list of IP addresses, one per line, of the worker and manager computers. They could be referenced symbolically in the mpd.hosts, for example, as:

MY_MANAGER_COMPUTER
WORKER_A_COMPUTER
WORKER_B_COMPUTER

So long as these symbolic names are listed in the /etc/hosts file with the IP address.

The number_of_computers is number of worker computers (not cores), plus the manager computer. If loading just on one computer, then

mpdboot -n 1

To unload MPI after your last NONMEM run,

mpdallexit

See section 5 of mpich2-1.2.1-userguide.pdf for a full description of using the man MPI program mpiexec or mpirun.

Next, replace libmpich.a, in the NONMEM 75 directory, with the one from the located in the MPICH2 directory, for example, for gfortran:

cd /opt/nm75/mpi/mpi_wing
cp libmpich.a libmpich.a.orig
cp /usr/local/mpi32/lib/libmpich.a libmpich.a

or for Intel fortran:

cd /opt/nm74/mpi/mpi_lini
cp libmpich.a libmpich.a.orig
cp /usr/local/mpi32/lib/libmpich.a libmpich.a

gfortran:

cd /nm75g64/mpi_mpi_ling/
cp /usr/bin/MPICH2/lib/mpi.mod .
cp mpi.mod ../../resource

intel fortran:

cd /nm75g64/mpi_mpi_lini/
cp /usr/bin/MPICH2/lib/mpi.mod .
cp mpi.mod ../../resource

Or you may have to build it from a file called mpi.h or mpi.f, probably located in the ..\include folder of the MPI installation.

gfortran:

gfortran -c -fno-range-check mpi.f90
cp mpi.mod ../../resource
gfortran -c -O3 -ffast-math -m64 -mpc64 -I../../resource -J../../resource PNM_MPI.f90

(note you may want to use whatever compile options you used for the main NONMEM installation).

Intel fortran:

ifort -c mpi.f90
cp mpi.mod ../../resource
ifort -c PNM_MPI.f90 /I:../../resource

(note you may want to use whatever compile options you used for the main NONMEM installation).

Once you have an MPI system set up, for a quick test on a single multi-core computer, try the following. Copy foce_parallel.ctl and example1b.csv from the NONMEM ..\examples directory, mpilinux8.pnm from the NONMEM ..\run directory, and psexec.exe from the NONMEM ..\run directory, into your standard run directory. Then, execute the following from your standard run directory:

./mfe75 foce_parallel.ctl foce_parallel.res -parafile=mpilinux8.pnm [nodes]=4

where the values of [nodes] should be no greater than the number of cores available on your computer.

A typical structure of a pnm file for running NONMEM/MPI/Linux (note TRANSFER_TYPE=1) is as follows:

$GENERAL              PARSE_NUM=50  TIMEOUTI=100       TIMEOUT=10   PARAPRINT=0
NODES=2 PARSE_TYPE=2
TRANSFER_TYPE=1

; NODES=number of nodes (that is process, whether cores or computers)
; SINGLE node: NODES=1
; MULTI node (node means process, whether cores or computers): NODES>1
; WORKER node: NODES=0
;
; parse_num=number of subjects to give to each node
; parse_type=0, give each node parse_num subjects
; parse_type=1, evenly distribute numbers of subjects among available nodes
; parse_type=2, load balance among nodes
; parse_type=3, assign subjects to nodes based on idranges
; parse_type=4, load balance among nodes, taking into account loading time.
; This setting of parse_type will assess ideal number of nodes.
; If loading time too costly, will eventually revert to single CPU mode.
;
; timeouti=seconds to wait for node to start. if not started in time,
; deassign node, and give its load to next worker, until next iteration
; timeout=minutes to wait for node to complete. As of nm75, only a warning
; is given if node takes longer than timeout minutes to return
; paraprint=1 print to console the parallel computing process. Can be
; modified at run-time with ctrl-B toggle.
; Regardless of paraprint setting, <control_stream>.log always records
; parallelization progress.
;
; transfer_type=0 for file transfer, unloading and reloading workers with
; each estimation
; transfer_type=1 for mpi
; transfer type=2 for file transfer, maintaining a single loaded process
; throughout the run.

;THE EXCLUDE/INCLUDE may be used to selectively use certain nodes,
; out of a large list.

 $EXCLUDE 5-7 ; exclude nodes 5-7
; or
;$EXCLUDE ALL
;$INCLUDE 1,4-6

$NAMES ; Give a name to each node, which is displayed
1:MANAGER
2:WORKER1
3:WORKER2

$COMMANDS ;each node gets a command line, used to launch the node session
; first one launches manager version
1:mpirun "$PWD" -n 1 ./nonmem $*
;
; This launches a worker process on the manager's computer
2:-wdir "$PWD"/nonmem/wrk_mpi -n 1 ./nonmem
; This launches a worker process on a separate computer
;
3:-wdir /home/myself/share/worker1 -n 1 -host any_worker ./nonmem

$DIRECTORIES
1:NONE ; FIRST DIRECTORY IS THE COMMON DIRECTORY
2:nonmem/wrk_mpi/ ; NEXT SET ARE THE WORKER directories
3:/mnt/share/worker1/

$IDRANGES ; USED IF PARSE_TYPE=3
1:1,50
2:51,100

You will want to modify the pnm file for your particular environment, and use some of the other options available in setting up the mpiexec/mpirun command line.

Unlike FPI, the MPI system can only use the starting PARAFILE specified at the command line, and it may not be easily switched later in the control stream. All processes remain resident throughout the entire job, although it will honor requests of parafile=off or parafile=on for individual $EST records, which allows you to have control of which estimation method will use parallel processing.

Earlier we show that the addresses to the worker computers listed in the file mpd.hosts could be loaded using the mpdboot -f command. The -f option is also available in mpirun, so this information may be supplied within the parafile, for example:

1:mpirun "$PWD" -n 1 0 -f mpd.hosts ./nonmem $*

Some Advanced Technics For Defining the PARAFILE for an MPI System.

Because the MPI system communicates completely via ports, and not via file transfer as the FPI system does, one can set up a parafile in which an MPI command is repeated for several nodes, even though they may point to the same directory. Here is an example which makes creating a PARAFILE for an MPI system versatile:

$GENERAL
NODES=8 PARSE_TYPE=2 TRANSFER_TYPE=1 PARAPRINT=0 COMPUTERS=2

$COMMANDS
1:mpiexec -wdir "$PWD" -n 1 ./nonmem $*
2-4: -wdir "$PWD" -n 1 -host MY_MANAGER_COMPUTER ./nonmem -wnf
5-8: -wdir $HOME -n 1 -host MY_WORKER_COMPUTER ./nonmem -wnf

$DIRECTORIES
1-8:NONE
5:/mnt/worker1

In this example, node 1 is defined as usual as the manager process. Then, processes 2 through 4 are defined using a command that is repeated for each of these processes (it is copied 3 times in the resulting nmmpi script file that is eventually executed). Yet processes 2-4 all point to the default current directory of the manager ("$PWD"). Furthermore, the $DIRECTORIES entries for these processes is NONE. That means the three worker processes which are loaded on the manager computer are sharing the same directory as the manager, and because of the NONE directory designation in $DIRECTORIES, the executable nonmem will not be copied, as it should not, since the worker processes are pointing to the manager directory, and therefore the nonmem executable in the manager directory is already available to worker processes as well. Furthermore, the option -wnf is given. This option tells the nonmem process that it is a worker, MPI method, and the nf tells it not to make any file buffers (nf=no files). The worker process has all the information it needs to launch without requiring any file based communication with the manager, and minimizes the footprint on the drive directory.

The next 4 processes are launched on a remote computer with similar settings. Notice that only one of the processes among the 5 to 8 had to have a $DIRECTORY defined, that of /mnt/worker1, which they all are pointing to. The $HOME directory of the worker computer is the directory /mnt/worker1 that the manager has a share connection to. This means that NONMEM has a path direction to copy the nonmem executable from its current directory to the $HOME directory on the worker computer. If all processes $DIRECTORIES entries were NONE, then the most recently built nonmem executable cannot be copied to the remote computer. You may want that, if for example, you have arranged for a nonmem executable to be there already that was previously built with the identical control stream file. Maybe the remote computer is a different platform than the manager computer, and needed a different executable. MPICH2 communication between a Linux and Windows operating system has not been attempted, so it is not known if this would work anyway.

Note that -host MY_MANAGER_COMPUTER had to be identified on the worker processes that were being launched locally. The mpiexec command gets confused if it has to deal with several lines containing different computer names. So it is best not to leave the -host switch up to default once you get past the manager processor line.

The -wnf switch must be carefully used. Make sure that LIM1, LIM3, LIM4, LIM13, and Lim15 are appropriately sized so that the buffer files (named FILEXX) do not have to be used. Or, as of NM73, you may set -maxlim=1 or higher on the nmfe75 command line. Then, LIM1, LIM3, LIM4, LIM13, and Lim15 (those used during estimation, and therefore by workers in a parallelization problem), will be set to the size needed to assure no buffer files are used, and everything is stored in memory, for the particular problem. If you set -maxlim=2, then LIM1, LIM2, LIM3, LIM4, LIM5, LIM6, LIM7, LIM8, LIM10, LIM13, LIM15, and LIM16 are also sized to what is needed to assure that buffer files are not needed.

If the buffer files do need to be used, then use switch -wf. Each worker process will make a series of files named WK1_FILE* for worker 1, WK2_FILE* for worker 2, etc. This way, even if the workers and manager share the same directory as a scratch pad, their files will be uniquely named, and there won’t be a file clobber.

An alternative method of launching mpi processes is to use its multiple process launch option -n xx, where xx is the number of processes to launch:

$GENERAL
NODES=8 PARSE_TYPE=2 TRANSFER_TYPE=1 PARAPRINT=0 COMPUTERS=2

$COMMANDS
1:mpiexec -wdir "$PWD" -n 1 ./nonmem $*
2: -wdir "$PWD" -n 3 -host MY_MANAGER_COMPUTER ./nonmem -wnf
3: -wdir $HOME -n 4 -host MY_WORKER_COMPUTER ./nonmem -wnf

$DIRECTORIES
1-8:NONE
3:/mnt/worker1

Command 2 launches 3 processes, and command 3 launches 4 processes, so there are still 8 processes launched.

As of nm75, any workers option switch that is prefixed with “a” (-aworker, -awnf etc), will be ignored by the first nonmem executable loaded by the MPI system (so it will be the manager), allowing the same arguments to be placed on all nodes. For example:

$DEFAULTS
[nodes]=8
GENERAL
NODES=[nodes] PARSE_TYPE=2 TRANSFER_TYPE=1 PARAPRINT=0 COMPUTERS=2

$COMMANDS
1:mpiexec -wdir "$PWD" -n [nodes] ./nonmem $* -awnf

$DIRECTORIES
1-[nodes]:NONE

9 Special Considerations for MAC OS X

The MAC OS X system uses a form of Linux as its operating system, but there are some differences in the environment compared to Linux on non-MAC systems.

9.1 Mounting file systems on MAC OS X

It is easier to use afp (Apple Filing Protocol) than nfs.

To export a file system or folder to another Mac:

Select the Apple menu / System Preferences / Sharing / File Sharing
Under “shared folders:” click + and select the folder e.g., mydir
Under “users:” click + and select the users.

To mount a file system or folder from another Mac:

Open a finder window.
You should see the hostname of the other computer listed under “Shared”
Click on it. Click on “connect as”
Enter the username and password.
Click on the folder, e.g., mydir
The file system or folder will be mounted as /Volumes/mydir

E.g., in a terminal window:

% ls /Volumes/mydir

9.2 Enabling ssh with no password on MAC OS X

Select the Apple menu / System Preferences / Sharing / Remote Login

The instructions for Linux (using ssh-keygen) should work on Mac OS X. There may be an interaction with keychain, and this may be problematic.

If “ssh -n” cannot be made to work, you can use the workaround for mpdboot described in the MPICH2 Installer’s Guide. See ‘start the daemons “by hand”’ on page 7 of mpich2-1.2.1-installguide.pdf

Installing MPICH2 on MAC OS X

MPICH2 must be compiled and installed for Mac OS X. Please look at mpich2/README_vin.mht and the other documents.

You may need to set CPATH as it was during the NONMEM install

First, see what kind of binaries have been installed, e.g.,

% cd /Users/Shared/nm750/mpi/mpi_ling (or mpi_lini, with ifort):
% file mpi.o

You will see either of the following:

mpi.o: Mach-O 64-bit object x86_64
mpi.o: Mach-O object i386

“i386” indicates 32 bit binaries.

Suggested options for the configure step with MacOS 10.15 Catalina:

If SETUP75 installed 64 bit binaries:

./configure --prefix=/usr/local/mpi64 CFLAGS="-m64" FFLAGS="-m64" --enable-f90
-disable-cxx --enable-timer-type=gettimeofday |& tee c.txt

If SETUP75 installed 32 bit binaries:

./configure --prefix=/usr/local/mpi32 --enable-f90
--enable-timer-type=gettimeofday |& tee c.txt

(With earlier releases than Catalina, option -enable-time-type may not be necessary) (With ifort rather than gfortran, -disable-cxx may not be necessary)

Either way, continue with

make |& tee m.txt
make install |& tee mi.txt

Then replace libmpich.a, in the NONMEM 75 directory, e.g, if 64 bit was installed:

cd /Users/Shared/nm750/mpi/mpi_ling
cp libmpich.a libmpich.a.orig
cp /usr/local/mpi64/lib/libmpich.a libmpich.a

gfortran:

cd /nm75g64/mpi_mpi_ling/
cp /usr/local/mpi64/lib/mpi.mod .
cp mpi.mod ../../resource

intel fortran:

cd /nm75g64/mpi_mpi_lini/
cp /usr/local/mpi64/lib/mpi.mod .
cp mpi.mod ../../resource

Or you may have to build it from a file called mpi.h or mpi.f, probably located in the ..\include folder of the MPI installation.

gfortran:

gfortran -c -fno-range-check mpi.f90
cp mpi.mod ../../resource
gfortran -c -O3 -ffast-math -m64 -mpc64 -I../../resource -J../../resource PNM_MPI.f90

(note you may want to use whatever compile options you used for the main NONMEM installation).

Intel fortran:

ifort -c mpi.f90
cp mpi.mod ../../resource
ifort -c PNM_MPI.f90 /I:../../resource

(note you may want to use whatever compile options you used for the main NONMEM installation).

9.3 Using the Correct MPI commands on MAC OS X

The user’s path should be set so that commands such as mpirun and mpf90 from MPICH2 are used instead of the corresponding Open MPI commands native to Mac OS X. For example, if 64 bit was installed, the following is suggested prior to doing one or more NONMEM runs with MPI in a csh window:

   % set path = (/usr/local/mpi64/bin $path)

If this is not done, the message may appear:

   Unfortunately, this installation of Open MPI was not compiled with
   Fortran 90 support. As such, the mpif90 compiler is non-functional.

9.4 Disabling Open MPI commands on MAC OS X

The Open MPI commands that are supplied with Mac OS X must be disabled. The following is suggested:

   % sudo -s
   # cd /usr/bin
   # mkdir default.mpi
   # mv mpi* default.mpi
   # exit

If this is not done, this message may appear:

   Unfortunately, this installation of Open MPI was not compiled with
   Fortran 90 support. As such, the mpif90 compiler is non-functional.