Structure of Xmipp protocols

Protocols are Python programs living in the directory $XMIPP_HOME/protocols. A protocol is composed by two parts: a header and a body. The header of the protocol defines the parameters of the protocol, while the body defines the actions that will be performed. For instance, the body of the protocol for screening micrographs is called protocol_screen_micrographs.py, and its header is called protocol_screen_micrographs_header.py. In the same directory you will find many useful routines under the modules protlib_....

Create a new protocol

To create a new protocol ( myProtocol) you need to create a file called protocol_myProtocol.py and protocol_myProtocol_header.py in $XMIPP_HOME/protocols. The minimum content of these files is

$XMIPP_HOME/protocols/protocol_myProtocol_header.py :

#!/usr/bin/env xmipp_python

# {begin_of_header} #{eval} expandCommentRun()

# {section} Input parameters # Number of times the text must be written NumberOfTimes = 6

# Text to print Text = 'Hello, world!'

# {end_of_header}

from protocol_myProtocol import * if __name__ == '__main__': protocolMain(ProtMyProtocol) 

$XMIPP_HOME/protocols/protocol_myProtocol.py :

#!/usr/bin/env xmipp_python # Protocol example # Author: Carlos Oscar Sorzano, June 2012

from protlib_base import *

class ProtMyProtocol(XmippProtocol): def __init__(self, scriptname, project): XmippProtocol.__init__(self, protDict.myProtocol.name, scriptname, project) self.Import = "from protocol_myProtocol import *"

def defineSteps(self): for i in range(1,self.NumberOfTimes): self.insertStep("printMessage",msg=self.Text)

def validate(self): errors = [] if self.NumberOfTimes<=1: errors.append("Number of times to print must be larger than 1"); return errors

def summary(self): message = [] message.append("Printing '%s' %d times" % (self.Text,self.NumberOfTimes)) return message

def visualize(self): pass

def printMessage(log,msg): print(msg) 

You need to register your new protocol. For doing so, edit the file $XMIPP_HOME/protocols/config_protocols.py and add the following lines:

$XMIPP_HOME/protocols/config_protocols.py :

protocols = { ... # All previous protocols here 'myProtocol': ('MyProtocol', 'MyProtocol') }

sections = [... ('Other', [['Extra','subtraction', 'mltomo', 'dummy', 'myProtocol']]) ] 

The result of this code is seen in this figure: the protocol is under the "Extra" menu. It opens a dialog to ask for the two input parameters.


Header definition

In the protocol header we will define the Python variables that will be used for protocol execution. The definition of such variables will contains some meta-information to help the automatic building of the protocol GUI. This will allow us easily define sections, file fields, list selection.

There are three main parts in the script:

  • pre-header: all lines before the {begin_of_header}, useful for import, comments, etc.
  • header: the variables definition, all lines between {begin_of_header} and {end_of_header}
  • post-header: all lines after the {end_of_header}, contains the main
An example of variable definition is as follows:

# {file}{validate}(PathExists) Input classes:
""" More description about input """
InputClasses = ''

The first comment line will contain a label that will be displayed on the gui. This line can also contains a lists of tags adding extra information to the variable definition, such as the expected input is a file and should exists. Following the first line could be a multi-line python string starting and ending with """ that will add extra help about the variable. In the GUI, a help button will appears to display this help.

The last line is the variable and its default value. The general form will be:

VarName = VALUE

  • If VALUE is True or False, the variable is considered as a Boolean(Yes/No button on GUI)
  • If VALUE starts with """ the variable is string( a text area on GUI)
  • If VALUE start with " or ' the variable is string (text entry on GUI) or a list option (see *{list} tag}
  • If VALUE is a number the variable will be int or float (text entry on GUI)
An special tag is {section}, it doesn't have a variable associated, just define that a new section will be created on the GUI.

Possible tags are:

  • {section} defines a section with the comment, not a variable
  • {has_question} this tag can be used with a section and take the next boolean variable as a checkbutton for show/hide the entire section box
  • {expert} mark this variable or section as "expert", not shown in the GUI by default, showed when pressed the "Show expert options" button
  • {file} or {dir} marks the option as a Filename or Directory, and will add a corresponding "Browse" button to that option
  • {view} this will add an extra button to view/explore some files or folders
  • {run} (prot1, prot2) Select as input previous runs from other(or same) protocol
  • {text} Will display a text box with more space for writing
  • {list} (option A, option B, option C) marks the option as a radio-list button. The selected value should be one of the options indicated.
  • {condition} (option=True) {condition} (selection=optionA) marks an option or section dependent of some condition, it can be used with boolean options (use square brackets instead parenthesis to group, e.g., option1="a" and [option2!="B" or option2!="C"]) or for list-selections
  • {cite} will display a message at the top of the protocol stating -If you publish results obtained with this protocol, please cite-, followed by the text on rest of the comment line. If more than one citation lines are present, they will all be displayed. DONT use very long citations, as this will results in an ugly gui.
  • {hidden} label on the comment line (#) marks the option as -hidden-
  • {wizard} (WizardFunction) this will serve to plugin a graphical wizard to select some parameter
  • {validate} (NonEmpty, PathExists, IsInt, IsFloat) Impose some validation on values entered by the user

Body definition

The body of a protocol is composed of several parts (see protocol_myProtocol.py above), note that the function names cannot change:

  • Constructor: Where some internal variables are defined.
  • defineSteps: Here you define the different steps of your protocol. The different steps are routines or programs that can be called from inside the python script
  • validate: Before running the protocol, this function checks the validity of the input parameters.
  • summary: It prints a summary of the protocol execution in the main project window.
  • visualize: It provides functions to analyze the output of the protocol.

Step definition

There are different ways of defining a step depending on whether it is a sequential step or a parallel step, and a Python routine or an external program. For the different steps there are different routines to define the step

Type Routine syntax
Sequential, Python routine self.insertStep
Sequential, External program self.insertRunJobStep
Parallel, Python routine self.insertParallelStep
Parallel, External program self.insertParallelRunJobStep

Filenames handling

There are several functions to try to standardize the generation of filenames. In the protlib_base.py you can find two function related to filename generations:

def createFilenameTemplates(self): ''' Each protocol should implement this function to create the dictionary with entries (alias, template) for filenames templates''' return {}

def createFilenameDict(self): ''' This will create some common templates and update with each protocol particular dictionary''' d = {

            $ 'acquisition': join('%(WorkingDir)s', 'acquisition_info.xmd'),
            $ 'extract_list': join('%(WorkingDir)s', "%(family)s_extract_list.xmd"),
            $ 'families': join('%(WorkingDir)s', 'families.xmd'),
            $ 'family': join('%(WorkingDir)s', '%(family)s.xmd'),
            $ 'macros': join('%(WorkingDir)s', 'macros.xmd'),
            $ 'micrographs': join('%(WorkingDir)s','micrographs.xmd'),
            $ 'microscope': join('%(WorkingDir)s','microscope.xmd'),
            $ 'tilted_pairs': join('%(WorkingDir)s','tilted_pairs.xmd') } d.update(self.createFilenameTemplates()) return d
In createFilenameDict is created a dictionary with templates for generating files names. These templates can be customized with the header variables or any variable that you include to the protocol ParamsDict. Or let some parameters that can be passed when the template will be used with the function getFilename. The function createFilenameTemplates should be used on each protocol to define its own templates, if the same template is defined, it will override the base one.

Other useful functions are:

  • workingDirPath for creating a filename inside the working dir of the protocol
  • tmpPath create a temporary filename
  • extraPath create a filename in the extra directory of the protocol
  • getFilename function the get the filename template using some parameter, can be used to query other protocols filenames

Create a Custom protocol

The idea of the custom protocols is a way of easily create a protocol without need of register or copy the files to the Xmipp installation folder. It is a good way of integrate you own scripts into the processing workflow and keep full trace of your work.

To simplify the inclusion of required files is recommended that for a custom protocol the header and implementation goes in the same file, for example: custom_create_stack.py:

Protocols Intercomunication

See here.

Example of a protocol workflow

The following example shows how to create a custom protocol in which several other protocols are called. This example is meant for a regular import and screening of micrographs that can be run within a cron schedule.

#!/usr/bin/env xmipp_python
# Protocol for regularly importing micrographs
# Author: Carlos Oscar Sorzano, August 2013

# {begin_of_header}

# {eval} expandCommentRun()

# {section} General parameters
# {dir} Directory with micrographs:
""" Directory with the input micrographs

# {wizard}(wizardMicrographExtension) Files to process
This is the micrographs files pattern, typically <*.tif> or <*.ser>, 
but may also be <*.mrc> or <*.spi> 
<Note:> you can use any wildcard like in shell expansion, e.g. <*3[1,2].tif>
ExtMicrographs = ''

#{run}(import_micrographs) Import Micrographs Example
""" Example import micrograph run"""

#{run}(screen_micrographs) Screen Micrographs Example
""" Example screen micrograph run"""

# {eval} expandParallel(threads=0,mpi=2)


import glob, os, sys, shutil,time
from protlib_base import *
from protlib_utils import runJob

class ProtRegularImport(CustomProtocol):
    def __init__(self, scriptname, project):
        CustomProtocol.__init__(self, scriptname, project)

    def defineSteps(self):
       Cs=importRun.SphericalAberration, Ts=importRun.SamplingRate,
       FastDefocus=screenRun.FastDefocus, MinFocus=screenRun.MinFocus,
    def summary(self):
        message.append("Importing micrographs from: ["+self.DirMicrographs+"]")
        return message
    def validate(self):
        errors = []
        return errors
def runProtocols(log, DirMicrographs, ExtMicrographs, Voltage, Cs, Ts, NumberOfMpi,
    MinFocus, MaxFocus, WinSize,AutomaticRejection):
    from protlib_base import ProtocolExecutor, XmippProject
    project = XmippProject()

    pe = ProtocolExecutor('import_micrographs', project)
    pe.setValues({'DirMicrographs': DirMicrographs, 'ExtMicrographs': ExtMicrographs,
                  'Voltage': Voltage, 'SphericalAberration': Cs,
                  'SamplingRate': Ts, 'NumberOfMpi': NumberOfMpi})

    pe2 = ProtocolExecutor('screen_micrographs', project)
    pe2.setValues({'ImportRun': pe.getExtendedRunName(),
                  'DownsampleFactor': DownsampleFactor,
        'AmplitudeContrast': AmplitudeContrast,
        'LowResolCutoff': LowResolCutoff,
        'HighResolCutoff': HighResolCutoff,
        'FastDefocus': FastDefocus,
        'MinFocus': MinFocus,
        'MaxFocus': MaxFocus,
        'WinSize': WinSize,
        'AutomaticRejection': AutomaticRejection,
        'NumberOfMpi': NumberOfMpi})

if __name__ == '__main__':

User's comments