xmipp_metadata_utilities (v3.0)


Perform several operation over the metadata files. If the -o option is not used the original metadata will be modified after applying some operations. Also you can use the --print option just to print out the result metadata to screen. The combination of -i and -o without other operations can serve to extract data blocks inside a medata and write to an independent one.

See also


-i <metadata>
Input metadata file
-o <metadata>
Output metadata file, if not provided result will overwrite input file
--set, -s <set_operation> <md2_file> <label=image>
Set operations
where <set_operation> can be:
  • union Union with metadata md2, duplicated values only will appear once
  • union_all Union with metadata md2, will repeat duplicated values
  • intersection Intersection with metadata md2
  • subtraction Subtraction with metadata md2
  • join Inner join with md2 using label l1
  • natural_join Natural join with md2 using all common labels
  • merge Merge columns with md2, label is ignored Both metadatas should have same size, and elements should be in same order, if not, you should use 'join' instead, but this constrain having a common label
or --operate, -e <operation>
Operations on the metadata structure
where <operation> can be:
  • sort <label=image> Sort metadata using a label as identifier for sorting according to a component of a vector label use label:col, e.g., NMADisplacements:0 The first column is column number 0
  • randomize Randomize elements of metadata
  • add_column <labels> Add some columns(label list) to metadata
  • drop_column <labels> Drop some columns(label list) from metadata
  • modify_values <expression> Use an SQLite expression to modify the metadata This option requires knowledge of basic SQL syntax(more specific SQLite
or --file, -f <file_operation>
File operations
where <file_operation> can be:
  • copy <directory> <label=image> Copy files in metadata md1 to directory path (file names at label column)
  • move <directory> <label=image> Move files in metadata md1 to directory path (file names at label column)
  • delete <label=image> Delete files in metadata md1 (file names at label column)
  • convert2db Convert metadata to sqlite database
  • convert2xml Convert metadata to xml file
  • import_txt <labels> Import a text file specifying its columns
or --query, -q <query_operation>
Query operations
where <query_operation> can be:
  • select <expression> Create new metadata with those entries that satisfy the expression
  • count <label> for each value of a given label create new metadata with the number of times the value appears
  • sum <label1> <label2> group metadata by label1 and add quantities in label2
  • size print Metadata size
or --fill, -l <labels> <fill_mode>
Fill a column values(should be of same type)
where <fill_mode> can be:
  • constant <value> Fill with a constant value
  • lineal <init_value> <step> Fill with a lineal serie starting at init_value with an step
  • rand_uniform <a=0.> <b=1.> Follow a uniform distribution between a and b
  • rand_gaussian <mean=0.> <stddev=1.> Follow a gaussian distribution with mean and stddev
  • rand_student <mean=0.> <stddev=1.> <df=3.> Follow a student distribution with mean, stddev and df degrees of freedom.
  • expand Treat the column as the filename of an row metadata and expand values
--print, -p
Just print medata to stdout, or if -o is specified written to disk. this option is useful for extrating data blocks inside a metadata.
--mode <mode=overwrite>
Metadata writing mode.
where <mode> can be:
  • overwrite Replace the content of the file with the Metadata
  • append Write the Metadata as a new block, removing the old one

Examples and notes

Concatenate two metadatas. If label is not provided, by default is 'image'
   xmipp_metadata_utilities -i mD1.doc --set union mD2.doc  -o out.doc
Intersect two metadatas using label 'order_'
   xmipp_metadata_utilities -i mD1.doc --set intersection mD2.doc order_ -o out.doc
Combine columns from two metadatas. Be sure of both have same number of rows and also there aren't common columns, in that case second metadata columns will be used
   xmipp_metadata_utilities -i mD1.doc --set merge mD2.doc -o out.doc
Sort the elements in metadata (using default label 'image').
   xmipp_metadata_utilities -i mD1.doc -s sort -o out.doc
Add columns 'shiftX' and 'shiftY' to metadata.
   xmipp_metadata_utilities -i mD1.doc --operate add_column "shiftX shiftY" -o out.doc
You can also add columns and 'filling' its values with different options By example, to add the column 'shiftX' with uniform random value between 0 and 10
   xmipp_metadata_utilities -i mD1.doc --fill shiftX rand_uniform 0 10 -o out.doc
Or for initialize metadata columns 'shiftX' and 'shiftY' with a constant value of 5
   xmipp_metadata_utilities -i mD1.doc -l "shiftX shiftY" constant 5 -o out.doc
If you have columns that represent the filename of a metadata with other data (ex CTFParams) you cant 'expand' the column with the values in that metadta
   xmipp_metadata_utilities -i mD1.doc --fill CTFParams expand -o outExpanded.doc
For check all options availables for 'filling' mode, use:
   xmipp_metadata_utilities --help fill
Dump metadata content to Sqlite3 database. (use xmipp_sqlite3 to visualize results)
   xmipp_metadata_utilities -i mD1.doc --operate convert2db -o mD1.db; xmipp_sqlite3 out.db
Dump metadata content to xml file.
   xmipp_metadata_utilities -i mD1.doc --operate convert2xml -o mD1.xml
Copy files in metadata to a location. The metadata will be also copied to new location
   xmipp_metadata_utilities -i mD1.doc --file copy /home/pepe/newLocation
Delete files in metadata.
   xmipp_metadata_utilities -i mD1.doc --file delete
Select elements in metadata that satisfy a given constrain.
   xmipp_metadata_utilities -i mD1.doc --query select "angleRot > 10 AND anglePsi < 0.5" -o out.doc
You can also modify your data using SQLite syntax expression
  xmipp_metadata_utilities  -i a.doc --operate modify_values "angleRot=(angleRot*3.1416/180.)" -o b.doc
  xmipp_metadata_utilities  -i a.doc --operate modify_values "image=replace(image, 'xmp','spi')" -o b.doc
  xmipp_metadata_utilities  -i a.doc --operate modify_values "image='new_prefix_dir/'||image" -o b.doc
Count number of images per CTF
   xmipp_metadata_utilities -i mD1.doc -q count CTFModel -o out.doc
images asigned a ctfgroup
   xmipp_metadata_utilities -i mD1.doc -q sum defocusGroup count -o out.doc
Print the metadata Size
   xmipp_metadata_utilities -i mD1.doc --query size

User's comments