merge

(1) spipy.merge.emc

--> " EMC merging algorithm implemetation " (Extension of Dragonfly software)

+ config_essential:dict

+ config_advanced:dict

+ :void = new_project (data_path:str, inh5:str, path:str/None, name:str/None)

+ :void = config (params:dict)

+ :void or str = run (num_proc:int, num_thread:int, iters:int, nohup:bool/False, resume:bool/False, cluster:bool/True)

+ :void = use_project (project_path:str)

--

• config_essential : dict of important parameters
  • keys : refer to Examples
    • 'parameters|detd' : distance between sample and detector [unit : mm]
    • 'parameters|lambda' : wave length of laser [unit : angstrom]
    • 'parameters|detsize' : detector size in width/height [unit : pixel]
    • 'parameters|pixsize' : pixel size of detector [unit : mm]
    • 'parameters|stoprad' : radius of a circle region at the center of pattern that will not be used in orientation recovery, but will be merged to final scattering volume [unit : pixel].This option only works when 'make_detector|in_mask_file'=None
    • 'parameters|polarization' : correction due to incident beam polarization, value from 'x', 'y' or 'none'
    • 'emc|num_div' : level that used to generate quaternions, also known as n, where M_rot=10(n+5n³)
    • 'emc|need_scaling' : whether need to scale patterns' intensities
    • 'emc|beta' : beta value of emc method
    • 'emc|beta_schedule' : for example, if 'emc|beta_schedule' = '1.414 10', that means for every 10 iterations, beta = beta * 1.414

--

• config_advanced : dict of advanced parameters
  • keys : refer to Example
    • 'parameters|ewald_rad' : Radius of curvature of the Ewald sphere in voxels, used to control oversampling rate in reciprocal space. For default, oversampling rate is controlled to be the same with experimental patterns. Larger ewald_rad means less oversampling and smaller reciprocal scattering matrix. details here
    • 'make_detector|in_mask_file' : path of mask file (a numpy/binary file, .npy, .byt, .bin) that operate on input patterns. Usually, most experiments need mask file. In your mask file, 0 marks unmasked area, 1 marks areas that will not be used for orientation recovery but will used in merging, 2 marks areas that will not be used for both orientation recovery and merging. For binary file the dtype should be 'uint8', for numpy file it can be any type. Default is None.
    • 'emc|sym_icosahedral' : bool (0/1), whether to force icosahedral symmetry in orientation recovery
    • 'emc|selection' : 'even_only', 'odd_only' and 'None', where 'even' / 'odd' means only patterns whose index is even / odd will be used. 'None' means all patterns will be used
    • 'emc|start_model_file' : path of file that store initiate model which will be used at the start of emc program. The format should be '.bin' binary file for C 'fopen' to read.

--

• new_project : create a new project at your given path, NO return
  • data_path : path of your dataset file, MUST be HDF5 file, and the data type must be 'int32' or 'int64'
  • inh5 : path of patterns inside h5 file, patterns should be stored in a numpy.ndarray, shape=(Nd,Nx,Ny)
  • path : create work directory at your give path, set None to use current dir, default=None
  • name : give a name to your project, set None to let program choose one for you, default=None

--

• config : edit configure file, NO return
  • params : dict, parameters that you want to modified. refer to example code

--

• run : start emc, return command string if cluster == True
  • num_proc : int, how many processes to run in parallel
  • num_thread : int, how many threads in each process
  • iters : int, how many reconstruction iterations
  • nohup : bool, whether run in the background, default=False
  • resume : bool, whether run from previous break point, default=False
  • cluster : bool, whether you will submit jobs using job scheduling system, if True, the function will only generate a command file at your work path without submitting it, and ignore nohup value; if False, the program will run directly. default=True
  • [Notice] As this program costs a lot of memories, use as less processes and much threads as possible. Recommended strategy : num_proc * num_thread ~ number of cores in all of your CPUs. Let one cluster node support 1~2 processes. (Mentioned, using too many processes may cause low precision in merging result)

--

• use_project : switch to a existing project, NO return
  • project_path : str, the path of project directory that you want to switch to

(2) spipy.merge.utils

--> " Utility functions to help you DIY your merging programs "

+ :numpy.3darray = get_slice (model:numpy.3darray, quaternions:numpy.2darray, det_size:list or tuple, det_center:list or tuple/None, mask:numpy.2darray/None)

+ :void = merge_slice (model:numpy.3darray, quaternions:numpy.2darray, slices:numpy.3darray, weights:numpy.3darray/None, det_center:list or tuple/None, mask:numpy.2darray/None)

+ :float = poisson_likelihood (W_j:numpy.1d or 2darray, K_k:numpy.1d or 2darray, beta:float/1, weight:float/None)

+ :numpy.1darray = maximization (K_ks:numpy.2darray, Prob_ks:numpy.1darray)

+ :numpy.2darray = get_quaternion (Num_level:int)

--

• get_slice : Generate slices from a 3D model according to given quaternions (orientations), output slices, shape=(Nq, Npx, Npy) , Nq is the number of quaternions
  • model : the model, a 3d numpy array
  • quaternions : a set of quaternions, np.array([[w,qx,qy,qz],...]), shape=(Nq,4)
  • det_size : the size of generated patterns (in pixels), [Npx, Npy]
  • det_center : the center of generated patterns (in pixels), default=None and the geometry center is used )
  • mask : pattern mask, 2d numpy array where 1 means masked area and 0 means useful area, default is None

--

• merge_slice : Merge sereval slices into a 3D model according to given quaternions (orientations), NO RETURN, input model is modified directly
  • model : the model, a 3d numpy array
  • quaternions : a set of quaternions, np.array([[w,qx,qy,qz],...]), shape=(Nq,4)
  • slices : slices to merge into model, numpy array, shape=(N,Npx,Npy)
  • weights : initial interpolation weights for every pixel of input model, shape=model.shape, default is None and weights=numpy.ones(model.shape) is used
  • det_center : the center of generated patterns (in pixels), default=None and the geometry center is used
  • mask : pattern mask, 2d numpy array where 1 means masked area and 0 means useful area, default is None

--

• poisson_likelihood : Calculate poisson likelihood between a model slice and an experiment pattern, return float, R_jk = weight * ( Product{ W_j^K_kexp(-W_j) } )^beta
  • W_j : model slice in orientation j, 2d pattern or flattened array, do masking in advance (set masked area to 0 or just give flattened array without masked points)
  • K_k : kth experiment pattern, 2d pattern or flattened array, do masking in advance
  • beta : float, suggested values are from 1 to 50, default=1
  • weight : float, the weight of orientation j, when orientations are not strictly uniformly sampled, default is None

--

• maximization : Calculate updated slice of orientation j after likelihood maximization, return W_j_prime, the updated slice in orientation j (flattened), length = K_ks.shape[1]
  • K_ks : ALL useful experiment patterns, numpy array, shape=(N,Np), where N is the number of patterns and Np is the number of useful pixels per pattern. Reshape pattern to array and do masking in advance !
  • Prob_ks : probabilities of ALL useful patterns (after normalizing in every orientation) in orientation j, shape=(N,)

--

• get_quaternion : Calculate quaternions which are uniformly distributed in orientation space (sampling weights are 1), return quaternions, shape=(2*Num_level³,4)
  • Num_level : controls the number of output quaternions ( 2*Num_level³ )