xptycho blueprints › Pre and Post Processing

Pre and Post Processing Blueprint

How raw detector frames become a Scan. The steps are those Qiuchen Zhai used for the gold-ball data in the two TCI papers, read from her script in ptycho_pmace_papers. Each step is a function of arrays in memory; one reader per file format does the file reading. Section 4 lists what was checked against her data, and section 5 gives notes on the design.

1. The goal

Take the raw file of an instrument to a Scan: frames that are intensities, centered so that zero frequency is at pixel \((N_p/2, N_p/2)\), with positions in meters and the instrument facts. The first test is to reproduce the frames the reference code reconstructs.

2. The script

import xptycho as xpt
import xptycho.preprocess as xpp

raw = xpp.cxi.load_raw(xpt.download(RAW_URL, DATA_DIR))       # counts, dark frames, translations, instrument facts

frames = xpp.subtract_dark(raw['frames'], raw['dark_frames'])  # 1. mean dark frame off; negatives to zero

outlier = xpp.find_outlier_frames(frames, threshold=2)         # 2. frames whose mean intensity is an outlier
frames, translations = frames[~outlier], raw['translations'][~outlier]

center = xpp.diffraction_center(frames)                        # 3. one center for the whole scan
frames = xpp.crop_frames(frames, center, 512)

frames = frames * xpp.tukey_window(512, 0.5) ** 2              # 4. taper; the window multiplies amplitudes

probe_positions = translations[:, [1, 0]]                            # as recorded: rows along y, columns along x
scan = xpt.Scan(frames, probe_positions, wavelength=raw['wavelength'],
                det_distance=raw['det_distance'], det_pixel_pitch=raw['det_pixel_pitch'])

Every intermediate is an array the user can look at. No step reads or writes a file except load_raw and download.

3. The functions

namewhat it does
preprocess.cxi.load_raw(path)Reads a CXI file (the HDF5 format of the Coherent X-ray Imaging Data Bank). Returns the detector counts, the dark frames, the recorded translations \((x, y, z)\) in meters, the wavelength from the recorded photon energy, the detector distance, and the detector pitch. Corrects nothing.
preprocess.subtract_dark(frames, dark_frames)Subtracts the mean of the dark frames from every frame and sets negative values to zero.
preprocess.find_outlier_frames(frames, threshold=2)Computes the mean intensity of each frame. A frame is an outlier when its mean is more than threshold standard deviations from the average of the means. Returns one True or False per frame; the user removes the frames and their positions.
preprocess.diffraction_center(frames)The intensity center of mass of the mean frame: one \((\text{row}, \text{col})\) for the whole scan.
preprocess.crop_frames(frames, center, size)Cuts a size by size square out of every frame with center, rounded to whole pixels, at pixel \((\text{size}/2, \text{size}/2)\).
preprocess.tukey_window(size, shape=0.5)A circular window: 1 in the middle, tapering to 0 at radius size/2, made by rotating a 1-D Tukey window. It multiplies amplitudes, so intensities are multiplied by its square.

A new instrument format gets its own reader module beside cxi. The general steps do not change. This is the layout of mbirtorch's preprocess package: general functions, and one module per instrument.

4. Checked against the reference data

The raw file is the gold-ball scan, 800 frames of 621 by 621 and 20 dark frames. The reference is the 794 preprocessed frames of 512 by 512 distributed with ptycho_pmace.

stepresult on the raw gold-ball file
outlier framesFrames 581, 648, 649, 723, 763, 764 are removed. These are the six frames absent from the reference data.
centerThe center of mass of the mean frame is (308.63, 295.18). Rounded, (309, 295), it gives the crop rows 53 to 564 and columns 39 to 550, the crop written by hand in the reference script.
framesAll 794 frames equal the reference frames to rounding: the largest difference is 0.0002 counts, of a largest count of 65,244.
windowEqual to the reference window to rounding (\(3 \times 10^{-8}\)).
reconstructionWith the known probe and these frames, 100 iterations end at data error 0.0906. On the reference frames and positions the same run ends at 0.090721, equal to the reference code, and the object differs from the reference code's by \(3 \times 10^{-6}\).

5. Notes

  1. The definition of the center. The center of mass of the mean frame reproduces the hand-chosen crop on this data. It is one center for all frames. A center per frame would be wrong: the center of mass of a single frame moves with the phase gradient of the object under the probe, so centering each frame would remove real signal. The fractional part of the center (0.37 and 0.18 pixel here) is dropped by the whole-pixel crop.
  2. Positions. The reference data carries positions that were refined with another program and rounded to whole pixels; they differ from the recorded ones by up to 3 pixels. The script above uses the recorded positions as recorded, with rows along \(y\) and columns along \(x\); the object then has the orientation of the SEM image supplied with the data set. This is written in the script, not in the reader.
  3. Saturated pixels. About 90 pixels per frame are at the detector's largest count, 65535. The reference preprocessing does nothing about them, and neither does this.
  4. The blind gold-ball experiment of the 2025 paper. Its script estimates the probe from one mode, adds a second at iteration 10, and at that iteration also replaces the modes by an orthogonal set (a singular value decomposition). The paper also refines the positions during the run. xptycho has the orthogonalization step (orthogonalize_modes), and demo 3 uses it.
  5. Dataset names inside a CXI file. load_raw uses the standard CXI names (entry_1/instrument_1/detector_1/data and so on). They are fixed in the reader, as the names of the format it reads.

Data