XPCS Data Formats and Conventions

This page explains the data formats and file conventions used by XPCS Viewer for X-ray Photon Correlation Spectroscopy data. Understanding these formats is important for interoperability with beamline data pipelines and third-party analysis tools.

What is XPCS Data?

X-ray Photon Correlation Spectroscopy (XPCS) measures temporal fluctuations of coherent X-ray speckle patterns scattered from a sample. The raw data consists of a time series of 2D detector images, from which correlation functions are computed to extract sample dynamics.

The primary data products of an XPCS experiment are:

  • G2 correlation functions: \(g_2(\tau)\) curves for each Q-bin, encoding the sample dynamics (see G2 Correlation Function Theory).

  • Q-maps: Pixel-to-Q mappings derived from detector geometry (see Mask and Q-Map Internals).

  • Two-time correlation matrices: \(C(t_1, t_2)\) for detecting non-equilibrium dynamics (see Two-Time Correlation Analysis).

  • Masks: Boolean arrays identifying valid detector pixels.

  • Metadata: Experimental geometry, beam parameters, and acquisition settings.

HDF5 File Structure

XPCS Viewer stores and reads data in HDF5 files. The standard hierarchy follows conventions from the APS 8-ID-I beamline, with extensions for mask and partition data from the SimpleMask module.

/
+-- xpcs/                      # Main XPCS data group
|   +-- qmap/                  # Q-space mapping
|   |   +-- sqmap              # float64, (H, W) -- static Q magnitude
|   |   +-- dqmap              # float64, (H, W) -- dynamic Q magnitude
|   |   +-- phis               # float64, (H, W) -- azimuthal angle
|   |   +-- mask               # int32, (H, W)   -- pixel mask (0/1)
|   |   +-- partition_map      # int32, (H, W)   -- Q-bin index per pixel
|   |
|   +-- g2/                    # G2 correlation results
|   |   +-- g2                 # float64, (n_delay, n_q)
|   |   +-- g2_err             # float64, (n_delay, n_q)
|   |   +-- delay_times        # float64, (n_delay,)
|   |   +-- q_values           # float64, (n_q,)
|   |
|   +-- metadata/              # Geometry and experimental parameters
|       +-- bcx                # float -- beam center X (column, pixels)
|       +-- bcy                # float -- beam center Y (row, pixels)
|       +-- det_dist           # float -- sample-to-detector distance (mm)
|       +-- lambda_            # float -- X-ray wavelength (Angstrom)
|       +-- pix_dim            # float -- pixel size (mm)
|       +-- shape              # (int, int) -- detector (height, width)
|
+-- simplemask/                # SimpleMask-generated data
|   +-- mask/                  # Mask with metadata and version
|   +-- partition/             # Q-bin partition with geometry
|
+-- exchange/                  # Two-time correlation data
    +-- C2T_all                # float64, (n_frames, n_frames) -- C(t1, t2)

Beamline Key Mapping

Different beamlines store XPCS results under different HDF5 paths. XPCS Viewer uses a key map (fileIO/aps_8idi.py) to translate logical data names to beamline-specific paths. For example, the G2 correlation data might reside at /xpcs/g2 at one beamline and /exchange/norm-0-g2 at another.

This abstraction allows XPCS Viewer to read data from multiple sources without modifying the analysis modules.

Schema Validation

Raw HDF5 data is untyped beyond basic array shape and dtype. To enforce correctness, XPCS Viewer uses schema validators – frozen dataclasses that validate data at I/O boundaries.

Schema

Validates

Key Constraints

QMapSchema

Q-map arrays and units

Matching shapes, float64 dtype, no NaN, valid unit strings

GeometryMetadata

Detector geometry

Positive distances and wavelength, beam center within shape

G2Data

Correlation data

Shape consistency, float64, non-negative errors, monotonic delay times

PartitionSchema

Q-bin partition

Integer partition map, matching list lengths, non-negative counts

MaskSchema

Pixel mask

2D integer array, values only 0 or 1

Schemas are constructed at the point of data ingestion (the I/O boundary), so invalid data is rejected immediately with a clear error message rather than causing subtle failures downstream.

Unit Conventions

XPCS Viewer follows these unit conventions:

Quantity

Unit

Notes

Q magnitude

nm-1 (preferred) or A-1

Internal default is nm^-1; legacy files may use A^-1

Azimuthal angle

radians or degrees

Internal computation uses radians; display may use degrees

Delay times

seconds

Logarithmically spaced in most experiments

Detector distance

mm

Sample-to-detector distance

Pixel size

mm

Physical size of one detector pixel

Wavelength

Angstrom

Converted from energy via \(\lambda = 12.398 / E_{\text{keV}}\)

X-ray energy

keV

Input parameter for Q-map computation

Array Conventions

  • Mask values: 0 = masked (invalid), 1 = valid pixel.

  • Array dtype: float64 for all scientific data; int32 for masks and partition maps.

  • Array orientation: (height, width) – row-major (C order), with (0, 0) at the top-left corner of the detector image.

  • Beam center: bcx is the column (X) index, bcy is the row (Y) index, both in pixel coordinates (0-indexed).

Connection Pooling and the HDF5 Facade

HDF5 files are accessed through the HDF5Facade class, which combines schema validation with connection pooling. The connection pool caches open file handles keyed by (file_path, mode) to avoid repeated open/close overhead during interactive analysis where the same file is read many times.

The facade provides a unified interface for all data types:

  • read_qmap() returns a validated QMapSchema

  • read_g2_data() returns a validated G2Data

  • read_geometry_metadata() returns a validated GeometryMetadata

  • write_mask() and write_partition() accept schema objects and write with compression and version metadata

This design ensures that all HDF5 access goes through a single, validated code path rather than being scattered across modules.