Data Flow

This document describes the primary data flows through XPCS Viewer, from HDF5 file ingestion through computation to GUI visualization.

XPCS Analysis Pipeline

The main data flow for G2 correlation analysis:

        flowchart TD
    HDF5["HDF5 File<br/>/xpcs/ hierarchy"]
    CP["HDF5ConnectionPool<br/>fileIO/hdf_reader.py"]
    XF["XpcsFile<br/>In-memory data model"]
    DC["DataCache<br/>LRU cache for arrays"]
    VK["ViewerKernel<br/>Analysis coordinator"]

    subgraph Backend["Backend Computation"]
        BE["get_backend()<br/>JAX or NumPy"]
        COMP["Array Operations<br/>backend.exp(), backend.mean(), ..."]
        JIT["JIT Cache<br/>(JAX only)"]
    end

    subgraph Fitting["Fitting Pipeline"]
        NLSQ["NLSQ 0.6.0<br/>Point estimates"]
        NUTS["NumPyro NUTS<br/>Posterior samples"]
        FRES["FitResult<br/>ArviZ diagnostics"]
    end

    subgraph Output["Output Boundaries"]
        PQG["PyQtGraph<br/>Interactive plots"]
        MPL["Matplotlib<br/>Publication plots"]
        H5W["HDF5 Write<br/>Fit results"]
    end

    EN["ensure_numpy()<br/>I/O boundary conversion"]

    HDF5 -->|"pool.get_connection()"| CP
    CP -->|"np.ndarray (float64)"| XF
    XF -->|"cached reads"| DC
    XF -->|"g2_data, qmap"| VK
    VK -->|"lazy module load"| COMP
    VK -->|"fit request"| NLSQ

    BE -->|"backend arrays"| COMP
    COMP -->|"JIT-compiled"| JIT

    NLSQ -->|"warm-start init"| NUTS
    NUTS -->|"posterior samples"| FRES

    COMP -->|"analysis results"| EN
    FRES -->|"plot data"| EN

    EN -->|"np.ndarray"| PQG
    EN -->|"np.ndarray"| MPL
    EN -->|"np.ndarray"| H5W

    style EN fill:#f96,stroke:#333,color:#000
    style BE fill:#69f,stroke:#333,color:#000
    style NLSQ fill:#9c6,stroke:#333,color:#000
    style NUTS fill:#9c6,stroke:#333,color:#000
    

Array Type Transitions

Stage	Array Type	Notes
HDF5 read	np.ndarray	Always NumPy from h5py
XpcsFile storage	np.ndarray	Cached as NumPy
Backend computation	JAX array or np.ndarray	Depends on active backend
JIT cache	JAX array	Compiled traces cached
NLSQ fitting	np.ndarray	NLSQ operates on NumPy
NumPyro NUTS	JAX array	Requires JAX backend
I/O boundary	np.ndarray	ensure_numpy() conversion
PyQtGraph/Matplotlib	np.ndarray	Display libraries require NumPy

SimpleMask Data Flow

Mask editing and Q-map computation flow:

        flowchart TD
    IMG["Detector Image<br/>np.ndarray (H, W)"]
    SMW["SimpleMaskWindow<br/>User draws ROIs"]
    SMK["SimpleMaskKernel<br/>Computation core"]

    subgraph MaskOps["Mask Operations"]
        AM["AreaMask<br/>Undo/redo history"]
        DT["DrawingTools<br/>Rect, Circle, Polygon, ..."]
        MASK["Final Mask<br/>int32, values 0/1"]
    end

    subgraph QMapComp["Q-Map Computation"]
        GEO["GeometryMetadata<br/>bcx, bcy, det_dist, ..."]
        QM["qmap.compute_qmap()<br/>JIT-compiled (JAX)"]
        QRES["QMapSchema<br/>sqmap, dqmap, phis"]
    end

    subgraph Partition["Partition Generation"]
        PG["utils.generate_partition()<br/>JIT-compiled (JAX)"]
        PS["PartitionSchema<br/>partition_map, val_list, num_list"]
    end

    subgraph Export["Signal Export"]
        SIG1["mask_exported<br/>Signal(np.ndarray)"]
        SIG2["qmap_exported<br/>Signal(dict)"]
    end

    XV["XPCS Viewer<br/>apply_mask(), apply_qmap_result()"]
    H5["HDF5 File<br/>Mask/Partition persistence"]

    IMG --> SMW
    SMW --> SMK
    SMK --> AM
    DT --> AM
    AM --> MASK

    SMK --> QM
    GEO --> QM
    QM --> QRES

    QRES --> PG
    MASK --> PG
    PG --> PS

    MASK --> SIG1
    PS --> SIG2

    SIG1 -.->|"loose coupling"| XV
    SIG2 -.->|"loose coupling"| XV

    SMK --> H5

    style SIG1 fill:#f96,stroke:#333,color:#000
    style SIG2 fill:#f96,stroke:#333,color:#000
    

Signal Payload Schemas

mask_exported(np.ndarray):

• Shape: (H, W), dtype: int32

• Values: 0 (masked) or 1 (valid)

qmap_exported(dict):

{
    "partition_map": np.ndarray,  # int32, (H, W), Q-bin indices
    "num_pts": int,               # Number of Q-bins
    "val_list": list[float],      # Q-bin center values
    "num_list": list[int],        # Pixels per Q-bin
}

Fitting Data Flow

The two-stage Bayesian fitting pipeline:

        flowchart TD
    G2["G2 Data<br/>delay_times, g2, g2_err"]

    subgraph Stage1["Stage 1: NLSQ Warm-Start"]
        NF["nlsq.nlsq_optimize()"]
        NR["NLSQResult<br/>Point estimates + CurveFitResult"]
        P0["Initial params<br/>tau, baseline, contrast"]
    end

    subgraph Stage2["Stage 2: Bayesian Inference"]
        MC["NumPyro Model<br/>single_exp_model()"]
        NU["NUTS Sampler<br/>4 chains x 1000 samples"]
        PS["Posterior Samples<br/>dict[str, ndarray]"]
    end

    subgraph Diagnostics["Convergence Diagnostics"]
        RH["R-hat < 1.01"]
        ESS["ESS bulk/tail > 400"]
        DIV["Divergences == 0"]
        BF["BFMI >= 0.2"]
        FD["FitDiagnostics"]
    end

    subgraph Results["Result Objects"]
        FR["FitResult<br/>samples + diagnostics"]
        AZ["xarray DataTree<br/>Posterior plots"]
    end

    G2 --> NF
    NF --> NR
    NR -->|"warm-start"| P0
    P0 --> MC
    MC --> NU
    NU --> PS

    PS --> RH
    PS --> ESS
    PS --> DIV
    PS --> BF
    RH --> FD
    ESS --> FD
    DIV --> FD
    BF --> FD

    PS --> FR
    FD --> FR
    FR --> AZ

    style NR fill:#9c6,stroke:#333,color:#000
    style FR fill:#69f,stroke:#333,color:#000
    style FD fill:#f96,stroke:#333,color:#000
    

Convergence Thresholds

Diagnostic	Threshold	Meaning
R-hat	< 1.01	All chains converge to same distribution
ESS bulk	> 400	Sufficient effective samples for mean/variance
ESS tail	> 400	Sufficient effective samples for quantiles
Divergences	== 0	No divergent transitions
BFMI	>= 0.2	Adequate energy exploration

HDF5 File Schema

The standard HDF5 file structure for XPCS data:

        graph LR
    subgraph HDF5["HDF5 File Structure"]
        ROOT["/"]
        XPCS["/xpcs/"]
        QMAP["/xpcs/qmap/"]
        G2["/xpcs/g2/"]
        META["/xpcs/metadata/"]
        SM["/simplemask/"]
        SMMASK["/simplemask/mask/"]
        SMPART["/simplemask/partition/"]
    end

    ROOT --> XPCS
    ROOT --> SM
    XPCS --> QMAP
    XPCS --> G2
    XPCS --> META
    SM --> SMMASK
    SM --> SMPART

    subgraph QMapDS["Q-Map Datasets"]
        SQ["sqmap<br/>float64, (H,W)"]
        DQ["dqmap<br/>float64, (H,W)"]
        PH["phis<br/>float64, (H,W)"]
        MK["mask<br/>int32, (H,W)"]
        PM["partition_map<br/>int32, (H,W)"]
    end

    subgraph G2DS["G2 Datasets"]
        G2V["g2<br/>float64, (n_delay, n_q)"]
        G2E["g2_err<br/>float64, (n_delay, n_q)"]
        DT["delay_times<br/>float64, (n_delay,)"]
        QV["q_values<br/>float64, (n_q,)"]
    end

    subgraph MetaDS["Metadata Attributes"]
        BCX["bcx: float"]
        BCY["bcy: float"]
        DD["det_dist: float (mm)"]
        LAM["lambda_: float (A)"]
        PD["pix_dim: float (mm)"]
        SHP["shape: (H, W)"]
    end

    QMAP --> QMapDS
    G2 --> G2DS
    META --> MetaDS
    

I/O Boundary Summary

All I/O boundaries where ensure_numpy() conversion occurs:

Boundary	Direction	Source Module	Target
PyQtGraph plotting	Backend -> NumPy	module/saxs1d, module/saxs2d, module/g2mod, module/twotime	plot.setData()
Matplotlib plotting	Backend -> NumPy	fitting/visualization	plt.plot()
HDF5 write	Backend -> NumPy	simplemask/area_mask, simplemask/simplemask_kernel	h5py.create_dataset()
HDF5 read	NumPy -> Backend	xpcs_file, fileIO/hdf_reader	backend.from_numpy()
Signal export	Backend -> NumPy	simplemask/qmap, simplemask/utils	Signal.emit()