How .DAT Files Are Organized
Imaging datasets are stored in binary files with the extension .dat. Each .dat file contains only a single channel. For example, in a recording where a fluorescence channel was acquired with red and green illuminations, each channel is stored separately in a dedicated .dat file. These files are created using one of the available Data Import functions. The binary files store only imaging data as 4-byte (32-bit) floating-point values (single precision). To properly read a .dat file, the necessary information about data dimensions and precision is stored in an associated .mat file with the same name as the .dat file. This metadata file also stores other relevant details, such as the frame rate and exposure time.
The Metadata File
As mentioned above, the .mat file associated with a binary .dat file contains all the metadata required to open and process the data in the toolbox. Below is an example of the metadata file contents:
| Field | Description |
|---|---|
| Datatype | Format of numerical data (e.g., single) |
| Freq | Recording sample rate in Hz |
| datSize | Size of the first and second dimensions of data |
| datLength | Size of the third and additional dimensions of data |
| dim_names | Cell array with dimension identifiers, such as Time, X, and Y |
| dataHistory | Structure array with metadata about the pipeline steps used to create the file. This is used by the Pipeline Manager Tool to skip steps during a pipeline execution |
| tExposure | Illumination exposure time (in milliseconds) |
Minimal Required Information
The minimal required metadata stored in the file should include:
- dim_names: A cell array containing dimension identifiers such as X, Y, T(ime), and E(vent).
- Datatype: The data's precision (e.g., single).
- datSize and datLength: Data's dimensions. Used to properly read the values from the binary .dat file
Metadata Changes During Data Processing
During a typical preprocessing pipeline, raw imaging data—typically stored as an image time series with dimensions Y, X, and T—can change. When a new file is created, its metadata is inherited from the input file. If a parameter is modified during processing, the corresponding field is updated in the new metadata file. For example, if temporal binning is applied, the Freq parameter is adjusted accordingly while the rest of the metadata remains unchanged.
A Note on Dimension Names
The dim_names variable is used by some analysis functions as well as visualization tools to define how the data dimensions are structured. For instance, if an image time series is split by events and averaged, its dimensions will transition from a 3D format (Y, X, T) to a 4D structure (E, Y, X, T). A dictionary of valid dimension names can be found in the .mat file: /subFunc/dimension_names.mat.
Resources for Creating Custom Data Import Functions
The toolbox provides helper functions for creating custom import functions that allow reading and saving .dat files. These functions are stored in the ../subFunc/ folder:
| loadDatFile | Reads a .dat file |
| mapDatFile | Maps the .dat file in RAM (see MATLAB's memmapfile documentation) |
| save2Dat | Saves data to a .dat file, including associated metadata |
| genMetaData | Helps generate valid metadata for use with the save2Dat function. Extra meta data can be included here. |