Organization of .dat files

Imaging data is stored in binary files with the extension .dat. Each .dat file holds a single channel and begins with a fixed 512-byte header that describes the data that follows: its numeric class, frame rate, exposure, channel name, and the size of each axis. A file can therefore be opened and reshaped from its own contents alone, without AcqInfos.mat or events.mat.

For example, in a recording where a fluorescence channel was acquired with red and green illuminations, each channel is stored in its own .dat file, and each file carries its own frame rate and size. The Data Import functions create these files, and so do the processing functions that save image results.

AcqInfos.mat does not describe .dat files
The shape, class, frame rate, and exposure of a .dat file come only from its header. AcqInfos.mat describes the raw acquisition, not the imported files; see Save Folder metadata files.

What a .dat file can hold

A .dat file holds image data and always has the axes Y and X (image rows and columns). It can also have:

Axis	Meaning
Y, X	Image rows and columns. Always present.
T	Time (frames).
E	Events: one slice per event instance, or one slice per condition after averaging over events.
F	Frequency.

The file can therefore be a single frame (Y-X), a time series (Y-X-T), an event-split array (Y-X-T-E or Y-X-E), or a frequency map (for example Y-X-F). Results that are not images, or that combine several named measurements, are saved as .umt files instead. Authors of custom functions can find the rules for choosing between the two formats in Choosing the output format.

The header

The header has a constant size of 512 bytes and every field is at a fixed position, so data always begins at byte 512, whatever the shape of the file.

Section	Contents
Identification (16 bytes)	A signature that marks the file as a umIT .dat file, the byte order, the header version, a write complete flag, and the position where the data starts.
Channel information (464 bytes)	Numeric class of the values, frame rate (Hz), exposure (ms), and channel name (up to 31 characters). Most of this section is reserved for future channel-level information.
Dimension table (32 bytes)	The size of the Y, X, T, E, and F axes. A size of 0 means the axis is absent. Three further slots are reserved.

The frame rate is a property of each file: channels of one recording can run at different rates, and processing steps that change the rate (for example temporal binning at import) write the new rate to the file they create. A file without a T axis stores no frame rate.

The write complete flag is set only after the last data value has been written. If umIT opens a file in which it is not set, it warns that the file may be incomplete, for example after an interrupted processing step.

Supported numeric classes are single, double, uint8, uint16, int16, uint32, and int32. Imported data and the files written by saveData use 32-bit single precision.

Ordering of the data

MATLAB array: data(Y, X, T, E, F)   (only the axes present in the file)
Serialized order, starting at byte 512:
  Y varies fastest (column X=1 from Y=1..Height), then X, then T, then E, then F

This is MATLAB column-major order over the axes that are present, taken in the order Y, X, T, E, F. An absent axis contributes nothing: a file with the axes Y, X, and E has the same layout as a Y-by-X-by-E array. Each value uses the number of bytes of the file's numeric class.

Event-split files

A .dat file split by event has an E axis whose size is the only event-related information in the file. It stores no event names or identities. umIT matches the E axis to events.mat in the same Save Folder by size: as many slices as there are events means one slice per event instance, and as many slices as there are conditions means one slice per condition after averaging. If neither matches, DataViewer shows E as repetitions of a single condition and warns. For this reason, do not purge events from events.mat while event-split files exist in the folder; the Events Manager prevents it.

How DataViewer reads data

DatImageSource opens files with the axes Y-X, Y-X-T, Y-X-E, and Y-X-T-E, reads frames and pixels directly from the file, and maintains an optional spatial-temporal cache. DataViewer does not require the whole recording in RAM.

Files from previous releases

Older .dat files have no header. Files created by previous releases of umIT, which are described by their own companion .mat file with the same base name (or <name>_info.mat), can still be opened. Their data starts at byte 0 and the companion file provides the size, class, and frame rate.

Headerless files that were described only by the AcqInfos.mat of their Save Folder, an intermediate format used in development versions, can no longer be read. umIT reports that the file must be re-imported from the raw data.

Programmatic access

Only four functions read the bytes of a .dat file, and every other part of the toolbox goes through them:

Function	Use
loadMetaData	Describes any readable .dat file (header or companion file).
loadData	Loads the file into a MATLAB array. Use it when the data fits in memory.
mapDat	Memory-maps the file for suitable programmatic workflows.
spatialSlabIO	Streams a file in parts (spatial slabs or selected frames), so files larger than the computer's memory can be processed.

loadMetaData returns a structure with these fields, for every kind of readable .dat file:

Field	Meaning
filePath	Full path of the file.
format	'header' for files with a header, or 'legacySidecar' for files described by a companion file.
dataOffset	Byte position where the data starts (512 for files with a header, 0 otherwise).
dataClass	MATLAB class of the values.
dimNames, dimSizes	Names and sizes of the axes present, in storage order.
frameRateHz, exposureMsec	Frame rate and exposure of this file (NaN when unknown or not applicable).
channelName	Channel label stored in the file.
writeComplete	Whether the file was completely written.

info = loadMetaData("C:\save\green.dat");
info.dimNames                            % for example {'Y','X','T'}
nFrames = datAxisSize(info, "T");        % size of an axis, or 0 when absent
data = loadData("C:\save\green.dat");    % use when it fits in memory
% For large files, use DatImageSource or spatialSlabIO.

Likewise, only two functions write the bytes of a .dat file: saveData (whole arrays and appending along the last axis) and spatialSlabIO (streamed writes). Both always write the header. Details for function authors are in the custom functions guide.

File naming

File names identify logical channels or processing results (for example green.dat or HbO.dat). Names never define the dimensions: those come from the header. Processing provenance is recorded in dataHistory.mat.

Never edit bytes independently
The header and the data belong together. Do not change precision, reshape assumptions, or file length with external tools, and do not replace or rename the files of one channel of a dual-camera recording individually. Use the importer or analysis function that owns the change.