Pipeline Manager

Pipeline Manager builds a directed acyclic graph (DAG): a sequence of analysis functions in which arrows show how data moves from one operation to the next. The included functions cover common widefield-imaging processing and analysis workflows. If you want to create a custom analysis function of your own, see Creating analysis functions for PipelineManager.

The manager records which functions and parameters produced each result, chooses an execution order, and reduces memory use by passing large data through files when appropriate. Its graph makes parallel and sequential processing paths visible before the pipeline runs.

Open the app

From DataViewer, load a .dat or image-kind .umt, open the Pipeline tab, and choose Pipeline Manager. The current file is already present as the protected input node.

To open the standalone app from MATLAB:

PipelineManagerTool;

Choose Select Data Folders before adding steps in standalone mode.

The current file is shown as the starting point in the graph. Function categories come from live PipelineManager discovery.

Interface layout

Available analysis functions are listed on the left. The middle shows the pipeline graph and execution order. The right side summarizes the selected step and its parameters. File, data, execution, and inspection controls are along the top; run progress and the activity log are at the bottom.

Area/control	Purpose
Available Functions + search	Filters discovered functions by category and name.
Add / Add as New Branch	Adds a compatible step and prompts for required sources.
Pipeline Graph	Shows sources, steps, connections, persistence markers, and selection.
Selected step details	Edits step tag and function options; summarizes inputs, outputs, and persistence.
Execution Order	Shows the runnable order and virtual inputs.
Load, Save, Clear Pipeline	Loads, saves, or clears the graph after applicable confirmation.
Auto-save final outputs	Controls automatic saving of outputs from the last steps. This choice is fixed to temporary viewer results when the app is opened from DataViewer.
RAM / Safety	Selects automatic or RAM-safe planning and the safety policy.
Reuse Existing Files	Allows compatible outputs already on disk to satisfy steps.
Run / Stop	Reviews and executes the graph, or requests a cooperative stop.

Build and run a first pipeline

1. Expand a function category on the left, select a function, and choose Add.
2. If the function needs an input that is not determined automatically, select the source in the dialog. Compatible outputs from earlier steps, files in the Save Folder, and the current DataViewer file are shown where applicable.
3. Select the new graph step. Right-click it and choose Edit parameters to set analysis options.
4. Add the next function. The manager connects compatible input and output types and updates the execution order.
5. Use Save Outputs on a step when a particular intermediate result must be written.
6. Choose Run, inspect the execution plan and warnings, then confirm.
7. Follow progress in the graph, execution-order panel, and activity log. After the run, inspect Tools > Show Latest Run Summary.

Graph interactions

Action	How and why
Select a step	Click a node to show its name, tag, inputs, outputs, parameters, and save choices on the right.
Edit parameters	Right-click a function node and choose Edit parameters.
Reconnect inputs	Right-click and choose Reconnect inputs, select the input to change, then click a highlighted compatible source or choose a file.
Save outputs	Right-click and choose Save Outputs to select which step outputs should persist.
Remove a step	Right-click and choose Remove step. Downstream connections are rechecked; the protected DataViewer source cannot be removed.
Clear the graph	Clear Pipeline removes all steps after confirmation but keeps the selected folders and execution settings.

Adding a parallel branch

Add as New Branch starts another path instead of extending the currently selected path. Use branches when one pipeline should process several input files or generate two independent results from the same source. The graph shows where branches share an input and where they remain separate.

Pipeline script generation

Choose File > Generate Script to export the current graph as a MATLAB script. The generated script reproduces the configured steps, connections, parameters, data folders, and execution settings. Review the file paths before running it on another computer.

Batch data folders

Data > Select Data Folders builds a table of Save Folder/Raw Folder pairs. Add folders one at a time or discover child Save Folders, filter the list, set a Raw Folder, use the Save Folder as its own raw source, mark unavailable raw data as Missing, and enable or disable rows. Invalid or duplicate Save Folders block the run. A step that requires raw data skips a row marked Missing rather than guessing a location.

Execution settings

Setting	Meaning
Reuse existing files	Uses an existing output only when its recorded pipeline history matches; disable it to force re-execution.
Avoid overwriting existing files	Chooses another available output filename when possible.
Save recovery data on failure	Saves the latest available data when a step fails, when possible.
Auto-save final outputs	Disabled saves only explicitly selected outputs; Enabled saves final branch outputs; Enabled as temporary files creates viewer-managed temporary results.

RAM settings

RAM: Auto lets the manager keep data in memory when estimates show that enough memory is available. RAM-safe prefers disk-backed processing for compatible functions. Strict blocks a plan that cannot meet its RAM rules; Best effort allows the manager to continue with warnings when a complete low-memory path is unavailable.

Execution > Advanced RAM Settings exposes four planning controls:

Control	Default	Meaning
Lookahead length	2	Number of downstream steps considered when deciding whether to retain or reload data.
Cooldown steps	2	Number of steps an intermediate result may remain in memory before release.
Safety overhead fraction	0.15	Fraction of available memory reserved as overhead (15% by default).
Data size multiplier	2.0	Multiplier applied to estimated data size to account for working copies.

Tools menu

Command	Purpose
Show Execution Plan	Lists executable steps, virtual inputs*, run order, and issues before execution.
Remove Invalid Steps	Removes steps that cannot be made runnable after validation.
Show Latest Run Summary	Displays completed, skipped, failed, and output rows from the most recent run.
View Folder Pipeline Log	Loads pipeLog.mat for a selected enabled Save Folder.
Export Error Log	Writes detailed diagnostic text for sharing or later review.

* Virtual input
A virtual input is a file or value that supplies a step without being produced by another step in the graph. Examples include the current DataViewer file and a file selected from the Save Folder.

Output files and control files

In standalone mode, saved final or explicitly selected outputs are written to each Save Folder using the function's declared file name. In DataViewer mode, compatible image results are offered for immediate display and are temporary by default; non-image reports and terminal results remain saved files when the function creates or requests them.

Item	Contents
Pipeline file	The saved graph, connections, parameters, and execution choices. Written only by Save Pipeline.
dataHistory.mat	For each output, the functions, versions, inputs, and parameters that produced it.
pipeLog.mat	Per-step execution status, timing, messages, inputs, and outputs for that Save Folder.
Temporary image results	Viewer-managed files removed when their lifecycle ends unless the user saves a permanent copy.

Read function-specific warnings before running.
Some functions change an entire folder. For example, data-import functions can replace classified output files, and applyRegistrationTformOnFolder rewrites every image .dat and image .umt file. Review the function documentation, execution plan, and confirmation dialog.