Workflow Caching
CausalIQ Workflow uses SQLite-based caches to store step results, enabling conservative execution and reproducibility.
What is a Workflow Cache?
A workflow cache is a .db file containing:
- Entries: Individual results from workflow steps
- Metadata: Key-value pairs describing each entry (algorithm, parameters, metrics)
- Objects: Named data objects (graphs, tables, traces) stored with each entry
Caches are the primary mechanism for passing results between workflow steps and enabling conservative execution.
Cache Entries
Each entry in a cache has:
| Component | Description |
|---|---|
| Key | Matrix values that uniquely identify the entry |
| Metadata | Dictionary of properties (algorithm, scores, timestamps) |
| Objects | Named data items (e.g., graph, trace, summary) |
Entry Keys
Entries are keyed by their matrix values. For a workflow with:
matrix:
network: [asia, cancer]
sample_size: [100, 1000]
Each entry is identified by a unique {network, sample_size} combination.
Entry Metadata
Metadata uses schema version 2 and four blocks: provenance, parameters,
result and history. The framework writes provenance, the action writes
result (and parameters when it supplies them), and history is reserved
for metadata inherited from earlier actions.
{
"provenance": {
"provider": "causaliq-discovery",
"provider_version": "0.2.0.dev1",
"workflow_version": "0.6.0.dev9",
"action": "learn_graph",
"started": "2026-10-22T06:06:16+01:00",
"elapsed": 225.0,
"environment": {
"os": "Linux v6.8.0",
"cpu": "AMD EPYC 7763",
"python": "3.12.4",
"ram": 64
}
},
"parameters": {},
"result": {
"status": "ok",
"algorithm": "pc",
"node_count": 8,
"edge_count": 8
},
"history": {}
}
During the structure phase, action fields live in result; the separate
values audit moves parameter-like fields into parameters. See
Metadata Structure for the full
schema.
Metadata Names in Filters and Plots
- Bare names refer to matrix values and result-object fields:
sample_size,f1,equiv.f1. - Metadata is producer-qualified:
learn_graph.status,learn_graph.hyperparameters.iss. - Explicit forms (
learn_graph.parameters.algorithm,learn_graph.result.status) and provider-qualified forms (causaliq_discovery.learn_graph.status) are available when needed. - A bare metadata name such as
statusis rejected; the reference must name the producer.
with:
action: plot
input: results/metrics.db
filter: learn_graph.algorithm_seconds < 3600
x: sample_size
y: f1
Entry Objects
Each cache holds one object type, and every successful entry holds exactly one object of that type:
| Name | Format | Description |
|---|---|---|
dag |
GraphML | Learned DAG |
pdg |
GraphML | Merged or LLM-sourced PDG |
metrics |
JSON | Evaluation scores |
trace |
JSON | Algorithm iteration history |
Objects are stored as content strings with a format identifier. See Cache Type Model for the full design.
Conservative Execution
By default, workflows execute conservatively — skipping work that has already been completed:
- Create steps: Skip if entry with matching key exists
- Aggregate steps: Skip if output entry with matching key exists
- Run-once steps: Execute on the first job only
This enables:
- Resumable workflows: Restart interrupted workflows without re-running completed steps
- Incremental analysis: Write new results to their own typed caches
- Efficient iteration: Modify workflow and re-run without starting from scratch
Bypassing Conservative Execution
Use --mode=force to re-run all steps regardless of existing results:
cqflow run workflow.yml --mode=force
Cache Files
Cache files are self-contained SQLite databases:
results/
├── graphs.db # Learned graphs from discovery
├── evaluated.db # Graphs with evaluation metrics
└── merged.db # Aggregated results
Cache Location
Cache paths in input: and output: resolve against the working
directory:
steps:
- name: "Learn"
uses: "causaliq-discovery"
with:
action: "learn_structure"
output: "results/graphs.db"
Use a workflow parameter when the location must change between runs:
output_root: "results"
steps:
- name: "Learn"
uses: "causaliq-discovery"
with:
action: "learn_structure"
output: "{{output_root}}/graphs.db"
cqflow run workflow.yml --param output_root=/experiments/project-001/results
Exporting and Importing
Caches can be exported to open formats for sharing and archival:
# Export to directory
cqflow export-cache -i results/graphs.db -o ./exported
# Export to zip
cqflow export-cache -i results/graphs.db -o results.zip
# Import from export
cqflow import-cache -i ./exported -o results/restored.db
Exported format uses:
- JSON for metadata
- GraphML for graph objects
- JSON for other objects
Each _meta.json records schema_version: 2, matrix_values and the
metadata envelope. Version 1 exports remain readable: import normalises
them to the current schema.
This enables interoperability with external tools and long-term archival in open formats.
Cache Schema Consistency
When using a cache across multiple workflow runs, the matrix dimensions must
remain consistent. Adding or removing matrix variables from a workflow that
writes to an existing cache will raise a MatrixSchemaError.
To change matrix dimensions, either:
- Use a new cache file
- Export, delete, and re-import the cache
- Delete the cache and regenerate
Python API
For programmatic cache access, see the Workflow Cache API.
from causaliq_workflow.cache import WorkflowCache
with WorkflowCache("results/graphs.db") as cache:
# Check if entry exists
key = {"network": "asia", "sample_size": 1000}
if cache.exists(key):
entry = cache.get(key)
print(entry.metadata["result"]["status"])