Skip to content

Summarisation Paradigm Design

Overview

The Summarisation Paradigm enables workflows to aggregate results from multiple matrix combinations into summary outputs. This is essential for causal discovery research where experiments are run across many parameter combinations (networks, sample sizes, algorithms) and results need to be combined for analysis, model averaging, or comparison.

Matrix Variable Handling

Matrix variables serve different purposes depending on the action pattern:

  • CREATE pattern: Matrix variables must be used in {{variable}} templates to produce distinct action invocations.
  • AGGREGATE pattern: Matrix variables define grouping dimensions for entry selection. They are not required in parameter templates since they control which cache entries are passed to the action.
  • Run-once steps: A step writing a plain file runs once, so matrix variables do not define its dimensionality.

Motivation

Research Workflow Pattern

A typical causal discovery workflow generates many individual results:

matrix:
  network: [asia, alarm, sachs]
  sample_size: [100, 500, 1000]
  seed: [1, 2, 3, 4, 5]

steps:
  - name: Generate Graph
    uses: causaliq/research
    with:
      action: generate_graph
      output: results/graphs.db

This produces 45 individual graphs (3 × 3 × 5). Researchers then need to:

  1. Aggregate by network: Combine all graphs for each network
  2. Filter by quality: Only include successful runs
  3. Compute summaries: Average structures, consensus edges, etc.

The Summarisation Paradigm provides first-class support for this pattern.

Architecture

Aggregation Mode Detection

Aggregation mode is automatically activated when a step has:

  1. A workflow matrix definition
  2. An input parameter with .db cache file(s)
def _is_aggregation_step(
    self,
    step: Dict[str, Any],
    matrix: Dict[str, List[Any]],
) -> bool:
    """Check if a step should execute in aggregation mode."""
    if not matrix:
        return False
    step_inputs = step.get("with", {})
    input_param = step_inputs.get("input")
    if input_param:
        inputs = (
            [input_param]
            if isinstance(input_param, str)
            else (list(input_param) if input_param else [])
        )
        return any(str(p).lower().endswith(".db") for p in inputs)
    return False

Aggregation Configuration

The AggregationConfig dataclass captures all aggregation parameters:

@dataclass
class AggregationConfig:
    input_caches: List[str]     # Cache paths to scan
    filter_expr: Optional[str]  # Filter expression
    matrix_vars: List[str]      # Grouping dimensions

Execution Phases

Aggregation execution has two phases:

Phase 1: Scan

The scan phase reads input caches, applies filters, and groups entries:

def _scan_aggregation_inputs(
    self,
    config: AggregationConfig,
    matrix_values: Dict[str, Any],
    logger: Optional[Callable[[str], None]] = None,
) -> List[Dict[str, Any]]:
    """Scan input caches and collect entries matching matrix values."""

Scan Process:

The scan runs once per matrix job, but each input cache is read only once per run:

  1. Build or reuse a memoised scan index for each input cache. The index holds each entry's parsed matrix values and flattened metadata, read via WorkflowCache.list_entry_summaries() without decompressing object payloads (e.g. graphs).
  2. Apply the filter expression (if present) against the flattened metadata; rejected entries are counted as filtered.
  3. Match surviving entries to the current matrix combination; non-matching entries are skipped.
  4. Load only the matched entries in full (via cache.get()) and return them with provenance.

The random() filter population and the matching step both read from the same index, so a large cache is not re-read for every matrix combination. A cache's index is discarded whenever that cache is written (a step store or an UPDATE), so later scans observe new entries.

Performance:

Scanning is O(cache entries) per input cache rather than O(matrix jobs × cache entries): the index is built once and reused across every matrix job and across the validation and execution passes. Only matched entries pay the cost of decompressing their object payloads.

Phase 2: Execute

The execute phase passes grouped entries to the action:

# In _execute_job:
agg_config = self._get_aggregation_config(step, matrix)
if agg_config is not None:
    matching_entries = self._scan_aggregation_inputs(
        agg_config,
        job,  # Current matrix values
    )
    resolved_inputs["_aggregation_entries"] = matching_entries

Actions receive entries via _aggregation_entries parameter.

Workflow Syntax

Basic Aggregation

matrix:
  network: [asia, alarm]

steps:
  # First step: generate individual graphs
  - name: Generate
    uses: causaliq/research
    with:
      action: generate_graph
      output: graphs.db

  # Second step: aggregate by network
  - name: Summarise
    uses: causaliq/analysis
    with:
      action: model_average
      input: graphs.db
      output: summaries.db

With Filter Expression

Filter entries before aggregation using matrix values, result-object fields and producer metadata:

- name: Summarise Successful
  uses: causaliq/analysis
  with:
    action: model_average
    input: graphs.db
    filter: "learn_graph.status == 'ok' and learn_graph.edge_count > 0"
    output: summaries.db

Multiple Input Caches

Combine results from multiple caches:

- name: Merge Results
  uses: causaliq/analysis
  with:
    action: merge_graphs
    input:
      - pc_results.db
      - ges_results.db
    output: merged.db

Filter Expression Syntax

Filter expressions use a safe subset of Python syntax via simpleeval:

Supported Operators

Operator Example Description
== learn_graph.status == 'ok' Equality
!= learn_graph.algorithm != 'pc' Inequality
>, < learn_graph.edge_count > 5 Comparison
>=, <= sample_size >= 100 Comparison
and a > 5 and b < 10 Logical AND
or a > 5 or b > 10 Logical OR
not not failed Logical NOT
in learn_graph.algorithm in ['pc', 'ges'] Membership

Available Variables

Filter expressions can access:

  1. Matrix values: network, sample_size, etc.
  2. Result-object fields: bare names such as f1 or equiv.f1.
  3. Metadata: producer-qualified names such as learn_graph.status or learn_graph.hyperparameters.iss.
# Filter by matrix variable
filter: "sample_size >= 500"

# Filter by result-object field
filter: "f1 > 0.8"

# Filter by metadata field
filter: "learn_graph.status == 'ok'"

# Filter by qualified path
filter: "causaliq_research.generate_graph.node_count > 5"

Reporting View

Filters, summarise and plot evaluate one flat reporting view of each entry:

  • matrix values and result-object fields by bare name (sample_size, f1);
  • metadata by producer-qualified name (generate_graph.node_count, learn_graph.hyperparameters.iss);
  • explicit generate_graph.parameters.* / generate_graph.result.* forms and provider-qualified causaliq_research.generate_graph.node_count forms when needed.

Version 1 provider -> action -> fields metadata is normalised before viewing. Provenance is recorded for reproducibility but is not exposed in filters by default. Bare metadata names are rejected: the reference must name the producer.

Entry Structure

Actions receive entries as dictionaries:

{
    "matrix_values": {"network": "asia", "sample_size": 100},
    "metadata": {...},      # Version 2 metadata envelope
    "cache_path": "graphs.db",
    "entry_hash": "abc123...",
    "entry": CacheEntry(...)  # Full entry object
}

Logging

Aggregation operations log statistics:

Aggregation scan: scanned=45, filtered=3, matched=15
  • scanned: Total entries examined
  • filtered: Entries rejected by filter
  • matched: Entries matching current matrix values

Error Handling

Missing Caches

Non-existent caches are skipped with warning:

Warning: Cache does not exist: missing.db

Filter Errors

Filter evaluation errors cause entry to be filtered:

try:
    if not evaluate_filter(filter_expr, flat_meta):
        total_filtered += 1
        continue
except Exception:
    total_filtered += 1
    continue

Cache Read Errors

Cache read failures are logged and skipped:

Warning: Failed to read cache graphs.db: database is locked

Integration with causaliq-core

The filter expression evaluation uses causaliq_core.utils:

from causaliq_core.utils import evaluate_filter

if config.filter_expr:
    if not evaluate_filter(config.filter_expr, flat_meta):
        continue

This provides consistent filter syntax across the CausalIQ ecosystem.

Future Enhancements

Weight-Based Aggregation

Support weighted combinations using compute_weight:

- name: Weighted Average
  uses: causaliq/analysis
  with:
    action: model_average
    aggregate: graphs.db
    weight: "1.0 / sample_size"  # Weight by inverse sample size

Hierarchical Aggregation

Multi-level aggregation for complex analyses:

- name: Per-Network Summary
  with:
    aggregate: graphs.db
    group_by: [network]

- name: Global Summary
  with:
    aggregate: network_summaries.db
    group_by: []  # Aggregate all

See Also