Skip to content

Evaluating Graphs

The evaluate_graph capability structurally evaluates a graph against a reference graph. Deterministic graphs (DAG, PDAG, CPDAG) and Probabilistic Dependency Graphs (PDGs) are supported; the latter are compared by edge probability. Note that comparisons between general SDG graphs are not supported.

This is a map action (see workflow patterns) when used within a CausalIQ workflow: it reads the learned graph from the input graph cache and writes a metrics object to the output cache. The metric values live only in that object; the entry's metadata records the effective parameters and the outcome status (see Workflow Metadata).


Parameters

Parameter CLI Flag Required Description
input -i/--input Yes Learned graph file (CLI) or workflow cache .db (action)
reference -r/--reference Yes Reference graph file (.csv, .graphml, .tetrad, .xdsl, .dsc) or workflow cache (.db)
metric -m/--metric Yes Metric(s) to compute (repeatable in CLI)
output -o/--output Workflow Metrics cache (.db) receiving the metrics object
filter — No Filter expression for cache entries (workflow only)

Supported Metrics: f1, shd, precision, recall, edge, node_count, equiv.f1, equiv.shd, equiv.edge

node_count reports the number of nodes in the evaluated graph. A successful comparison requires the learned graph and the reference to share the same node set, so it is also the reference's node count; it is useful for tables and for plots that relate error metrics to network size.

Notes:

  • In CLI mode, input is a graph file (.csv, .graphml, .tetrad, .xdsl, .dsc) and output is a directory where _meta.json will be written.
  • IN CLI mode you can request multiple metrics by repeating the -m/00metric option e.g. "-m f1 -m shd"
  • In workflows, input is a workflow cache (.db) and output is a workflow cache (.db) receiving a metrics object. The filter parameter can select specific cache entries.
  • In workflows, reference can be another workflow cache (.db) whose entries contain reference graphs with the same key structure as the input cache. Each input entry is compared against the reference entry with identical matrix variable values.

CLI Usage

Basic Comparison

Compare a learned graph against a reference:

causaliq-analysis evaluate-graph -i learned.graphml -r ground_truth.graphml \
    -m f1 -m shd -o results/eval

This creates results/eval/_meta.json containing:

{
  "f1": 0.7,
  "shd": 3
}

All Metrics

Request all available metrics:

causaliq-analysis evaluate-graph -i learned.graphml -r ground_truth.graphml \
    -m f1 -m shd -m precision -m recall -m edge -m node_count \
    -m equiv.f1 -m equiv.shd -m equiv.edge -o results/eval

Equivalence Class Metrics

Compare equivalence classes (CPDAGs) rather than raw graphs:

causaliq-analysis evaluate-graph -i learned.graphml -r ground_truth.graphml \
    -m equiv.f1 -m equiv.shd -o results/eval

Low-level Edge Counts

The edge metric expands to the nine low-level edge categories reported by pdag_compare, plus missing_matched which completes the 2x2 confusion matrix:

Count Meaning
arc_matched Arc present with the same orientation in both graphs
arc_reversed Arc present in both graphs but oppositely orientated
edge_not_arc Undirected edge in the graph, arc in the reference
arc_not_edge Arc in the graph, undirected edge in the reference
edge_matched Undirected edge present in both graphs
arc_extra Arc in the graph with no counterpart in the reference
edge_extra Undirected edge in the graph with no counterpart
arc_missing Arc in the reference with no counterpart in the graph
edge_missing Undirected edge in the reference with no counterpart
missing_matched Edges absent from both graphs (correctly rejected)
causaliq-analysis evaluate-graph -i learned.graphml -r ground_truth.graphml \
    -m edge -o results/eval

This writes results/eval/_meta.json containing the counts:

{
  "arc_matched": 2,
  "arc_reversed": 0,
  "edge_not_arc": 0,
  "arc_not_edge": 0,
  "edge_matched": 0,
  "arc_extra": 0,
  "edge_extra": 0,
  "arc_missing": 0,
  "edge_missing": 0,
  "missing_matched": 4
}

missing_matched is the number of edges absent from both graphs. It is derived as the maximum possible number of edges minus the sum of the other counts, so it is usually the largest of the counts.

When either graph is a PDG these counts are fractional: each node pair contributes the sixteen products of the reference and graph edge-state probabilities, and the counts sum to the number of node pairs. Because a PDG has no single equivalence class, equiv.* metrics are unavailable for PDG inputs and are omitted with a warning.

equiv.edge reports the same counts computed after converting both graphs to CPDAGs, exactly as equiv.f1 and equiv.shd do. Its keys are prefixed with equiv.:

causaliq-analysis evaluate-graph -i learned.graphml -r ground_truth.graphml \
    -m equiv.edge -o results/eval
{
  "equiv.arc_matched": 0,
  "equiv.arc_reversed": 0,
  "equiv.edge_not_arc": 0,
  "equiv.arc_not_edge": 0,
  "equiv.edge_matched": 2,
  "equiv.arc_extra": 0,
  "equiv.edge_extra": 0,
  "equiv.arc_missing": 0,
  "equiv.edge_missing": 0,
  "equiv.missing_matched": 1
}

Because the counts are stored as flat metadata keys, they can be aggregated directly with the summarise action:

causaliq-analysis summarise -i results/eval/_meta.json \
    -m arc_matched.mean -m equiv.missing_matched.sd -o results/summary.csv

Workflow Usage

In a CausalIQ workflow, evaluate_graph operates as a MAP action: it reads the matched entry of the input cache and writes a metrics object to its own output cache. The metrics cache has the same dimensions as the input cache, so the workflow matrix may be omitted and is taken from the input cache.

steps:
  - name: "Evaluate Graphs"
    uses: "causaliq-analysis"
    with:
      action: "evaluate_graph"
      input: "results/graphs.db"
      output: "results/metrics.db"
      reference: "reference/asia_true.graphml"
      metric:
        - f1
        - shd
        - precision
        - recall

Each entry in metrics.db holds a metrics object with the requested values; its provenance metadata records the status, the reference and the evaluated graph type. summarise and plot read the values from that object.

Workflow Cache Reference

Instead of a single ground-truth graph file, reference can point to another workflow cache (.db) containing reference graphs. This enables comparing graphs across caches that share identical cache keys, for example when checking that causal discovery produces the same graphs as legacy BNSL code:

steps:
  - name: "Compare against legacy BNSL graphs"
    uses: "causaliq-analysis"
    with:
      action: "evaluate_graph"
      input: "results/causaliq_graphs.db"
      reference: "results/legacy_bnsl_graphs.db"
      metric:
        - f1
        - shd

The reference cache must use the same key structure (matrix variable names) as the input cache. Each input entry is evaluated against the reference entry with identical matrix variable values (e.g. network='asia', sample_size=100). If the key structures differ, no matching reference entry exists, or a reference entry does not contain a graph, an error is reported.


Supported Metrics

Available Metrics Summary

Metric Description
f1 F1 score from direct graph comparison
shd Structural Hamming Distance
precision Precision from direct comparison
recall Recall from direct comparison
equiv.f1 F1 comparing equivalence classes (CPDAGs)
equiv.shd SHD comparing equivalence classes (CPDAGs)
edge Low-level edge comparison counts (see above)
node_count Number of nodes in the evaluated graph
equiv.edge Low-level edge counts from CPDAG comparison

Metric Naming in CausalIQ

Many different structural metrics are used to evaluate graphs in causal discovery. Common ones are F1, Precision, Recall and Structural Hamming Distance (SHD), but others specific to causal discovery, such as Structural Intervention Distance (SID), are also employed.

Critical differences in structural evaluation include:

  • Whether the raw graphs (e.g., a learned DAG and a reference DAG) are compared, or whether the equivalence classes (CPDAGs or PAGs) to which they belong are compared. The former is generally more appropriate in causal discovery where orientation of arcs is critical.
  • Many structural metrics are built upon true/false positive/negative counts, and different authors take different approaches to computing these counts for arcs which have an orientation property.
  • Some authors report the raw metric but others normalise it (e.g., SHD divided by the number of variables or edges).

CausalIQ uses the following naming structure for metrics:

[<preprocessing>].<metric>.[<semantics>].[<postprocessing>].[<statistic>]

Element Optional Description Supported Values
<preprocessing> Yes Preprocessing before comparison equiv (convert to CPDAGs first)
<metric> No The basic metric f1, shd, precision, recall, and the low-level edge counts (arc_matched, arc_reversed, edge_not_arc, arc_not_edge, edge_matched, arc_extra, edge_extra, arc_missing, edge_missing, missing_matched)
<scheme> Yes Alternative computation semantics not currently supported
<postprocessing> Yes Postprocessing, e.g., normalisation not currently supported
<statistic> Yes Statistic over multiple values see summarise action

Legacy Support

The core module which provides structural comparisons between PDAGs (mixed directed and undirected edge graphs, a superset of DAGs and CPDAGs) is pdag_compare in metrics.py. It implements the comparison semantics used consistently in CausalIQ papers and the legacy discovery repository.

Comparison Semantics

To be completed — will describe in detail how the CausalIQ code computes the confusion matrix counts that underlie the structural metrics.

Failed Runs

A failed input run holds provenance only — no object — so it cannot be scored. evaluate_graph therefore writes a provenance-only entry to the metrics cache, carrying result.status: "input_failed" and result.error: "No graph present". The upstream reason the graph was not produced stays with the producing entry rather than being copied here. This keeps one entry per attempted job, so failure cases can be analysed (e.g. failure incidence by sample size or algorithm), whilst no f1, precision or shd value is ever produced for them. Summaries count only the successful jobs for those metrics.

Reference cache entries must still hold an evaluable graph; a failure in the reference cache is an error.


Workflow Metadata

Each cache entry records two metadata blocks alongside the framework provenance. Metric values are never duplicated into metadata: summarise and plot read the metrics object.

Parameters — every user-facing parameter is recorded, including those the workflow does not supply (as null):

Key Value
input Input cache path, or null when omitted
reference Reference graph file or cache path
metric Requested metrics as a list
output Output cache path, resolved from the open step cache when the executor removes output
filter Filter expression, or null when omitted

Result — the status of the evaluation:

Case Result block
Success {"status": "ok"}
No learnt graph {"status": "input_failed", "error": "No graph present"}
Dry run {"status": "skipped"}

For example, a successful evaluation of the Asia network records:

{
  "parameters": {
    "input": "results/learn.db",
    "reference": "networks/asia/asia.xdsl",
    "metric": ["f1", "shd"],
    "output": "results/metrics.db",
    "filter": null
  },
  "result": {"status": "ok"}
}

The evaluated graph type is not recorded: it can be recovered from the metadata history once that feature is implemented.

See Also