Skip to content

CausalIQ Analysis Metrics

This module provides functions for analysing and comparing causal graphs, including structural metrics, Kullback-Leibler divergence calculations, and Bayesys compatibility metrics.

Core Functions

pdag_compare

pdag_compare(
    graph: Any,
    reference: Any,
    bayesys: Optional[str] = None,
    identify_edges: bool = False,
) -> Dict[str, Any]

Compare a pdag with a reference pdag.

Retained for compatibility; delegates to :func:pdg_compare, which converts both graphs to PDGs before comparing them. For deterministic graphs the low-level metrics are integers, matching historical output.

Parameters:

  • graph

    (PDAG) –

    graph which is to be compared

  • reference

    (PDAG) –

    reference graph for comparison

  • bayesys

    (str, default: None ) –

    version of Bayesys metrics to return, or None if not required

  • identify_edges

    (bool, default: False ) –

    whether edges in each low level category (e.g. arc_missing) are to be included in metrics returned.

Raises:

  • TypeError –

    if bad argument types

Returns:

  • dict ( Dict[str, Any] ) –

    structural comparison metrics

Compare a PDAG with a reference PDAG to compute structural comparison metrics including precision, recall, F1 score, and Structural Hamming Distance (SHD).

pdg_compare

pdg_compare(
    graph: Any,
    reference: Any,
    bayesys: Optional[str] = None,
    identify_edges: bool = False,
) -> Dict[str, Any]

Compare a PDG with a reference PDG.

Both arguments may be PDGs or deterministic graphs (DAG/PDAG/CPDAG), which are converted to PDGs first, each edge state then having probability 1.0. Every node pair contributes the sixteen products of the reference and graph edge-state probabilities, so the low-level edge metrics are fractional for probabilistic graphs and add up to 1.0 for each node pair.

Parameters:

  • graph

    (PDG) –

    graph which is to be compared

  • reference

    (PDG) –

    reference graph for comparison

  • bayesys

    (str, default: None ) –

    version of Bayesys metrics to return, or None if not required

  • identify_edges

    (bool, default: False ) –

    whether edges in each low level category (e.g. arc_missing) are to be included in metrics returned.

Raises:

  • TypeError –

    if bad argument types

  • ValueError –

    if bad bayesys value or the graphs have different nodes

Returns:

  • dict ( Dict[str, Any] ) –

    structural comparison metrics

Compare a PDG with a reference PDG to compute structural comparison metrics. Deterministic DAG/PDAG/CPDAG inputs are converted to PDGs first, so this provides a single probabilistic comparison path for every supported graph type.

kl

kl(dist: Series, ref_dist: Series) -> float

Returns the Kullback-Liebler Divergence of dist from ref_dist.

Parameters:

  • dist

    (Series) –

    distribution to compute KL from ...

  • ref_dist

    (Series) –

    ... the reference/theoretical distribution

Raises:

  • TypeError –

    if both arguments not Panda Series

  • ValueError –

    if dists have different indices or bad values

Returns:

  • float ( float ) –

    divergence value

Compute the Kullback-Leibler divergence of one probability distribution from another reference distribution.

bayesys_metrics

bayesys_metrics(
    metrics: Dict[str, Union[int, float, None]], max_edges: int, num_ref_edges: float
) -> Dict[str, float]

Compute Bayesys-compatible metrics from structural metrics.

If the reference has an edge but the graph has an arc this is counted as a match, and a "half-match" is recorded when the graph has an edge, or an oppositely oriented arc, compared to the reference. Note that edge_matched is not counted as a true positive, which would otherwise give an incorrectly high shd-b for CPDAG comparisons.

Parameters:

  • metrics

    (Dict[str, Union[int, float, None]]) –

    Low-level edge metrics from a PDG comparison.

  • max_edges

    (int) –

    Number of possible node pairs.

  • num_ref_edges

    (float) –

    Number of edges in the reference graph.

Returns:

  • Dict[str, float] –

    Dictionary of Bayesys-compatible metrics.

Compute Bayesys-compatible metrics from structural comparison results, including precision, recall, F1 with half-match support, Bayesys Scoring Function (BSF), and Delta Dependency Measure (DDM).

Overview

PDAG Comparison

The pdag_compare function provides comprehensive structural comparison between two Partially Directed Acyclic Graphs (PDAGs). It computes detailed edge-level metrics including:

  • Arc metrics: arc_matched, arc_reversed, arc_missing, arc_extra
  • Edge metrics: edge_matched, edge_missing, edge_extra, edge_not_arc, arc_not_edge
  • Confusion matrix: missing_matched counts the edges absent from both graphs, completing the 2x2 confusion matrix
  • Summary metrics: precision, recall, F1 score, Structural Hamming Distance (SHD)
  • Bayesys compatibility: Optional Bayesys v1.5+ metrics

The constant EDGE_METRICS lists all ten low-level count names and is used by the evaluate_graph CLI command and workflow action to expand the edge and equiv.edge metric requests.

The function includes built-in sanity checks to ensure metric consistency and can optionally identify specific edges in each category for detailed analysis.

PDG Comparison

The pdg_compare function extends the comparison to Probabilistic Dependency Graphs (PDGs), which assign a probability to each possible edge state between a pair of variables: forward (A -> B), backward (A <- B), undirected (A -- B) and none (no edge). Deterministic graphs (DAG/PDAG/CPDAG) are converted to PDGs first, so pdg_compare compares every supported graph type.

For each node pair the four reference states and four graph states are combined in sixteen products, for example:

Reference Graph Metric Contribution
A -> B 0.6 A -> B 0.4 arc_matched 0.6 x 0.4 = 0.24
A -> B 0.6 A <- B 0.3 arc_reversed 0.6 x 0.3 = 0.18
no edge 0.3 no edge 0.2 missing_matched 0.3 x 0.2 = 0.06

The fractional contributions add up to 1.0 for each node pair. Deterministic inputs (probabilities of 0.0/1.0) therefore reproduce the integer counts of pdag_compare, which is retained as a wrapper around pdg_compare for compatibility.

The derived metrics (precision, recall, F1 and SHD) are computed from the fractional counts in exactly the same way as for pdag_compare.

Distribution Analysis

The kl function computes Kullback-Leibler divergence for comparing probability distributions, commonly used in causal discovery for independence testing and model comparison.

Bayesys Metrics

The bayesys_metrics function converts structural metrics to Bayesys-compatible format, supporting the half-match concept where reversed arcs and edge/arc mismatches are treated as partial matches. This enables comparison with results from the Bayesys causal discovery software.

Usage Examples

Basic PDAG Comparison

from causaliq_analysis.metrics import pdag_compare
from causaliq_core.graph import PDAG

# Compare two PDAGs
result = pdag_compare(learned_graph, reference_graph)
print(f"F1 Score: {result['f1']}")
print(f"SHD: {result['shd']}")

PDG Comparison

from causaliq_analysis.metrics import pdg_compare
from causaliq_core.graph import PDG, EdgeProbabilities

learned = PDG(
    ["A", "B"],
    {("A", "B"): EdgeProbabilities(forward=0.4, backward=0.3, none=0.3)},
)
reference = PDG(
    ["A", "B"],
    {("A", "B"): EdgeProbabilities(forward=0.6, backward=0.1, none=0.3)},
)

# Compare edge probabilities (fractional low-level counts)
result = pdg_compare(learned, reference)
print(f"Matched: {result['arc_matched']}")
print(f"F1: {result['f1']}")

With Bayesys Compatibility

# Include Bayesys v1.5+ metrics
result = pdag_compare(
    learned_graph, reference_graph, bayesys="v1.5+"
)
print(f"Bayesys F1: {result['f1-b']}")
print(f"BSF Score: {result['bsf']}")

KL Divergence

from causaliq_analysis.metrics import kl
import pandas as pd

# Compare two probability distributions
dist1 = pd.Series([0.4, 0.3, 0.3], index=['A', 'B', 'C'])
dist2 = pd.Series([0.33, 0.33, 0.34], index=['A', 'B', 'C'])
divergence = kl(dist1, dist2)
print(f"KL Divergence: {divergence}")