Skip to content

Data Pipeline API Reference

For usage examples and getting started, see the Data Pipeline Guide.

CLI Usage

dataset load                  # Load dataset (uses caches when available)
dataset load --fetch-override # Re-fetch from source, skip all caches
dataset load --process-override # Re-run clean/transform, skip processed caches
dataset status                # Per-source file status (path, size, modified time)
dataset graph                 # Show pipeline dependency graph

Programmatic equivalents:

from project.dataset import pipeline

result = pipeline.load_dataset()
result = pipeline.load_dataset(fetch_override=True)
result = pipeline.load_dataset(process_override=True)
pipeline.show_status()
pipeline.show_graph()

DataSource

Pydantic BaseModel representing a single data source. A source is either primary (fetches data independently) or derived (computes from other sources via required_sources).

class DataSource(BaseModel):
    name: str
    fetch_fn: Callable
    raw_file: Optional[Path] = None
    processed_file: Optional[Path] = None
    read_fn: Optional[Callable[[Union[str, Path]], DataFrame]] = None
    write_fn: Optional[Callable[[DataFrame, Union[str, Path]], Path]] = None
    clean_fn: Optional[Callable[[DataFrame], DataFrame]] = None
    transform_fn: Optional[Callable[[DataFrame], DataFrame]] = None
    required_sources: List[str] = []
    output: bool = True

Attributes

Attribute Description
name Unique identifier for the source. Appears in logs, status output, and result dictionaries.
fetch_fn Produces this source's data. Primary: () -> DataFrame. Derived: (Dict[str, DataFrame]) -> DataFrame, where keys are the names from required_sources.
raw_file Path to cache fetched data. Primary sources only (no required_sources). When the file exists, fetch_fn is skipped and data is loaded from disk. On a process_override run, cached raw data is loaded instead of re-fetching.
processed_file Path to cache data after clean and transform. Output sources only (output=True). When the file exists and no override is active, the entire fetch/clean/transform pipeline is skipped.
read_fn Function to read data from storage. Overrides the config-level default. Signature: (Union[str, Path]) -> DataFrame.
write_fn Function to write data to storage. Overrides the config-level default. Signature: (DataFrame, Union[str, Path]) -> Path.
clean_fn Applied to data after fetching. Signature: (DataFrame) -> DataFrame.
transform_fn Applied to data after cleaning. Signature: (DataFrame) -> DataFrame.
required_sources Names of upstream sources this depends on. When non-empty, makes this a derived source. Processing order is determined via topological sort.
output Whether to include this source in the final result. Set to False for intermediate sources that only feed downstream consumers.

DatasetConfig

Pydantic BaseModel that ties together sources and provides default I/O functions.

class DatasetConfig(BaseModel):
    sources: Union[DataSource, List[DataSource]]
    read_fn: Optional[Callable[[Union[str, Path]], DataFrame]] = None
    write_fn: Optional[Callable[[DataFrame, Union[str, Path]], Path]] = None

Attributes

Attribute Description
sources One or more DataSource definitions. A single DataSource is normalized to a list internally.
read_fn Default read function. Used by sources that don't define their own read_fn.
write_fn Default write function. Used by sources that don't define their own write_fn.

Validation Rules

DatasetConfig validates the following on construction, raising ValueError on failure:

  • Source names must be unique.
  • All required_sources entries must reference defined source names.
  • No circular dependencies between sources.
  • raw_file is not allowed on derived sources (sources with required_sources).
  • processed_file is not allowed on non-output sources (output=False).
  • write_fn must be available (source-level or config-level) when raw_file or processed_file is set.
  • read_fn must be available when cached files already exist on disk.

DatasetManager

Manages the complete dataset pipeline, processing sources in dependency order.

class DatasetManager:
    def __init__(self, config: DatasetConfig)
    def load_dataset(
        self,
        fetch_override: bool = False,
        process_override: bool = False,
    ) -> Union[DataFrame, Dict[str, DataFrame]]
    def show_status(self) -> None
    def show_graph(self) -> None

Methods

__init__(config) : Initialize the manager with a DatasetConfig.

load_dataset(fetch_override=False, process_override=False) : Load and process the complete dataset. Returns a single DataFrame if exactly one source has output=True, otherwise returns a Dict[str, DataFrame] keyed by source name.

show_status() : Display a table showing per-source file status (existence, size, modification time) for both raw and processed files.

show_graph() : Display the pipeline dependency graph. In Jupyter notebooks, renders an interactive Mermaid diagram. In the terminal, displays a text-based graph showing sources, dependencies, processing functions, and output status.


Derived Sources and Dependency Ordering

Merging and feature engineering are expressed as derived sources -- sources whose required_sources list names one or more upstream sources. The framework determines processing order via topological sort.

How topological sort works:

  1. Primary sources (no required_sources) are processed first.
  2. Derived sources are processed after all their dependencies are ready.
  3. Each derived source's fetch_fn receives a Dict[str, DataFrame] containing only its declared dependencies.

Merge example:

def merge_sales_customers(deps: Dict[str, DataFrame]) -> DataFrame:
    """Merge sales and customer data on customer_id."""
    return deps["sales"].join(deps["customers"], on="customer_id", how="left")

config = DatasetConfig(
    sources=[
        DataSource(
            name="sales",
            fetch_fn=fetch_sales,
            raw_file=RAW_DATA_DIR / "sales_raw.parquet",
            output=False,
        ),
        DataSource(
            name="customers",
            fetch_fn=fetch_customers,
            raw_file=RAW_DATA_DIR / "customers_raw.parquet",
            output=False,
        ),
        DataSource(
            name="merged",
            fetch_fn=merge_sales_customers,
            required_sources=["sales", "customers"],
            processed_file=PROCESSED_DATA_DIR / "merged.parquet",
            clean_fn=clean_merged,
            transform_fn=add_features,
            output=True,
        ),
    ],
    read_fn=pl.read_parquet,
    write_fn=lambda df, path: df.write_parquet(path),
)

Here sales and customers are primary sources with output=False (intermediate only). The merged source depends on both, performs a join in its fetch_fn, and is the sole output. load_dataset() returns a single DataFrame.

Multi-stage pipeline chaining:

sources = [
    DataSource(name="raw_a", fetch_fn=fetch_a, output=False),
    DataSource(name="raw_b", fetch_fn=fetch_b, output=False),
    DataSource(
        name="combined",
        fetch_fn=combine_a_b,
        required_sources=["raw_a", "raw_b"],
        output=False,
    ),
    DataSource(
        name="final",
        fetch_fn=enrich_combined,
        required_sources=["combined"],
        processed_file=PROCESSED_DATA_DIR / "final.parquet",
        output=True,
    ),
]

Derived sources can depend on other derived sources, forming arbitrary DAGs. The topological sort resolves the full chain: raw_a, raw_b -> combined -> final.


Function Requirements

IO Functions

read_fn and write_fn must accept the file paths specified in raw_file and processed_file:

  • read_fn: (file_path: Union[str, Path]) -> DataFrame
  • write_fn: (df: DataFrame, file_path: Union[str, Path]) -> Path

The raw_file and processed_file paths are passed directly to these functions. No wrapper is needed.

Data Fetching

Primary sources (no required_sources):

def fetch_fn() -> DataFrame

Derived sources (with required_sources):

def fetch_fn(deps: Dict[str, DataFrame]) -> DataFrame

Keys in deps correspond to the names listed in required_sources. Only declared dependencies are included.

Data Processing

clean_fn and transform_fn both follow the same signature:

def clean_fn(df: DataFrame) -> DataFrame
def transform_fn(df: DataFrame) -> DataFrame

clean_fn runs first, then transform_fn. Both are optional.

Function Overrides with partial

Functions that need additional parameters can be wrapped with functools.partial or lambdas to conform to the required signatures:

from functools import partial
from datetime import datetime

def fetch_data_with_dates(start_date: datetime, end_date: datetime) -> DataFrame:
    """Fetch data for a specific date range."""
    ...

fetch_recent = partial(
    fetch_data_with_dates,
    start_date=datetime(2026, 1, 1),
    end_date=datetime(2026, 3, 31),
)

source = DataSource(name="sales", fetch_fn=fetch_recent, ...)

Custom parameters from a notebook:

notebook.ipynb
from functools import partial
from datetime import datetime
from project.dataset import fetch_data_with_dates, clean_data, read_dataframe, write_dataframe
from project.dataset_utils import DataSource, DatasetConfig, DatasetManager
from project.config import RAW_DATA_DIR, PROCESSED_DATA_DIR

source = DataSource(
    name="sales",
    fetch_fn=partial(fetch_data_with_dates, start_date=datetime(2025, 6, 1), end_date=datetime(2025, 12, 31)),
    raw_file=RAW_DATA_DIR / "sales_h2_2025.parquet",
    processed_file=PROCESSED_DATA_DIR / "sales_h2_2025.parquet",
    clean_fn=clean_data,
)

config = DatasetConfig(sources=[source], read_fn=read_dataframe, write_fn=write_dataframe)
df = DatasetManager(config).load_dataset()

This reuses existing functions from dataset.py while controlling parameters at call time. fetch_fn is just a callable -- no framework changes needed.


Data Loading Precedence

For each source, processed in topological (dependency) order:

  1. Processed file cache: If processed_file exists on disk and no overrides are active, load and return it. The entire fetch/clean/transform pipeline is skipped.
  2. Raw file cache (primary sources only): If raw_file exists and no fetch override, load from disk instead of calling fetch_fn.
  3. Fetch via fetch_fn: If no cache is available (or overrides are active), call fetch_fn. Primary sources: fetch_fn(). Derived sources: fetch_fn(deps).
  4. Apply clean_fn then transform_fn: Run each (if provided) on the fetched data.
  5. Persist to processed_file: If processed_file is set, write the result to disk.
  6. Collect output sources: After all sources are processed, collect those with output=True. One output source returns a DataFrame; multiple return a Dict[str, DataFrame].

Override Behavior

Flag Effect
process_override=True Skips processed_file caches. Raw data is still loaded from raw_file if available, then clean and transform are re-applied.
fetch_override=True Skips both raw_file and processed_file caches. Data is re-fetched and fully reprocessed. Passing process_override=True alongside has no additional effect.

Function Precedence and Fallbacks

  1. Source-level: read_fn and write_fn defined on DataSource take precedence.
  2. Config-level: read_fn and write_fn on DatasetConfig are used as fallbacks.
  3. Required functions:
  4. write_fn is required if raw_file or processed_file is specified.
  5. read_fn is required when cached files already exist on disk.
  6. fetch_fn is always required for each source.

Best Practices

Source Design

  • Use output=False for intermediate sources that only feed downstream consumers.
  • Keep derived sources focused -- one join or transformation per source.
  • Use processed_file on output sources to cache expensive results. Output sources without processed_file log a reminder on every pipeline run.

Function Implementation

  • Keep functions pure and side-effect free.
  • Use type hints for better IDE support.
  • Use docstrings to document function behavior.

File Management

  • Use consistent file extensions (.csv or .parquet).
  • Implement proper error handling in read/write functions.
  • Set raw_file on primary sources where fetching is expensive (API calls, database queries).

Dependency Management

  • Keep dependency chains shallow where possible.
  • Name sources descriptively so the dependency graph is self-documenting.
  • Use show_status() to verify cache state before running the pipeline.

Configuration

  • Provide read_fn and write_fn at the config level to avoid repeating them on every source.
  • Use meaningful source names -- they appear in logs, status output, and result dictionaries.

Common Pitfalls

  1. raw_file on derived sources: Setting raw_file on a source with required_sources raises ValueError. Raw file caching is only for primary sources.

  2. Missing I/O functions: Not providing write_fn when raw_file or processed_file is set, or not providing read_fn when cached files exist on disk. Caught at config validation time with clear error messages.

  3. Circular dependencies: Source A requiring source B which requires source A raises ValueError during config validation.

  4. Duplicate source names: Every source must have a unique name. Duplicates are caught at validation time.

  5. Forgetting output=False on intermediates: If you don't set output=False on intermediate sources, they are included in the result dict, which may not be intended and may have performance implications based on dataset size.

  6. processed_file on non-output sources: Setting processed_file on a source with output=False raises ValueError. Since overrides are pipeline-wide, intermediate sources are always recomputed -- caching them to disk provides no benefit.

  7. Undefined required_sources references: Listing a source name in required_sources that doesn't exist in the config raises ValueError.

  8. Missing processed_file on output sources: Output sources without processed_file are recomputed on every load_dataset() call. The pipeline logs a reminder. Set processed_file to enable caching unless you intentionally want fresh computation every time.

  9. Unexpected return type: load_dataset() returns a single DataFrame when exactly one source has output=True, and a Dict[str, DataFrame] otherwise. Design your pipeline with the expected return type in mind.