Data Pipeline API Reference¶
For usage examples and getting started, see the Data Pipeline Guide.
CLI Usage¶
dataset load # Load dataset (uses caches when available)
dataset load --fetch-override # Re-fetch from source, skip all caches
dataset load --process-override # Re-run clean/transform, skip processed caches
dataset status # Per-source file status (path, size, modified time)
dataset graph # Show pipeline dependency graph
Programmatic equivalents:
from project.dataset import pipeline
result = pipeline.load_dataset()
result = pipeline.load_dataset(fetch_override=True)
result = pipeline.load_dataset(process_override=True)
pipeline.show_status()
pipeline.show_graph()
DataSource¶
Pydantic BaseModel representing a single data source. A source is either primary (fetches data independently) or derived (computes from other sources via required_sources).
class DataSource(BaseModel):
name: str
fetch_fn: Callable
raw_file: Optional[Path] = None
processed_file: Optional[Path] = None
read_fn: Optional[Callable[[Union[str, Path]], DataFrame]] = None
write_fn: Optional[Callable[[DataFrame, Union[str, Path]], Path]] = None
clean_fn: Optional[Callable[[DataFrame], DataFrame]] = None
transform_fn: Optional[Callable[[DataFrame], DataFrame]] = None
required_sources: List[str] = []
output: bool = True
Attributes¶
| Attribute | Description |
|---|---|
name |
Unique identifier for the source. Appears in logs, status output, and result dictionaries. |
fetch_fn |
Produces this source's data. Primary: () -> DataFrame. Derived: (Dict[str, DataFrame]) -> DataFrame, where keys are the names from required_sources. |
raw_file |
Path to cache fetched data. Primary sources only (no required_sources). When the file exists, fetch_fn is skipped and data is loaded from disk. On a process_override run, cached raw data is loaded instead of re-fetching. |
processed_file |
Path to cache data after clean and transform. Output sources only (output=True). When the file exists and no override is active, the entire fetch/clean/transform pipeline is skipped. |
read_fn |
Function to read data from storage. Overrides the config-level default. Signature: (Union[str, Path]) -> DataFrame. |
write_fn |
Function to write data to storage. Overrides the config-level default. Signature: (DataFrame, Union[str, Path]) -> Path. |
clean_fn |
Applied to data after fetching. Signature: (DataFrame) -> DataFrame. |
transform_fn |
Applied to data after cleaning. Signature: (DataFrame) -> DataFrame. |
required_sources |
Names of upstream sources this depends on. When non-empty, makes this a derived source. Processing order is determined via topological sort. |
output |
Whether to include this source in the final result. Set to False for intermediate sources that only feed downstream consumers. |
DatasetConfig¶
Pydantic BaseModel that ties together sources and provides default I/O functions.
class DatasetConfig(BaseModel):
sources: Union[DataSource, List[DataSource]]
read_fn: Optional[Callable[[Union[str, Path]], DataFrame]] = None
write_fn: Optional[Callable[[DataFrame, Union[str, Path]], Path]] = None
Attributes ¶
| Attribute | Description |
|---|---|
sources |
One or more DataSource definitions. A single DataSource is normalized to a list internally. |
read_fn |
Default read function. Used by sources that don't define their own read_fn. |
write_fn |
Default write function. Used by sources that don't define their own write_fn. |
Validation Rules¶
DatasetConfig validates the following on construction, raising ValueError on failure:
- Source names must be unique.
- All
required_sourcesentries must reference defined source names. - No circular dependencies between sources.
raw_fileis not allowed on derived sources (sources withrequired_sources).processed_fileis not allowed on non-output sources (output=False).write_fnmust be available (source-level or config-level) whenraw_fileorprocessed_fileis set.read_fnmust be available when cached files already exist on disk.
DatasetManager¶
Manages the complete dataset pipeline, processing sources in dependency order.
class DatasetManager:
def __init__(self, config: DatasetConfig)
def load_dataset(
self,
fetch_override: bool = False,
process_override: bool = False,
) -> Union[DataFrame, Dict[str, DataFrame]]
def show_status(self) -> None
def show_graph(self) -> None
Methods¶
__init__(config)
: Initialize the manager with a DatasetConfig.
load_dataset(fetch_override=False, process_override=False)
: Load and process the complete dataset. Returns a single DataFrame if exactly one source has output=True, otherwise returns a Dict[str, DataFrame] keyed by source name.
show_status()
: Display a table showing per-source file status (existence, size, modification time) for both raw and processed files.
show_graph()
: Display the pipeline dependency graph. In Jupyter notebooks, renders an interactive Mermaid diagram. In the terminal, displays a text-based graph showing sources, dependencies, processing functions, and output status.
Derived Sources and Dependency Ordering¶
Merging and feature engineering are expressed as derived sources -- sources whose required_sources list names one or more upstream sources. The framework determines processing order via topological sort.
How topological sort works:
- Primary sources (no
required_sources) are processed first. - Derived sources are processed after all their dependencies are ready.
- Each derived source's
fetch_fnreceives aDict[str, DataFrame]containing only its declared dependencies.
Merge example:
def merge_sales_customers(deps: Dict[str, DataFrame]) -> DataFrame:
"""Merge sales and customer data on customer_id."""
return deps["sales"].join(deps["customers"], on="customer_id", how="left")
config = DatasetConfig(
sources=[
DataSource(
name="sales",
fetch_fn=fetch_sales,
raw_file=RAW_DATA_DIR / "sales_raw.parquet",
output=False,
),
DataSource(
name="customers",
fetch_fn=fetch_customers,
raw_file=RAW_DATA_DIR / "customers_raw.parquet",
output=False,
),
DataSource(
name="merged",
fetch_fn=merge_sales_customers,
required_sources=["sales", "customers"],
processed_file=PROCESSED_DATA_DIR / "merged.parquet",
clean_fn=clean_merged,
transform_fn=add_features,
output=True,
),
],
read_fn=pl.read_parquet,
write_fn=lambda df, path: df.write_parquet(path),
)
Here sales and customers are primary sources with output=False (intermediate only). The merged source depends on both, performs a join in its fetch_fn, and is the sole output. load_dataset() returns a single DataFrame.
Multi-stage pipeline chaining:
sources = [
DataSource(name="raw_a", fetch_fn=fetch_a, output=False),
DataSource(name="raw_b", fetch_fn=fetch_b, output=False),
DataSource(
name="combined",
fetch_fn=combine_a_b,
required_sources=["raw_a", "raw_b"],
output=False,
),
DataSource(
name="final",
fetch_fn=enrich_combined,
required_sources=["combined"],
processed_file=PROCESSED_DATA_DIR / "final.parquet",
output=True,
),
]
Derived sources can depend on other derived sources, forming arbitrary DAGs. The topological sort resolves the full chain: raw_a, raw_b -> combined -> final.
Function Requirements¶
IO Functions¶
read_fn and write_fn must accept the file paths specified in raw_file and processed_file:
read_fn:(file_path: Union[str, Path]) -> DataFramewrite_fn:(df: DataFrame, file_path: Union[str, Path]) -> Path
The raw_file and processed_file paths are passed directly to these functions. No wrapper is needed.
Data Fetching¶
Primary sources (no required_sources):
def fetch_fn() -> DataFrame
Derived sources (with required_sources):
def fetch_fn(deps: Dict[str, DataFrame]) -> DataFrame
Keys in deps correspond to the names listed in required_sources. Only declared dependencies are included.
Data Processing¶
clean_fn and transform_fn both follow the same signature:
def clean_fn(df: DataFrame) -> DataFrame
def transform_fn(df: DataFrame) -> DataFrame
clean_fn runs first, then transform_fn. Both are optional.
Function Overrides with partial¶
Functions that need additional parameters can be wrapped with functools.partial or lambdas to conform to the required signatures:
from functools import partial
from datetime import datetime
def fetch_data_with_dates(start_date: datetime, end_date: datetime) -> DataFrame:
"""Fetch data for a specific date range."""
...
fetch_recent = partial(
fetch_data_with_dates,
start_date=datetime(2026, 1, 1),
end_date=datetime(2026, 3, 31),
)
source = DataSource(name="sales", fetch_fn=fetch_recent, ...)
Custom parameters from a notebook:
from functools import partial
from datetime import datetime
from project.dataset import fetch_data_with_dates, clean_data, read_dataframe, write_dataframe
from project.dataset_utils import DataSource, DatasetConfig, DatasetManager
from project.config import RAW_DATA_DIR, PROCESSED_DATA_DIR
source = DataSource(
name="sales",
fetch_fn=partial(fetch_data_with_dates, start_date=datetime(2025, 6, 1), end_date=datetime(2025, 12, 31)),
raw_file=RAW_DATA_DIR / "sales_h2_2025.parquet",
processed_file=PROCESSED_DATA_DIR / "sales_h2_2025.parquet",
clean_fn=clean_data,
)
config = DatasetConfig(sources=[source], read_fn=read_dataframe, write_fn=write_dataframe)
df = DatasetManager(config).load_dataset()
This reuses existing functions from dataset.py while controlling parameters at call time. fetch_fn is just a callable -- no framework changes needed.
Data Loading Precedence¶
For each source, processed in topological (dependency) order:
- Processed file cache: If
processed_fileexists on disk and no overrides are active, load and return it. The entire fetch/clean/transform pipeline is skipped. - Raw file cache (primary sources only): If
raw_fileexists and no fetch override, load from disk instead of callingfetch_fn. - Fetch via
fetch_fn: If no cache is available (or overrides are active), callfetch_fn. Primary sources:fetch_fn(). Derived sources:fetch_fn(deps). - Apply
clean_fnthentransform_fn: Run each (if provided) on the fetched data. - Persist to
processed_file: Ifprocessed_fileis set, write the result to disk. - Collect output sources: After all sources are processed, collect those with
output=True. One output source returns aDataFrame; multiple return aDict[str, DataFrame].
Override Behavior¶
| Flag | Effect |
|---|---|
process_override=True |
Skips processed_file caches. Raw data is still loaded from raw_file if available, then clean and transform are re-applied. |
fetch_override=True |
Skips both raw_file and processed_file caches. Data is re-fetched and fully reprocessed. Passing process_override=True alongside has no additional effect. |
Function Precedence and Fallbacks¶
- Source-level:
read_fnandwrite_fndefined onDataSourcetake precedence. - Config-level:
read_fnandwrite_fnonDatasetConfigare used as fallbacks. - Required functions:
write_fnis required ifraw_fileorprocessed_fileis specified.read_fnis required when cached files already exist on disk.fetch_fnis always required for each source.
Best Practices¶
Source Design¶
- Use
output=Falsefor intermediate sources that only feed downstream consumers. - Keep derived sources focused -- one join or transformation per source.
- Use
processed_fileon output sources to cache expensive results. Output sources withoutprocessed_filelog a reminder on every pipeline run.
Function Implementation¶
- Keep functions pure and side-effect free.
- Use type hints for better IDE support.
- Use docstrings to document function behavior.
File Management¶
- Use consistent file extensions (
.csvor.parquet). - Implement proper error handling in read/write functions.
- Set
raw_fileon primary sources where fetching is expensive (API calls, database queries).
Dependency Management¶
- Keep dependency chains shallow where possible.
- Name sources descriptively so the dependency graph is self-documenting.
- Use
show_status()to verify cache state before running the pipeline.
Configuration¶
- Provide
read_fnandwrite_fnat the config level to avoid repeating them on every source. - Use meaningful source names -- they appear in logs, status output, and result dictionaries.
Common Pitfalls¶
-
raw_fileon derived sources: Settingraw_fileon a source withrequired_sourcesraisesValueError. Raw file caching is only for primary sources. -
Missing I/O functions: Not providing
write_fnwhenraw_fileorprocessed_fileis set, or not providingread_fnwhen cached files exist on disk. Caught at config validation time with clear error messages. -
Circular dependencies: Source A requiring source B which requires source A raises
ValueErrorduring config validation. -
Duplicate source names: Every source must have a unique
name. Duplicates are caught at validation time. -
Forgetting
output=Falseon intermediates: If you don't setoutput=Falseon intermediate sources, they are included in the result dict, which may not be intended and may have performance implications based on dataset size. -
processed_fileon non-output sources: Settingprocessed_fileon a source withoutput=FalseraisesValueError. Since overrides are pipeline-wide, intermediate sources are always recomputed -- caching them to disk provides no benefit. -
Undefined
required_sourcesreferences: Listing a source name inrequired_sourcesthat doesn't exist in the config raisesValueError. -
Missing
processed_fileon output sources: Output sources withoutprocessed_fileare recomputed on everyload_dataset()call. The pipeline logs a reminder. Setprocessed_fileto enable caching unless you intentionally want fresh computation every time. -
Unexpected return type:
load_dataset()returns a singleDataFramewhen exactly one source hasoutput=True, and aDict[str, DataFrame]otherwise. Design your pipeline with the expected return type in mind.