Package Structure

healpyxel/
├── nbs/                                            # Notebooks (source of truth)
│   ├── index.ipynb                                 # Package homepage
│   ├── 00_core.ipynb                               # Core utilities
│   ├── 01_sidecar.ipynb                            # Sidecar generation
│   ├── 02_aggregate.ipynb                          # Batch aggregation
│   ├── 03_accumulator.ipynb                        # Streaming accumulation
│   ├── 04_finalize.ipynb                           # Finalization
│   ├── 05_cli.ipynb                                # CLI entry points
│   ├── 06_visualization.ipynb                      # visualization helpers
│   ├── 07_geospatial.ipynb                         # geospatial conversion 
│   ├── 8X_ : Examples
│   │   ├── 80_example_quickstart.ipynb             # jump in 
│   │   ├── 81_example_visualization_workflow.ipynb # complete vis workflow
│   │   ├── 82_example_visualization_psf.ipynb      # WIP : Poinst Spread function over pixels
│   │   ├── 83_example_accumulation.ipynb           # WIP : Accumlate data
│   │   └── 90_example_streaming.ipynb              # WIP : Streaming data
│   └── 90_examples.ipynb                           # Usage examples
│
├── healpyxel/                                      # Auto-generated Python package
├── tests/                                          # Auto-generated tests
├── docs/                                           # Auto-generated documentation
└── test_data/                                      # Test datasets

Test Data Location

Test data has been copied to test_data/ directory:

• test_data/batches/ - Sequential batches for streaming tests
• test_data/samples/ - Size-based samples for quick tests
• test_data/validation/ - Validation datasets

All notebooks should reference test data using relative paths from package root.

from pathlib import Path
import os

# Get package root
pkg_root = Path().absolute()
if pkg_root.name == 'nbs':
    pkg_root = pkg_root.parent

# Test data paths
test_data_dir = pkg_root / 'test_data'
batches_dir = test_data_dir / 'batches'
samples_dir = test_data_dir / 'samples'
validation_dir = test_data_dir / 'validation'

print(f"Package root: {pkg_root}")
print(f"Test data: {test_data_dir}")
print(f"\nAvailable test files:")

if batches_dir.exists():
    batch_files = sorted(batches_dir.glob('*.parquet'))
    print(f"  Batches: {len(batch_files)} files")
    
if samples_dir.exists():
    sample_files = sorted(samples_dir.glob('*.parquet'))
    print(f"  Samples: {len(sample_files)} files")
    
if validation_dir.exists():
    val_files = sorted(validation_dir.glob('*.parquet'))
    print(f"  Validation: {len(val_files)} files")

Package root: /home/mariodamore/Documents/work/MESSENGER/MASCS_gn_dlr_processed/healpyxel
Test data: /home/mariodamore/Documents/work/MESSENGER/MASCS_gn_dlr_processed/healpyxel/test_data

Available test files:
  Batches: 10 files
  Samples: 3 files
  Validation: 2 files

Migration Steps

1. Create Core Utilities (00_core.ipynb)

Extract common functions used across multiple modules: - HEALPix validation - Statistics functions (MAD, robust_std) - File I/O utilities - Logging setup

2. Convert Scripts to Notebooks

For each script (healpix_*.py), create corresponding notebook:

1. Read the script to understand structure
2. Create notebook with appropriate number
3. Add markdown cells explaining functionality
4. Split code into cells by logical sections
5. Add #| export directive to cells that should be in the module
6. Add tests as assert statements in separate cells
7. Add examples showing how to use the functions

3. nbdev Directives

Key directives to use in cells:

#| default_exp module_name    # Define which module this nb creates
#| export                      # Export this cell to the module
#| hide                        # Hide this cell from docs
#| echo: false                 # Don't show code in docs

4. Build and Test

After creating notebooks:

nbdev_export   # Convert notebooks → Python modules
nbdev_test     # Run tests
nbdev_docs     # Generate documentation
nbdev_preview  # Preview docs locally

Example: Loading Test Data

Here’s how to load test data in any notebook:

import pandas as pd
from pathlib import Path

# Load small sample for quick testing
sample_file = test_data_dir / 'samples' / 'sample_5k.parquet'

if sample_file.exists():
    df = pd.read_parquet(sample_file)
    print(f"Loaded {len(df):,} observations")
    print(f"Columns: {len(df.columns)}")
    print(f"\nSpectral columns:")
    spectral_cols = [col for col in df.columns if col.startswith('r') and col[1:4].isdigit()]
    print(f"  {spectral_cols}")
else:
    print(f"⚠️  Test file not found: {sample_file}")

Loaded 5,000 observations
Columns: 61

Spectral columns:
  ['r310', 'r390', 'r750', 'r950', 'r1050', 'r1400', 'r415', 'r433_2', 'r479_9', 'r556_9', 'r628_8', 'r748_7', 'r828_4', 'r898_8', 'r996_2']

Quick Start: Create Core Notebook

Let’s create the core utilities notebook (00_core.ipynb) first:

# This will be done manually in Jupyter
# See the notebook 00_core.ipynb for the template
print("✓ Ready to create notebooks!")
print("\nNext: Open Jupyter and create 00_core.ipynb")

✓ Ready to create notebooks!

Next: Open Jupyter and create 00_core.ipynb

Installation for Development

Make sure you have the package installed in editable mode:

cd healpyxel
pip install -e ".[dev,tdigest,duckdb]"
nbdev_install_hooks