Troubleshooting Guide

This guide provides solutions to common issues encountered when using the ADCIRC Subgrid Preprocessor.

Installation Issues

GDAL Installation Problems

Problem: GDAL fails to install or import with errors like:

ImportError: No module named 'osgeo'
ERROR: Could not install packages due to an OSError

Solutions:

  1. Use Conda for GDAL Installation:

    # Remove existing installation
pip uninstall gdal

# Install via conda
conda install -c conda-forge gdal

  2. System-Level GDAL Installation (Linux):

    # Ubuntu/Debian
sudo apt-get update
sudo apt-get install gdal-bin libgdal-dev
export GDAL_CONFIG=/usr/bin/gdal-config
pip install gdal==$(gdal-config --version)

# CentOS/RHEL
sudo yum install gdal gdal-devel
pip install gdal

  3. macOS Specific:

    # Using Homebrew (may conflict with conda)
brew install gdal

# Set environment variables
export GDAL_CONFIG=/usr/local/bin/gdal-config
pip install gdal

Verification:

python -c "from osgeo import gdal; print('GDAL version:', gdal.__version__)"

Memory Errors During Installation

Problem: Installation fails with memory-related errors:

MemoryError: Unable to allocate array
ERROR: Failed building wheel for package

Solutions:

  1. Increase Virtual Memory:

    # Linux
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

  2. Use No-Cache Installation:

    pip install --no-cache-dir .

  3. Install Dependencies Individually:

    pip install numpy
pip install scipy
pip install pandas
# Continue with other packages...

Dependency Conflicts

Problem: Conflicting package versions:

ERROR: pip's dependency resolver does not currently take into account
all the packages that are installed

Solutions:

  1. Create Fresh Environment:

    conda deactivate
conda env remove -n adcirc-subgrid
conda create -n adcirc-subgrid python=3.9

  2. Use Conda-Lock Files:

    conda install --file requirements/adcirc-subgrid-conda-linux-64.yaml

  3. Pin Specific Versions:

    pip install "numpy>=1.20,<1.25" "pandas>=1.3,<2.0"

Runtime Errors

Configuration File Errors

Problem: YAML parsing errors:

yaml.scanner.ScannerError: while scanning a simple key
could not find expected ':'

Solutions:

  1. Check YAML Syntax:

    # Use online YAML validator or
python -c "import yaml; yaml.safe_load(open('config.yaml'))"

  2. Common YAML Issues:

    # Incorrect (missing space after colon)
input:
  adcirc_mesh:./fort.14

# Correct
input:
  adcirc_mesh: ./fort.14

# Incorrect (inconsistent indentation)
options:
 n_subgrid_levels: 50
   n_phi_levels: 50

# Correct
options:
  n_subgrid_levels: 50
  n_phi_levels: 50

File Path Issues

Problem: File not found errors:

FileNotFoundError: No such file or directory: './fort.14'

Solutions:

  1. Use Absolute Paths:

    input:
  adcirc_mesh: /full/path/to/fort.14
  dem: /full/path/to/elevation.tif

  2. Verify File Existence:

    ls -la fort.14
file fort.14  # Check file type

  3. Check Current Directory:

    pwd  # Print working directory
ls   # List files in current directory

Memory and Performance Issues

Problem: Out of memory errors during processing:

MemoryError: Unable to allocate 2.3 GiB for an array with shape...

Solutions:

  1. Reduce Window Memory:

    adcirc-subgrid prep config.yaml --window-memory 32

  2. Process Smaller Regions:

    # Use GDAL to clip input data
gdal_translate -projwin xmin ymin xmax ymax input.tif subset.tif

  3. System Memory Settings:

    # Linux: Check available memory
free -h

# Increase swap space if needed
sudo swapon --show

  4. Environment Variables:

    export GDAL_CACHEMAX=256  # MB
export OMP_NUM_THREADS=2  # Reduce parallelism

Coordinate System Issues

Problem: Coordinate system mismatches:

ValueError: Incompatible coordinate reference systems

Solutions:

  1. Check Coordinate Systems:

    gdalsrsinfo elevation.tif
gdalsrsinfo landcover.tif

  2. Reproject Data:

    # Reproject to WGS84
gdalwarp -t_srs EPSG:4326 input.tif output_wgs84.tif

  3. Verify ADCIRC Mesh CRS:

    Check ADCIRC documentation or mesh metadata for coordinate system information.

Data Quality Issues

Invalid φ Values

Problem: φ values outside the valid range [0,1]:

Warning: Found phi values > 1.0 or < 0.0

Diagnosis:

import xarray as xr
import numpy as np

ds = xr.open_dataset('subgrid.nc')
phi = ds['phi']

# Check for invalid values
invalid_high = (phi > 1.0).any()
invalid_low = (phi < 0.0).any()

print(f"Values > 1.0: {invalid_high.sum().item()}")
print(f"Values < 0.0: {invalid_low.sum().item()}")

Solutions:

  1. Check DEM Quality:

    gdalinfo -stats elevation.tif
# Look for unusual min/max values

  2. Verify Datum Consistency:

    Ensure DEM and mesh use consistent vertical datums.

  3. Adjust Processing Parameters:

    options:
  distribution_factor: 0.8  # Reduce range
  n_subgrid_levels: 30      # Fewer levels

Non-Monotonic φ Relationships

Problem: φ decreases with increasing water level:

Warning: Non-monotonic phi relationship detected

Diagnosis:

# Check monotonicity
phi_diff = ds['phi'].diff(dim='phi_level')
non_monotonic = (phi_diff < 0).any(dim='phi_level')
problem_nodes = non_monotonic.sum().item()

Solutions:

  1. Increase Processing Levels:

    options:
  n_subgrid_levels: 75  # More levels for smoother relationships

  2. Use Histogram Distribution:

    options:
  subgrid_level_distribution: histogram  # More adaptive

  3. Clean Input DEM:

    # Remove spikes and fill holes
gdal_fillnodata.py elevation.tif elevation_cleaned.tif

Visualization Problems

Matplotlib Display Issues

Problem: Plots don’t display or show blank windows:

UserWarning: Matplotlib is currently using agg, which is a non-GUI backend

Solutions:

  1. Set Backend:

    import matplotlib
matplotlib.use('TkAgg')  # or 'Qt5Agg'
import matplotlib.pyplot as plt

  2. Install GUI Backend:

    # For Tkinter backend
sudo apt-get install python3-tk

# For Qt backend
pip install PyQt5

  3. Use X11 Forwarding (SSH):

    ssh -X username@hostname

Large Dataset Visualization

Problem: Visualization crashes with large datasets:

MemoryError during plotting

Solutions:

  1. Plot Subsets:

    adcirc-subgrid plot-mesh subgrid.nc --variable percent_wet \
    --bbox -95.0 29.0 -94.0 30.0

  2. Reduce Point Density:

    # In Python visualization scripts
stride = 10  # Plot every 10th point
plt.scatter(x[::stride], y[::stride], c=phi[::stride])

  3. Use Vector Graphics:

    adcirc-subgrid plot-mesh subgrid.nc --output-filename plot.svg

Performance Optimization

Slow Processing

Problem: Preprocessing takes excessively long:

Diagnosis:

  1. Profile Processing Steps:

    adcirc-subgrid --verbose prep config.yaml 2>&1 | tee processing.log

  2. Check Resource Usage:

    # Monitor during processing
htop  # or top
iostat -x 1  # I/O monitoring

Solutions:

  1. Optimize Configuration:

    options:
  n_subgrid_levels: 30    # Reduce from 50+
  subgrid_level_distribution: histogram  # More efficient

  2. Increase Memory Allocation:

    adcirc-subgrid prep config.yaml --window-memory 128

  3. Use Faster Storage:

    Move input/output files to SSD storage for better I/O performance.

  4. Parallel Processing:

    export OMP_NUM_THREADS=4  # Use available cores

Large File Handling

Problem: Processing very large DEM files:

Solutions:

  1. Use Pyramids/Overviews:

    gdaladdo -r average elevation.tif 2 4 8 16

  2. Tile Large Files:

    gdal_retile.py -ps 2048 2048 -targetDir tiles/ elevation.tif

  3. Use Cloud-Optimized GeoTIFF:

    gdal_translate -of COG -co COMPRESS=LZW input.tif output_cog.tif

Integration Issues

ADCIRC Compatibility

Problem: ADCIRC doesn’t read subgrid file:

ERROR in READ_SFLUX_DATA: NetCDF file not found or corrupted

Solutions:

  1. Verify File Format:

    ncdump -h subgrid.nc | head -20

  2. Check File Permissions:

    ls -la subgrid.nc
chmod 644 subgrid.nc

  3. Verify Node Count Match:

    # Compare mesh and subgrid node counts
mesh_nodes=$(head -2 fort.14 | tail -1 | awk '{print $2}')
subgrid_nodes=$(ncdump -h subgrid.nc | grep "node =" | cut -d= -f2 | cut -d' ' -f2)
echo "Mesh: $mesh_nodes, Subgrid: $subgrid_nodes"

ADCIRC fort.15 Configuration

Problem: ADCIRC run fails with subgrid enabled:

Solutions:

  1. Check NWP Setting:

    NWP = 1  ! Must be 1 to enable subgrid

  2. Verify File Path:

    ! Use relative or absolute path
NCFILE = './subgrid.nc'
! or
NCFILE = '/full/path/to/subgrid.nc'

  3. Adjust Timestep:

    DTDP = 0.5  ! May need smaller timestep for stability

Getting Help

Log Information

When reporting issues, include:

  1. Verbose Log Output:

    adcirc-subgrid --verbose prep config.yaml > processing.log 2>&1

  2. System Information:

    python --version
conda list | grep -E "(gdal|numpy|scipy|pandas|netcdf4)"
uname -a  # Linux/macOS
# or
systeminfo  # Windows

  3. Configuration File:

    Include your YAML configuration (remove sensitive paths if needed).

  4. Error Messages:

    Copy complete error messages, including stack traces.

Community Resources

When submitting issues, provide: - Minimal reproducible example - Complete error messages - System and environment information - Steps already attempted

This troubleshooting guide should resolve most common issues encountered during subgrid preprocessing workflows.