MTH5 File Validator

A comprehensive validation tool for MTH5 (Magnetotelluric HDF5) files that checks file format, structure, and metadata compliance.

Features¶

File Format Validation: Verify HDF5 file attributes (type, version, data level)
Structure Validation: Check group hierarchy based on file version (v0.1.0 or v0.2.0)
Metadata Validation: Validate metadata using mt_metadata schemas
Data Validation: Optional check for channel data integrity
Multiple Interfaces: Use programmatically or via command-line
Flexible Output: Human-readable reports or JSON for integration

Installation¶

The validator is included with mth5. After installing mth5, the validator is automatically available:

pip install mth5

For development installation:

cd mth5
pip install -e .

Quick Start¶

Command-Line Usage¶

# Basic validation
mth5-cli validate myfile.mth5

# Verbose output
mth5-cli validate myfile.mth5 --verbose

# Check data integrity (slower)
mth5-cli validate myfile.mth5 --check-data

# JSON output
mth5-cli validate myfile.mth5 --json

Python API Usage¶

from mth5.utils.mth5_validator import validate_mth5_file

# Quick validation
results = validate_mth5_file('myfile.mth5')

if results.is_valid:
    print("File is valid!")
else:
    results.print_report()

Validation Checks¶

File Format Checks¶

file.type: Must be “MTH5”
file.version: Must be “0.1.0” or “0.2.0”
data_level: Must be 0, 1, or 2

Structure Checks (v0.1.0)¶

/Survey
  ├── Stations/
  ├── Reports/
  ├── Filters/
  ├── Standards/
  ├── channel_summary (dataset)
  └── tf_summary (dataset)

Structure Checks (v0.2.0)¶

/Experiment
  ├── Surveys/
  │   └── {survey_id}/
  │       ├── Stations/
  │       ├── Reports/
  │       ├── Filters/
  │       └── Standards/
  ├── Reports/
  ├── Standards/
  ├── channel_summary (dataset)
  └── tf_summary (dataset)

Station/Run Structure¶

Each station should contain:

One or more run groups
Each run should contain channel datasets

Metadata Checks¶

Validates metadata attributes exist
Checks for required mth5_type attributes
Uses mt_metadata schemas for validation

Data Checks (Optional)¶

Verifies channels contain data
Detects empty or all-zero channels
Samples data without loading full arrays

Command-Line Interface¶

mth5-cli validate¶

Validate an MTH5 file.

Usage:

mth5-cli validate FILE [OPTIONS]

Arguments:

FILE: Path to MTH5 file to validate

Options:

-v, --verbose: Enable verbose output with detailed information
--skip-metadata: Skip metadata validation (structure only)
--check-data: Check that channels contain data (slower)
--json: Output results as JSON

Exit Codes:

0: File is valid
1: File has errors or validation failed

Examples:

# Basic validation
mth5-cli validate data.mth5

# Detailed validation report
mth5-cli validate data.mth5 --verbose

# Full validation including data
mth5-cli validate data.mth5 --check-data --verbose

# JSON output for scripting
mth5-cli validate data.mth5 --json > report.json

# Batch validation
for file in data/*.mth5; do
    mth5-cli validate "$file" || echo "Failed: $file"
done

Python API¶

validate_mth5_file()¶

Convenience function for quick validation.

from mth5.utils import validate_mth5_file

results = validate_mth5_file(
    file_path='data.mth5',
    verbose=False,
    validate_metadata=True,
    check_data=False
)

print(f"Valid: {results.is_valid}")
print(f"Errors: {results.error_count}")

MTH5Validator Class¶

Full-featured validator class for advanced usage.

from mth5.utils.mth5_validator import MTH5Validator

# Create validator
validator = MTH5Validator(
    file_path='data.mth5',
    verbose=True,
    validate_metadata=True,
    check_data=False
)

# Run validation
results = validator.validate()

# Access results
print(f"Valid: {results.is_valid}")
print(f"Errors: {results.error_count}")
print(f"Warnings: {results.warning_count}")

# Print report
results.print_report(include_info=True)

# Get JSON
json_str = results.to_json()

# Get dictionary
data_dict = results.to_dict()

ValidationResults Object¶

Results object returned by validation.

Properties:

is_valid (bool): True if no errors
error_count (int): Number of errors
warning_count (int): Number of warnings
info_count (int): Number of info messages
messages (list): All validation messages
checked_items (dict): Dictionary of validation checks performed

Methods:

print_report(include_info=False): Print formatted report
to_dict(): Convert to dictionary
to_json(**kwargs): Convert to JSON string
add_error(category, message, path=None, **details): Add error message
add_warning(category, message, path=None, **details): Add warning message
add_info(category, message, path=None, **details): Add info message

Use Cases¶

Pre-Processing Validation¶

Validate files before processing:

from mth5.utils.mth5_validator import validate_mth5_file

def process_mth5(filepath):
    # Validate first
    results = validate_mth5_file(filepath)
    
    if not results.is_valid:
        print(f"Cannot process {filepath}:")
        results.print_report()
        return False
    
    # Process file
    # ...
    return True

Archive Quality Control¶

Check files meet archive standards:

from pathlib import Path
from mth5.utils.mth5_validator import MTH5Validator

def qa_check_archive(archive_dir):
    """Quality check all files in archive."""
    failed = []
    
    for mth5_file in Path(archive_dir).glob("**/*.mth5"):
        validator = MTH5Validator(
            mth5_file,
            validate_metadata=True,
            check_data=True
        )
        results = validator.validate()
        
        if not results.is_valid:
            failed.append(mth5_file)
    
    return failed

Automated Testing¶

Use in test suites:

import pytest
from mth5.utils.mth5_validator import validate_mth5_file

def test_mth5_file_valid(mth5_file):
    """Test that generated MTH5 file is valid."""
    results = validate_mth5_file(mth5_file)
    
    assert results.is_valid, f"Validation failed:\n{results.messages}"
    assert results.error_count == 0

CI/CD Integration¶

GitHub Actions example:

name: Validate MTH5 Files

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: '3.10'
      - name: Install dependencies
        run: |
          pip install mth5
      - name: Validate MTH5 files
        run: |
          for file in data/*.mth5; do
            mth5-cli validate "$file" --json > "${file%.mth5}_validation.json"
            mth5-cli validate "$file" || exit 1
          done

Validation Levels¶

ERROR¶

Critical issues that indicate an invalid file:

Missing required file attributes
Invalid file version or type
Missing required groups
Corrupted file structure

WARNING¶

Issues that should be reviewed but don’t prevent usage:

Missing optional metadata
Empty summary tables
Runs with no channels
Missing recommended attributes

INFO¶

Informational messages:

File version and type
Number of surveys/stations/runs
Summary of validation checks
Data statistics

Performance¶

Speed Considerations¶

Basic validation (format + structure): Very fast, <1 second
With metadata validation: Fast, 1-5 seconds
With data checking: Slower, depends on file size (samples data efficiently)

Large Files¶

For large files (>1GB), consider:

# Skip data checking for speed
validator = MTH5Validator(
    file_path='large_file.mth5',
    check_data=False  # Much faster
)

Creating a Standalone Executable¶

For users without Python, create a standalone executable using PyInstaller:

# Install PyInstaller
pip install pyinstaller

# Create standalone executable
pyinstaller --onefile \
    --name mth5-validator \
    --add-data "mth5:mth5" \
    mth5/utils/cli.py

# Executable will be in dist/
./dist/mth5-validator validate data.mth5

Troubleshooting¶

Import Errors¶

If you get import errors:

pip install --upgrade mth5 h5py mt_metadata

File Access Errors¶

Ensure file is not open in another program:

# Close any open references
import h5py
h5py.File.close_open_files()

Validation Too Strict¶

Skip certain checks:

validator = MTH5Validator(
    file_path='data.mth5',
    validate_metadata=False  # Skip metadata validation
)

Contributing¶

To add new validation checks:

Add check method to MTH5Validator class in mth5_validator.py
Call from validate() method
Add results to ValidationResults
Update tests
Document in this README

License¶

MIT License - See LICENSE file in mth5 repository.

Support¶

Documentation: https://mth5.readthedocs.io/
Issues: https://github.com/kujaku11/mth5/issues
Discussions: https://github.com/kujaku11/mth5/discussions