Python PyYAML Variable Substitution: Easy Implementation Guide

Variable substitution in YAML files is a powerful feature that allows you to create dynamic configuration files. Let's explore how to implement this functionality using PyYAML in Python.

Basic Variable Substitution Setup

First, install PyYAML using pip and create a simple YAML file with variables. Environment variables are commonly used in configuration files.

# Install PyYAML
# pip install pyyaml

import yaml
import os

# Sample YAML content
yaml_text = """
database:
  host: ${DB_HOST}
  port: ${DB_PORT}
  username: ${DB_USER}
"""

# Environment variables
os.environ['DB_HOST'] = 'localhost'
os.environ['DB_PORT'] = '5432'
os.environ['DB_USER'] = 'admin'

Implementing the Substitution

Create a custom loader class to handle variable substitution. This approach allows you to change variable values dynamically.

import re
from yaml import SafeLoader

class EnvVarLoader(SafeLoader):
    def __init__(self, stream):
        super().__init__(stream)

    def construct_scalar(self, node):
        value = super().construct_scalar(node)
        if isinstance(value, str):
            # Pattern to match ${VARIABLE_NAME}
            pattern = r'\$\{([^}^{]+)\}'
            matches = re.finditer(pattern, value)
            for match in matches:
                env_var = match.group(1)
                value = value.replace(f'${{{env_var}}}', os.environ.get(env_var, ''))
        return value

# Load YAML with custom loader
config = yaml.load(yaml_text, Loader=EnvVarLoader)

Using the Substituted Values

Now we can access our configuration with substituted values. The variables are replaced with their corresponding environment variable values.

# Print the loaded configuration
print(config)

# Access specific values
print(f"Database Host: {config['database']['host']}")
print(f"Database Port: {config['database']['port']}")
print(f"Database User: {config['database']['username']}")

{'database': {'host': 'localhost', 'port': '5432', 'username': 'admin'}}
Database Host: localhost
Database Port: 5432
Database User: admin

Default Values and Error Handling

It's important to handle cases where environment variables might not be set. Printing variable values can help with debugging.

def get_env_value(env_var, default=''):
    """Helper function to get environment variable with default value"""
    return os.environ.get(env_var, default)

# Example with default values
yaml_text_with_defaults = """
database:
  host: ${DB_HOST:-localhost}
  port: ${DB_PORT:-5432}
  username: ${DB_USER:-default_user}
"""

Conclusion

PyYAML variable substitution provides a flexible way to manage configuration files in Python applications. By using custom loaders and proper error handling, you can create robust configuration systems.

Remember to always validate your configuration values and provide meaningful default values for better application reliability. This approach is particularly useful in containerized environments.