Schedule Python Tasks with Airflow

Python scripts are great for automation. But running them manually is not scalable. You need a scheduler that runs them on time, every time.

Apache Airflow is the industry standard for workflow scheduling. It runs your Python tasks on cron schedules, handles retries, and gives you a full UI for monitoring.

This guide walks you through scheduling Python tasks with Airflow. You will learn cron expressions, schedule intervals, backfills, and how to avoid the most common pitfalls.

Why Use Airflow for Scheduling?

You could use cron or Windows Task Scheduler. But those tools have limits. They do not track history, handle dependencies, or retry on failure.

Airflow solves all of that. It stores every run in a database. It retries failed tasks automatically. It shows you exactly what ran, when, and why it failed.

Airflow also handles dependencies. Task B waits for Task A to finish. This makes complex pipelines reliable.

If you have not installed Airflow yet, start with our guide on how to install Apache Airflow in Python. It covers the virtual environment, database setup, and starting the scheduler.

Understanding Schedule Intervals

The schedule_interval parameter controls when your DAG runs. It accepts several formats.

A cron expression is the most flexible. Five fields define minute, hour, day of month, month, and day of week.

 # Run at 2:30 AM every day schedule_interval='30 2 * * *' # Run every Monday at 6 AM schedule_interval='0 6 * * 1' # Run every 15 minutes schedule_interval='*/15 * * * *' # Run at midnight on the first day of each month schedule_interval='0 0 1 * *' 

Airflow also provides presets. These are shortcuts for common schedules.

 schedule_interval='@hourly' # every hour schedule_interval='@daily' # every day at midnight schedule_interval='@weekly' # every Sunday at midnight schedule_interval='@monthly' # first day of each month schedule_interval='@yearly' # January 1st each year 

For simple intervals, use a timedelta object. This is useful when you want to run every N minutes or hours.

 from datetime import timedelta # Run every 30 minutes schedule_interval=timedelta(minutes=30) # Run every 6 hours schedule_interval=timedelta(hours=6) 

You can also set schedule_interval=None. This creates a DAG that only runs when triggered manually.

Create a Scheduled DAG

Here is a complete DAG that runs a Python function every day at 3 AM.

 from datetime import datetime, timedelta from airflow import DAG from airflow.operators.python import PythonOperator def daily_report(): print("Generating daily report...") # Your logic here return "Report complete" default_args = { 'owner': 'admin', 'retries': 2, 'retry_delay': timedelta(minutes=5), 'email_on_failure': False, } with DAG( dag_id='daily_report_dag', default_args=default_args, description='Runs a daily report at 3 AM', start_date=datetime(2024, 5, 1), schedule_interval='0 3 * * *', catchup=False, tags=['reporting'], ) as dag: report_task = PythonOperator( task_id='generate_report', python_callable=daily_report, ) 

Save this file in your ~/airflow/dags folder. The scheduler picks it up within a minute.

If you are new to writing DAGs, see our guide on writing your first Airflow DAG. It explains the structure and core classes.

Understanding the start_date

The start_date is when your DAG becomes active. Airflow schedules runs from this date forward.

Important: Always set start_date to a past date. If you set it to a future date, the DAG will not run until that date arrives.

The first run happens at the end of the first interval. For a daily DAG with start_date of May 1, the first run executes on May 2 at midnight.

 # First run: May 2 at midnight start_date=datetime(2024, 5, 1), schedule_interval='@daily', 

This is a common source of confusion. Airflow runs the interval that just ended, not the one starting.

Handling Backfills with catchup

The catchup parameter controls backfilling. When True, Airflow runs every missed interval from start_date to now.

When False, Airflow only runs the most recent interval. Older intervals are skipped.

 # This will run every day from May 1 until today with DAG( dag_id='backfill_dag', start_date=datetime(2024, 5, 1), schedule_interval='@daily', catchup=True, # creates many runs ) as dag: pass 

Warning: Setting catchup=True with an old start_date triggers thousands of runs. This can overwhelm your system and hit API rate limits.

Set catchup=False for new DAGs. Enable backfills only when you intentionally need to reprocess historical data.

Passing Dates to Tasks

Scheduled tasks often need to know which date they are processing. Airflow provides this through the context.

The ds variable gives you the logical date as a string. The execution_date gives you a datetime object.

 def process_date(**context): ds = context['ds'] print(f"Processing data for {ds}") # Use ds to query the right partition process_task = PythonOperator( task_id='process', python_callable=process_date, ) 

 [2024-05-02 00:00:00,000] {python.py:177} INFO - Processing data for 2024-05-01 

The logical date is the start of the interval, not the wall clock time. This is important for idempotent pipelines.

Use the provide_context=True argument if you need the full context. In newer Airflow versions, this is automatic.

Manual Triggers and CLI Commands

Scheduled DAGs also support manual triggers. This is useful for testing and ad-hoc runs.

Use the CLI to trigger a DAG run.

 airflow dags trigger daily_report_dag 

 [2024-05-01 10:00:00,000] {dag.py:1234} INFO - Created 

You can also trigger with a specific logical date.

 airflow dags trigger daily_report_dag --exec-date "2024-04-15T00:00:00" 

This is useful for reprocessing a specific day without running the full backfill.

Pause and Unpause DAGs

New DAGs are paused by default. They will not run until you unpause them.

Toggle the switch in the UI. Or use the CLI.

 airflow dags unpause daily_report_dag airflow dags pause daily_report_dag 

 Dag: daily_report_dag, paused: False 

Tip: Keep DAGs paused during development. Unpause only when you are ready for scheduled runs.

Common Scheduling Errors

DAG never runs: The DAG is paused. Check the UI toggle or run airflow dags list to see the paused state.

DAG runs immediately on creation: The start_date is in the past and catchup is True. Set catchup=False to avoid backfills.

Schedule interval not firing: The scheduler is not running. Start it with airflow scheduler in a separate terminal.

Wrong time of day: Cron uses UTC by default. Set your timezone in the DAG or change the Airflow default.

 from pendulum import timezone with DAG( dag_id='local_time_dag', start_date=datetime(2024, 5, 1, tzinfo=timezone('America/New_York')), schedule_interval='0 9 * * *', # 9 AM Eastern ) as dag: pass 

Task runs twice: Two schedulers are running. Airflow uses a database lock, but misconfigured setups can cause duplicates.

Backfill overload: An old start_date with catchup=True creates thousands of runs. Pause the DAG, fix the start_date, and clear the unwanted runs.

Best Practices for Scheduling

Use cron expressions for precise control. Use presets for common cases. Use timedelta for simple intervals.

Always set catchup=False for new DAGs. Enable it only when you need historical backfills.

Set retries and retry_delay in default_args. Transient failures are common in data pipelines.

Add alerts for failures. You can send HTML emails with SendGrid and Python from an on_failure_callback to notify your team instantly.

Use timezone-aware start_dates. This avoids confusion when your team spans multiple regions.

Keep your schedule simple. If you need complex timing logic, run a task every hour and branch inside it.

Monitor the scheduler health. It is the heart of Airflow. If it stops, no DAGs run.

If your pipeline generates reports, produce them inside a task and email them as attachments. A common pattern is to generate PDFs with ReportLab, then attach the result to a SendGrid message.

Conclusion

Scheduling Python tasks with Airflow is powerful and flexible. Use cron expressions for precise control. Use presets for common cases. Set catchup=False to avoid unwanted backfills.

Always set start_date to a past date. Pass the logical date to your tasks with the context. Test with manual triggers before enabling the schedule.

Follow the examples in this guide and your tasks will run reliably on time. Airflow handles retries, logging, and monitoring so you can focus on the logic. Happy scheduling!