From AWS SWF to Apache Airflow: Fixing Data Pipeline Failures

We built our provider coordination platform using AWS Simple Workflow Service (SWF) to orchestrate data pipelines. It worked at first, but as pipelines grew in number and complexity, SWF's limitations became painful: no monitoring, hard to debug, timeouts on long-running tasks, retry logic was manual.

We switched to Apache Airflow + Docker. Errors dropped 80%. Debugging became trivial. We went from 3-4 pipelines to running more reliably with full visibility.

This is the real story of that migration.

The Problem with AWS SWF

What is SWF?

AWS Simple Workflow Service is a task coordinator. You define workflows (sequences of tasks), and SWF orchestrates them. Sounds perfect for data pipelines.

Until it's not.

The Actual Problems

1. No Monitoring Visibility

When a workflow failed, you had to dig through CloudWatch logs manually
No UI showing workflow status in real-time
We'd find out about failures hours later when customers reported stale data

2. Timeout Issues on Long-Running Tasks

Provider syncs sometimes took 10+ minutes
SWF tasks had configurable timeouts, but we kept guessing wrong values
No way to extend a timeout mid-execution
When a timeout happened, the whole workflow died

3. Manual Retry Logic

SWF has retry support, but it's clunky to configure
Most of our tasks had custom retry code (error-prone)
No exponential backoff built-in
Retried tasks still timed out (defeating the purpose)

4. Data Validation Was Afterthought

No built-in data quality checks at each step
Bad data from task A would silently propagate to task B
By the time we noticed, multiple pipelines had failed

5. Debugging Was a Nightmare

No easy way to rerun a single task (you'd have to trigger entire workflow)
Logs were scattered across CloudWatch, not centralized
Testing workflow logic locally was hard

        The Result: Timeout errors, data errors, retry failures. We estimated 80% of pipeline failures came from these three categories.
      

Switching to Apache Airflow + Docker

Why Airflow?

Airflow is purpose-built for data orchestration:

DAG-native: Define workflows as Python code (easy to test, version control)
Web UI: Real-time visibility into pipeline status
Built-in retry: Exponential backoff, max retries, email alerts
Task isolation: Each task runs in a container (reproducibility)
Logging: Centralized, searchable logs per task
Monitoring: Native integrations with monitoring systems

The Migration (3-4 pipelines initially)

Step 1: Set up Airflow infrastructure

Hosted Airflow on EC2
PostgreSQL for metadata (Airflow's internal DB)
Configured Docker as the task executor (each task = one container)

Step 2: Write Airflow DAGs

Example: A provider data sync pipeline:

from airflow import DAG
from airflow.operators.docker_operator import DockerOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'data-eng',
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
    'email_on_failure': True,
    'email': ['ops@company.com'],
}

dag = DAG(
    'provider_sync_daily',
    default_args=default_args,
    schedule_interval='0 2 * * *',  # 2 AM daily
)

# Task 1: Extract data from providers
extract = DockerOperator(
    task_id='extract_providers',
    image='data-pipeline:latest',
    command=['python', '/app/extract.py'],
    dag=dag,
)

# Task 2: Validate data quality
validate = DockerOperator(
    task_id='validate_data',
    image='data-pipeline:latest',
    command=['python', '/app/validate.py'],
    dag=dag,
)

# Task 3: Load to our graph database
load = DockerOperator(
    task_id='load_graph',
    image='data-pipeline:latest',
    command=['python', '/app/load.py'],
    dag=dag,
)

# Define dependencies
extract >> validate >> load

Step 3: Migrate SWF workflows to Airflow DAGs

Converted 3-4 SWF workflows to Airflow DAGs
Tested each DAG locally (Airflow has a local mode)
Ran parallel for 1 week (SWF + Airflow both executing)
Compared outputs (must match exactly)

Step 4: Cutover to Airflow

Disabled SWF workflows
Enabled Airflow DAGs
Monitored via Airflow Web UI (huge improvement)

Problems We Solved

Problem 1: Timeouts on Long Tasks

Solution: Set generous timeouts per task, add heartbeat mechanism (Docker keeps connection alive).

Now provider syncs that took 10 minutes run without timing out.

Problem 2: Data Errors

Solution: Added explicit validation step between every task.

def validate_step_output(output_file):
    df = pd.read_csv(output_file)
    assert len(df) > 0, "No data loaded"
    assert df['provider_id'].notna().all(), "Missing provider_ids"
    assert df['date'].dtype == 'datetime64', "Invalid date format"
    return True

Bad data is caught immediately (not hours later).

Problem 3: Retry Logic

Solution: Airflow's built-in retry with exponential backoff.

default_args = {
    'retries': 3,
    'retry_delay': timedelta(minutes=5),  # First retry: 5 min
    # Airflow auto-multiplies: 10 min, 20 min, 40 min...
}

Most transient errors (network hiccups, temporary service outages) are automatically resolved by retry.

Problem 4: Debugging

Solution: Airflow Web UI + centralized logging.

Click on a task, see full logs (no CloudWatch hunting)
Rerun failed task without rerunning entire pipeline
See which tasks passed/failed in one view
Access task history (what ran, when, how long)

Results

Metric	With SWF	With Airflow
Pipeline failures/week	3-4	0-1
Time to debug failure	30-60 min	5 min
Timeout errors	5-10/week	0
Data quality issues caught	Usually downstream	At source immediately
Visibility into pipelines	Poor (CloudWatch logs)	Excellent (Web UI)

Overall: 80% reduction in errors (from 3-4 failures/week to 0-1).

What We Learned

1. Purpose-Built Tools > Generic Services

SWF is a generic workflow engine. Airflow is built specifically for data pipelines. The difference shows.

2. Visibility Prevents Most Failures

The Airflow Web UI caught issues we didn't know about. Early visibility = early fixes.

3. Retry + Validation = Reliability

Built-in retry + explicit validation at each step eliminated 80% of our error categories.

4. Docker Isolation = Reproducibility

Every task runs in identical Docker environment. No "it works on my machine" surprises.

Conclusion

Switching from AWS SWF to Apache Airflow was one of the best decisions we made for data reliability. It took initial setup effort, but now we can confidently scale pipelines without ops burden.

If you're managing data pipelines and seeing timeout errors, data quality issues, or mysterious failures, Airflow is worth evaluating. The learning curve is real, but the payoff is huge.