AWS Elastic Beanstalk Container Commands Failing: Troubleshooting Django App Deployment Issues

Table of Contents#

1. Understanding AWS Elastic Beanstalk Container Commands
2. Common Causes of Container Command Failures
3. Step-by-Step Troubleshooting Methodology
4. Django-Specific Container Command Issues
5. Advanced Troubleshooting Techniques
6. Conclusion
7. References

1. Understanding AWS Elastic Beanstalk Container Commands#

Before diving into failures, let’s clarify what container commands are and when they run.

What Are Container Commands?#

Container commands are scripts defined in your Elastic Beanstalk configuration (via .ebextensions/*.config files) that execute on EC2 instances after your application code is extracted (to /var/app/staging/) but before the application server (e.g., Gunicorn, uWSGI) starts. They’re ideal for pre-deployment setup tasks like:

• Running Django migrations (python manage.py migrate).
• Collecting static files (python manage.py collectstatic).
• Setting file permissions or environment variables.

Key Characteristics#

• Execution Context: Run as the ec2-user by default (not root—use sudo for elevated privileges).
• Working Directory: Defaults to /var/app/staging/ (your app’s codebase).
• Order of Execution:
  • Files in .ebextensions/ are processed alphabetically (e.g., 01_django.config runs before 02_db.config).
  • Within a file, commands run in the order they’re listed.
• Leader-Only Execution: Use leader_only: true to run a command on only one instance (critical for migrations to avoid race conditions).

Example Configuration#

A typical .ebextensions/django.config file for a Django app might look like this:

container_commands:
  01_migrate:
    command: "python manage.py migrate"
    leader_only: true  # Run migrations on only one instance
  02_collectstatic:
    command: "python manage.py collectstatic --noinput"
    leader_only: false  # Run on all instances to collect static files

2. Common Causes of Container Command Failures#

Container commands fail for predictable reasons. Let’s break down the most frequent culprits.

2.1 YAML Syntax Errors#

Elastic Beanstalk uses YAML for .ebextensions configs, and YAML is unforgiving of syntax mistakes. Common issues:

• Missing Indentation: YAML relies on consistent spaces (not tabs). For example:

  # BROKEN: Missing indent for "command"
  container_commands:
  01_migrate:
  command: "python manage.py migrate"  # ❌ Indentation error

• Incorrect Key Names: Typos like container_command (singular) instead of container_commands (plural).
• Unquoted Special Characters: Commands with spaces or special characters (e.g., --noinput) must be quoted.

2.2 Incorrect Working Directory or Paths#

Container commands run in /var/app/staging/, but if your Django project uses a subdirectory (e.g., myproject/manage.py instead of manage.py), relative paths will fail:

# BROKEN: Assumes manage.py is in /var/app/staging/, but it’s in /var/app/staging/myproject/
container_commands:
  01_migrate:
    command: "python manage.py migrate"  # ❌ File not found

2.3 Permission Issues#

By default, commands run as ec2-user, which lacks root privileges. Tasks like writing to system directories (e.g., /var/www/static/) or installing system packages will fail without sudo:

# BROKEN: collectstatic tries to write to /var/www/static/ (owned by root)
container_commands:
  02_collectstatic:
    command: "python manage.py collectstatic --noinput"  # ❌ Permission denied

2.4 Timing/Dependency Issues#

Commands often depend on external services (e.g., RDS databases) that aren’t ready when the command runs. For example:

• Running migrate before the RDS instance initializes results in OperationalError: could not connect to server.

2.5 Missing Environment Variables#

Django commands like migrate or collectstatic rely on environment variables (e.g., DATABASE_URL, SECRET_KEY). If these aren’t set in Elastic Beanstalk (via Configuration > Software > Environment properties), commands fail:

# Error in logs: django.core.exceptions.ImproperlyConfigured: SECRET_KEY not set

2.6 Resource Limitations#

Commands like collectstatic or heavy migrations may exhaust memory or CPU on t2.micro instances, causing the OS to kill the process (look for Killed in logs).

3. Step-by-Step Troubleshooting Methodology#

When a container command fails, follow this workflow to diagnose the issue.

Step 1: Check Elastic Beanstalk Deployment Logs#

The first stop is Elastic Beanstalk’s eb-activity.log, which captures deployment steps, including container command output.

How to Access Logs:

• Via EB CLI: Run eb logs --all to download logs to your local machine.
• Via AWS Console: Go to your EB environment > Logs > Request Logs > Last 100 Lines (or Full Logs for more detail).

Key Log File: /var/log/eb-activity.log contains container command execution details. Look for lines like:

2024-03-01 12:34:56,789 [INFO] Running container command: 01_migrate
2024-03-01 12:34:57,123 [ERROR] Command failed: python manage.py migrate

Step 2: Validate Container Command Configuration#

Ensure your .ebextensions/*.config files are valid YAML and correctly structured.

Checklist:

• Use a YAML linter to catch syntax errors.
• Verify container_commands (plural) is used, not container_command.
• Confirm command paths: If manage.py is in a subdirectory (e.g., myapp/manage.py), update the command:

  # FIXED: Specify subdirectory path
  container_commands:
    01_migrate:
      command: "python myapp/manage.py migrate"

Step 3: Test Commands Locally#

Replicate the Elastic Beanstalk environment locally to isolate issues:

1. Use the EB Local Run tool to simulate deployment:

   eb local run

2. Manually run the failing command in the local container to see errors in real time:

   docker exec -it <container_id> bash  # Enter the local EB container
   cd /var/app/staging/
   python manage.py migrate  # Replicate the command

Step 4: Inspect the EC2 Instance via SSH#

If local testing doesn’t reveal the issue, SSH into the EB instance to debug live:

1. Enable SSH access via the EB CLI:

   eb ssh

2. Navigate to the staging directory:

   cd /var/app/staging/

3. Check file permissions, environment variables, and run commands manually:

   # Check environment variables (Elastic Beanstalk sets these)
   echo $DATABASE_URL
    
   # Test the failing command with verbose output
   python manage.py migrate --traceback  # Show full error stack

Step 5: Check for Resource Limits#

Use top or htop on the EC2 instance to monitor CPU/memory usage during deployment. If commands are killed, upgrade to a larger instance type (e.g., t2.small) or optimize the command (e.g., split large migrations).

4. Django-Specific Container Command Issues#

Django apps have unique dependencies that often cause container command failures. Let’s address the most common.

4.1 Django Migrations Failing#

python manage.py migrate is the most critical (and error-prone) container command.

Common Causes & Fixes#

• Missing Environment Variables: Migrations require database credentials (e.g., DATABASE_URL). Ensure these are set in EB’s Environment Properties.

  # In EB Configuration > Software > Environment properties:
  DATABASE_URL: postgres://user:[email protected]:5432/mydb

• Database Not Ready: RDS or Aurora instances may take 1–2 minutes to initialize. Add a retry loop to wait for the database:

  container_commands:
    01_migrate:
      command: |
        until python manage.py migrate; do
          echo "Migration failed. Retrying in 5 seconds..."
          sleep 5
        done
      leader_only: true

• Corrupted Migration Files: If a migration file is missing or invalid, migrate will fail. Run python manage.py showmigrations via SSH to check for gaps.

4.2 Static File Collection (collectstatic) Failing#

Django’s collectstatic gathers static assets (CSS, JS) into a single directory, but it’s prone to permission and configuration issues.

Common Causes & Fixes#

• Missing STATIC_ROOT: Django requires STATIC_ROOT in settings.py to know where to collect files:

  # settings.py (add this)
  STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')  # Or '/var/www/static/'

• Permission Denied: If STATIC_ROOT is a system directory (e.g., /var/www/static/), use sudo and set permissions:

  container_commands:
    01_create_static_dir:
      command: "sudo mkdir -p /var/www/static && sudo chown ec2-user:ec2-user /var/www/static"
    02_collectstatic:
      command: "python manage.py collectstatic --noinput"

• Missing STATIC_URL: While not required for collectstatic, STATIC_URL misconfiguration can break static file serving post-deployment.

4.3 Django Settings Not Loading#

Commands may fail if Django can’t load settings due to missing environment variables or incorrect DJANGO_SETTINGS_MODULE.

Fix#

Ensure DJANGO_SETTINGS_MODULE is set (default: myproject.settings), and all required variables are in EB’s environment properties:

# In EB Environment Properties:
DJANGO_SETTINGS_MODULE: myproject.settings.production
SECRET_KEY: your_secure_key
DEBUG: false

5. Advanced Troubleshooting Techniques#

For stubborn issues, use these pro tips.

5.1 Enable Verbose Logging#

Modify container commands to output more details. For Django commands, use --traceback or redirect output to a log file:

container_commands:
  01_migrate:
    command: "python manage.py migrate --traceback > /var/log/migrate.log 2>&1"
    leader_only: true

Then SSH into the instance and check /var/log/migrate.log.

5.2 Use leader_only for Single-Instance Commands#

Migrations, database seeding, or cache clearing should run on only one instance to avoid race conditions. Always add leader_only: true to these commands:

container_commands:
  01_migrate:
    command: "python manage.py migrate"
    leader_only: true  # Critical for multi-instance environments

5.3 Override the Working Directory#

If your Django project isn’t in the root of /var/app/staging/, explicitly set the working directory for commands:

container_commands:
  01_migrate:
    command: "python manage.py migrate"
    cwd: "/var/app/staging/myproject/"  # Set custom working directory

5.4 Retry Flaky Commands#

For commands that fail intermittently (e.g., due to network/database latency), use a retry loop with a timeout:

container_commands:
  01_migrate:
    command: |
      MAX_RETRIES=5
      RETRY_DELAY=10
      for ((i=1; i<=$MAX_RETRIES; i++)); do
        if python manage.py migrate; then
          echo "Migration succeeded on attempt $i"
          exit 0
        fi
        echo "Migration failed. Retrying in $RETRY_DELAY seconds (attempt $i/$MAX_RETRIES)..."
        sleep $RETRY_DELAY
      done
      echo "Migration failed after $MAX_RETRIES attempts"
      exit 1
    leader_only: true

6. Conclusion#

Container command failures in Elastic Beanstalk are rarely mysterious—they boil down to syntax errors, permission issues, missing dependencies, or environment misconfigurations. By methodically checking logs, testing locally, and debugging on EC2 instances, you can resolve most issues quickly.

For Django apps, focus on migration/database readiness, static file permissions, and environment variables. Use leader_only for single-instance commands and retry loops for flaky dependencies like RDS. With these tools, you’ll turn deployment headaches into smooth sailing.

7. References#

• AWS Elastic Beanstalk Container Commands Documentation
• Django Deployment Checklist
• YAML Syntax Reference
• Elastic Beanstalk Logs Guide