Argo Workflows: Complete Kubernetes-Native Workflow Engine Guide

Argo Workflows is an open-source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Implemented as a Kubernetes CRD, it enables complex workflow orchestration including DAGs, sequential and parallel execution, and parameter passing. This comprehensive guide covers everything from basic workflows to advanced production patterns.

What is Argo Workflows?

Argo Workflows provides:

Key Features

Kubernetes Native: Workflows as Kubernetes resources
DAG Support: Complex directed acyclic graphs
Parallelism: Run steps in parallel
Artifacts: Pass data between steps
Conditional Execution: Dynamic workflow paths
Retry/Timeout: Built-in error handling
Cron Workflows: Scheduled workflow execution
UI Dashboard: Web interface for monitoring
CLI Tool: Command-line workflow management
Event-Driven: Trigger workflows from events

Architecture

┌─────────────────────────────────────────────────────────┐
│                  Argo Workflows UI                       │
│              (Web Dashboard & CLI)                       │
└────────────────────┬────────────────────────────────────┘
                     │
┌────────────────────┴────────────────────────────────────┐
│            Workflow Controller                           │
│                                                           │
│  • Watches Workflow CRDs                                 │
│  • Schedules workflow steps                              │
│  • Manages artifacts                                     │
│  • Handles retries and timeouts                          │
└────────────────────┬────────────────────────────────────┘
                     │
┌────────────────────┴────────────────────────────────────┐
│              Kubernetes API                              │
└────────────────────┬────────────────────────────────────┘
                     │
        ┌────────────┴────────────┐
        ▼                         ▼
┌──────────────┐          ┌──────────────┐
│   Pod 1      │          │   Pod 2      │
│  (Step 1)    │          │  (Step 2)    │
└──────────────┘          └──────────────┘

Installation

Using kubectl

# Install Argo Workflows
kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.5.4/install.yaml

# Wait for pods to be ready
kubectl wait --for=condition=Ready pods --all -n argo --timeout=300s

# Patch server authentication (for quick start)
kubectl patch deployment argo-server \
  --namespace argo \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": [
  "server",
  "--auth-mode=server"
]}]'

# Port forward to access UI
kubectl -n argo port-forward deployment/argo-server 2746:2746

# Access at https://localhost:2746

Using Helm

# Add Argo Helm repository
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

# Create values file
cat > argo-workflows-values.yaml << 'EOF'
controller:
  replicas: 3
  resources:
    limits:
      cpu: 1000m
      memory: 1Gi
    requests:
      cpu: 500m
      memory: 512Mi

server:
  replicas: 2

  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    hosts:
      - argo-workflows.example.com
    tls:
      - secretName: argo-workflows-tls
        hosts:
          - argo-workflows.example.com

  # SSO configuration
  sso:
    enabled: true
    issuer: https://accounts.google.com
    clientId:
      name: argo-workflows-sso
      key: client-id
    clientSecret:
      name: argo-workflows-sso
      key: client-secret
    redirectUrl: https://argo-workflows.example.com/oauth2/callback

useDefaultArtifactRepo: true
artifactRepository:
  s3:
    bucket: argo-artifacts
    endpoint: s3.amazonaws.com
    region: us-east-1
    insecure: false
    accessKeySecret:
      name: argo-artifacts-s3
      key: accesskey
    secretKeySecret:
      name: argo-artifacts-s3
      key: secretkey

# Workflow defaults
workflowDefaults:
  spec:
    ttlStrategy:
      secondsAfterCompletion: 86400  # 24 hours
    podGC:
      strategy: OnWorkflowSuccess
EOF

# Install with Helm
helm install argo-workflows argo/argo-workflows \
  -n argo \
  --create-namespace \
  -f argo-workflows-values.yaml

Install CLI

# macOS
brew install argo

# Linux
curl -sLO https://github.com/argoproj/argo-workflows/releases/download/v3.5.4/argo-linux-amd64.gz
gunzip argo-linux-amd64.gz
chmod +x argo-linux-amd64
sudo mv argo-linux-amd64 /usr/local/bin/argo

# Windows
$version = "v3.5.4"
$url = "https://github.com/argoproj/argo-workflows/releases/download/$version/argo-windows-amd64.gz"
Invoke-WebRequest -Uri $url -OutFile argo.gz

# Verify
argo version

Basic Workflows

Hello World

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["Hello Argo Workflows!"]

Submit workflow:

argo submit -n argo hello-world.yaml --watch

Parameters

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: parameters-
spec:
  entrypoint: print-message
  arguments:
    parameters:
    - name: message
      value: "Hello World"
    - name: name
      value: "Argo"

  templates:
  - name: print-message
    inputs:
      parameters:
      - name: message
      - name: name
    container:
      image: alpine:latest
      command: [echo]
      args: ["{{inputs.parameters.message}} from {{inputs.parameters.name}}"]

Submit with custom parameters:

argo submit -n argo parameters.yaml \
  -p message="Hello Kubernetes" \
  -p name="User" \
  --watch

Sequential Steps

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: steps-
spec:
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: step1
        template: print-message
        arguments:
          parameters:
          - name: message
            value: "Step 1"

    - - name: step2a
        template: print-message
        arguments:
          parameters:
          - name: message
            value: "Step 2a (parallel)"
      - name: step2b
        template: print-message
        arguments:
          parameters:
          - name: message
            value: "Step 2b (parallel)"

    - - name: step3
        template: print-message
        arguments:
          parameters:
          - name: message
            value: "Step 3"

  - name: print-message
    inputs:
      parameters:
      - name: message
    container:
      image: alpine:latest
      command: [echo]
      args: ["{{inputs.parameters.message}}"]

DAG (Directed Acyclic Graph)

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-
spec:
  entrypoint: main
  templates:
  - name: main
    dag:
      tasks:
      - name: A
        template: echo
        arguments:
          parameters:
          - name: message
            value: "Task A"

      - name: B
        dependencies: [A]
        template: echo
        arguments:
          parameters:
          - name: message
            value: "Task B (after A)"

      - name: C
        dependencies: [A]
        template: echo
        arguments:
          parameters:
          - name: message
            value: "Task C (after A)"

      - name: D
        dependencies: [B, C]
        template: echo
        arguments:
          parameters:
          - name: message
            value: "Task D (after B and C)"

  - name: echo
    inputs:
      parameters:
      - name: message
    container:
      image: alpine:latest
      command: [echo]
      args: ["{{inputs.parameters.message}}"]

Advanced Patterns

Artifacts

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifacts-
spec:
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: generate
        template: generate-artifact
    - - name: consume
        template: consume-artifact
        arguments:
          artifacts:
          - name: message
            from: "{{steps.generate.outputs.artifacts.result}}"

  - name: generate-artifact
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo 'Hello from artifact' > /tmp/result.txt"]
    outputs:
      artifacts:
      - name: result
        path: /tmp/result.txt

  - name: consume-artifact
    inputs:
      artifacts:
      - name: message
        path: /tmp/message.txt
    container:
      image: alpine:latest
      command: [cat]
      args: [/tmp/message.txt]

Loops

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-
spec:
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: process-items
        template: print-item
        arguments:
          parameters:
          - name: item
            value: "{{item}}"
        withItems:
        - apple
        - banana
        - orange

  - name: print-item
    inputs:
      parameters:
      - name: item
    container:
      image: alpine:latest
      command: [echo]
      args: ["Processing {{inputs.parameters.item}}"]

Conditional Execution

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: conditionals-
spec:
  entrypoint: main
  arguments:
    parameters:
    - name: environment
      value: "production"

  templates:
  - name: main
    steps:
    - - name: check-env
        template: echo
        arguments:
          parameters:
          - name: message
            value: "Environment: {{workflow.parameters.environment}}"

    - - name: deploy-dev
        template: echo
        arguments:
          parameters:
          - name: message
            value: "Deploying to dev"
        when: "{{workflow.parameters.environment}} == dev"

      - name: deploy-prod
        template: echo
        arguments:
          parameters:
          - name: message
            value: "Deploying to production"
        when: "{{workflow.parameters.environment}} == production"

  - name: echo
    inputs:
      parameters:
      - name: message
    container:
      image: alpine:latest
      command: [echo]
      args: ["{{inputs.parameters.message}}"]

Retry and Timeout

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: retry-timeout-
spec:
  entrypoint: main
  templates:
  - name: main
    retryStrategy:
      limit: 3
      retryPolicy: "Always"
      backoff:
        duration: "10s"
        factor: 2
        maxDuration: "1m"

    timeout: "5m"

    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["exit 1"]  # This will fail and retry

CI/CD Pipeline Example

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: cicd-pipeline-
spec:
  entrypoint: main
  volumeClaimTemplates:
  - metadata:
      name: workspace
    spec:
      accessModes: [ReadWriteOnce]
      resources:
        requests:
          storage: 1Gi

  templates:
  - name: main
    dag:
      tasks:
      - name: checkout
        template: git-clone
        arguments:
          parameters:
          - name: repo
            value: "https://github.com/example/app.git"
          - name: branch
            value: "main"

      - name: test
        dependencies: [checkout]
        template: run-tests

      - name: build
        dependencies: [test]
        template: build-image
        arguments:
          parameters:
          - name: tag
            value: "{{workflow.parameters.version}}"

      - name: scan
        dependencies: [build]
        template: security-scan

      - name: deploy-dev
        dependencies: [scan]
        template: deploy
        arguments:
          parameters:
          - name: environment
            value: "dev"

      - name: integration-test
        dependencies: [deploy-dev]
        template: run-integration-tests

      - name: deploy-prod
        dependencies: [integration-test]
        template: deploy
        arguments:
          parameters:
          - name: environment
            value: "production"

  - name: git-clone
    inputs:
      parameters:
      - name: repo
      - name: branch
    container:
      image: alpine/git:latest
      command: [sh, -c]
      args:
      - |
        cd /workspace
        git clone {{inputs.parameters.repo}} .
        git checkout {{inputs.parameters.branch}}
      volumeMounts:
      - name: workspace
        mountPath: /workspace

  - name: run-tests
    container:
      image: node:18
      command: [sh, -c]
      args:
      - |
        cd /workspace
        npm install
        npm test
      volumeMounts:
      - name: workspace
        mountPath: /workspace

  - name: build-image
    inputs:
      parameters:
      - name: tag
    container:
      image: gcr.io/kaniko-project/executor:latest
      command: [/kaniko/executor]
      args:
      - --context=/workspace
      - --dockerfile=/workspace/Dockerfile
      - --destination=myregistry/app:{{inputs.parameters.tag}}
      - --cache=true
      volumeMounts:
      - name: workspace
        mountPath: /workspace

  - name: security-scan
    container:
      image: aquasec/trivy:latest
      command: [trivy]
      args:
      - image
      - --severity HIGH,CRITICAL
      - myregistry/app:{{workflow.parameters.version}}

  - name: deploy
    inputs:
      parameters:
      - name: environment
    container:
      image: bitnami/kubectl:latest
      command: [kubectl]
      args:
      - apply
      - -f
      - /workspace/k8s/{{inputs.parameters.environment}}
      volumeMounts:
      - name: workspace
        mountPath: /workspace

  - name: run-integration-tests
    container:
      image: curlimages/curl:latest
      command: [sh, -c]
      args:
      - |
        for i in $(seq 1 10); do
          if curl -f http://app-service/health; then
            echo "Health check passed"
            exit 0
          fi
          sleep 5
        done
        echo "Health check failed"
        exit 1

Cron Workflows

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: nightly-backup
  namespace: argo
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  timezone: "America/New_York"

  startingDeadlineSeconds: 0
  concurrencyPolicy: "Forbid"  # Don't run if previous job still running
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1

  workflowSpec:
    entrypoint: backup
    templates:
    - name: backup
      steps:
      - - name: database-backup
          template: backup-database
      - - name: upload-to-s3
          template: upload-backup
          arguments:
            artifacts:
            - name: backup-file
              from: "{{steps.database-backup.outputs.artifacts.backup}}"

    - name: backup-database
      container:
        image: postgres:15
        command: [sh, -c]
        args:
        - |
          pg_dump -h db-host -U postgres mydb > /tmp/backup.sql
      outputs:
        artifacts:
        - name: backup
          path: /tmp/backup.sql

    - name: upload-backup
      inputs:
        artifacts:
        - name: backup-file
          path: /tmp/backup.sql
      container:
        image: amazon/aws-cli:latest
        command: [sh, -c]
        args:
        - |
          aws s3 cp /tmp/backup.sql s3://backups/$(date +%Y%m%d).sql

Workflow Templates

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: build-deploy-template
  namespace: argo
spec:
  entrypoint: main
  arguments:
    parameters:
    - name: repo
    - name: branch
      value: "main"
    - name: environment

  templates:
  - name: main
    dag:
      tasks:
      - name: checkout
        template: git-clone
        arguments:
          parameters:
          - name: repo
            value: "{{workflow.parameters.repo}}"
          - name: branch
            value: "{{workflow.parameters.branch}}"

      - name: build
        dependencies: [checkout]
        template: build-image

      - name: deploy
        dependencies: [build]
        template: deploy-app
        arguments:
          parameters:
          - name: environment
            value: "{{workflow.parameters.environment}}"

  # ... template definitions ...

Use template:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: deploy-app-
spec:
  workflowTemplateRef:
    name: build-deploy-template
  arguments:
    parameters:
    - name: repo
      value: "https://github.com/example/app.git"
    - name: environment
      value: "production"

CLI Commands

# Submit workflow
argo submit workflow.yaml -n argo

# Submit with parameters
argo submit workflow.yaml -n argo \
  -p param1=value1 \
  -p param2=value2

# List workflows
argo list -n argo

# Get workflow details
argo get <workflow-name> -n argo

# Get workflow logs
argo logs <workflow-name> -n argo

# Follow logs
argo logs <workflow-name> -n argo --follow

# Watch workflow
argo watch <workflow-name> -n argo

# Delete workflow
argo delete <workflow-name> -n argo

# Terminate running workflow
argo terminate <workflow-name> -n argo

# Retry failed workflow
argo retry <workflow-name> -n argo

# Resubmit workflow
argo resubmit <workflow-name> -n argo

# List cron workflows
argo cron list -n argo

# Suspend cron workflow
argo cron suspend <cron-name> -n argo

# Resume cron workflow
argo cron resume <cron-name> -n argo

Event-Driven Workflows

Argo Events Integration

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: webhook
  namespace: argo-events
spec:
  webhook:
    github:
      port: "12000"
      endpoint: /push
      method: POST
---
# sensor.yaml
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: github-workflow-trigger
  namespace: argo-events
spec:
  dependencies:
  - name: github-dep
    eventSourceName: webhook
    eventName: github

  triggers:
  - template:
      name: github-workflow-trigger
      k8s:
        operation: create
        source:
          resource:
            apiVersion: argoproj.io/v1alpha1
            kind: Workflow
            metadata:
              generateName: ci-pipeline-
              namespace: argo
            spec:
              workflowTemplateRef:
                name: build-deploy-template
              arguments:
                parameters:
                - name: repo
                  value: "{{inputs.parameters.repo}}"

Monitoring

Prometheus Metrics

# ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argo-workflows
  namespace: argo
spec:
  selector:
    matchLabels:
      app: argo-workflows-metrics
  endpoints:
  - port: metrics

Key metrics:

argo_workflow_count: Total workflows
argo_workflow_status_counter: Workflows by status
argo_workflow_duration_seconds: Workflow duration
argo_workflow_pod_count: Pods per workflow

Best Practices

Resource Management

spec:
  templates:
  - name: resource-limits
    container:
      image: alpine:latest
      resources:
        limits:
          cpu: 1000m
          memory: 1Gi
        requests:
          cpu: 500m
          memory: 512Mi

    # Pod-level settings
    podSpecPatch: |
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000

Cleanup Policies

spec:
  # TTL for workflow after completion
  ttlStrategy:
    secondsAfterCompletion: 86400
    secondsAfterSuccess: 43200
    secondsAfterFailure: 172800

  # Pod garbage collection
  podGC:
    strategy: OnWorkflowSuccess
    # or: OnWorkflowCompletion, OnPodCompletion, OnPodSuccess

Security

spec:
  # Service account
  serviceAccountName: workflow-sa

  # Security context
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 1000

  templates:
  - name: secure-container
    container:
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
          - ALL
        readOnlyRootFilesystem: true

Troubleshooting

# Check controller logs
kubectl logs -n argo deployment/workflow-controller

# Check server logs
kubectl logs -n argo deployment/argo-server

# Describe workflow
kubectl describe workflow <name> -n argo

# Get workflow events
kubectl get events -n argo --field-selector involvedObject.name=<workflow-name>

# Debug failed step
argo logs <workflow-name> -n argo <step-name>

Conclusion

Argo Workflows provides a powerful, Kubernetes-native solution for orchestrating complex workflows. From simple sequential tasks to complex DAGs and CI/CD pipelines, Argo Workflows enables teams to build reliable, scalable workflow automation on Kubernetes.

Master Kubernetes workflow orchestration with our training programs. Contact us for customized training.