k8s-maestro Documentation

Welcome to the k8s-maestro documentation. k8s-maestro is a Kubernetes workflow orchestrator with minimal requirements and full power.

What is k8s-maestro?

k8s-maestro provides a high-level, type-safe Rust API for orchestrating complex workflows on Kubernetes. Built with test-driven development principles, it offers a clean builder pattern for creating multi-step workflows with dependencies, conditional execution, and powerful networking capabilities.

Key Features

Multi-step Workflows: Define complex workflows with multiple steps and dependencies
Conditional Execution: Execute steps based on conditions (success, failure, output values)
Multiple Step Types: Support for Kubernetes jobs, exec steps, WASM, and custom step types
Services & Ingress: Built-in support for exposing services and configuring ingress
Sidecar Containers: Easily add sidecar containers to workflow steps
File Observer: Monitor file changes and trigger workflow execution
Checkpointing: Automatic checkpointing and recovery for long-running workflows
Multi-tenant Security: Role-based access control and namespace isolation
Builder Pattern: Fluent API for easy workflow and resource construction
TDD Approach: Extensive test coverage with unit, integration, and E2E tests

Quick Links

Getting Started

Installation - Install and configure k8s-maestro
Quick Start - Run your first workflow in 5 minutes
Concepts - Understand workflows, steps, and dependencies

Guides

Basic Workflows - Create your first workflow
Dependencies - Configure step dependencies and execution order
Services & Ingress - Expose services and configure ingress
Multi-language Steps - Use Python, Rust, WASM in workflows
Checkpointing - Enable workflow checkpointing and recovery
Security - Configure multi-tenant security and RBAC

API Reference

Client API - MaestroClient reference
Workflow API - Workflow and WorkflowBuilder reference
Steps API - Available step types and APIs
Networking API - Services, Ingress, and DNS utilities

Examples

Spark Cluster - Orchestrate Apache Spark on Kubernetes
ML Pipeline - Build machine learning pipelines
Data Processing - ETL and data transformation workflows

Reference

Configuration - Configuration options and environment variables
Troubleshooting - Common issues and solutions

Example Usage

use k8s_maestro::{MaestroClientBuilder, WorkflowBuilder};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = MaestroClientBuilder::new()
        .with_namespace("default")
        .build()?;

    let workflow = WorkflowBuilder::new()
        .with_name("my-workflow")
        .add_step(JobStep::new("my-job", "nginx:latest"))
        .build()?;

    let execution = client.execute_workflow(&workflow).await?;
    println!("Workflow executed: {:?}", execution);

    Ok(())
}

Community & Support

GitHub Repository: https://github.com/andreclaudino/k8s-maestro
Issues: Report bugs and request features
Contributing: See the contributing guidelines in the main repository

License

This project is dual-licensed under:

Getting Started

Welcome to the k8s-maestro getting started guide. This section will help you install, configure, and start using k8s-maestro.

Overview

k8s-maestro is a Kubernetes workflow orchestrator for Rust with a type-safe, high-level API. It allows you to create complex workflows with multiple steps, dependencies, and conditional execution.

Prerequisites

Rust 1.70 or later
Access to a Kubernetes cluster
Basic knowledge of Kubernetes concepts

What You'll Learn

How to install k8s-maestro
Basic workflow concepts
How to create your first workflow
Common patterns and best practices

Installation

This guide will help you install and configure k8s-maestro.

Prerequisites

Before installing k8s-maestro, ensure you have:

Rust 1.70 or later (Install Rust)
Cargo (comes with Rust)
Kubernetes cluster (local or remote)
kubectl configured to access your cluster

Installing k8s-maestro

Using Cargo

Add k8s-maestro to your Cargo.toml:

[dependencies]
k8s-maestro = "0.3"

Run cargo build to download and compile the crate:

cargo build --release

Enabling Kubernetes Support

k8s-maestro supports multiple Kubernetes versions. Enable the appropriate feature flag:

k8s-maestro = { version = "0.3", features = ["k8s_v1_28"] }

Available features:

k8s_v1_28 - Kubernetes 1.28 (default)
k8s_v1_29 - Kubernetes 1.29
k8s_v1_30 - Kubernetes 1.30
k8s_v1_31 - Kubernetes 1.31
k8s_v1_32 - Kubernetes 1.32

Enabling Additional Features

k8s-maestro provides optional features for extended functionality:

# Enable exec steps (for running scripts directly)
k8s-maestro = { version = "0.3", features = ["exec-steps"] }

# Combine features
k8s-maestro = { version = "0.3", features = ["k8s_v1_30", "exec-steps"] }

Kubernetes Cluster Setup

Using Kind (Local Development)

For local development and testing, use Kind to create a local Kubernetes cluster:

# Install Kind (if not already installed)
go install sigs.k8s.io/kind@v0.20.0

# Create a Kind cluster
kind create cluster --name maestro-test

# Verify cluster is running
kubectl cluster-info

Using Minikube

Alternatively, use Minikube:

# Install Minikube
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Start Minikube
minikube start

# Verify cluster is running
kubectl cluster-info

Using a Remote Cluster

Configure kubectl to access your remote cluster:

# Update kubeconfig
kubectl config use-context your-remote-cluster

# Verify connection
kubectl cluster-info

Verifying Installation

Create a test file test_installation.rs:

use k8s_maestro::MaestroClientBuilder;

fn main() -> anyhow::Result<()> {
    let client = MaestroClientBuilder::new()
        .with_namespace("default")
        .build()?;

    println!("k8s-maestro installed successfully!");
    println!("Namespace: {}", client.namespace());

    Ok(())
}

Run the test:

cargo run --bin test_installation

Troubleshooting

Connection Issues

If you encounter connection errors to the Kubernetes cluster:

Verify kubectl is configured correctly:
```
kubectl cluster-info
```
Check your kubeconfig file:
```
kubectl config view
```
Ensure the kubeconfig file is in the default location:
```
echo $KUBECONFIG
# Usually: ~/.kube/config
```

Feature Conflicts

When using multiple features, ensure compatibility:

# Correct - compatible features
k8s-maestro = { version = "0.3", features = ["k8s_v1_30", "exec-steps"] }

# Incorrect - only one k8s version feature should be enabled
k8s-maestro = { version = "0.3", features = ["k8s_v1_28", "k8s_v1_30"] }

Build Errors

If you encounter build errors, try:

# Clean and rebuild
cargo clean
cargo build --release

# Update dependencies
cargo update

# Check Rust version
rustc --version  # Should be 1.70 or later

Next Steps

After installation, continue to Quick Start to run your first workflow.

Quick Start

This guide will help you run your first workflow in just 5 minutes.

Prerequisites

Ensure you have completed the Installation guide and have:

k8s-maestro installed
A running Kubernetes cluster
kubectl configured

Your First Workflow

Create a new Rust project:

cargo new my-first-workflow
cd my-first-workflow

Add k8s-maestro to Cargo.toml:

[package]
name = "my-first-workflow"
version = "0.1.0"
edition = "2021"

[dependencies]
k8s-maestro = { version = "1.0", features = ["k8s_v1_30"] }
tokio = { version = "1", features = ["full"] }
anyhow = "1.0"

Update src/main.rs:

use k8s_maestro::{MaestroClientBuilder, WorkflowBuilder};
use k8s_maestro::steps::KubeJobStep;
use k8s_maestro::clients::MaestroK8sClient;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    println!("Creating Kubernetes client...");

    let k8s_client = MaestroK8sClient::new().await?;

    println!("Creating k8s-maestro client...");

    let client = MaestroClientBuilder::new()
        .with_namespace("default")
        .with_client(k8s_client)
        .build()?;

    println!("Building workflow...");

    let workflow = WorkflowBuilder::new()
        .with_name("hello-maestro")
        .add_step(KubeJobStep::new("hello-job", "nginx:latest", k8s_client.clone()))
        .build()?;

    println!("Executing workflow...");

    let created = client.create_workflow(workflow)?;
    println!("Workflow created with ID: {}", created.id());
    println!("Workflow name: {}", created.name());
    println!("Namespace: {}", created.namespace());

    Ok(())
}

Run the workflow:

cargo run

Verifying Execution

Check the workflow status with kubectl:

# List pods in the workflow
kubectl get pods

# View pod logs
kubectl logs -l app=hello-maestro

# Check workflow status
kubectl describe jobs

Example Output

You should see output similar to:

Creating k8s-maestro client...
Building workflow...
Executing workflow...
Workflow created with ID: 550e8400-e29b-41d4-a716-446655440000
Workflow name: hello-maestro
Namespace: default

Next Steps

Congratulations! You've successfully run your first workflow. Continue to:

Basic Workflows - Learn workflow building patterns
Dependencies - Configure step execution order
Examples - Explore more workflow examples

Troubleshooting

"Namespace not found" Error

Create the namespace if it doesn't exist:

kubectl create namespace default

"Connection refused" Error

Verify your Kubernetes cluster is running:

kubectl cluster-info

Workflow stuck in "Pending" state

Check the pod status for more details:

kubectl describe pod <pod-name>

Common issues:

Insufficient resources: Increase cluster capacity
Image pull errors: Check image availability
Node affinity: Ensure nodes match the workflow requirements

Concepts

This guide explains the core concepts of k8s-maestro.

Workflows

A Workflow is a collection of steps that execute in a specific order or in parallel. Workflows represent a logical unit of work that can be scheduled, executed, monitored, and managed in Kubernetes.

Workflow Properties

Name: Unique identifier for the workflow
Namespace: Kubernetes namespace where the workflow runs
Steps: Individual tasks to execute
Metadata: Labels, annotations, and custom metadata
Execution Mode: Sequential or parallel execution
Parallelism: Maximum number of steps running simultaneously
Checkpointing: Automatic checkpoint and recovery configuration

Steps

A Step is the smallest unit of work in a workflow. Each step represents a task that can be executed independently.

Step Types

k8s-maestro supports multiple step types:

JobStep: Kubernetes Job - runs containers to completion
ExecStep: Execute commands directly (local execution)
WasmStep: WebAssembly module execution
PythonStep: Python script execution (aspirational)
RustStep: Rust code execution (aspirational)

Step Properties

Step ID: Unique identifier within the workflow
Container Image: Docker image to run
Command/Arguments: Command to execute in the container
Environment Variables: Configuration for the step
Resource Limits: CPU, memory, and GPU constraints
Volume Mounts: Storage attachments
Dependencies: Other steps that must complete first

Dependencies

Dependencies define the execution order of steps. A step can depend on one or more other steps, creating a Directed Acyclic Graph (DAG).

Dependency Types

Simple Dependency: Step B runs after Step A completes
Multiple Dependencies: Step C runs after Steps A and B complete
Any Dependency: Step D runs when ANY of A, B, or C completes
Conditional Dependency: Step E runs only if a condition is met

Dependency Conditions

All Success: Run only if all dependencies succeed
Any Success: Run if any dependency succeeds
Output Check: Run based on dependency output values
Custom Condition: User-defined condition logic

Execution Modes

Sequential

Steps execute one after another:

Step A -> Step B -> Step C -> Step D

Parallel

Multiple steps execute simultaneously:

Step A
Step B
Step C

Mixed

Combination of sequential and parallel:

    Step A
   /      \
Step B    Step C
   \      /
    Step D

Client

The MaestroClient is the main interface for interacting with Kubernetes workflows.

Client Capabilities

Create workflows
Retrieve workflow status
List workflows
Delete workflows
Watch workflow execution
Configure default settings (namespace, timeout, logging)

Client Configuration

#![allow(unused)]
fn main() {
let client = MaestroClientBuilder::new()
    .with_namespace("production")
    .with_default_timeout(Duration::from_secs(300))
    .with_dry_run(false)
    .with_log_level("info")
    .build()?;
}

Checkpointing

Checkpointing enables workflow recovery from failures. When enabled, k8s-maestro periodically saves the workflow state.

Checkpoint Benefits

Resume from last checkpoint after failures
Avoid re-executing completed steps
Debug workflow execution
Audit workflow history

Checkpoint Configuration

#![allow(unused)]
fn main() {
let checkpoint_config = LegacyCheckpointConfig::new()
    .enabled(true)
    .with_interval_secs(60)
    .with_retention_count(10)
    .with_storage_path("/checkpoints");

let workflow = WorkflowBuilder::new()
    .with_checkpointing(checkpoint_config)
    // ...
}

Services and Ingress

Services

A Service exposes a workflow step to network traffic within the cluster.

#![allow(unused)]
fn main() {
let service = ServiceBuilder::new()
    .with_name("my-service")
    .with_port(80, 8080, "TCP")
    .with_type(ServiceType::ClusterIP)
    .build()?;
}

Service Types

ClusterIP: Internal cluster access only
NodePort: Accessible via node IPs
LoadBalancer: External load balancer
Headless: DNS returns pod IPs directly

Ingress

Ingress provides external HTTP/HTTPS access to services.

#![allow(unused)]
fn main() {
let ingress = IngressBuilder::new()
    .with_name("my-ingress")
    .with_host("example.com")
    .with_path("/", "my-service", 80)
    .with_tls_secret("tls-secret")
    .build()?;
}

Sidecars

A Sidecar is an auxiliary container that runs alongside the main container in the same pod.

Common Sidecar Use Cases

Logging: Fluent Bit, log collectors
Monitoring: Prometheus exporters, statsd
Proxies: Envoy, NGINX sidecars
Debugging: Debug containers, network tools

#![allow(unused)]
fn main() {
let logging_sidecar = SidecarContainer::new("fluent/fluent-bit:2.2", "log-collector")
    .with_shared_volume("/var/log", "/var/log");
}

Next Steps

Basic Workflows - Create workflows step by step
Dependencies - Configure complex execution flows
API Reference - Detailed API documentation

Guides

This section contains step-by-step guides for common k8s-maestro workflows and patterns.

Available Guides

Basic Workflow

Learn how to create and run a basic workflow with multiple steps.

Advanced Patterns

Multi-workflow orchestration
Service exposure and ingress configuration
Sidecar containers
Conditional execution
File observer integration

Best Practices

Workflow design patterns
Error handling strategies
Performance optimization
Testing workflows

Basic Workflows

This guide teaches you how to build and execute basic workflows with k8s-maestro.

Creating a Simple Workflow

The simplest workflow consists of a single step that runs a container to completion.

use k8s_maestro::{MaestroClientBuilder, MaestroK8sClient, WorkflowBuilder};
use k8s_maestro::steps::kubernetes::JobStep;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let k8s_client = MaestroK8sClient::new().await?;
    
    let client = MaestroClientBuilder::new()
        .with_namespace("default")
        .with_client(k8s_client.clone())
        .build()?;

    let workflow = WorkflowBuilder::new()
        .with_name("simple-workflow")
        .add_step(JobStep::new("hello-job", "nginx:latest", k8s_client))
        .build()?;

    let created = client.create_workflow(workflow)?;
    println!("Workflow created: {}", created.id());

    Ok(())
}

Configuring Container Options

Customize the step's container with arguments, environment variables, and resource limits.

#![allow(unused)]
fn main() {
use k8s_maestro::{MaestroK8sClient, WorkflowBuilder};
use k8s_maestro::entities::MaestroContainer;
use k8s_maestro::steps::kubernetes::JobStep;
use std::collections::BTreeMap;

let k8s_client = MaestroK8sClient::new().await?;

let container = MaestroContainer::new("python:3.11", "data-processor")
    .set_arguments(&vec![
        "python".to_string(),
        "-c".to_string(),
        "print('Processing data...')".to_string(),
    ])
    .set_environment_variables(vec![
        ("LOG_LEVEL".to_string(), "info".to_string()),
    ].into_iter().collect());

let mut resource_limits = BTreeMap::new();
resource_limits.insert("cpu".to_string(), "500m".to_string());
resource_limits.insert("memory".to_string(), "512Mi".to_string());

let workflow = WorkflowBuilder::new()
    .with_name("configured-workflow")
    .add_step(JobStep::new("process-job", "python:3.11", k8s_client.clone())
        .with_container(container)
        .with_resource_limits(resource_limits))
    .build()?;
}

Adding Multiple Steps

Create workflows with multiple independent steps that run in parallel.

#![allow(unused)]
fn main() {
use k8s_maestro::{MaestroK8sClient, WorkflowBuilder};
use k8s_maestro::steps::kubernetes::JobStep;

let k8s_client = MaestroK8sClient::new().await?;

let workflow = WorkflowBuilder::new()
    .with_name("multi-step-workflow")
    .with_parallelism(3)
    .add_step(JobStep::new("fetch-data", "curlimages/curl", k8s_client.clone()))
    .add_step(JobStep::new("process-a", "python:3.11", k8s_client.clone()))
    .add_step(JobStep::new("process-b", "python:3.11", k8s_client))
    .build()?;
}

Setting Workflow Metadata

Add labels and annotations for organization and tracking.

#![allow(unused)]
fn main() {
use k8s_maestro::{MaestroK8sClient, WorkflowBuilder};
use k8s_maestro::steps::kubernetes::JobStep;

let k8s_client = MaestroK8sClient::new().await?;

let workflow = WorkflowBuilder::new()
    .with_name("labeled-workflow")
    .with_label("environment", "production")
    .with_label("team", "data-science")
    .with_label("cost-center", "engineering")
    .with_annotation("owner", "team-lead@example.com")
    .with_annotation("jira-ticket", "OPS-1234")
    .add_step(JobStep::new("main-job", "python:3.11", k8s_client))
    .build()?;
}

Configuring Execution Mode

Control whether steps run sequentially or in parallel.

Sequential Execution

#![allow(unused)]
fn main() {
use k8s_maestro::{MaestroK8sClient, WorkflowBuilder};
use k8s_maestro::steps::kubernetes::JobStep;

let k8s_client = MaestroK8sClient::new().await?;

let workflow = WorkflowBuilder::new()
    .with_name("sequential-workflow")
    .with_execution_mode(ExecutionMode::Sequential)
    .add_step(JobStep::new("step-1", "python:3.11", k8s_client.clone()))
    .add_step(JobStep::new("step-2", "python:3.11", k8s_client.clone()))
    .add_step(JobStep::new("step-3", "python:3.11", k8s_client))
    .build()?;
}

Parallel Execution

#![allow(unused)]
fn main() {
use k8s_maestro::{MaestroK8sClient, WorkflowBuilder};
use k8s_maestro::steps::kubernetes::JobStep;

let k8s_client = MaestroK8sClient::new().await?;

let workflow = WorkflowBuilder::new()
    .with_name("parallel-workflow")
    .with_execution_mode(ExecutionMode::Parallel(5))
    .add_step(JobStep::new("worker-1", "python:3.11", k8s_client.clone()))
    .add_step(JobStep::new("worker-2", "python:3.11", k8s_client.clone()))
    .add_step(JobStep::new("worker-3", "python:3.11", k8s_client.clone()))
    .add_step(JobStep::new("worker-4", "python:3.11", k8s_client.clone()))
    .add_step(JobStep::new("worker-5", "python:3.11", k8s_client))
    .build()?;
}

Working with Workflows

Creating a Workflow

#![allow(unused)]
fn main() {
use k8s_maestro::{MaestroK8sClient, WorkflowBuilder};
use k8s_maestro::steps::kubernetes::JobStep;

let k8s_client = MaestroK8sClient::new().await?;

let workflow = WorkflowBuilder::new()
    .with_name("my-workflow")
    .add_step(JobStep::new("job-1", "nginx:latest", k8s_client))
    .build()?;

let created = client.create_workflow(workflow)?;
}

Retrieving a Workflow

#![allow(unused)]
fn main() {
if let Some(workflow) = client.get_workflow(&created.id())? {
    println!("Workflow: {}", workflow.name());
    println!("Status: {:?}", workflow.status());
}
}

Listing Workflows

#![allow(unused)]
fn main() {
let workflows = client.list_workflows()?;

for workflow in workflows {
    println!("{}: {} ({})", workflow.id(), workflow.name(), workflow.namespace());
}
}

Deleting a Workflow

#![allow(unused)]
fn main() {
client.delete_workflow(&workflow_id)?;
}

Best Practices

Use descriptive names for workflows and steps
Add labels for organization and filtering
Set appropriate resource limits to prevent resource exhaustion
Choose the right execution mode for your use case
Monitor workflow execution with kubectl
Use dry-run mode for testing without execution

Next Steps

Dependencies - Configure step execution order
Services & Ingress - Expose workflow services
Examples - More workflow examples

Reference

This section provides detailed reference documentation for k8s-maestro configuration, API, and troubleshooting.

Configuration

This guide covers k8s-maestro configuration options and environment variables.

Client Configuration

Configure the MaestroClient with various options for your use case.

Namespace

Set the default namespace for workflow operations.

#![allow(unused)]
fn main() {
let client = MaestroClientBuilder::new()
    .with_namespace("production")
    .build()?;
}

Dry Run

Enable dry run mode to validate workflows without execution.

#![allow(unused)]
fn main() {
let client = MaestroClientBuilder::new()
    .with_dry_run(true)
    .build()?;

// Workflow will be validated but not executed
let created = client.create_workflow(workflow)?;
assert!(created.is_dry_run());
}

Timeout

Set a default timeout for workflow operations.

#![allow(unused)]
fn main() {
use std::time::Duration;

let client = MaestroClientBuilder::new()
    .with_default_timeout(Duration::from_secs(300))
    .build()?;
}

Logging

Configure the log level for client operations.

#![allow(unused)]
fn main() {
let client = MaestroClientBuilder::new()
    .with_log_level("debug")
    .build()?;
}

Available log levels:

trace - Most verbose
debug - Detailed debugging
info - Informational (default)
warn - Warnings
error - Errors only

Resource Limits

Set default resource limits for all workflows.

#![allow(unused)]
fn main() {
use k8s_maestro::steps::traits::ResourceLimits;

let limits = ResourceLimits::new()
    .with_cpu("500m")
    .with_memory("512Mi")
    .with_gpu("0"); // No GPU by default

let client = MaestroClientBuilder::new()
    .with_default_resource_limits(limits)
    .build()?;
}

Kubeconfig Path

Specify a custom kubeconfig file location.

#![allow(unused)]
fn main() {
use std::path::PathBuf;

let client = MaestroClientBuilder::new()
    .with_kube_config_path(PathBuf::from("/custom/path/to/kubeconfig"))
    .build()?;
}

Environment Variables

k8s-maestro can be configured using environment variables.

KUBECONFIG

Path to the kubeconfig file.

export KUBECONFIG=/path/to/kubeconfig

MAESTRO_NAMESPACE

Default namespace for workflows.

export MAESTRO_NAMESPACE=production

MAESTRO_LOG_LEVEL

Log level for k8s-maestro operations.

export MAESTRO_LOG_LEVEL=debug

MAESTRO_DRY_RUN

Enable dry run mode (1) or disable (0).

export MAESTRO_DRY_RUN=0

MAESTRO_TIMEOUT

Default timeout in seconds for operations.

export MAESTRO_TIMEOUT=300

MAESTRO_DEFAULT_CPU

Default CPU limit for workflows.

export MAESTRO_DEFAULT_CPU=500m

MAESTRO_DEFAULT_MEMORY

Default memory limit for workflows.

export MAESTRO_DEFAULT_MEMORY=512Mi

MAESTRO_DEFAULT_GPU

Default GPU limit for workflows.

export MAESTRO_DEFAULT_GPU=0

Workflow Configuration

Configure individual workflow settings.

Checkpointing

Enable automatic checkpointing and recovery.

#![allow(unused)]
fn main() {
use k8s_maestro::workflows::LegacyCheckpointConfig;

let checkpoint_config = LegacyCheckpointConfig::new()
    .enabled(true)
    .with_interval_secs(60)
    .with_retention_count(10)
    .with_storage_path("/checkpoints");

let workflow = WorkflowBuilder::new()
    .with_name("checkpointed-workflow")
    .with_checkpointing(checkpoint_config)
    .add_step(JobStep::new("long-running", "python:3.11"))
    .build()?;
}

Parallelism

Set the maximum number of parallel steps.

#![allow(unused)]
fn main() {
let workflow = WorkflowBuilder::new()
    .with_name("parallel-workflow")
    .with_parallelism(5)
    .add_step(JobStep::new("worker-1", "python:3.11"))
    .add_step(JobStep::new("worker-2", "python:3.11"))
    .add_step(JobStep::new("worker-3", "python:3.11"))
    .add_step(JobStep::new("worker-4", "python:3.11"))
    .add_step(JobStep::new("worker-5", "python:3.11"))
    .build()?;
}

Execution Mode

Configure how steps execute (sequential or parallel).

#![allow(unused)]
fn main() {
use k8s_maestro::workflows::ExecutionMode;

let workflow = WorkflowBuilder::new()
    .with_name("execution-workflow")
    .with_execution_mode(ExecutionMode::Sequential)
    .add_step(JobStep::new("step-1", "python:3.11"))
    .add_step(JobStep::new("step-2", "python:3.11"))
    .build()?;
}

Kubernetes Configuration

Context Selection

Select a specific Kubernetes context from kubeconfig.

kubectl config use-context production-cluster

Namespace Creation

Create a namespace if it doesn't exist.

kubectl create namespace production

Resource Quotas

Set resource quotas to limit resource usage.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: maestro-quota
  namespace: production
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi

Configuration File (Future)

k8s-maestro plans to support configuration files in the future:

# maestro.yaml
client:
  namespace: production
  dry_run: false
  timeout: 300
  log_level: info

defaults:
  resources:
    cpu: 500m
    memory: 512Mi

checkpointing:
  enabled: true
  interval: 60
  retention: 10
  storage_path: /checkpoints

Best Practices

Use environment variables for sensitive data (API keys, passwords)
Set appropriate resource limits to prevent resource exhaustion
Enable checkpointing for long-running workflows
Use dry run mode for testing in production environments
Organize with namespaces for different environments
Monitor resource usage with Kubernetes metrics

Next Steps

Troubleshooting - Common configuration issues
API Reference - Detailed API documentation

Troubleshooting

This guide helps you diagnose and fix common issues with k8s-maestro.

Connection Issues

"Connection refused" or "No route to host"

Symptoms:

Cannot connect to Kubernetes cluster
Timeout errors during client creation

Solutions:

Verify cluster is running:
```
kubectl cluster-info
```
Check kubeconfig:
```
kubectl config view
```
Test connection:
```
kubectl get nodes
```

Verify context:

kubectl config current-context
kubectl config use-context correct-context

"Unauthorized" or "Forbidden"

Symptoms:

Authentication errors
Permission denied errors

Solutions:

Check current user:
```
kubectl auth whoami
```

Verify permissions:

kubectl auth can-i create jobs --namespace=default

Update credentials:

aws eks update-kubeconfig --name my-cluster  # EKS
gcloud container clusters get-credentials my-cluster  # GKE
az aks get-credentials --resource-group myRG --name myCluster  # AKS

Workflow Execution Issues

Workflow stuck in "Pending" state

Symptoms:

Workflow created but not executing
Pods remain in Pending state

Solutions:

Check pod status:
```
kubectl describe pod <pod-name>
```
Common causes:
- Insufficient resources: Add more nodes or reduce resource limits
- Image pull errors: Verify image name and registry access
- Node affinity: Ensure nodes match the workflow requirements
- Taints and tolerations: Check if nodes have taints blocking the pod

Check resource usage:

kubectl top nodes
kubectl describe nodes

Workflow failed with "CrashLoopBackOff"

Symptoms:

Pod repeatedly crashes
Container exits immediately

Solutions:

Check pod logs:
```
kubectl logs <pod-name> --previous
```
Common causes:
- Application errors: Fix the application code
- Missing dependencies: Ensure all required files/packages are present
- Configuration errors: Verify environment variables and arguments
- Health check failures: Adjust probe configurations

Debug with interactive shell:

kubectl exec -it <pod-name> -- /bin/bash

Workflow execution timeout

Symptoms:

Workflow takes longer than expected
Timeout errors

Solutions:

Check pod logs for slow operations:
```
kubectl logs -f <pod-name>
```
Common causes:
- Inefficient algorithms: Optimize the workflow logic
- Insufficient resources: Increase CPU/memory limits
- Network bottlenecks: Check network connectivity
- Large data sets: Consider data partitioning

Increase timeout:

#![allow(unused)]
fn main() {
let client = MaestroClientBuilder::new()
    .with_default_timeout(Duration::from_secs(600))
    .build()?;
}

Resource Issues

"Insufficient cpu" or "Insufficient memory"

Symptoms:

Pods not scheduled
Resource quota exceeded errors

Solutions:

Check cluster capacity:

kubectl describe nodes | grep -A 3 "Allocated resources"

Check resource quotas:

kubectl describe resourcequota -n <namespace>

Solutions:
- Scale cluster: Add more nodes
- Reduce limits: Lower resource requirements
- Delete unused resources: Free up capacity
- Request quota increase: If using managed quotas

"OOMKilled" (Out of Memory)

Symptoms:

Pod killed due to memory exhaustion
Container restarts

Solutions:

Check pod events:
```
kubectl describe pod <pod-name>
```
Solutions:
- Increase memory limit: Allocate more memory
- Optimize memory usage: Reduce memory footprint
- Enable swap: If supported by your runtime

Update workflow:

#![allow(unused)]
fn main() {
let workflow = WorkflowBuilder::new()
    .with_name("memory-workflow")
    .add_step(JobStep::new("job-1", "python:3.11")
        .with_resource_limits(ResourceLimits::new()
            .with_memory("2Gi")))
    .build()?;
}

Image Issues

"ImagePullBackOff" or "ErrImagePull"

Symptoms:

Cannot pull container image
Image pull errors

Solutions:

Check image name:

# Verify image exists
docker pull <image-name>

Check registry access:

# Test registry authentication
docker login <registry-url>

Solutions:
- Fix image name: Ensure correct registry and tag
- Configure image pull secret: Add secrets for private registries
- Use public image: If registry issues persist

Configure image pull secret:

#![allow(unused)]
fn main() {
let workflow = WorkflowBuilder::new()
    .with_name("private-image-workflow")
    .with_image_pull_secret("my-registry-secret")
    .add_step(JobStep::new("job-1", "my-registry.com/my-image:latest"))
    .build()?;
}

Dependency Issues

"Cycle detected" in dependencies

Symptoms:

Workflow creation fails
Cycle detection error

Solutions:

Review dependency graph:

#![allow(unused)]
fn main() {
let graph = chain.build_dag();
match graph.detect_cycles() {
    Ok(()) => println!("No cycles"),
    Err(e) => println!("Cycle: {}", e),
}
}

Common causes:
- Circular references: A -> B -> C -> A
- Self-dependencies: Step depends on itself

Fix by removing the cycle:

#![allow(unused)]
fn main() {
// Incorrect (creates cycle)
chain.add_step("A");
chain.add_step("B").with_dependency("A");
chain.add_step("C").with_dependency("B");
chain.add_step("A").with_dependency("C"); // Cycle!

// Correct
chain.add_step("A");
chain.add_step("B").with_dependency("A");
chain.add_step("C").with_dependency("B");
// Step A is the root, no self-dependency
}

Step never runs (stuck waiting for dependencies)

Symptoms:

Step remains in pending state
Dependencies never complete

Solutions:

Check dependency status:

kubectl get pods -l workflow=<workflow-id>

Common causes:
- Failed dependencies: Check if dependencies succeeded
- Incorrect dependency names: Verify step IDs
- Missing dependencies: Ensure all dependencies exist

Use conditional dependencies:

#![allow(unused)]
fn main() {
chain.add_step("validate");
chain.add_step("process")
    .with_conditional_dependency("validate", ConditionBuilder::all_success());
chain.add_step("cleanup")
    .with_conditional_dependency_any(vec!["validate", "process"], ConditionBuilder::any_failure());
}

Debugging Tips

Enable Verbose Logging

#![allow(unused)]
fn main() {
let client = MaestroClientBuilder::new()
    .with_log_level("trace")
    .build()?;
}

Or use environment variable:

export RUST_LOG=trace
export MAESTRO_LOG_LEVEL=trace

Inspect Created Resources

# List all resources created by the workflow
kubectl get all -l app=<workflow-name>

# Describe a specific resource
kubectl describe pod <pod-name>

# View events
kubectl get events --sort-by='.lastTimestamp'

Test with Dry Run

#![allow(unused)]
fn main() {
let client = MaestroClientBuilder::new()
    .with_dry_run(true)
    .build()?;

let created = client.create_workflow(workflow)?;
// No resources are actually created
}

Use Kind for Local Testing

# Create a local cluster
kind create cluster --name test

# Run your workflow
cargo run

# Debug with kubectl
kubectl get pods
kubectl logs -f <pod-name>

Getting Help

If you're still experiencing issues:

Check the GitHub Issues
Search for similar issues
Create a new issue with:
- k8s-maestro version
- Kubernetes version
- Full error message
- Steps to reproduce
- Relevant logs and configuration

Next Steps

Configuration - Configuration options
API Reference - Detailed API documentation

Testing Guide

This guide explains how to write and run tests for k8s-maestro.

Test Categories

k8s-maestro uses a three-tier test organization:

Category	Purpose	Cluster Required	Speed
Unit	Test individual functions and logic	No	Fast (< 10s)
Integration	Test Kubernetes API interactions	Yes (Kind)	Medium (< 5min)
E2E	Test complete workflow scenarios	Yes (Kind)	Medium (< 5min)

Running Tests

Unit Tests

# Run unit tests only (no Docker needed)
cargo test --lib

# Run a specific unit test
cargo test test_create_configmap

Integration Tests

# Run integration tests (requires Docker)
cargo test --test '*' -- --ignored

# Run specific integration test
cargo test --test kind_cluster_lifecycle -- --ignored

All Tests

# Run all tests including ignored ones
cargo test -- --include-ignored

Writing Tests

Unit Tests with Mocking

Use the mocking module for unit tests that don't need a real cluster:

#![allow(unused)]
fn main() {
use k8s_maestro::tests::common::mocking::{MockK8sClient, mock_error, mock_resource_response};

#[test]
fn test_create_resource_success() {
    let mut client = MockK8sClient::new()
        .add_create_response(Ok(mock_resource_response("ConfigMap", "test-cm", "default")));
    
    let response = client.next_create_response().unwrap();
    assert_eq!(response["kind"], "ConfigMap");
}

#[test]
fn test_create_resource_already_exists() {
    let mut client = MockK8sClient::new()
        .add_create_response(Err(mock_error("AlreadyExists", "resource exists")));
    
    let result = client.next_create_response();
    assert!(result.is_err());
}
}

Integration Tests with Kind

Use the Kind cluster module for integration tests:

#![allow(unused)]
fn main() {
use k8s_maestro::tests::common::kind_cluster::KindCluster;
use k8s_maestro::tests::common::utilities::{
    create_configmap, apply_resource, verify_resource_exists, delete_resource_by_name,
};
use k8s_openapi::api::core::v1::ConfigMap;

#[tokio::test]
#[ignore = "Requires Docker"]
async fn test_configmap_lifecycle() {
    // Setup cluster
    let cluster = KindCluster::new().await.expect("Failed to create cluster");
    
    // Create and apply ConfigMap
    let cm = create_configmap("test-cm", "default", std::collections::BTreeMap::new());
    apply_resource(&client, &cm, "default").await.expect("Failed to apply");
    
    // Verify it exists
    assert!(verify_resource_exists::<ConfigMap>(&client, "test-cm", "default").await);
    
    // Cleanup
    delete_resource_by_name::<ConfigMap>(&client, "test-cm", "default").await.ok();
}
}

E2E Tests for Workflows

Use the E2E helpers for complete workflow scenarios:

#![allow(unused)]
fn main() {
use k8s_maestro::tests::e2e::setup_e2e_test;
use k8s_maestro::tests::common::fixtures::load_workflow_fixture;

#[tokio::test]
#[ignore = "Requires Docker"]
async fn e2e_workflow_execution() {
    let (client, _cluster) = setup_e2e_test().await.expect("Failed to setup");
    
    // Load and apply workflow
    let workflow = load_workflow_fixture("simple-workflow").expect("Failed to load");
    
    // Verify workflow execution...
}
}

Test Utilities

Fixtures

Load pre-defined YAML fixtures:

#![allow(unused)]
fn main() {
use k8s_maestro::tests::common::fixtures::{
    load_configmap_fixture,
    load_secret_fixture,
    load_pvc_fixture,
};

let cm = load_configmap_fixture("test-configmap").unwrap();
let secret = load_secret_fixture("test-secret").unwrap();
}

Resource Helpers

Create resources programmatically:

#![allow(unused)]
fn main() {
use k8s_maestro::tests::common::utilities::{
    create_configmap, create_secret, create_pvc, create_namespace,
};
use std::collections::BTreeMap;

let cm = create_configmap("my-cm", "default", BTreeMap::new());
let secret = create_secret("my-secret", "default", BTreeMap::new());
let ns = create_namespace("test");
}

Validation Helpers

Verify resource states:

#![allow(unused)]
fn main() {
use k8s_maestro::tests::common::utilities::{
    verify_resource_exists,
    verify_resource_state,
    wait_for_resource_ready,
};

// Check if resource exists
if verify_resource_exists::<ConfigMap>(&client, "my-cm", "default").await {
    println!("ConfigMap exists!");
}

// Wait for resource to be ready
wait_for_resource_ready::<Pod>(&client, "my-pod", "default", |pod| {
    pod.status.as_ref().map(|s| s.phase.as_deref() == Some("Running")).unwrap_or(false)
}).await.expect("Pod never became ready");
}

Best Practices

Use the right test category: Unit tests for logic, integration for K8s API, E2E for workflows
Clean up resources: Always clean up created resources in tests
Use unique names: Create resources with unique names to avoid conflicts
Mark tests appropriately: Use #[ignore] for tests requiring Docker
Keep fixtures minimal: Only include necessary fields in fixtures