Job Workflows | Sylphx Documentation

Job Chaining

Sequential step execution

Fan-Out

Parallel job execution

Fan-In

Collect parallel results

Saga Pattern

Distributed transactions

Overview

Workflows allow you to orchestrate multiple jobs as a single unit of work. Each step in a workflow is a job that can depend on previous steps, run in parallel, and pass data to subsequent steps.

import { platform } from '@/lib/platform'

// Create a simple sequential workflow
const workflow = await platform.workflows.create({
  id: 'order-processing',
  steps: [
    { id: 'validate', url: '/api/jobs/validate-order' },
    { id: 'charge', url: '/api/jobs/charge-payment', dependsOn: ['validate'] },
    { id: 'fulfill', url: '/api/jobs/fulfill-order', dependsOn: ['charge'] },
    { id: 'notify', url: '/api/jobs/send-confirmation', dependsOn: ['fulfill'] },
  ],
  context: {
    orderId: 'order_123',
    userId: 'user_456',
  },
})

Workflow Configuration

Configure workflow behavior and step execution.

Property	Type	Description
`id`required	`string`	Unique workflow identifier
`steps`required	`WorkflowStep[]`	Array of workflow steps to execute
`context`	`object`	Initial context passed to all steps
`onComplete`	`string`	Webhook URL called when workflow completes
`onError`	`string`	Webhook URL called when workflow fails
`timeout`	`number`= 86400 (24h)	Maximum workflow duration in seconds

Step Configuration

Property	Type	Description
`id`required	`string`	Unique step identifier within workflow
`url`required	`string`	Job handler URL for this step
`dependsOn`	`string[]`	Step IDs that must complete before this step
`parallel`	`boolean`= false	Run with other parallel steps concurrently
`retries`	`RetryConfig`	Retry configuration for this step
`timeout`	`number`= 300	Step timeout in seconds
`compensation`	`string`	URL to call if workflow needs rollback (saga pattern)

Chaining Jobs

Chain jobs together to execute sequentially. Each step receives the context and results from previous steps.

Create workflow

import { platform } from '@/lib/platform'

// Sequential order processing workflow
const workflow = await platform.workflows.create({
  id: 'order-processing',
  steps: [
    {
      id: 'validate',
      url: 'https://myapp.com/api/jobs/validate-order',
    },
    {
      id: 'reserve-inventory',
      url: 'https://myapp.com/api/jobs/reserve-inventory',
      dependsOn: ['validate'],
    },
    {
      id: 'process-payment',
      url: 'https://myapp.com/api/jobs/process-payment',
      dependsOn: ['reserve-inventory'],
      retries: {
        maxAttempts: 3,
        backoff: 'exponential',
      },
    },
    {
      id: 'create-shipment',
      url: 'https://myapp.com/api/jobs/create-shipment',
      dependsOn: ['process-payment'],
    },
    {
      id: 'send-confirmation',
      url: 'https://myapp.com/api/jobs/send-confirmation',
      dependsOn: ['create-shipment'],
    },
  ],
  context: {
    orderId: 'order_123',
    userId: 'user_456',
    items: [
      { sku: 'SKU001', quantity: 2 },
      { sku: 'SKU002', quantity: 1 },
    ],
  },
  onComplete: 'https://myapp.com/api/webhooks/workflow-complete',
  onError: 'https://myapp.com/api/webhooks/workflow-error',
})

Fan-Out Patterns

Execute multiple jobs in parallel for improved throughput. Fan-out is useful when steps are independent and can run concurrently.

Fan-out workflow

import { platform } from '@/lib/platform'

// Process multiple items in parallel
const workflow = await platform.workflows.create({
  id: 'batch-process',
  steps: [
    // Initial step
    {
      id: 'prepare',
      url: 'https://myapp.com/api/jobs/prepare-batch',
    },
    // Parallel processing steps (all have same dependency)
    {
      id: 'process-item-1',
      url: 'https://myapp.com/api/jobs/process-item',
      dependsOn: ['prepare'],
      parallel: true, // Run in parallel with other parallel steps
    },
    {
      id: 'process-item-2',
      url: 'https://myapp.com/api/jobs/process-item',
      dependsOn: ['prepare'],
      parallel: true,
    },
    {
      id: 'process-item-3',
      url: 'https://myapp.com/api/jobs/process-item',
      dependsOn: ['prepare'],
      parallel: true,
    },
    // Final step waits for all parallel steps
    {
      id: 'finalize',
      url: 'https://myapp.com/api/jobs/finalize-batch',
      dependsOn: ['process-item-1', 'process-item-2', 'process-item-3'],
    },
  ],
  context: {
    batchId: 'batch_123',
    items: ['item_1', 'item_2', 'item_3'],
  },
})

Fan-In Patterns

Collect and aggregate results from parallel jobs. The fan-in step receives all results from its dependencies.

Fan-in handler

// app/api/jobs/finalize-batch/route.ts
import { platform } from '@/lib/platform'
import { NextRequest } from 'next/server'

export async function POST(req: NextRequest) {
  const isValid = await platform.jobs.verifyRequest(req)
  if (!isValid) {
    return new Response('Unauthorized', { status: 401 })
  }

  const { context, previousSteps } = await req.json()

  // Collect all results from parallel steps
  const processResults = Object.entries(previousSteps)
    .filter(([stepId]) => stepId.startsWith('process-'))
    .map(([stepId, result]) => ({
      stepId,
      ...result.result,
    }))

  // Aggregate statistics
  const stats = {
    total: processResults.length,
    successful: processResults.filter(r => r.success).length,
    failed: processResults.filter(r => !r.success).length,
    totalProcessed: processResults.reduce((sum, r) => sum + (r.processedCount || 0), 0),
  }

  // Store aggregated results
  await db.batchResults.create({
    data: {
      batchId: context.batchId,
      stats,
      results: processResults,
      completedAt: new Date(),
    },
  })

  return Response.json({
    success: true,
    result: stats,
  })
}

Handling Partial Failures

Configure how the workflow should behave when some parallel steps fail. You can fail the entire workflow, continue with partial results, or retry only failed steps.

Job Dependencies

Define complex dependency graphs to control execution order. Steps wait for all their dependencies to complete before running.

import { platform } from '@/lib/platform'

// Complex dependency graph
//
//          ┌─── process-a ───┐
// prepare ─┼─── process-b ───┼─── aggregate ─── notify
//          └─── process-c ───┘
//                   │
//               validate ─────────────────────┘

const workflow = await platform.workflows.create({
  id: 'complex-pipeline',
  steps: [
    { id: 'prepare', url: '/api/jobs/prepare' },

    // Parallel processing
    { id: 'process-a', url: '/api/jobs/process-a', dependsOn: ['prepare'], parallel: true },
    { id: 'process-b', url: '/api/jobs/process-b', dependsOn: ['prepare'], parallel: true },
    { id: 'process-c', url: '/api/jobs/process-c', dependsOn: ['prepare'], parallel: true },

    // Validation runs after process-c specifically
    { id: 'validate', url: '/api/jobs/validate', dependsOn: ['process-c'] },

    // Aggregate waits for all processing
    { id: 'aggregate', url: '/api/jobs/aggregate', dependsOn: ['process-a', 'process-b', 'process-c'] },

    // Notify waits for both aggregate AND validate
    { id: 'notify', url: '/api/jobs/notify', dependsOn: ['aggregate', 'validate'] },
  ],
  context: { pipelineId: 'pipeline_123' },
})

Workflow State Management

Track and manage workflow state throughout execution.

Get workflow status

import { platform } from '@/lib/platform'

// Get workflow status
const status = await platform.workflows.getStatus('workflow_abc123')

console.log({
  id: status.id,
  state: status.state, // 'pending' | 'running' | 'completed' | 'failed'
  startedAt: status.startedAt,
  completedAt: status.completedAt,
  currentStep: status.currentStep,
  completedSteps: status.completedSteps,
  failedSteps: status.failedSteps,
  context: status.context,
  results: status.results,
})

// Get detailed step information
for (const step of status.steps) {
  console.log({
    stepId: step.id,
    state: step.state,
    startedAt: step.startedAt,
    completedAt: step.completedAt,
    attempts: step.attempts,
    result: step.result,
    error: step.error,
  })
}

Saga Patterns

Implement distributed transactions with compensating actions. If a step fails, the workflow can automatically roll back previous steps.

With compensation

import { platform } from '@/lib/platform'

// Order saga with compensating transactions
const workflow = await platform.workflows.create({
  id: 'order-saga',
  saga: true, // Enable saga mode
  steps: [
    {
      id: 'reserve-inventory',
      url: 'https://myapp.com/api/jobs/reserve-inventory',
      compensation: 'https://myapp.com/api/jobs/release-inventory',
    },
    {
      id: 'charge-payment',
      url: 'https://myapp.com/api/jobs/charge-payment',
      dependsOn: ['reserve-inventory'],
      compensation: 'https://myapp.com/api/jobs/refund-payment',
    },
    {
      id: 'create-shipment',
      url: 'https://myapp.com/api/jobs/create-shipment',
      dependsOn: ['charge-payment'],
      compensation: 'https://myapp.com/api/jobs/cancel-shipment',
    },
    {
      id: 'notify-customer',
      url: 'https://myapp.com/api/jobs/notify-customer',
      dependsOn: ['create-shipment'],
      // No compensation needed - notification is not reversible
    },
  ],
  context: {
    orderId: 'order_123',
    userId: 'user_456',
  },
})

// If create-shipment fails:
// 1. cancel-shipment is NOT called (step didn't complete)
// 2. refund-payment IS called (reverses charge-payment)
// 3. release-inventory IS called (reverses reserve-inventory)
// Compensations run in reverse order

Compensation Design

Design compensation handlers to be idempotent. They may be called multiple times if retries are configured. Always check if the action has already been reversed before performing the rollback.

Best Practices

Keep Steps Small

Each step should do one thing well. Smaller steps are easier to retry and debug.

Minimize Dependencies

Only add dependencies where truly needed. More parallelism means faster completion.

Design for Failure

Every step can fail. Use saga patterns for critical workflows that need rollback.

Idempotent Handlers

Steps may run multiple times due to retries. Design handlers to handle duplicate execution.

Workflow Visibility

Use the Sylphx console to visualize workflow execution, view step dependencies as a graph, and debug failed workflows with detailed logs for each step.

Job Components

Pre-built React components for job management

Background Jobs

Return to the jobs overview