A/B Testing | Sylphx Documentation

Multi-Variant

Test control vs multiple variants

Bayesian Stats

Probability-based analysis

Goal Tracking

Track conversions and metrics

Winner Detection

Know when to conclude

Creating Experiments

Create experiments via the console or API to compare different versions of a feature. Define your hypothesis, variants, and success metrics.

Console UI

// Experiments are created via the Sylphx Console:
// 1. Navigate to your app → Feature Flags → Experiments
// 2. Click "Create Experiment"
// 3. Define:
//    - Key: unique identifier (e.g., "checkout_redesign")
//    - Variants: control + treatment(s) with weights
//    - Primary metric: event to track for success
//    - Statistical settings: confidence level, sample size

// Or create via tRPC API:
const experiment = await trpc.experiments.create({
  appId: "your-app-id",
  key: "checkout_redesign",
  name: "Checkout Page Redesign",
  hypothesis: "Simplified checkout will increase conversion by 10%",

  // Define variants with weights (must sum to 100)
  variants: [
    { key: "control", name: "Current Checkout", weight: 50 },
    { key: "treatment", name: "New Checkout", weight: 50 },
  ],

  // Primary metric to track
  primaryMetric: {
    event: "purchase_completed",
    type: "conversion",
  },

  // Statistical settings
  confidenceLevel: 95,
  minimumSampleSize: 1000,
  minimumDurationDays: 7,
})

Define Clear Hypotheses

A good hypothesis is specific and measurable: "Changing X will improve Y by Z%". This helps you know what success looks like before you start.

Getting User Variants

When a user encounters your experiment, get their assigned variant. The assignment is deterministic (same user always gets same variant) and exposure is automatically tracked.

import { sylphx } from '@sylphx/sdk'

// Get the user's assigned variant
const { variant, inExperiment, payload } = await sylphx.experiments.getVariant({
  experimentKey: 'checkout_redesign',
  userId: user.id,
  // Optional context for targeting
  context: {
    plan: user.plan,
    country: user.country,
  },
})

// Render based on variant
if (variant === 'treatment') {
  return <NewCheckout />
} else {
  return <CurrentCheckout />
}

Variant Assignment Rules

Users are randomly assigned once and stay in their variant
Assignment is deterministic based on user ID hash (MurmurHash3)
Exposures are automatically tracked and deduplicated
Returns "control" if experiment is not running

Multi-Variant Testing

Test more than two variants simultaneously. Useful for testing multiple approaches or fine-tuning values.

Multi-Variant Experiment

// Create experiment with multiple variants
const experiment = await trpc.experiments.create({
  appId: "your-app-id",
  key: "pricing_test",
  name: "Pricing Page Variants",

  variants: [
    { key: "control", name: "Current Pricing", weight: 25 },
    { key: "simple", name: "Simplified (2 tiers)", weight: 25 },
    { key: "detailed", name: "Detailed (5 tiers)", weight: 25 },
    { key: "value", name: "Value-focused Copy", weight: 25 },
  ],

  primaryMetric: {
    event: "subscription_started",
    type: "conversion",
  },
})

// In your component
function PricingPage() {
  const { variant } = useExperiment('pricing_test')

  switch (variant) {
    case 'simple':
      return <SimplePricing />
    case 'detailed':
      return <DetailedPricing />
    case 'value':
      return <ValuePricing />
    default:
      return <CurrentPricing />
  }
}

Sample Size Consideration

Multi-variant tests require more traffic to reach statistical significance. With 4 variants, you need roughly 4x the sample size of a simple A/B test. Ensure you have enough traffic before running multi-variant experiments.

Tracking Conversions

Track conversion events to measure experiment success. Conversions are automatically linked to the user's assigned variant.

import { sylphx } from '@sylphx/sdk'

// Track conversion when user completes the goal
await sylphx.experiments.trackConversion({
  experimentKey: 'checkout_redesign',
  event: 'purchase_completed',
  userId: user.id,

  // Optional properties for analysis
  properties: {
    total: order.total,
    items: order.items.length,
    paymentMethod: 'card',
  },
})

Conversion Attribution

Conversions are only attributed to users who have been exposed to the experiment. If a user converts without seeing the experiment, the conversion is not counted.

Statistical Analysis

Sylphx uses Bayesian statistics to calculate the probability that each variant is the best. This approach gives you faster, more intuitive results.

95%

Confidence Level

Probability the result is real, not chance

Minimum Effect

Smallest improvement worth detecting

1000+

Sample Size

Users per variant for reliable results

7+ days

Test Duration

Account for weekly patterns

Get Results

// Get experiment results with statistical analysis
const results = await trpc.experiments.getResults({
  experimentId: "experiment-uuid",
})

// Results structure
{
  experiment: {
    id: "...",
    key: "checkout_redesign",
    status: "running",
    startedAt: "2024-01-15T...",
  },

  primaryMetric: {
    event: "purchase_completed",
    type: "conversion",
  },

  variants: [
    {
      key: "control",
      name: "Current Checkout",
      exposures: 5234,
      conversions: 523,
      conversionRate: 0.0999, // 9.99%
      confidenceInterval: { lower: 0.092, upper: 0.108 },
      relativeLift: 0, // Baseline
      probabilityToBeBest: 0.033, // 3.3%
    },
    {
      key: "treatment",
      name: "New Checkout",
      exposures: 5198,
      conversions: 572,
      conversionRate: 0.1100, // 11.00%
      confidenceInterval: { lower: 0.102, upper: 0.119 },
      relativeLift: 0.101, // +10.1% vs control
      probabilityToBeBest: 0.967, // 96.7%
    },
  ],

  analysis: {
    totalExposures: 10432,
    isSignificant: true,
    confidence: 0.967,
    winner: "treatment",
    canConclude: true,
    blockers: [], // Empty when can conclude
  },
}

Bayesian Interpretation

The "probability to be best" directly tells you the chance each variant is the winner. A 96.7% probability means treatment has a 96.7% chance of being truly better than control — much more intuitive than p-values!

When to Conclude

Knowing when to stop is as important as knowing when to start. The results API tells you when your experiment is ready to conclude.

Ready to Conclude

Reached minimum sample size per variant
Running for at least minimum duration
Confidence level reached
analysis.canConclude === true

Wait Before Concluding

analysis.blockers lists what's needed
Sample size not yet reached
Less than minimum days of data
Confidence below threshold

Conclude Experiment

// When ready, conclude the experiment
await trpc.experiments.conclude({
  experimentId: "experiment-uuid",
  winningVariant: "treatment", // Optional: declare winner
})

// The experiment is marked as concluded
// No more users will be assigned variants
// Results are preserved for reference

Experiment Lifecycle

Experiments go through defined states: Draft → Running → (Paused) → Concluded.

Lifecycle Management

// Create experiment (starts as "draft")
const experiment = await trpc.experiments.create({ ... })
// experiment.status === "draft"

// Start the experiment
await trpc.experiments.start({ experimentId: experiment.id })
// experiment.status === "running"

// Optionally pause if issues arise
await trpc.experiments.pause({ experimentId: experiment.id })
// experiment.status === "paused"

// Resume the experiment
await trpc.experiments.start({ experimentId: experiment.id })
// experiment.status === "running"

// Conclude with a winner
await trpc.experiments.conclude({
  experimentId: experiment.id,
  winningVariant: "treatment",
})
// experiment.status === "concluded"

Best Practices

One Change at a Time

Test single variables to understand what drives results

Adequate Sample Size

Wait for statistical power before drawing conclusions

Full Business Cycles

Run tests through weekends and different periods

Document Everything

Record hypothesis, results, and learnings for future reference

Trust the Stats

Don't peek and stop early — let the numbers guide you

Segment Analysis

Check if results differ by user segment (mobile, plan, etc.)

API Reference

Method	Description
experiments.create(config)	Create a new experiment
experiments.list(appId)	List experiments for an app
experiments.get(id)	Get experiment details
experiments.start(id)	Start running an experiment
experiments.pause(id)	Pause an experiment
experiments.conclude(id)	Mark experiment as concluded
experiments.getVariant(key, ctx)	Get user's variant assignment
experiments.trackConversion()	Track a conversion event
experiments.getResults(id)	Get results with statistics

Local Evaluation

Edge caching and offline mode for low-latency flag evaluation

Analytics

Deep dive into event tracking and user analytics