Migrating from BullMQ to flashQ: Complete Guide

If you're running BullMQ in production and considering a switch to flashQ, you're in the right place. This guide will walk you through the entire migration process, from updating your dependencies to handling edge cases. The good news? flashQ's API is designed to be BullMQ-compatible, making the migration straightforward.

Why Migrate from BullMQ?

Before diving into the how, let's briefly cover the why:

Aspect	BullMQ + Redis	flashQ
Infrastructure	Requires Redis server	Single binary, no dependencies
Performance	~50K jobs/sec	1.9M jobs/sec
Latency	5-10ms (network hop)	<1ms
Payload size	~512KB practical	10MB
Memory cost	Redis RAM pricing	Included in server

Migration Checklist

Here's what we'll cover:

1. Setting up the flashQ server
2. Updating dependencies
3. Updating import statements
4. Migrating Queue instances
5. Migrating Worker instances
6. Handling API differences
7. Testing the migration
8. Production cutover strategy

Step 1: Set Up flashQ Server

First, you'll need the flashQ server running. Unlike Redis, flashQ is a single binary:

# Option 1: Docker (recommended)
docker run -d --name flashq -p 6789:6789 flashq/flashq

# Option 2: Download binary
curl -L https://github.com/egeominotti/flashq/releases/latest/download/flashq-linux -o flashq
chmod +x flashq
./flashq

# Option 3: With persistence (PostgreSQL)
docker run -d --name flashq \
  -p 6789:6789 \
  -e DATABASE_URL=postgres://user:pass@host/db \
  flashq/flashq

Verify it's running:

curl http://localhost:6790/health
# {"status":"healthy","version":"0.1.5"}

Step 2: Update Dependencies

Replace BullMQ with flashQ in your package.json:

# Remove BullMQ
npm uninstall bullmq

# Install flashQ
npm install flashq

If you were using ioredis directly for Redis connections, you can remove that too:

npm uninstall ioredis

Step 3: Update Import Statements

This is often the only code change needed. Find and replace your imports:

// Before (BullMQ)
import { Queue, Worker, QueueEvents } from 'bullmq';

// After (flashQ)
import { Queue, Worker } from 'flashq';

For TypeScript projects, update your types too:

// Before
import type { Job, JobsOptions } from 'bullmq';

// After
import type { Job, JobOptions } from 'flashq';

Step 4: Migrate Queue Instances

Queue initialization is almost identical, but connection options differ:

// Before (BullMQ)
import { Queue } from 'bullmq';

const queue = new Queue('my-queue', {
  connection: {
    host: 'localhost',
    port: 6379,
  }
});

// After (flashQ)
import { Queue } from 'flashq';

const queue = new Queue('my-queue', {
  connection: {
    host: 'localhost',
    port: 6789,  // flashQ default port
  }
});

Adding Jobs

The add() method is compatible:

// Works the same in both!
await queue.add('process-order', {
  orderId: '12345',
  items: [...]
}, {
  priority: 10,
  delay: 5000,
  attempts: 3,
  backoff: {
    type: 'exponential',
    delay: 1000
  }
});

Bulk Operations

// BullMQ
await queue.addBulk([
  { name: 'job1', data: {...} },
  { name: 'job2', data: {...} }
]);

// flashQ - same API!
await queue.addBulk([
  { name: 'job1', data: {...} },
  { name: 'job2', data: {...} }
]);

Step 5: Migrate Worker Instances

Workers are also compatible:

// Before (BullMQ)
import { Worker } from 'bullmq';

const worker = new Worker('my-queue', async (job) => {
  console.log(`Processing ${job.id}`);
  await job.updateProgress(50);
  return { success: true };
}, {
  connection: { host: 'localhost', port: 6379 },
  concurrency: 5
});

// After (flashQ)
import { Worker } from 'flashq';

const worker = new Worker('my-queue', async (job) => {
  console.log(`Processing ${job.id}`);
  await job.updateProgress(50);
  return { success: true };
}, {
  connection: { host: 'localhost', port: 6789 },
  concurrency: 5
});

Worker Events

Event handling works the same way:

worker.on('completed', (job, result) => {
  console.log(`Job ${job.id} completed with result:`, result);
});

worker.on('failed', (job, error) => {
  console.error(`Job ${job.id} failed:`, error.message);
});

worker.on('progress', (job, progress) => {
  console.log(`Job ${job.id} progress: ${progress}%`);
});

Step 6: Handle API Differences

While most APIs are compatible, there are some differences to be aware of:

QueueEvents (Not Needed)

BullMQ uses a separate QueueEvents class for global events. In flashQ, events are built into the Queue class:

// BullMQ - separate QueueEvents instance
const queueEvents = new QueueEvents('my-queue', { connection });
queueEvents.on('completed', ({ jobId, returnvalue }) => {...});

// flashQ - events on Queue instance
queue.on('completed', (job, result) => {...});

Flow Producer

flashQ has a different syntax for job dependencies:

// BullMQ FlowProducer
const flow = new FlowProducer({ connection });
await flow.add({
  name: 'parent',
  queueName: 'my-queue',
  data: {...},
  children: [
    { name: 'child1', queueName: 'my-queue', data: {...} }
  ]
});

// flashQ - use depends_on option
const child1 = await queue.add('child1', {...});
const parent = await queue.add('parent', {...}, {
  depends_on: [child1.id]
});

Repeatable Jobs

flashQ uses a dedicated cron API:

// BullMQ - repeat option
await queue.add('cleanup', {}, {
  repeat: { cron: '0 0 * * *' }
});

// flashQ - cron API
await queue.addCron('daily-cleanup', {
  queue: 'my-queue',
  schedule: '0 0 0 * * *',  // 6-field cron (includes seconds)
  data: {}
});

Rate Limiting

Rate limiting is simpler in flashQ:

// BullMQ - limiter option on worker
const worker = new Worker('my-queue', processor, {
  limiter: { max: 100, duration: 60000 }
});

// flashQ - queue-level rate limit
await queue.setRateLimit(100);  // 100 jobs per minute

Step 7: Test the Migration

Before going to production, test thoroughly:

// test/queue.test.ts
import { Queue, Worker } from 'flashq';

describe('Queue Migration', () => {
  let queue: Queue;
  let worker: Worker;

  beforeAll(async () => {
    queue = new Queue('test-queue');
    await queue.connect();
  });

  afterAll(async () => {
    await worker?.close();
    await queue.close();
  });

  it('should process jobs', async () => {
    const results: any[] = [];

    worker = new Worker('test-queue', async (job) => {
      results.push(job.data);
      return { processed: true };
    });

    await queue.add('test', { value: 1 });
    await queue.add('test', { value: 2 });

    // Wait for processing
    await new Promise(r => setTimeout(r, 1000));

    expect(results).toHaveLength(2);
  });

  it('should handle retries', async () => {
    let attempts = 0;

    worker = new Worker('test-queue', async (job) => {
      attempts++;
      if (attempts < 3) throw new Error('Retry me');
      return { success: true };
    });

    await queue.add('retry-test', {}, { attempts: 3 });

    await new Promise(r => setTimeout(r, 3000));

    expect(attempts).toBe(3);
  });
});

Step 8: Production Cutover

For a safe production migration, follow these steps:

Strategy 1: Blue-Green Deployment

// 1. Deploy flashQ server alongside existing Redis
// 2. Update application to use flashQ
// 3. Let existing BullMQ jobs drain
// 4. Switch traffic to new deployment
// 5. Decommission Redis when empty

Strategy 2: Gradual Migration

Migrate queue by queue:

// config/queues.ts
const USE_FLASHQ = {
  'email-queue': true,      // Migrated
  'payment-queue': false,   // Still on BullMQ
  'analytics-queue': true,  // Migrated
};

export function createQueue(name: string) {
  if (USE_FLASHQ[name]) {
    return new FlashQueue(name, { connection: flashQConfig });
  }
  return new BullMQQueue(name, { connection: redisConfig });
}

Monitoring the Migration

// Monitor both systems during migration
const flashQStats = await flashQueue.getJobCounts();
const bullMQStats = await bullQueue.getJobCounts();

console.log('flashQ:', flashQStats);
// { waiting: 45, active: 5, completed: 1230, failed: 2 }

console.log('BullMQ:', bullMQStats);
// { waiting: 0, active: 0, completed: 5000, failed: 10 }

Common Issues and Solutions

Connection Errors

// Ensure flashQ server is running
const queue = new Queue('my-queue', {
  connection: {
    host: process.env.FLASHQ_HOST || 'localhost',
    port: parseInt(process.env.FLASHQ_PORT || '6789'),
  }
});

// Add connection error handling
queue.on('error', (err) => {
  console.error('Queue connection error:', err);
});

TypeScript Errors

If you encounter type errors, ensure you're using the correct imports:

// Correct type imports
import type { Job, JobOptions, QueueOptions } from 'flashq';

Conclusion

Migrating from BullMQ to flashQ is straightforward thanks to the compatible API. The key steps are:

1. Start the flashQ server
2. Replace the npm package
3. Update import statements
4. Change connection port from 6379 to 6789
5. Test thoroughly
6. Deploy with a safe cutover strategy

After migration, you'll benefit from:

• No Redis: One less service to manage
• 10x Performance: 1.9M jobs/sec vs 50K
• Lower Latency: Sub-millisecond response times
• Larger Payloads: 10MB vs practical 512KB limit
• Cost Savings: No Redis infrastructure costs