Working with Streaming LLMs

DialogueDB stores complete messages — it doesn't proxy or generate LLM responses. Stream from your LLM provider as usual, then call saveMessage() with the full response after the stream finishes.

Streaming Chat with OpenAI + DialogueDB

A complete Next.js route handler showing the pattern:

import { DialogueDB } from 'dialogue-db';
import OpenAI from 'openai';

const db = new DialogueDB({ apiKey: process.env.DIALOGUE_DB_API_KEY });
const openai = new OpenAI();

export async function POST(req: Request) {
  const { messages, dialogueId } = await req.json();
  const dialogue = await db.getDialogue(dialogueId);

  // 1. Save user message immediately
  await dialogue.saveMessage({ role: 'user', content: messages.at(-1).content });

  // 2. Stream from YOUR LLM (OpenAI, Anthropic, etc.)
  const stream = await openai.chat.completions.create({
    model: 'gpt-4',
    messages,
    stream: true,
  });

  // 3. Collect chunks while streaming to client
  let fullResponse = '';
  const encoder = new TextEncoder();
  const readable = new ReadableStream({
    async start(controller) {
      for await (const chunk of stream) {
        const text = chunk.choices[0]?.delta?.content || '';
        fullResponse += text;
        controller.enqueue(encoder.encode(`data: ${text}\n\n`));
      }
      // 4. Save complete response to DialogueDB AFTER stream finishes
      await dialogue.saveMessage({ role: 'assistant', content: fullResponse });
      controller.close();
    }
  });

  return new Response(readable, { headers: { 'Content-Type': 'text/event-stream' } });
}

Key Points

• Save the user message before streaming — don't lose it if the stream fails
• Save the assistant message after the stream completes — you need the full content
• Load messages from DialogueDB for LLM context — this is your conversation history
• DialogueDB handles persistence, your LLM provider handles generation — pair it with any streaming provider (OpenAI, Anthropic, Cohere, etc.)