Streaming

Overview

For Chat Completions, set stream: true to receive Server-Sent Events. Messages and Responses use different event formats described in their guides. Each data: event carries a JSON chunk, and a normal stream ends with data: [DONE]. Network reads do not necessarily align with complete events; use an SDK or an SSE parser.

import os
import sys
from openai import OpenAI

client = OpenAI(
    base_url="https://api.netraruntime.com/v1",
    api_key=os.environ["NETRA_API_KEY"],
    max_retries=0,
)
with client.chat.completions.create(
    model=os.environ["NETRA_MODEL"],
    messages=[{"role": "user", "content": "Explain tracing briefly."}],
    max_tokens=256,
    stream=True,
    stream_options={"include_usage": True},
) as stream:
    reasoning_details = []
    for chunk in stream:
        # Usage may arrive with empty choices; inspect it separately.
        if chunk.usage is not None:
            details = getattr(chunk.usage, "completion_tokens_details", None)
            count = getattr(details, "reasoning_tokens", None)
            if count is not None:
                print(f"Reasoning tokens: {count}", file=sys.stderr)
        if chunk.choices:
            delta = chunk.choices[0].delta
            details = getattr(delta, "reasoning_details", None)
            if details is not None:
                reasoning_details.extend(details)
            if details:
                reasoning = "".join(
                    d.get("text", "") if d.get("type") == "reasoning.text"
                    else d.get("summary", "") if d.get("type") == "reasoning.summary"
                    else "" for d in details
                )
            else:
                reasoning = getattr(delta, "reasoning", None) or getattr(delta, "reasoning_content", None)
            if reasoning:
                print(reasoning, end="", flush=True, file=sys.stderr)
            text = delta.content
            if text:
                print(text, end="", flush=True)
print()

Chunks can contain role metadata, empty choices, reasoning deltas, tool-call fragments, usage, or finish information rather than display text. Accumulate content, reasoning, and tool arguments separately. choices[0].delta.reasoning can arrive while delta.content is null. The legacy reasoning_content alias can appear in the same chunk. Render one channel: nonempty reasoning_details first, otherwise reasoning, otherwise reasoning_content. Do not concatenate these aliases. Retain every detail object in receive order for continuation, including empty arrays when supplied. Display text/summary only; encrypted data and signatures are opaque. A provider may expose final usage when supported, but it is not guaranteed on every chunk or stream.

With stream_options: {"include_usage": true}, a supported provider can report final usage in an event such as this illustrative example:

{
  "choices": [],
  "usage": {
    "prompt_tokens": 30,
    "completion_tokens": 12,
    "completion_tokens_details": { "reasoning_tokens": 8 },
    "total_tokens": 42
  }
}

Read usage.completion_tokens_details.reasoning_tokens independently of choices and display deltas. It is a subset of completion tokens, including when reasoning output is excluded; do not add it to the completion total or infer it from text length. Nested counts take precedence over the compatible flat usage.reasoning_tokens alias. Missing provider counts stay absent, and an interrupted stream may never deliver final usage. Do not sum repeated cumulative usage snapshots as if they were per-chunk token counts.

Once streaming headers have been sent, a later failure cannot change the original HTTP status. A balance exhaustion can appear as a terminal error event followed by [DONE]. Treat an error event or unexpected connection closure as an incomplete answer even when the initial HTTP status was 200. Preserve already received output and record the request ID for investigation.

Cancelling or disconnecting does not guarantee zero cost; output already generated or delivered can still be settled. A retry can start another billable generation. For command-line inspection, add -N to cURL to disable its output buffering. For hosted clients, ensure your proxy permits streaming and has a timeout appropriate for generation.