# Codex CLI 0.159.2: API key vs ChatGPT OAuth reasoning effort

**Question.** Does the same client, sending the same request, get the same amount of
reasoning from OpenAI models when it authenticates with an **API key** as when it uses
a **ChatGPT (Codex) OAuth login**?

**Answer (so far).** No, not for `gpt-6-luna`. With the official Codex CLI as the
client, the OAuth route produced about **0.39×** the reasoning tokens of the API route
at medium effort (n=50 per arm), and was slower. `gpt-6.1-sol` showed no difference at
medium effort and a smaller gap (about 0.78×) at high effort (n=8).

![Luna medium reasoning tokens](results/luna-medium-n50/plots/reasoning_tokens.png)

## Update, 9 October 2026: reproduced on Codex CLI 0.161.0 and 0.162.0

I reran the luna-medium-n50 experiment with the same scripts and prompt on two newer
CLI versions. **The gap is unchanged at 0.38×. What's new is that the OAuth route
increasingly skips reasoning altogether, and those runs are often wrong.**

![Luna medium reasoning tokens by CLI version](results/comparison/luna-medium-reasoning-by-version.png)

| Codex CLI (date) | API median reasoning | OAuth median reasoning | OAuth/API (95% CI) | OAuth runs with 0 reasoning | wrong answers API / OAuth | median wall s API / OAuth | output tok per wall s API / OAuth |
|---|---|---|---|---|---|---|---|
| [0.159.2](results/luna-medium-n50/report.md) (2 Oct) | 2,674 | 1,034 | 0.39 (0.35–0.41) | 1 / 50 | 0 / 1 | 29.3 / 35.9 | 111 / 42 |
| [0.161.0](results/luna-medium-n50-codex0.161.0/report.md) (9 Oct) | 2,577 | 982 | **0.38** (0.34–0.41) | **7 / 50** | 0 / **3** | 27.2 / 17.1 | 111 / 83 |
| [0.162.0](results/luna-medium-n50-codex0.162.0/report.md) (9 Oct) | 2,588 | 982 | **0.38** (0.24–0.42) | **17 / 50** | 0 / **10** | 29.4 / 17.1 | 107 / 76 |

All runs completed. Mann-Whitney p < 1e-16 for reasoning tokens on every version.

- **The reasoning gap hasn't changed.** On both routes, the runs that do reason match
  2 October: the API route centres around 2,600 tokens and the OAuth route around 1,000.
- **The OAuth route now has a second cluster at zero reasoning.** It went from 1 run to
  7 and then 17 out of 50. The API route had **no** zero-reasoning runs in 150 tries.
  **Every wrong answer, across all three versions, was a zero-reasoning OAuth run**
  (14 of 25). Those answers listed only some of the pairs or included wrong pairs.
- **The OAuth route is now faster per run,** about 17 s against 28 s for the API route.
  Its throughput went from 42 to about 80 output tokens per wall second, while the API
  route stayed at about 110. The API route is still faster per token. The OAuth route
  finishes sooner only because it writes about half as many tokens. That suggests the
  queueing on 2 October has mostly gone, while the reasoning cut has stayed the same.
- **Caveat: version and time are confounded.** 0.161.0 ran from 19:09 to 19:18 UTC and
  0.162.0 from 19:19 to 19:28 UTC, one after the other. The zero-reasoning runs on
  0.162.0 cluster in the first half of that run. The rise from 7 to 17 may come from a
  server-side change over time rather than the CLI version. Alternating versions within
  one run would separate the two.
- Input sizes differ between versions because the CLI instructions changed. The API
  route grew from 12,128 tokens on 0.161.0 to 12,901 on 0.162.0. The OAuth route stayed
  at about 13,890. One OAuth run on each version ran a shell command and took two turns.
- `luna-medium-oauth-only-n4-codex0.16x/` contains an earlier OAuth-only smoke test
  (4 runs per version, 2 workers, both versions at the same time). Each version had one
  zero-reasoning run, and all answers were correct.

## Method

- Client: Codex CLI `0.159.2`, `codex exec --json --ephemeral --ignore-user-config
  --ignore-rules --skip-git-repo-check -s read-only -m <model> -c
  'model_reasoning_effort="<effort>"'`. The prompt is read from stdin, and the CLI runs
  in an empty working directory.
- The only difference between arms is authentication. Each arm runs in its own
  throwaway `CODEX_HOME`:
  - **api:** `codex login --with-api-key`. Standard tier, with no service-tier override.
  - **oauth:** a copy of a ChatGPT-mode `~/.codex/auth.json`.
  - Provider environment variables are removed from `codex exec`, so neither arm can
    fall back to the other's credentials.
- Prompt: a single number-theory proof task with a known answer: all (x, y) with
  1 ≤ x ≤ y ≤ 100 where xy divides x² + y² + 1. The answer is
  `(1,1),(1,2),(2,5),(5,13),(13,34),(34,89)`. The prompt is in each results folder as
  `prompt.txt` (sha256 `4adea627…`).
- Runs alternate between the arms (`api,oauth`, then `oauth,api`, and so on). The n=50
  run used 4 parallel workers, shared by both arms. All runs took place on 2 October
  2026 between 08:03 and 08:47 UTC.
- Metrics come from the CLI's `turn.completed` usage events: input, cached input,
  output and `reasoning_output_tokens`. Output includes reasoning. Wall-clock time
  includes CLI start-up.

## Results

| run | model / effort | n per arm | API reasoning (median) | OAuth reasoning (median) | OAuth/API (95% CI) | Mann-Whitney p | correct API / OAuth | median wall s API / OAuth |
|---|---|---|---|---|---|---|---|---|
| [luna-medium-n50](results/luna-medium-n50/report.md) | gpt-6-luna medium | 50 | 2,674 | 1,034 | **0.39** (0.35–0.41) | 7e-18 | 50 / 49 | 29.3 / 35.9 |
| [luna-medium-n4](results/luna-medium-n4/report.md) | gpt-6-luna medium | 4 | 2,287 | 1,325 | 0.58 (0.39–0.85) | 0.06 | 4 / 4 | 30.4 / 46.3 |
| [luna-high-n8](results/luna-high-n8/report.md) | gpt-6-luna high | 8 | 3,445 | 2,053 | 0.60 (0.42–0.71) | 0.0009 | 8 / 8 | 39.7 / 52.7 |
| [sol61-medium-n4](results/sol61-medium-n4/report.md) | gpt-6.1-sol medium | 4 | 423 | 427 | 1.01 (0.82–1.24) | 1.0 | 4 / 4 | 23.5 / 50.7 |
| [sol61-high-n8](results/sol61-high-n8/report.md) | gpt-6.1-sol high | 8 | 957 | 745 | 0.78 (0.66–0.87) | 0.004 | 8 / 8 | 37.1 / 63.7 |

- Confidence intervals are bootstrap intervals for the ratio of medians.
- The OAuth arm consistently sends about 15% more input tokens (11.8k against 13.6k
  for Luna). The CLI adds different instructions when logged in with ChatGPT.
- The OAuth route is slower even though it produces fewer tokens. For Luna at n=50,
  the median throughput was 111 output tokens per wall-clock second on the API arm
  and 42 on the OAuth arm.
- The one wrong answer (luna-medium-n50, OAuth, round 39) used **0** reasoning tokens.

## Caveats

- Only one prompt was used, and the small runs (n=4 and n=8) are indicative only.
  The n=50 Luna run is the robust result.
- The OAuth arm reflects one ChatGPT account and plan, observed at one point in time.
  Server-side policy can change.
- The comparison is API standard tier against the ChatGPT/Codex backend. The
  difference is attributed to the server side because the client, prompt, model and
  requested effort are identical. This matches an earlier independent observation with
  a different client (fast-agent): about 0.40× reasoning on the Codex route for the
  same prompt.

## Layout

```
scripts/
  codex_route_ab.py          runner (stdlib only; needs `codex` on PATH)
  codex_route_ab_report.py   report.md: medians, p10/p90, ratios, bootstrap CIs, p-values
  codex_route_ab_plot.py     plots/reasoning_tokens.png, plots/wall_seconds.png (uv run; matplotlib)
  codex_route_ab_compare_plot.py  one row per CLI version (uv run; LABEL=RESULTS_DIR ...)
results/comparison/          cross-version charts
results/<model>-<effort>-n<runs per arm>[-codex<version>]/   (no suffix = 0.159.2)
  report.md       summary tables and plots
  summary.json    run metadata (CLI version, model, effort, prompt sha256, auth method) and medians
  results.jsonl   one row per generation: route, round, exit, seconds, usage, answer, correct
  prompt.txt      exact prompt
  plots/          reasoning-token and wall-time distributions (kernel density, median, one tick per run)
  raw/            per-generation `codex exec --json` event logs and final answers
```

## Reproduce

```bash
export OPENAI_API_KEY=...                         # API arm
python3 scripts/codex_route_ab.py --model gpt-6-luna --effort medium --rounds 50 --parallel 4 \
  --oauth-codex-auth ~/.codex/auth.json           # OAuth arm (copied; refuses logins >7 days old)
python3 scripts/codex_route_ab_report.py codex-route-ab-<timestamp>
uv run scripts/codex_route_ab_plot.py codex-route-ab-<timestamp>
```

Add `--login-only` to check both logins without calling a model. Use `--prompt-file` and
`--expected` to run a different task. Credentials reach the CLI only through stdin and
are deleted with the temporary homes when the run ends.