OpenAI's Hosted Evals Goes Read-Only Oct 31. What Now?

What's actually shutting down, and when

Editorial correction (October 14, 2026): an earlier version conflated hosted Evals with all trace grading and described a local assertion as equivalent to a hosted grader. The shutdown dates apply to the Evals platform; the example below is one narrower self-owned check.

If your bot's test suite runs through OpenAI's hosted Evals dashboard or API, you're on a clock. OpenAI's deprecations page lists three dates for the Evals platform: the deprecation was announced June 3, 2026, existing evals become read-only on October 31, 2026, and the dashboard and API are scheduled to shut down on November 30, 2026. Today is October 14, 2026 — that gives you roughly two and a half weeks before the read-only cutover, and not quite seven before the full shutdown.

The same deprecation page confirms this is specifically the hosted product, not every form of OpenAI evaluation support. OpenAI's own guide for the platform says it is deprecating the Evals platform, that existing content stays available during a transition window, but that it becomes read-only on October 31, 2026 and closes November 30, 2026. Read-only means you can still look at past runs and results after October 31, but you can't create a new eval, edit a grader, or launch a new run through the dashboard or API after that date.

Sources: Deprecations, Evaluate external models.

The hosted dashboard is not the same as the GitHub framework

Worth separating clearly: the thing being retired is the web dashboard and the Evals REST API you configure at platform.openai.com. The openai/evals repository on GitHub is a different artifact — an open-source framework and benchmark registry that you install and run yourself, with no dependency on the hosted dashboard. Its own README describes it as a framework for evaluating LLMs and LLM systems, with a registry of evals and the ability to write custom ones, run locally against your own data.

That distinction matters for the decision you're about to make. If your tests currently live only as eval objects and run configs inside the dashboard, you have real migration work before October 31. If you've already been running the open-source framework locally — pulling results into your own CI rather than the hosted UI — this deprecation doesn't touch you directly, though you should still confirm your pipeline never depended on dashboard-only features like hosted report URLs.

Sources: evals/README.md at main · openai/evals.

What to pull out before October 31

Treat October 31 as the real deadline, not November 30 — a read-only dashboard can still let you copy things out, but it won't let you re-run a grader against new data to confirm your export is complete. OpenAI's working-with-evals guide documents the eval object as a dataset schema plus one or more testing criteria, and the eval run as the execution of that definition against a data source, producing graded results you can read back through the API. Pull all three pieces: the dataset file you uploaded, the grader/testing-criteria configuration attached to each eval, and the run history with scores, not just the headline pass rate.

If a grader or trace evaluation was configured inside the retiring Evals product, save its criteria and representative graded runs alongside the dataset. OpenAI's trace-grading guide also describes grading traces from SDK-based agents in the dashboard; the deprecation notice does not say that all SDK tracing or every trace-grading feature shuts down with the Evals API. Inventory which product owns each grader before planning its migration. The guide describes run filters such as model, date range, and tool calls, but those filters are not themselves a specification of correct tool arguments. Record your actual grader criteria separately.

Sources: Working with evals, Trace grading.

A worked example: one local agent check

Suppose a user asks for an order status. In your own test fixture, the expected behavior is one call to get_order with the right user_id, no destructive tool call, and a final answer containing the returned status. Capture the run's tool calls, tool result, and final answer. A local, version-controlled check can then compare that evidence with the fixture. This is an illustrative new test, not a recreation of an unspecified hosted grader.

The check can assert the tool name and user_id, reject destructive calls, enforce a step budget, and compare the final answer with the fixture's expected status. If a test requires a particular call order, assert that order too. Run those assertions over representative captured traces in CI. This catches specific regressions you define; it does not reproduce every score, model-based judgment, dashboard view, or report from OpenAI's hosted Evals product.

The tradeoff is explicit: you must maintain the fixtures and checks yourself, and a simple assertion cannot replace a grader that judged open-ended answers. The benefit is that the behavior you actually assert is visible in your repository and can run without the retiring Evals dashboard.

The decision to make this week

If your tests depend on the hosted Evals dashboard or API, inventory their definitions, data, grader criteria, and run history before October 31, 2026. Confirm the available export or API readback for your own account while the service is still writable; the official deprecation notice promises a read-only transition, not a one-click export of every artifact. If you use the separate open-source openai/evals framework or your own harness, check whether any part of your pipeline still relies on hosted Evals objects or report URLs.

For each dependency you find, choose a replacement you can verify. OpenAI's deprecation page points to a Promptfoo migration guide; a small local assertion such as the example above is another option for narrow deterministic checks. Preserve the original test cases and compare results before retiring the old path.

Sources: Deprecations.