OpenAI recently released a significant new feature in response to user demand: structured output for their API. This functionality aims to address issues with large language models (LLMs) generating JSON content, such as hallucinations or incomplete parsing.
The new structured output feature ensures that model-generated output matches a specified JSON schema. This is crucial for maintaining consistency across applications and has been a top requested feature from developers. OpenAI CEO Sam Altman noted that this release "meets an urgent need for many users."
Key points about the structured output feature:
- Available for GPT-4o-mini, GPT-4o, and their fine-tuned versions
- Compatible with Chat Completions API, Assistant API, Batch API, and visual inputs
- Allows developers to constrain OpenAI models to match data schemas
- Improves model understanding of complex data schemas
- Ensures adherence to data schemas, preventing missing keys or invalid enum values
- Supports step-by-step answer generation
- Simplifies prompts and provides clear rejection statements
- Follows existing OpenAI safety policies
OpenAI claims their latest GPT-4o model achieved a "100% perfect" evaluation score for structured output.
The company acknowledges drawing inspiration from open-source projects like outlines, jsonformer, instructor, guidance, and lark libraries.
While this feature addresses a significant need, some developers note potential limitations:
- Slow generation of the first token due to schema compilation
- Limited acceptance of JSON schemas in the API
- Incomplete implementation of advertised changes in the Python SDK
Some developers suggest that combinations like Instructor + Pydantic may still be the simplest method for achieving structured output with OpenAI and other LLM solutions.
Overall, the introduction of native structured output support in OpenAI's API represents a significant step forward, though it may require further refinement before widespread adoption in production environments.