Text-to-SQL Fails Quietly: What a Semantic Layer Catches Before the Query Runs

A text-to-SQL query that fails loudly is the easy case. The syntax is wrong, the database returns an error, and someone fixes it. The harder case is a query that runs, returns a number, and gives no sign that the number is wrong.

The independent auto-ontology-mcp project describes this directly: a query can execute successfully while producing a misleading result. [S2] NVIDIA makes the same point from the business side: a faster revenue calculation has no value if the number is wrong. [S1]

Semantic layers address different parts of that problem. NVIDIA Auto Ontology and auto-ontology-mcp describe checking generated queries against reviewed definitions and stored values. [S1][S2] The dbt Semantic Layer takes a narrower governance approach: it centralizes metric definitions so downstream tools and AI clients use the same business logic. [S4]

The query ran, and the number is wrong

The auto-ontology-mcp README illustrates three broad kinds of quiet failure that can occur when a model sees only table and column names. [S2]

Meaning errors. The model may choose billing location when the question means delivery location, or attempt to calculate return on ad spend when the database contains revenue but no advertising-cost data.
Value errors. A filter can miss because the stored capitalization differs, or because a percentage is represented as 0.2 rather than 20.
Relationship errors. Coincidentally similar integer IDs can produce an invalid join, while an otherwise valid one-to-many join can inflate an aggregate by counting the same value repeatedly.

Each mistake can produce a plausible result rather than a database error. A careful analyst catches it by knowing what the data means; a model working from names and types alone may not.

Why it is hard: values, not just schemas

The BIRD benchmark, published at NeurIPS 2023, was created to narrow the gap between academic text-to-SQL benchmarks and real-world databases. Its abstract says earlier benchmarks such as Spider and WikiSQL focused on schemas with few rows of database contents. BIRD contains 12,751 question-SQL pairs over 95 databases totalling 33.4 GB across 37 domains, and emphasizes dirty contents, external knowledge connecting questions to stored values, and SQL efficiency. [S3]

The paper's abstract reports that the strongest model tested, ChatGPT, reached 40.08% execution accuracy, compared with 92.96% for humans. [S3] Because that benchmark result is from 2023, it should be treated as historical rather than as a current accuracy estimate. It demonstrates that schema parsing alone is insufficient: accurate text-to-SQL also requires understanding stored values, domain knowledge, and the relationship between a question and the database contents.

What a semantic layer adds

NVIDIA Auto Ontology builds a layer of three kinds of definition over a database: Terms, which map business concepts such as Customer to columns; SQL attributes, which preserve reviewed expressions so measures are computed consistently; and Relationships, which describe how tables join, including bridge tables. [S1]

It works in three stages. First it generates a draft from source metadata by reading schemas, profiling columns, naming entities and inferring joins. People then test the draft with real questions, refine it, and certify names, descriptions and SQL attributes once they are correct. Finally, the agent answers from the ontology, checks filter values and the query against the database, repairs SQL when a check fails, and reports when the available data cannot answer the question. [S1]

The idea is not unique to NVIDIA. The dbt Semantic Layer centralizes metric definitions in the modeling layer so different business units work from the same definitions. AI tools such as Claude and ChatGPT can connect through the dbt MCP server so their answers use governed metrics rather than guessing at raw tables. [S4]

Model proposes, server checks

auto-ontology-mcp is an independent personal open-source project that brings Auto Ontology's approach to MCP clients. It states that it is not affiliated with, sponsored by or endorsed by NVIDIA. [S2]

Its design draws a clear line. The model in the user's MCP client handles the language steps: reading, extracting, naming, judging and writing SQL. The server performs repeatable work such as schema extraction, profiling, key inference, validation, storage and read-only execution. [S2] NVIDIA's version checks SQL using database probes plus a language-model review; the independent port uses database probes and ontology checks in validate_sql. [S2]

This design assigns open-ended interpretation to the model and repeatable checks to code. Whether that split improves accuracy still needs independent measurement.

Read-only by construction, with one caveat

The project says it enforces read-only access to connected databases in three layers. [S2]

A SQL guard accepts one SELECT, WITH or set operation per call and refuses data changes, schema changes, and functions that reach files, other servers or server state.
Read-only connections open each supported database engine in its read-only mode.
Rollback ends every transaction without committing it.

Queries also time out after 30 seconds and return at most 500 rows. [S2]

The caveat is that datasource URLs, including passwords, are stored in plain text in the project's local database. [S2] The README recommends using a read-only database account and warns that data returned by the server is passed to the model behind the user's MCP client under that provider's terms. [S2]

How this maps to the trusted-agents checklist

Our earlier article, What Makes an AI Agent Trustworthy?, proposed five questions for evaluating an agent. A semantic layer answers several of them for one specific task:

What evidence grounds each decision? Reviewed Terms and relationships, plus the database's stored values rather than column names alone. [S1][S2]
Which steps are deterministic? Validation, key inference and read-only execution remain in server code rather than the model. [S2]
What can the agent change? In auto-ontology-mcp's connected databases, nothing; the server still writes its own local knowledge and ontology files. [S2]
What record remains? NVIDIA's agent returns the SQL and the steps it took with each answer. [S1]
Where must a human approve? During certification, people decide when definitions, descriptions and SQL attributes are correct. [S1]

Honest limits

The evidence is mostly self-reported. The claims about what Auto Ontology, auto-ontology-mcp and the dbt Semantic Layer catch come from the projects themselves. [S1][S2][S4] The independent BIRD paper shows that database values create real text-to-SQL difficulty; it does not measure a semantic layer. [S3]
auto-ontology-mcp is very new. Its repository was created on November 6, 2026, has one contributor, and describes the software as provided without warranty. [S2] Its README demonstration uses a synthetic store and labels the displayed model replies as illustrative. [S2]
The port is not NVIDIA's product. It replaces several of NVIDIA's language-model and embedding-search steps with the client's model plus server-side name rules and value checks. [S1][S2] Results from one should not be assumed for the other.
Someone still has to review the ontology. A generated draft is not a certified one. [S1] These sources do not document the ongoing effort required to recertify definitions as schemas change.

What these projects describe is specific: checking a query against reviewed definitions and stored data before execution, and reporting when the available data cannot answer a question. [S1][S2] That is a useful design change, provided teams treat the ontology as something people own and review—and ask for independent measurements before relying on its accuracy claims.