Queries returning duplicate column names fail with ArrowInvalid: Can't unify schema with duplicate field names since 4.2.6 #989

Summary

Since 4.2.6, any query whose result has two columns with the same name fails on fetch. This includes fetchall(), fetchone() and fetchall_arrow(). 4.2.5 handles the same queries correctly.

Duplicate names are valid SQL and common in practice, for example SELECT * FROM a JOIN b where both tables have an id column.

Reproduction

from databricks import sql

with sql.connect(server_hostname=..., http_path=..., access_token=...) as conn:
    with conn.cursor() as cursor:
        cursor.execute("SELECT 1 AS a, 2 AS a")
        print(cursor.fetchall())

The same happens with fetchone() and fetchall_arrow(). Tested against a SQL warehouse with pyarrow 26.0.0 and Python 3.12.

Offline, without a warehouse:

import pyarrow as pa
from databricks.sql.utils import concat_table_chunks

chunk = pa.Table.from_arrays([pa.array([1]), pa.array([2])], names=["a", "a"])
concat_table_chunks([chunk])  # 4.2.5: OK; 4.2.6+: ArrowInvalid

Cause

#751 (for #418) changed concat_table_chunks to call pyarrow.concat_tables(..., promote_options="default"). #814 later changed the same path. Arrow's schema unification rejects duplicate field names, even when every chunk has the same schema and nothing needs promoting. It fails even with a single chunk, and every Arrow fetch on the Thrift backend goes through concat_table_chunks.

This looks unintended. #418 and #751 are only about nullability differences between chunks. Elsewhere the connector explicitly supports duplicate names: the Row docstring documents them, and _convert_arrow_table renames columns before to_pandas so they survive, with a unit test (test_duplicate_column_names). That test passes because it calls _convert_arrow_table directly and never goes through the concat step.

Suggested fix

Promote only when chunk schemas actually differ:

def _concat_arrow_tables(table_chunks):
    if all(t.schema.equals(table_chunks[0].schema) for t in table_chunks[1:]):
        return pyarrow.concat_tables(table_chunks)
    # existing promote_options="default" / promote=True fallback

This restores duplicate-name results and keeps the #418 nullability case (string vs string not null) working. I checked both. A result that has duplicate names and differing chunk schemas would still fail, but that's a much narrower case than today's.

It would also be worth adding a fetch-level test with duplicate column names, so this path stays covered.