Skip to content

FIX: Inconsistent retrieval of CP1252 encoded data in VARCHAR columns - Windows vs. Linux #468 #495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
subrata-ms wants to merge 3 commits into
base: main
Choose a base branch
from
Open

FIX: Inconsistent retrieval of CP1252 encoded data in VARCHAR columns - Windows vs. Linux #468 #495

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
14d1f1d
base code changes
subrata-ms Apr 3, 2026
8817007
Merge branch 'main' into subrata-ms/CP1252Encoding
subrata-ms Apr 3, 2026
597ee4c
code fix for linux based system
subrata-ms Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions mssql_python/connection.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,10 +261,14 @@ def __init__(
}

# Initialize decoding settings with Python 3 defaults
# SQL_CHAR default uses SQL_WCHAR ctype so the ODBC driver returns
# UTF-16 data for VARCHAR columns. This avoids encoding mismatches on
# Windows where the driver returns raw bytes in the server's native
# code page (e.g. CP-1252) that may fail to decode as UTF-8.
self._decoding_settings = {
ConstantsDDBC.SQL_CHAR.value: {
"encoding": "utf-8",
"ctype": ConstantsDDBC.SQL_CHAR.value,
"encoding": "utf-16le",
"ctype": ConstantsDDBC.SQL_WCHAR.value,
},
Comment on lines 268 to 272
Copy link

Copilot AI Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Connection.init now sets SQL_CHAR decoding defaults to encoding='utf-16le' with ctype=SQL_WCHAR, but setdecoding() still defaults SQL_CHAR to 'utf-8' when encoding is None. This makes it impossible to reliably “reset to defaults” via setdecoding(SQL_CHAR) and creates inconsistent documented behavior. Update setdecoding’s default-encoding branch (and its docstring/examples) so SQL_CHAR defaults match the new connection defaults (utf-16le + SQL_WCHAR).

Copilot uses AI. Check for mistakes.
ConstantsDDBC.SQL_WCHAR.value: {
"encoding": "utf-16le",
Expand Down
9 changes: 6 additions & 3 deletions mssql_python/cursor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2368,8 +2368,9 @@ def fetchone(self) -> Union[None, Row]:
ret = ddbc_bindings.DDBCSQLFetchOne(
self.hstmt,
row_data,
char_decoding.get("encoding", "utf-8"),
char_decoding.get("encoding", "utf-16le"),
wchar_decoding.get("encoding", "utf-16le"),
char_decoding.get("ctype", ddbc_sql_const.SQL_WCHAR.value),
)

if self.hstmt:
Expand Down Expand Up @@ -2434,8 +2435,9 @@ def fetchmany(self, size: Optional[int] = None) -> List[Row]:
self.hstmt,
rows_data,
size,
char_decoding.get("encoding", "utf-8"),
char_decoding.get("encoding", "utf-16le"),
wchar_decoding.get("encoding", "utf-16le"),
char_decoding.get("ctype", ddbc_sql_const.SQL_WCHAR.value),
)

if self.hstmt:
Expand Down Expand Up @@ -2492,8 +2494,9 @@ def fetchall(self) -> List[Row]:
ret = ddbc_bindings.DDBCSQLFetchAll(
self.hstmt,
rows_data,
char_decoding.get("encoding", "utf-8"),
char_decoding.get("encoding", "utf-16le"),
wchar_decoding.get("encoding", "utf-16le"),
char_decoding.get("ctype", ddbc_sql_const.SQL_WCHAR.value),
)

# Check for errors
Expand Down
Loading
Loading