smartlab-br · thyarles · Jun 7, 2023 · Aug 1, 2023 · Oct 24, 2023 · Nov 22, 2023
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,104 @@
 Changelog
 =========
 
+0.24
+------
+* **Breaking changes**
+  - Remove Python 2.7 and <3.8 support (#606)
+
+* **Improvements**
+  - Switch to pyproject.toml from setup.py (#607)
+  - Upgrade Thrift to 0.23.0 (#609)
+    - also unpinned Thrift dependency (>=0.23.0)
+
+* **Bug Fixes**
+  - Fix non-http SSL connections with verify_cert=True and
+    ca_cert=None (these were always rejected in 0.23) (#609)
+
+0.23
+------
+* **Improvements**
+  - Add HTTP server verification for hs2-http (#598)
+  - Allow verifying server using default certificates (#601)
+    - enabled with verify_cert (default False) arg in connect()
+    - may be turned on by default in a future release
+
+* **Bug Fixes**
+  - handle expect_more_rows in fetchcolumnar() (#596)
+  - handle STILL_EXECUTING_STATUS during fetch (#594)
+
+0.22
+------
+* **Improvements**
+  - Fallback to winkerberos if kerberos is not available (#587)
+  - Update build_summary_table function to match impala-shell (#578)
+  - Unpin bitarray dependency on Python 3 (#588)
+
+* **Bug Fixes**
+  - sqlalchemy 2 related fixes #580, #582
+
+0.21
+------
+* **Improvements**
+  - Remove versioneer and add Python3.12/3.13 testing (#572)
+    - this is a temporary solution till Python2 support is dropped
+      as no version of versioneer handles both Python 2.7 and 3.12
+  - Allow users to add custom http headers when using hs2-http (#557)
+    - this change is intended to help with testing the server side
+  - Update Impala Thrift definitions. (#575)
+    - this helps in testing newer features in Impala
+
+* **Bug Fixes**
+  - Fix IPv6 address handling in hs2-http protocol
+  - Fix proxy-authentication headers for Python 3.* and long basic
+    credential encodings (#562)
+  - Fix passing retry count configuration to rpc operations (#564)
+  - Fix has_table() with sqlalchemy2 (#568)
+
+Note that this may be the last release with Python 2.7 support.
+
+0.20
+------
+* **Improvements**
+  - Support wildcard http_cookie_names (#509)
+  - Add Knox cookies in default cookies list (#525)
+  - Support CHAR type in SQLAlchemy (#516)
+  - Support Cursor.rowcount and close finished queries (#528)
+    Note that this is a potentially breaking change. See the PR
+    for details about the side-effects.
+    The old behavior can be restored by setting close_finished_queries=False
+    when creating a Cursor.
+    Also note that Cursor.rowcount only works with Impala server - with
+    Hive it will always return -1.
+  - Allow skipping utf8 conversion in Python3 (#548)
+  - Subtract RPC time from sleep in _wait_to_finish (#551)
+  - Reduced logging:
+    - Log "Closing operation" at debug level (#539)
+    - Never log passwords in http connections (#545)
+      - Before the fix passwords were logged at debug level
+
+* **Bug Fixes**
+  - Avoid retrying non-idempotent RPCs in binary connections (#549)
+  - Always set ImpalaHttpClient.__preserve_all_cookies (#553)
+  - Fix https connection with Python 3.12 (#531)
+    Note that Python 3.12 support is not complete yet.
+    A known issue is that installing with setuptools fails with Python 3.12.
+  - Fix SQLAlchemy support for Impala on Python 3.10 (#538)
+  - Turn regex strings into raw strings (#535)
+
+Note that this may be the last release with Python 2.7 support.
+
+0.19.0
+------
+* **Improvements**
+  - Add get_view_name support to SQLAlchemy (#511)
+    SHOW VIEWS is expected to be supported in Impala soon.
+  - Add additional checks to ensure connection arguments (#515)
+
+* **Bug Fixes**
+  - Fix Cookie handling with Python 3 (#518)
+  - Fix numeric parameter substitution bug (#508)
+
 0.18.0
 ------
 * **Improvements**

diff --git a/DEVELOP.md b/DEVELOP.md
@@ -14,12 +14,17 @@ Fork the repo and send a pull request against `master`.  Contributions welcome!
     rm -rf $IMPYLA_REPO/impala/thrift/*.thrift
     ```
 
-1. Execute `$IMPYLA_REPO/impala/thrift/process_thrift.sh`
+1. `cp $IMPALA_REPO/common/thrift/ImpalaService.thrift $IMPYLA_REPO/impala/thrift`
+
+Hand edit ImpalaService.thrift to exclude files, API and definitions unrelated to query
+profile such as Frontend.thrift, BackendGflags.thrift, and Query.thrift.
+
+2. Execute `$IMPYLA_REPO/impala/thrift/process_thrift.sh`
 
 This should only need to be done very irregularly, as the generated code is
 committed to the repo.  Only when the original thrift IDL files change. People
 checking out the repo to develop on it do NOT need to run the codegen.  Codegen
-performed with Thrift 0.9.x.
+performed with Thrift 0.16.x.
 
 
 #### UDF maintenance

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -2,5 +2,4 @@ include ez_setup.py
 include README.md
 include LICENSE.txt
 include CHANGELOG.md
-include versioneer.py
 include impala/_version.py
diff --git a/README.md b/README.md
@@ -11,9 +11,9 @@ distributed data sets, see the [Ibis project][ibis].
 * HiveServer2 compliant; works with Impala and Hive, including nested data
 
 * Fully [DB API 2.0 (PEP 249)][pep249]-compliant Python client (similar to
-sqlite or MySQL clients) supporting Python 2.6+ and Python 3.3+.
+sqlite or MySQL clients) supporting Python 3.8+.
 
-* Works with Kerberos, LDAP, SSL
+* Works with Kerberos, LDAP, SSL, JWT
 
 * [SQLAlchemy][sqlalchemy] connector
 
@@ -26,24 +26,25 @@ experience
 
 Required:
 
-* Python 2.7+ or 3.5+
+* Python 3.8+ (Python 2.7 was supported up to Impyla 0.23.0)
 
-* `six`, `bitarray`
+* `bitarray`
 
-* `thrift==0.16.0`
+* `thrift>=0.23.0`
 
 * `thrift_sasl==0.4.3`
 
 Optional:
 
 * `kerberos>=1.3.0` for Kerberos over HTTP support. This also requires Kerberos libraries
    to be installed on your system - see [System Kerberos](#system-kerberos)
+  * On Windows operating systems an alternative is 'winkerberos' which can be installed with pip
 
 * `pandas` for conversion to `DataFrame` objects; but see the [Ibis project][ibis] instead
 
 * `sqlalchemy` for the SQLAlchemy engine
 
-* `pytest` for running tests; `unittest2` for testing on Python 2.6
+* `pytest` and `requests` for running tests; `unittest2` for testing on Python 2.6
 
 
 #### System Kerberos
@@ -80,7 +81,7 @@ or clone the repo:
 ```bash
 git clone https://github.com/cloudera/impyla.git
 cd impyla
-python setup.py install
+python -m pip install .
 ```
 
 #### Running the tests
@@ -103,6 +104,17 @@ py.test --connect impala
 
 Leave out the `--connect` option to skip tests for DB API compliance.
 
+To test impyla with different Python versions [tox] can be used.
+The commands below will run all impyla tests with all supported and
+installed Python versions:
+```bash
+cd path/to/impyla
+tox
+```
+To filter environments / tests use `-e` and [pytest] arguments after `--`:
+```bash
+tox -e py310 -- -ktest_utf8_strings
+```
 
 ### Usage
 
@@ -111,7 +123,7 @@ Impyla implements the [Python DB API v2.0 (PEP 249)][pep249] database interface
 
 ```python
 from impala.dbapi import connect
-conn = connect(host='my.host.com', port=21050)
+conn = connect(host='my.host.com', port=21050) # auth_mechanism='PLAIN' for unsecured Hive connection, see function doc
 cursor = conn.cursor()
 cursor.execute('SELECT * FROM mytable LIMIT 100')
 print cursor.description  # prints the result set's schema
@@ -152,6 +164,16 @@ df = as_pandas(cur)
 # carry df through scikit-learn, for example
 ```
 
+For secure connection set use_ssl=True in connect(). Warning: this doesn't verify the server
+by default! To verify server set verify_cert or ca_cart perameter:
+
+```python
+# Verify server using system CA certificates:
+conn = connect(host='my.host.com', use_ssl=True, verify_cert=True)
+
+# Verify server using custom CA certificate:
+conn = connect(host='my.host.com', use_ssl=True, ca_cart="/tmp/my_cert.pem")
+```
 
 [pep249]: http://legacy.python.org/dev/peps/pep-0249/
 [pandas]: http://pandas.pydata.org/
@@ -160,6 +182,7 @@ df = as_pandas(cur)
 [pytest]: http://pytest.org/latest/
 [sqlalchemy]: http://www.sqlalchemy.org/
 [ibis]: http://www.ibis-project.org/
+[tox]: http://tox.wiki/
 
 # How do I contribute code?
 You need to first sign and return an

diff --git a/bin/bootstrap_test_env.sh b/bin/bootstrap_test_env.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+# Copyright 2026 Cloudera Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Install dependencies to run tests on each supported Python version using tox.
+
+sudo apt install --yes -q python3-pip
+python3 -m pip install tox
+
+# Install various python versions.
+# Some dependencies in Python 3.8 and 3.9 also need distutils.
+sudo add-apt-repository ppa:deadsnakes/ppa
+sudo apt update
+sudo apt install --yes -q python3.8 python3.8-dev python3.8-distutils
+sudo apt install --yes -q python3.9 python3.9-dev python3.9-distutils
+sudo apt install --yes -q python3.10 python3.10-dev
+sudo apt install --yes -q python3.11 python3.11-dev
+sudo apt install --yes -q python3.12 python3.12-dev
+sudo apt install --yes -q python3.13 python3.13-dev
+sudo apt install --yes -q python3.14 python3.14-dev
diff --git a/bin/run_tests_in_impala_dev_env.sh b/bin/run_tests_in_impala_dev_env.sh
@@ -0,0 +1,77 @@
+#!/bin/bash
+# Copyright 2026 Cloudera Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Run all tests by restarting Impala cluster with different authentication flags.
+# TODO: move this to tox or something like CustomClusterTest in Impala?
+
+set -eu
+
+TOX_ENV_ARGS=""
+if [[ -n "${1:-}" ]]; then
+    TOX_ENV_ARGS="-e $1"
+fi
+
+if [[ -z "$IMPALA_HOME" ]]; then
+    echo "Must provide IMPALA_HOME in environment" 1>&2
+    exit 1
+fi
+if [[ -z "$IMPYLA_HOME" ]]; then
+    echo "Must provide IMPYLA_HOME in environment" 1>&2
+    exit 1
+fi
+
+. "$IMPALA_HOME/bin/impala-config.sh"
+. "$IMPALA_HOME/bin/set-pythonpath.sh"
+
+unset IMPYLA_TEST_AUTH_MECH
+unset IMPYLA_SSL_CERT
+unset IMPYLA_SSL_WRONG_CERT
+
+
+# Run tests without SSL and authentication.
+$IMPALA_HOME/bin/start-impala-cluster.py
+unset IMPYLA_REPORT_PREFIX
+set +e
+python3 -m tox $TOX_ENV_ARGS
+ret=$?
+set -e
+
+# Run tests with SSL enabled.
+export IMPALA_SSL_CERT_DIR=$IMPALA_HOME/be/src/testutil
+export IMPYLA_SSL_CERT=$IMPALA_SSL_CERT_DIR/server-cert.pem
+export IMPALA_SSL_ARGS="--ssl_client_ca_certificate=$IMPYLA_SSL_CERT --ssl_server_certificate=$IMPYLA_SSL_CERT --ssl_private_key=$IMPALA_SSL_CERT_DIR/server-key.pem --hostname=localhost"
+export IMPYLA_SSL_WRONG_CERT=$IMPALA_SSL_CERT_DIR/incorrect-commonname-cert.pem
+$IMPALA_HOME/bin/start-impala-cluster.py --impalad_args="$IMPALA_SSL_ARGS" --catalogd_args="$IMPALA_SSL_ARGS" --state_store_args="$IMPALA_SSL_ARGS"
+export IMPYLA_REPORT_PREFIX="ssl-"
+set +e
+python3 -m tox $TOX_ENV_ARGS -- -m ssl
+ret=$(( ret != 0 ? ret : $? ))
+set -e
+
+
+unset IMPYLA_SSL_CERT
+unset IMPYLA_SSL_WRONG_CERT
+
+export IMPYLA_TEST_AUTH_MECH=JWT
+$IMPALA_HOME/bin/start-impala-cluster.py --impalad_args="--jwt_token_auth=true --jwt_validate_signature=false --jwt_allow_without_tls=true"
+export IMPYLA_REPORT_PREFIX="jwt-"
+set +e
+python3 -m tox $TOX_ENV_ARGS -- -m jwt_auth
+ret=$(( ret != 0 ? ret : $? ))
+set -e
+
+exit $ret
+
+