diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e839bc03..761a05bb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,104 @@ Changelog ========= +0.24 +------ +* **Breaking changes** + - Remove Python 2.7 and <3.8 support (#606) + +* **Improvements** + - Switch to pyproject.toml from setup.py (#607) + - Upgrade Thrift to 0.23.0 (#609) + - also unpinned Thrift dependency (>=0.23.0) + +* **Bug Fixes** + - Fix non-http SSL connections with verify_cert=True and + ca_cert=None (these were always rejected in 0.23) (#609) + +0.23 +------ +* **Improvements** + - Add HTTP server verification for hs2-http (#598) + - Allow verifying server using default certificates (#601) + - enabled with verify_cert (default False) arg in connect() + - may be turned on by default in a future release + +* **Bug Fixes** + - handle expect_more_rows in fetchcolumnar() (#596) + - handle STILL_EXECUTING_STATUS during fetch (#594) + +0.22 +------ +* **Improvements** + - Fallback to winkerberos if kerberos is not available (#587) + - Update build_summary_table function to match impala-shell (#578) + - Unpin bitarray dependency on Python 3 (#588) + +* **Bug Fixes** + - sqlalchemy 2 related fixes #580, #582 + +0.21 +------ +* **Improvements** + - Remove versioneer and add Python3.12/3.13 testing (#572) + - this is a temporary solution till Python2 support is dropped + as no version of versioneer handles both Python 2.7 and 3.12 + - Allow users to add custom http headers when using hs2-http (#557) + - this change is intended to help with testing the server side + - Update Impala Thrift definitions. (#575) + - this helps in testing newer features in Impala + +* **Bug Fixes** + - Fix IPv6 address handling in hs2-http protocol + - Fix proxy-authentication headers for Python 3.* and long basic + credential encodings (#562) + - Fix passing retry count configuration to rpc operations (#564) + - Fix has_table() with sqlalchemy2 (#568) + +Note that this may be the last release with Python 2.7 support. + +0.20 +------ +* **Improvements** + - Support wildcard http_cookie_names (#509) + - Add Knox cookies in default cookies list (#525) + - Support CHAR type in SQLAlchemy (#516) + - Support Cursor.rowcount and close finished queries (#528) + Note that this is a potentially breaking change. See the PR + for details about the side-effects. + The old behavior can be restored by setting close_finished_queries=False + when creating a Cursor. + Also note that Cursor.rowcount only works with Impala server - with + Hive it will always return -1. + - Allow skipping utf8 conversion in Python3 (#548) + - Subtract RPC time from sleep in _wait_to_finish (#551) + - Reduced logging: + - Log "Closing operation" at debug level (#539) + - Never log passwords in http connections (#545) + - Before the fix passwords were logged at debug level + +* **Bug Fixes** + - Avoid retrying non-idempotent RPCs in binary connections (#549) + - Always set ImpalaHttpClient.__preserve_all_cookies (#553) + - Fix https connection with Python 3.12 (#531) + Note that Python 3.12 support is not complete yet. + A known issue is that installing with setuptools fails with Python 3.12. + - Fix SQLAlchemy support for Impala on Python 3.10 (#538) + - Turn regex strings into raw strings (#535) + +Note that this may be the last release with Python 2.7 support. + +0.19.0 +------ +* **Improvements** + - Add get_view_name support to SQLAlchemy (#511) + SHOW VIEWS is expected to be supported in Impala soon. + - Add additional checks to ensure connection arguments (#515) + +* **Bug Fixes** + - Fix Cookie handling with Python 3 (#518) + - Fix numeric parameter substitution bug (#508) + 0.18.0 ------ * **Improvements** diff --git a/DEVELOP.md b/DEVELOP.md index 86252426e..b6c9f1ee9 100644 --- a/DEVELOP.md +++ b/DEVELOP.md @@ -14,12 +14,17 @@ Fork the repo and send a pull request against `master`. Contributions welcome! rm -rf $IMPYLA_REPO/impala/thrift/*.thrift ``` -1. Execute `$IMPYLA_REPO/impala/thrift/process_thrift.sh` +1. `cp $IMPALA_REPO/common/thrift/ImpalaService.thrift $IMPYLA_REPO/impala/thrift` + +Hand edit ImpalaService.thrift to exclude files, API and definitions unrelated to query +profile such as Frontend.thrift, BackendGflags.thrift, and Query.thrift. + +2. Execute `$IMPYLA_REPO/impala/thrift/process_thrift.sh` This should only need to be done very irregularly, as the generated code is committed to the repo. Only when the original thrift IDL files change. People checking out the repo to develop on it do NOT need to run the codegen. Codegen -performed with Thrift 0.9.x. +performed with Thrift 0.16.x. #### UDF maintenance diff --git a/MANIFEST.in b/MANIFEST.in index 51ef6c843..b5dda2e37 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -2,5 +2,4 @@ include ez_setup.py include README.md include LICENSE.txt include CHANGELOG.md -include versioneer.py include impala/_version.py diff --git a/README.md b/README.md index 315517f8c..ca106ba7c 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,9 @@ distributed data sets, see the [Ibis project][ibis]. * HiveServer2 compliant; works with Impala and Hive, including nested data * Fully [DB API 2.0 (PEP 249)][pep249]-compliant Python client (similar to -sqlite or MySQL clients) supporting Python 2.6+ and Python 3.3+. +sqlite or MySQL clients) supporting Python 3.8+. -* Works with Kerberos, LDAP, SSL +* Works with Kerberos, LDAP, SSL, JWT * [SQLAlchemy][sqlalchemy] connector @@ -26,11 +26,11 @@ experience Required: -* Python 2.7+ or 3.5+ +* Python 3.8+ (Python 2.7 was supported up to Impyla 0.23.0) -* `six`, `bitarray` +* `bitarray` -* `thrift==0.16.0` +* `thrift>=0.23.0` * `thrift_sasl==0.4.3` @@ -38,12 +38,13 @@ Optional: * `kerberos>=1.3.0` for Kerberos over HTTP support. This also requires Kerberos libraries to be installed on your system - see [System Kerberos](#system-kerberos) + * On Windows operating systems an alternative is 'winkerberos' which can be installed with pip * `pandas` for conversion to `DataFrame` objects; but see the [Ibis project][ibis] instead * `sqlalchemy` for the SQLAlchemy engine -* `pytest` for running tests; `unittest2` for testing on Python 2.6 +* `pytest` and `requests` for running tests; `unittest2` for testing on Python 2.6 #### System Kerberos @@ -80,7 +81,7 @@ or clone the repo: ```bash git clone https://github.com/cloudera/impyla.git cd impyla -python setup.py install +python -m pip install . ``` #### Running the tests @@ -103,6 +104,17 @@ py.test --connect impala Leave out the `--connect` option to skip tests for DB API compliance. +To test impyla with different Python versions [tox] can be used. +The commands below will run all impyla tests with all supported and +installed Python versions: +```bash +cd path/to/impyla +tox +``` +To filter environments / tests use `-e` and [pytest] arguments after `--`: +```bash +tox -e py310 -- -ktest_utf8_strings +``` ### Usage @@ -111,7 +123,7 @@ Impyla implements the [Python DB API v2.0 (PEP 249)][pep249] database interface ```python from impala.dbapi import connect -conn = connect(host='my.host.com', port=21050) +conn = connect(host='my.host.com', port=21050) # auth_mechanism='PLAIN' for unsecured Hive connection, see function doc cursor = conn.cursor() cursor.execute('SELECT * FROM mytable LIMIT 100') print cursor.description # prints the result set's schema @@ -152,6 +164,16 @@ df = as_pandas(cur) # carry df through scikit-learn, for example ``` +For secure connection set use_ssl=True in connect(). Warning: this doesn't verify the server +by default! To verify server set verify_cert or ca_cart perameter: + +```python +# Verify server using system CA certificates: +conn = connect(host='my.host.com', use_ssl=True, verify_cert=True) + +# Verify server using custom CA certificate: +conn = connect(host='my.host.com', use_ssl=True, ca_cart="/tmp/my_cert.pem") +``` [pep249]: http://legacy.python.org/dev/peps/pep-0249/ [pandas]: http://pandas.pydata.org/ @@ -160,6 +182,7 @@ df = as_pandas(cur) [pytest]: http://pytest.org/latest/ [sqlalchemy]: http://www.sqlalchemy.org/ [ibis]: http://www.ibis-project.org/ +[tox]: http://tox.wiki/ # How do I contribute code? You need to first sign and return an diff --git a/bin/bootstrap_test_env.sh b/bin/bootstrap_test_env.sh new file mode 100755 index 000000000..5e66b529d --- /dev/null +++ b/bin/bootstrap_test_env.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# Copyright 2026 Cloudera Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Install dependencies to run tests on each supported Python version using tox. + +sudo apt install --yes -q python3-pip +python3 -m pip install tox + +# Install various python versions. +# Some dependencies in Python 3.8 and 3.9 also need distutils. +sudo add-apt-repository ppa:deadsnakes/ppa +sudo apt update +sudo apt install --yes -q python3.8 python3.8-dev python3.8-distutils +sudo apt install --yes -q python3.9 python3.9-dev python3.9-distutils +sudo apt install --yes -q python3.10 python3.10-dev +sudo apt install --yes -q python3.11 python3.11-dev +sudo apt install --yes -q python3.12 python3.12-dev +sudo apt install --yes -q python3.13 python3.13-dev +sudo apt install --yes -q python3.14 python3.14-dev diff --git a/bin/run_tests_in_impala_dev_env.sh b/bin/run_tests_in_impala_dev_env.sh new file mode 100755 index 000000000..d43cefef4 --- /dev/null +++ b/bin/run_tests_in_impala_dev_env.sh @@ -0,0 +1,77 @@ +#!/bin/bash +# Copyright 2026 Cloudera Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Run all tests by restarting Impala cluster with different authentication flags. +# TODO: move this to tox or something like CustomClusterTest in Impala? + +set -eu + +TOX_ENV_ARGS="" +if [[ -n "${1:-}" ]]; then + TOX_ENV_ARGS="-e $1" +fi + +if [[ -z "$IMPALA_HOME" ]]; then + echo "Must provide IMPALA_HOME in environment" 1>&2 + exit 1 +fi +if [[ -z "$IMPYLA_HOME" ]]; then + echo "Must provide IMPYLA_HOME in environment" 1>&2 + exit 1 +fi + +. "$IMPALA_HOME/bin/impala-config.sh" +. "$IMPALA_HOME/bin/set-pythonpath.sh" + +unset IMPYLA_TEST_AUTH_MECH +unset IMPYLA_SSL_CERT +unset IMPYLA_SSL_WRONG_CERT + + +# Run tests without SSL and authentication. +$IMPALA_HOME/bin/start-impala-cluster.py +unset IMPYLA_REPORT_PREFIX +set +e +python3 -m tox $TOX_ENV_ARGS +ret=$? +set -e + +# Run tests with SSL enabled. +export IMPALA_SSL_CERT_DIR=$IMPALA_HOME/be/src/testutil +export IMPYLA_SSL_CERT=$IMPALA_SSL_CERT_DIR/server-cert.pem +export IMPALA_SSL_ARGS="--ssl_client_ca_certificate=$IMPYLA_SSL_CERT --ssl_server_certificate=$IMPYLA_SSL_CERT --ssl_private_key=$IMPALA_SSL_CERT_DIR/server-key.pem --hostname=localhost" +export IMPYLA_SSL_WRONG_CERT=$IMPALA_SSL_CERT_DIR/incorrect-commonname-cert.pem +$IMPALA_HOME/bin/start-impala-cluster.py --impalad_args="$IMPALA_SSL_ARGS" --catalogd_args="$IMPALA_SSL_ARGS" --state_store_args="$IMPALA_SSL_ARGS" +export IMPYLA_REPORT_PREFIX="ssl-" +set +e +python3 -m tox $TOX_ENV_ARGS -- -m ssl +ret=$(( ret != 0 ? ret : $? )) +set -e + + +unset IMPYLA_SSL_CERT +unset IMPYLA_SSL_WRONG_CERT + +export IMPYLA_TEST_AUTH_MECH=JWT +$IMPALA_HOME/bin/start-impala-cluster.py --impalad_args="--jwt_token_auth=true --jwt_validate_signature=false --jwt_allow_without_tls=true" +export IMPYLA_REPORT_PREFIX="jwt-" +set +e +python3 -m tox $TOX_ENV_ARGS -- -m jwt_auth +ret=$(( ret != 0 ? ret : $? )) +set -e + +exit $ret + + diff --git a/build-dists.sh b/build-dists.sh deleted file mode 100755 index 54e48b3fc..000000000 --- a/build-dists.sh +++ /dev/null @@ -1,70 +0,0 @@ -#!/bin/bash -# Copyright 2015 Cloudera Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -x - -# Usage info -show_help() { -cat << EOF -Usage: ${0##*/} [-h] [-a GITHUB_ACCOUNT] GIT_VERSION_TAG - -h display this help and exit - -a GITHUB_ACCOUNT use GITHUB_ACCOUNT instead of 'cloudera' -EOF -} - -# Parse command line options -GITHUB_ACCOUNT="cloudera" -GIT_VERSION_TAG="" - -OPTIND=1 -while getopts ha: opt; do - case $opt in - h) - show_help - exit 0 - ;; - a) GITHUB_ACCOUNT=$OPTARG - ;; - *) - show_help >&2 - exit 1 - ;; - esac -done -shift "$((OPTIND-1))" # Discard the options and sentinel -- - -GIT_VERSION_TAG="$1" -if [ -z "$GIT_VERSION_TAG" ] || [ "$#" -gt 1 ]; then - show_help >&2 - exit 1 -fi - -# Start build script in manylinux docker container -DOCKER_IMAGE='quay.io/pypa/manylinux2010_x86_64' - -docker pull "$DOCKER_IMAGE" -docker container run -t --rm -v "$(pwd)/io:/io" "$DOCKER_IMAGE" \ - "/io/manylinux/build.sh" \ - "/io/pip-dists-build" \ - "$GIT_VERSION_TAG" \ - "$GITHUB_ACCOUNT" - -RETVAL="$?" -if [[ "$RETVAL" != "0" ]]; then - echo "Failed with $RETVAL" -else - echo "Succeeded" -fi -exit $RETVAL diff --git a/dev/merge-pr.py b/dev/merge-pr.py index d76b22e9a..7361f27a4 100644 --- a/dev/merge-pr.py +++ b/dev/merge-pr.py @@ -22,8 +22,6 @@ # # Lightly modified from version of this script in incubator-parquet-format -from __future__ import print_function - from requests.auth import HTTPBasicAuth import requests diff --git a/ez_setup.py b/ez_setup.py deleted file mode 100644 index 1b8fd9597..000000000 --- a/ez_setup.py +++ /dev/null @@ -1,332 +0,0 @@ -#!/usr/bin/env python -"""Bootstrap setuptools installation - -To use setuptools in your package's setup.py, include this -file in the same directory and add this to the top of your setup.py:: - - from ez_setup import use_setuptools - use_setuptools() - -To require a specific version of setuptools, set a download -mirror, or use an alternate download directory, simply supply -the appropriate options to ``use_setuptools()``. - -This file can also be run as a script to install or upgrade setuptools. -""" -import os -import shutil -import sys -import tempfile -import zipfile -import optparse -import subprocess -import platform -import textwrap -import contextlib - -from distutils import log - -try: - from site import USER_SITE -except ImportError: - USER_SITE = None - -DEFAULT_VERSION = "3.4.4" -DEFAULT_URL = "https://pypi.python.org/packages/source/s/setuptools/" - -def _python_cmd(*args): - """ - Return True if the command succeeded. - """ - args = (sys.executable,) + args - return subprocess.call(args) == 0 - - -def _install(archive_filename, install_args=()): - with archive_context(archive_filename): - # installing - log.warn('Installing Setuptools') - if not _python_cmd('setup.py', 'install', *install_args): - log.warn('Something went wrong during the installation.') - log.warn('See the error message above.') - # exitcode will be 2 - return 2 - - -def _build_egg(egg, archive_filename, to_dir): - with archive_context(archive_filename): - # building an egg - log.warn('Building a Setuptools egg in %s', to_dir) - _python_cmd('setup.py', '-q', 'bdist_egg', '--dist-dir', to_dir) - # returning the result - log.warn(egg) - if not os.path.exists(egg): - raise IOError('Could not build the egg.') - - -def get_zip_class(): - """ - Supplement ZipFile class to support context manager for Python 2.6 - """ - class ContextualZipFile(zipfile.ZipFile): - def __enter__(self): - return self - def __exit__(self, type, value, traceback): - self.close - return zipfile.ZipFile if hasattr(zipfile.ZipFile, '__exit__') else \ - ContextualZipFile - - -@contextlib.contextmanager -def archive_context(filename): - # extracting the archive - tmpdir = tempfile.mkdtemp() - log.warn('Extracting in %s', tmpdir) - old_wd = os.getcwd() - try: - os.chdir(tmpdir) - with get_zip_class()(filename) as archive: - archive.extractall() - - # going in the directory - subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0]) - os.chdir(subdir) - log.warn('Now working in %s', subdir) - yield - - finally: - os.chdir(old_wd) - shutil.rmtree(tmpdir) - - -def _do_download(version, download_base, to_dir, download_delay): - egg = os.path.join(to_dir, 'setuptools-%s-py%d.%d.egg' - % (version, sys.version_info[0], sys.version_info[1])) - if not os.path.exists(egg): - archive = download_setuptools(version, download_base, - to_dir, download_delay) - _build_egg(egg, archive, to_dir) - sys.path.insert(0, egg) - - # Remove previously-imported pkg_resources if present (see - # https://bitbucket.org/pypa/setuptools/pull-request/7/ for details). - if 'pkg_resources' in sys.modules: - del sys.modules['pkg_resources'] - - import setuptools - setuptools.bootstrap_install_from = egg - - -def use_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL, - to_dir=os.curdir, download_delay=15): - to_dir = os.path.abspath(to_dir) - rep_modules = 'pkg_resources', 'setuptools' - imported = set(sys.modules).intersection(rep_modules) - try: - import pkg_resources - except ImportError: - return _do_download(version, download_base, to_dir, download_delay) - try: - pkg_resources.require("setuptools>=" + version) - return - except pkg_resources.DistributionNotFound: - return _do_download(version, download_base, to_dir, download_delay) - except pkg_resources.VersionConflict as VC_err: - if imported: - msg = textwrap.dedent(""" - The required version of setuptools (>={version}) is not available, - and can't be installed while this script is running. Please - install a more recent version first, using - 'easy_install -U setuptools'. - - (Currently using {VC_err.args[0]!r}) - """).format(VC_err=VC_err, version=version) - sys.stderr.write(msg) - sys.exit(2) - - # otherwise, reload ok - del pkg_resources, sys.modules['pkg_resources'] - return _do_download(version, download_base, to_dir, download_delay) - -def _clean_check(cmd, target): - """ - Run the command to download target. If the command fails, clean up before - re-raising the error. - """ - try: - subprocess.check_call(cmd) - except subprocess.CalledProcessError: - if os.access(target, os.F_OK): - os.unlink(target) - raise - -def download_file_powershell(url, target): - """ - Download the file at url to target using Powershell (which will validate - trust). Raise an exception if the command cannot complete. - """ - target = os.path.abspath(target) - cmd = [ - 'powershell', - '-Command', - "(new-object System.Net.WebClient).DownloadFile(%(url)r, %(target)r)" % vars(), - ] - _clean_check(cmd, target) - -def has_powershell(): - if platform.system() != 'Windows': - return False - cmd = ['powershell', '-Command', 'echo test'] - devnull = open(os.path.devnull, 'wb') - try: - try: - subprocess.check_call(cmd, stdout=devnull, stderr=devnull) - except Exception: - return False - finally: - devnull.close() - return True - -download_file_powershell.viable = has_powershell - -def download_file_curl(url, target): - cmd = ['curl', url, '--silent', '--output', target] - _clean_check(cmd, target) - -def has_curl(): - cmd = ['curl', '--version'] - devnull = open(os.path.devnull, 'wb') - try: - try: - subprocess.check_call(cmd, stdout=devnull, stderr=devnull) - except Exception: - return False - finally: - devnull.close() - return True - -download_file_curl.viable = has_curl - -def download_file_wget(url, target): - cmd = ['wget', url, '--quiet', '--output-document', target] - _clean_check(cmd, target) - -def has_wget(): - cmd = ['wget', '--version'] - devnull = open(os.path.devnull, 'wb') - try: - try: - subprocess.check_call(cmd, stdout=devnull, stderr=devnull) - except Exception: - return False - finally: - devnull.close() - return True - -download_file_wget.viable = has_wget - -def download_file_insecure(url, target): - """ - Use Python to download the file, even though it cannot authenticate the - connection. - """ - try: - from urllib.request import urlopen - except ImportError: - from urllib2 import urlopen - src = dst = None - try: - src = urlopen(url) - # Read/write all in one block, so we don't create a corrupt file - # if the download is interrupted. - data = src.read() - dst = open(target, "wb") - dst.write(data) - finally: - if src: - src.close() - if dst: - dst.close() - -download_file_insecure.viable = lambda: True - -def get_best_downloader(): - downloaders = [ - download_file_powershell, - download_file_curl, - download_file_wget, - download_file_insecure, - ] - - for dl in downloaders: - if dl.viable(): - return dl - -def download_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL, - to_dir=os.curdir, delay=15, downloader_factory=get_best_downloader): - """ - Download setuptools from a specified location and return its filename - - `version` should be a valid setuptools version number that is available - as an egg for download under the `download_base` URL (which should end - with a '/'). `to_dir` is the directory where the egg will be downloaded. - `delay` is the number of seconds to pause before an actual download - attempt. - - ``downloader_factory`` should be a function taking no arguments and - returning a function for downloading a URL to a target. - """ - # making sure we use the absolute path - to_dir = os.path.abspath(to_dir) - zip_name = "setuptools-%s.zip" % version - url = download_base + zip_name - saveto = os.path.join(to_dir, zip_name) - if not os.path.exists(saveto): # Avoid repeated downloads - log.warn("Downloading %s", url) - downloader = downloader_factory() - downloader(url, saveto) - return os.path.realpath(saveto) - -def _build_install_args(options): - """ - Build the arguments to 'python setup.py install' on the setuptools package - """ - return ['--user'] if options.user_install else [] - -def _parse_args(): - """ - Parse the command line for options - """ - parser = optparse.OptionParser() - parser.add_option( - '--user', dest='user_install', action='store_true', default=False, - help='install in user site package (requires Python 2.6 or later)') - parser.add_option( - '--download-base', dest='download_base', metavar="URL", - default=DEFAULT_URL, - help='alternative URL from where to download the setuptools package') - parser.add_option( - '--insecure', dest='downloader_factory', action='store_const', - const=lambda: download_file_insecure, default=get_best_downloader, - help='Use internal, non-validating downloader' - ) - parser.add_option( - '--version', help="Specify which version to download", - default=DEFAULT_VERSION, - ) - options, args = parser.parse_args() - # positional arguments are ignored - return options - -def main(): - """Install or upgrade setuptools and EasyInstall""" - options = _parse_args() - archive = download_setuptools( - version=options.version, - download_base=options.download_base, - downloader_factory=options.downloader_factory, - ) - return _install(archive, _build_install_args(options)) - -if __name__ == '__main__': - sys.exit(main()) diff --git a/impala/__init__.py b/impala/__init__.py index b5e289d77..74349916e 100644 --- a/impala/__init__.py +++ b/impala/__init__.py @@ -12,8 +12,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -from __future__ import absolute_import - -from ._version import get_versions -__version__ = get_versions()['version'] -del get_versions +# pyproject.toml takes version from this. +__version__ = u'v0.24.0' diff --git a/impala/_thrift_api.py b/impala/_thrift_api.py index bd1e8906e..f605f36b3 100644 --- a/impala/_thrift_api.py +++ b/impala/_thrift_api.py @@ -19,8 +19,6 @@ # pylint: disable=wrong-import-position -from __future__ import absolute_import - import base64 import datetime import getpass @@ -29,16 +27,18 @@ from collections import namedtuple from io import BytesIO -from six.moves import urllib, http_client +import urllib.parse +import urllib.request +import http.client as http_client import warnings -import six import ssl import sys from impala.error import HttpError +from impala.util import get_basic_credentials_for_request_headers from impala.util import get_logger_and_init_null -from impala.util import get_all_matching_cookies, get_cookie_expiry +from impala.util import get_all_matching_cookies, get_all_cookies, get_cookie_expiry # Declare namedtuple for Cookie with named fields - cookie and expiry_time Cookie = namedtuple('Cookie', ['cookie', 'expiry_time']) @@ -68,12 +68,14 @@ class ImpalaHttpClient(TTransportBase): MIN_REQUEST_SIZE_FOR_EXPECT = 1024 def __init__(self, uri_or_host, port=None, path=None, cafile=None, cert_file=None, - key_file=None, ssl_context=None, http_cookie_names=None): + key_file=None, ssl_context=None, http_cookie_names=None, + get_user_custom_headers_func=None): """ImpalaHttpClient supports two different types of construction: ImpalaHttpClient(host, port, path) - deprecated ImpalaHttpClient(uri, [port=, path=, cafile=, cert_file=, - key_file=, ssl_context=, http_cookie_names=]) + key_file=, ssl_context=, http_cookie_names=], + get_user_custom_headers_func=) Only the second supports https. To properly authenticate against the server, provide the client's identity by specifying cert_file and key_file. To properly @@ -83,7 +85,12 @@ def __init__(self, uri_or_host, port=None, path=None, cafile=None, cert_file=Non cookie-based authentication or session management. If there's only one name in the cookie name list, a str value can be specified instead of the list. If a cookie with one of these names is returned in an http response by the server or an intermediate - proxy then it will be included in each subsequent request for the same connection. + proxy then it will be included in each subsequent request for the same connection. If + it is set as wildcards, all cookies in an http response will be preserved. + The optional get_user_custom_headers_func parameter can be used to add http headers + to outgoing http messages when using hs2-http protocol. The parameter should be a + function returning a list of tuples, each tuple containing a key-value pair + representing the header name and value. """ if port is not None: warnings.warn( @@ -129,10 +136,16 @@ def __init__(self, uri_or_host, port=None, path=None, cafile=None, cert_file=Non self.realhost = self.realport = self.proxy_auth = None if not http_cookie_names: # 'http_cookie_names' was explicitly set as an empty value ([], or '') in connect(). + self.__preserve_all_cookies = False self.__http_cookie_dict = None self.__auth_cookie_names = None + elif str(http_cookie_names).strip() == "*": + self.__preserve_all_cookies = True + self.__http_cookie_dict = dict() + self.__auth_cookie_names = set() else: - if isinstance(http_cookie_names, six.string_types): + self.__preserve_all_cookies = False + if isinstance(http_cookie_names, str): http_cookie_names = [http_cookie_names] # Build a dictionary that maps cookie name to namedtuple. self.__http_cookie_dict = \ @@ -140,7 +153,7 @@ def __init__(self, uri_or_host, port=None, path=None, cafile=None, cert_file=Non # Store the auth cookie names in __auth_cookie_names. # Assume auth cookie names end with ".auth". self.__auth_cookie_names = \ - [ cn for cn in http_cookie_names if cn.endswith(".auth") ] + { cn for cn in http_cookie_names if cn.endswith(".auth") } # Set __are_matching_cookies_found as True if matching cookies are found in response. self.__are_matching_cookies_found = False self.__wbuf = BytesIO() @@ -151,6 +164,12 @@ def __init__(self, uri_or_host, port=None, path=None, cafile=None, cert_file=Non # new request. self.__custom_headers = None self.__get_custom_headers_func = None + # __user_custom_headers is a list of tuples, each tuple contains a key-value pair. + self.__user_custom_headers = None + if get_user_custom_headers_func: + self.__get_user_custom_headers_func = get_user_custom_headers_func + else: + self.__get_user_custom_headers_func = None # the default user agent if none is provied self.__custom_user_agent = 'Python/ImpylaHttpClient' @@ -158,10 +177,10 @@ def __init__(self, uri_or_host, port=None, path=None, cafile=None, cert_file=Non def basic_proxy_auth_header(proxy): if proxy is None or not proxy.username: return None - ap = "%s:%s" % (urllib.parse.unquote(proxy.username), - urllib.parse.unquote(proxy.password)) - cr = base64.b64encode(ap).strip() - return "Basic " + cr + return "Basic " + get_basic_credentials_for_request_headers( + user=urllib.parse.unquote(proxy.username), + password=urllib.parse.unquote(proxy.password), + ) def using_proxy(self): return self.realhost is not None @@ -172,8 +191,6 @@ def open(self): timeout=self.__timeout) elif self.scheme == 'https': self.__http = http_client.HTTPSConnection(self.host, self.port, - key_file=self.keyfile, - cert_file=self.certfile, timeout=self.__timeout, context=self.context) if self.using_proxy(): @@ -207,12 +224,19 @@ def setCustomUserAgent(self, user_agent): def setGetCustomHeadersFunc(self, func): self.__get_custom_headers_func = func - # Update HTTP headers based on the saved cookies and auth mechanism. + # Update outgoing HTTP headers. + # This is done by two callback functions, if present + # __get_custom_headers_func adds headers based on the saved cookies and auth + # mechanism. + # __get_user_custom_headers_func adds custom user-supplied http headers. def refreshCustomHeaders(self): if self.__get_custom_headers_func: cookie_header, has_auth_cookie = self.getHttpCookieHeaderForRequest() self.__custom_headers = \ self.__get_custom_headers_func(cookie_header, has_auth_cookie) + if self.__get_user_custom_headers_func: + self.__user_custom_headers = \ + self.__get_user_custom_headers_func() # Return first value as a cookie list for Cookie header. It's a list of name-value # pairs in the form of =. Pairs in the list are separated by @@ -242,13 +266,20 @@ def getHttpCookieHeaderForRequest(self): # Extract cookies from response and save those cookies for which the cookie names # are in the cookie name list specified in the connect() API. def extractHttpCookiesFromResponse(self): - if self.__http_cookie_dict is not None: + if self.__preserve_all_cookies: + matching_cookies = get_all_cookies(self.path, self.headers) + elif self.__http_cookie_dict is not None: matching_cookies = get_all_matching_cookies( self.__http_cookie_dict.keys(), self.path, self.headers) - if matching_cookies: - self.__are_matching_cookies_found = True - for c in matching_cookies: - self.__http_cookie_dict[c.key] = Cookie(c, get_cookie_expiry(c)) + else: + matching_cookies = None + + if matching_cookies: + self.__are_matching_cookies_found = True + for c in matching_cookies: + self.__http_cookie_dict[c.key] = Cookie(c, get_cookie_expiry(c)) + if c.key.endswith(".auth"): + self.__auth_cookie_names.add(c.key) # Return True if there are any saved cookies which are sent in previous request. def areHttpCookiesSaved(self): @@ -307,7 +338,10 @@ def sendRequestRecvResp(data): self.__http.putheader('User-Agent', user_agent) if self.__custom_headers: - for key, val in six.iteritems(self.__custom_headers): + for key, val in self.__custom_headers.items(): + self.__http.putheader(key, val) + if self.__user_custom_headers: + for key, val in self.__user_custom_headers: self.__http.putheader(key, val) self.__http.endheaders() @@ -347,25 +381,25 @@ def sendRequestRecvResp(data): raise HttpError(self.code, self.message, body, self.headers) -def get_socket(host, port, use_ssl, ca_cert): +def get_ssl_context(ca_cert, verify_cert): + ssl_ctx = ssl.create_default_context(cafile=ca_cert) + if ca_cert or verify_cert: + ssl_ctx.check_hostname = True + ssl_ctx.verify_mode = ssl.CERT_REQUIRED + else: + ssl_ctx.check_hostname = False # Mandated by the SSL lib for CERT_NONE mode. + ssl_ctx.verify_mode = ssl.CERT_NONE + return ssl_ctx + +def get_socket(host, port, use_ssl, ca_cert, verify_cert): # based on the Impala shell impl - log.debug('get_socket: host=%s port=%s use_ssl=%s ca_cert=%s', - host, port, use_ssl, ca_cert) + log.debug('get_socket: host=%s port=%s use_ssl=%s ca_cert=%s verify_cert=%s', + host, port, use_ssl, ca_cert, verify_cert) if use_ssl: from thrift.transport.TSSLSocket import TSSLSocket - - # This copies the solution in IMPALA-11343. - # TODO: remove once Thrit 0.17.0 is released - class ImpalaTSSLSocket(TSSLSocket): - # THRIFT-5595: override TSocket.isOpen because it's broken for TSSLSocket - def isOpen(self): - return self.handle is not None - - if ca_cert is None: - return ImpalaTSSLSocket(host, port, validate=False) - else: - return ImpalaTSSLSocket(host, port, validate=True, ca_certs=ca_cert) + ssl_ctx = get_ssl_context(ca_cert, verify_cert) + return TSSLSocket(host, port, ssl_context=ssl_ctx) else: return TSocket(host, port) @@ -373,27 +407,28 @@ def isOpen(self): def get_http_transport(host, port, http_path, timeout=None, use_ssl=False, ca_cert=None, auth_mechanism='NOSASL', user=None, password=None, kerberos_host=None, kerberos_service_name=None, - http_cookie_names=None, jwt=None, user_agent=None): + http_cookie_names=None, jwt=None, user_agent=None, + get_user_custom_headers_func=None, verify_cert=False): + host_url = "[%s]" % host if ":" in host else host # add brackets for ipv6 address # TODO: support timeout if timeout is not None: log.error('get_http_transport does not support a timeout') if use_ssl: - ssl_ctx = ssl.create_default_context(cafile=ca_cert) - if ca_cert: - ssl_ctx.verify_mode = ssl.CERT_REQUIRED - else: - ssl_ctx.check_hostname = False # Mandated by the SSL lib for CERT_NONE mode. - ssl_ctx.verify_mode = ssl.CERT_NONE + ssl_ctx = get_ssl_context(ca_cert, verify_cert) - url = 'https://%s:%s/%s' % (host, port, http_path) + url = 'https://%s:%s/%s' % (host_url, port, http_path) log.debug('get_http_transport url=%s', url) # TODO(#362): Add server authentication with thrift 0.12. - transport = ImpalaHttpClient(url, ssl_context=ssl_ctx, - http_cookie_names=http_cookie_names) + transport = ImpalaHttpClient( + url, ssl_context=ssl_ctx, + http_cookie_names=http_cookie_names, + get_user_custom_headers_func=get_user_custom_headers_func) else: - url = 'http://%s:%s/%s' % (host, port, http_path) + url = 'http://%s:%s/%s' % (host_url, port, http_path) log.debug('get_http_transport url=%s', url) - transport = ImpalaHttpClient(url, http_cookie_names=http_cookie_names) + transport = ImpalaHttpClient( + url, http_cookie_names=http_cookie_names, + get_user_custom_headers_func=get_user_custom_headers_func) # set custom user agent if provided by user if user_agent: @@ -410,14 +445,12 @@ def get_http_transport(host, port, http_path, timeout=None, use_ssl=False, else: # PLAIN always requires a password for HS2. password = 'password' - log.debug('get_http_transport: password=%s', password) + log.debug('get_http_transport: password=%s', password) + else: + log.debug('get_http_transport: password=fuggetaboutit') auth_mechanism = 'PLAIN' # sasl doesn't know mechanism LDAP # Set the BASIC auth header - user_password = '%s:%s'.encode() % (user.encode(), password.encode()) - try: - auth = base64.encodebytes(user_password).decode().strip('\n') - except AttributeError: - auth = base64.encodestring(user_password).decode().strip('\n') + auth = get_basic_credentials_for_request_headers(user, password) def get_custom_headers(cookie_header, has_auth_cookie): custom_headers = {} @@ -436,7 +469,15 @@ def get_custom_headers(cookie_header, has_auth_cookie): elif auth_mechanism == 'GSSAPI': # For GSSAPI over http we need to dynamically generate custom request headers. def get_custom_headers(cookie_header, has_auth_cookie): - import kerberos + # Try importing kerberos, then winkerberos as fallback. Report the original exception for kerberos. + try: + import kerberos + except ImportError as original_ex: + try: + log.debug('importing kerberos failed, falling back to winkerberos') + import winkerberos as kerberos + except ImportError: + raise original_ex custom_headers = {} if cookie_header: log.debug('add cookies to HTTP header') diff --git a/impala/_thrift_gen/ErrorCodes/constants.py b/impala/_thrift_gen/ErrorCodes/constants.py index 1708ecdbf..89d0677f5 100644 --- a/impala/_thrift_gen/ErrorCodes/constants.py +++ b/impala/_thrift_gen/ErrorCodes/constants.py @@ -29,7 +29,7 @@ "Metadata states that in group $0($1) there are $2 rows, but $3 rows were read.", "(unused)", "File '$0' column '$1' does not have the decimal precision set.", - "File '$0' column '$1' has a precision that does not match the table metadata precision. File metadata precision: $2, table metadata precision: $3.", + "File '$0' column '$1' has a precision that does not match the table metadata precision. File metadata precision: $2, table metadata precision: $3.", "File '$0' column '$1' does not have converted type set to DECIMAL", "File '$0' column '$1' contains decimal data but the table metadata has type $2", "Problem parsing file $0 at $1$2", @@ -72,7 +72,7 @@ "Temporary file $0 is blacklisted from a previous error and cannot be expanded.", "RPC client failed to connect: $0", "Metadata for file '$0' appears stale. Try running \"refresh $1\" to reload the file metadata.", - "File '$0' has an invalid version number: $1\nThis could be due to stale metadata. Try running \"refresh $2\".", + "File '$0' has an invalid Parquet version number: $1.\nPlease check that it is a valid Parquet file. This error can also occur due to stale metadata. If you believe this is a valid Parquet file, try running \"refresh $2\".", "Tried to read $0 bytes but could only read $1 bytes. This may indicate data file corruption. (file $2, byte offset: $3)", "Invalid read of $0 bytes. This may indicate data file corruption. (file $1, byte offset: $2)", "File '$0' has an invalid version header: $1\nMake sure the file is an Avro data file.", @@ -86,7 +86,7 @@ "Sender$0 timed out waiting for receiver fragment instance: $1, dest node: $2", "Kudu type $0 is not available in Impala.", "Impala type $0 is not available in Kudu.", - "Kudu is not supported on this operating system.", + "Not in use.", "Kudu features are disabled by the startup flag --disable_kudu.", "Cannot perform hash join at node with id $0. Repartitioning did not reduce the size of a spilled partition. Repartitioning level $1. Number of rows $2:\n$3\n$4", "Not in use.", @@ -113,14 +113,14 @@ "Column '$0': invalid Avro decimal type with precision = '$1' scale = '$2'", "Row with null value violates nullability constraint on table '$0'.", "Parquet file '$0' column '$1' contains an out of range timestamp. The valid date range is 1400-01-01..9999-12-31.", - "Could not create files in any configured scratch directories (--scratch_dirs=$0) on backend '$1'. $2 of scratch is currently in use by this Impala Daemon ($3 by this query). See logs for previous errors that may have prevented creating or writing scratch files.", + "Could not create files in any configured scratch directories (--scratch_dirs=$0) on backend '$1'. $2 of scratch is currently in use by this Impala Daemon ($3 by this query). See logs for previous errors that may have prevented creating or writing scratch files. The following directories were at capacity: $4", "Error reading $0 bytes from scratch file '$1' on backend $2 at offset $3: could only read $4 bytes", "Kudu table '$0' column '$1' contains an out of range timestamp. The valid date range is 1400-01-01..9999-12-31.", - "Row of size $0 could not be materialized in plan node with id $1. Increase the max_row_size query option (currently $2) to process larger rows.", + "Row of size $0 could not be materialized by $1. Increase the max_row_size query option (currently $2) to process larger rows.", "Failed to verify generated IR function $0, see log for more details.", "Failed to get minimum memory reservation of $0 on daemon $1:$2 for query $3 due to following error: $4Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.", "Rejected query from pool $0: $1", - "Admission for query exceeded timeout $0ms in pool $1. Queued reason: $2", + "Admission for query exceeded timeout $0ms in pool $1. Queued reason: $2 Additional Details: $3", "Failed to create thread $0 in category $1: $2", "Disk I/O error on $0: $1", "DataStreamRecvr for fragment=$0, node=$1 is closed already", @@ -144,4 +144,34 @@ "Query $0 terminated due to CPU limit of $1", "Query $0 terminated due to scan bytes limit of $1", "Query $0 terminated due to rows produced limit of $1. Unset or increase NUM_ROWS_PRODUCED_LIMIT query option to produce more rows.", + "Expression rewrite rejected due to result size ($0) exceeding the limit ($1).", + "Query $0 cancelled due to unresponsive backend: $1 has not sent a report in $2ms (max allowed lag is $3ms)", + "Parquet file '$0' column '$1' contains an out of range date. The valid date range is 0001-01-01..9999-12-31.", + "Session closed because it has no active connections", + "The user authorized on the connection '$0' does not match the session username '$1'", + "$0 failed with error: $1", + "LZ4Block: Decompressed size is not correct.", + "LZ4Block: Invalid input length.", + "LZ4Block: Invalid compressed length. Data is likely corrupt.", + "LZ4: LZ4_decompress_safe failed", + "LZ4: LZ4_compress_default failed", + "Statement length of $0 bytes exceeds the maximum statement length ($1 bytes)", + "Avro file '$0' is corrupt: out of range date value $1 at offset $2. The valid date range is -719162..2932896 (0001-01-01..9999-12-31).", + "ORC file '$0' column '$1' contains an out of range timestamp. The valid date range is 1400-01-01..9999-12-31.", + "ORC file '$0' column '$1' contains an out of range date. The valid date range is 0001-01-01..9999-12-31.", + "File '$0' has an incompatible ORC schema for column '$1', Column type: $2, ORC schema: $3", + "Root of the $0 type returned by the ORC lib is not STRUCT: $1. Either there are bugs in the ORC lib or ORC file '$2' is corrupt.", + "Unable to perform Null-Aware Anti-Join. Could not get enough reservation to fit all rows with NULLs from the build side in memory. Memory required for $0 rows was $1. $2/$3 of the join's reservation was available for the rows.", + "Invalid or unknown query handle: $0.", + "Query $0 terminated due to join rows produced exceeds the limit of $1 at node with id $2. Unset or increase JOIN_ROWS_PRODUCED_LIMIT query option to produce more rows.", + "Query execution failure caused by local disk IO fatal error on backend: $0.", + "Error parsing JWKS: $0.", + "Error verifying JWT Token: $0.", + "Couldn't skip rows in column '$0' in file '$1'.", + "Failed to parse query option '$0': $1", + "Client has incompatible protocol version V$0 conflicting with catalogd's version V$1", + "Subscriber '$0' has incompatible protocol version V$1 conflicting with statestored's version V$2", + "Error in JDBC table configuration: $0.", + "Inconsistent tuple cache found: $0.", + "Error verifying OAuth Token: $0.", ] diff --git a/impala/_thrift_gen/ErrorCodes/ttypes.py b/impala/_thrift_gen/ErrorCodes/ttypes.py index df10cb06c..0d7df435e 100644 --- a/impala/_thrift_gen/ErrorCodes/ttypes.py +++ b/impala/_thrift_gen/ErrorCodes/ttypes.py @@ -148,6 +148,36 @@ class TErrorCode(object): CPU_LIMIT_EXCEEDED = 129 SCAN_BYTES_LIMIT_EXCEEDED = 130 ROWS_PRODUCED_LIMIT_EXCEEDED = 131 + EXPR_REWRITE_RESULT_LIMIT_EXCEEDED = 132 + UNRESPONSIVE_BACKEND = 133 + PARQUET_DATE_OUT_OF_RANGE = 134 + DISCONNECTED_SESSION_CLOSED = 135 + UNAUTHORIZED_SESSION_USER = 136 + ZSTD_ERROR = 137 + LZ4_BLOCK_DECOMPRESS_DECOMPRESS_SIZE_INCORRECT = 138 + LZ4_BLOCK_DECOMPRESS_INVALID_INPUT_LENGTH = 139 + LZ4_BLOCK_DECOMPRESS_INVALID_COMPRESSED_LENGTH = 140 + LZ4_DECOMPRESS_SAFE_FAILED = 141 + LZ4_COMPRESS_DEFAULT_FAILED = 142 + MAX_STATEMENT_LENGTH_EXCEEDED = 143 + AVRO_INVALID_DATE = 144 + ORC_TIMESTAMP_OUT_OF_RANGE = 145 + ORC_DATE_OUT_OF_RANGE = 146 + ORC_NESTED_TYPE_MISMATCH = 147 + ORC_TYPE_NOT_ROOT_AT_STRUCT = 148 + NAAJ_OUT_OF_MEMORY = 149 + INVALID_QUERY_HANDLE = 150 + JOIN_ROWS_PRODUCED_LIMIT_EXCEEDED = 151 + LOCAL_DISK_FAULTY = 152 + JWKS_PARSE_ERROR = 153 + JWT_VERIFY_FAILED = 154 + PARQUET_ROWS_SKIPPING = 155 + QUERY_OPTION_PARSE_FAILED = 156 + CATALOG_INCOMPATIBLE_PROTOCOL = 157 + STATESTORE_INCOMPATIBLE_PROTOCOL = 158 + JDBC_CONFIGURATION_ERROR = 159 + TUPLE_CACHE_INCONSISTENCY = 160 + OAUTH_VERIFY_FAILED = 161 _VALUES_TO_NAMES = { 0: "OK", @@ -282,6 +312,36 @@ class TErrorCode(object): 129: "CPU_LIMIT_EXCEEDED", 130: "SCAN_BYTES_LIMIT_EXCEEDED", 131: "ROWS_PRODUCED_LIMIT_EXCEEDED", + 132: "EXPR_REWRITE_RESULT_LIMIT_EXCEEDED", + 133: "UNRESPONSIVE_BACKEND", + 134: "PARQUET_DATE_OUT_OF_RANGE", + 135: "DISCONNECTED_SESSION_CLOSED", + 136: "UNAUTHORIZED_SESSION_USER", + 137: "ZSTD_ERROR", + 138: "LZ4_BLOCK_DECOMPRESS_DECOMPRESS_SIZE_INCORRECT", + 139: "LZ4_BLOCK_DECOMPRESS_INVALID_INPUT_LENGTH", + 140: "LZ4_BLOCK_DECOMPRESS_INVALID_COMPRESSED_LENGTH", + 141: "LZ4_DECOMPRESS_SAFE_FAILED", + 142: "LZ4_COMPRESS_DEFAULT_FAILED", + 143: "MAX_STATEMENT_LENGTH_EXCEEDED", + 144: "AVRO_INVALID_DATE", + 145: "ORC_TIMESTAMP_OUT_OF_RANGE", + 146: "ORC_DATE_OUT_OF_RANGE", + 147: "ORC_NESTED_TYPE_MISMATCH", + 148: "ORC_TYPE_NOT_ROOT_AT_STRUCT", + 149: "NAAJ_OUT_OF_MEMORY", + 150: "INVALID_QUERY_HANDLE", + 151: "JOIN_ROWS_PRODUCED_LIMIT_EXCEEDED", + 152: "LOCAL_DISK_FAULTY", + 153: "JWKS_PARSE_ERROR", + 154: "JWT_VERIFY_FAILED", + 155: "PARQUET_ROWS_SKIPPING", + 156: "QUERY_OPTION_PARSE_FAILED", + 157: "CATALOG_INCOMPATIBLE_PROTOCOL", + 158: "STATESTORE_INCOMPATIBLE_PROTOCOL", + 159: "JDBC_CONFIGURATION_ERROR", + 160: "TUPLE_CACHE_INCONSISTENCY", + 161: "OAUTH_VERIFY_FAILED", } _NAMES_TO_VALUES = { @@ -417,6 +477,36 @@ class TErrorCode(object): "CPU_LIMIT_EXCEEDED": 129, "SCAN_BYTES_LIMIT_EXCEEDED": 130, "ROWS_PRODUCED_LIMIT_EXCEEDED": 131, + "EXPR_REWRITE_RESULT_LIMIT_EXCEEDED": 132, + "UNRESPONSIVE_BACKEND": 133, + "PARQUET_DATE_OUT_OF_RANGE": 134, + "DISCONNECTED_SESSION_CLOSED": 135, + "UNAUTHORIZED_SESSION_USER": 136, + "ZSTD_ERROR": 137, + "LZ4_BLOCK_DECOMPRESS_DECOMPRESS_SIZE_INCORRECT": 138, + "LZ4_BLOCK_DECOMPRESS_INVALID_INPUT_LENGTH": 139, + "LZ4_BLOCK_DECOMPRESS_INVALID_COMPRESSED_LENGTH": 140, + "LZ4_DECOMPRESS_SAFE_FAILED": 141, + "LZ4_COMPRESS_DEFAULT_FAILED": 142, + "MAX_STATEMENT_LENGTH_EXCEEDED": 143, + "AVRO_INVALID_DATE": 144, + "ORC_TIMESTAMP_OUT_OF_RANGE": 145, + "ORC_DATE_OUT_OF_RANGE": 146, + "ORC_NESTED_TYPE_MISMATCH": 147, + "ORC_TYPE_NOT_ROOT_AT_STRUCT": 148, + "NAAJ_OUT_OF_MEMORY": 149, + "INVALID_QUERY_HANDLE": 150, + "JOIN_ROWS_PRODUCED_LIMIT_EXCEEDED": 151, + "LOCAL_DISK_FAULTY": 152, + "JWKS_PARSE_ERROR": 153, + "JWT_VERIFY_FAILED": 154, + "PARQUET_ROWS_SKIPPING": 155, + "QUERY_OPTION_PARSE_FAILED": 156, + "CATALOG_INCOMPATIBLE_PROTOCOL": 157, + "STATESTORE_INCOMPATIBLE_PROTOCOL": 158, + "JDBC_CONFIGURATION_ERROR": 159, + "TUPLE_CACHE_INCONSISTENCY": 160, + "OAUTH_VERIFY_FAILED": 161, } fix_spec(all_structs) del all_structs diff --git a/impala/_thrift_gen/ExecStats/ttypes.py b/impala/_thrift_gen/ExecStats/ttypes.py index 233de3fe7..5c106c545 100644 --- a/impala/_thrift_gen/ExecStats/ttypes.py +++ b/impala/_thrift_gen/ExecStats/ttypes.py @@ -148,11 +148,12 @@ class TPlanNodeExecSummary(object): - estimated_stats - exec_stats - is_broadcast + - num_hosts """ - def __init__(self, node_id=None, fragment_idx=None, label=None, label_detail=None, num_children=None, estimated_stats=None, exec_stats=None, is_broadcast=None,): + def __init__(self, node_id=None, fragment_idx=None, label=None, label_detail=None, num_children=None, estimated_stats=None, exec_stats=None, is_broadcast=None, num_hosts=None,): self.node_id = node_id self.fragment_idx = fragment_idx self.label = label @@ -161,6 +162,7 @@ def __init__(self, node_id=None, fragment_idx=None, label=None, label_detail=Non self.estimated_stats = estimated_stats self.exec_stats = exec_stats self.is_broadcast = is_broadcast + self.num_hosts = num_hosts def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -218,6 +220,11 @@ def read(self, iprot): self.is_broadcast = iprot.readBool() else: iprot.skip(ftype) + elif fid == 9: + if ftype == TType.I32: + self.num_hosts = iprot.readI32() + else: + iprot.skip(ftype) else: iprot.skip(ftype) iprot.readFieldEnd() @@ -263,6 +270,10 @@ def write(self, oprot): oprot.writeFieldBegin('is_broadcast', TType.BOOL, 8) oprot.writeBool(self.is_broadcast) oprot.writeFieldEnd() + if self.num_hosts is not None: + oprot.writeFieldBegin('num_hosts', TType.I32, 9) + oprot.writeI32(self.num_hosts) + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -294,13 +305,17 @@ class TExecProgress(object): Attributes: - total_scan_ranges - num_completed_scan_ranges + - total_fragment_instances + - num_completed_fragment_instances """ - def __init__(self, total_scan_ranges=None, num_completed_scan_ranges=None,): + def __init__(self, total_scan_ranges=None, num_completed_scan_ranges=None, total_fragment_instances=None, num_completed_fragment_instances=None,): self.total_scan_ranges = total_scan_ranges self.num_completed_scan_ranges = num_completed_scan_ranges + self.total_fragment_instances = total_fragment_instances + self.num_completed_fragment_instances = num_completed_fragment_instances def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -321,6 +336,16 @@ def read(self, iprot): self.num_completed_scan_ranges = iprot.readI64() else: iprot.skip(ftype) + elif fid == 3: + if ftype == TType.I64: + self.total_fragment_instances = iprot.readI64() + else: + iprot.skip(ftype) + elif fid == 4: + if ftype == TType.I64: + self.num_completed_fragment_instances = iprot.readI64() + else: + iprot.skip(ftype) else: iprot.skip(ftype) iprot.readFieldEnd() @@ -339,6 +364,14 @@ def write(self, oprot): oprot.writeFieldBegin('num_completed_scan_ranges', TType.I64, 2) oprot.writeI64(self.num_completed_scan_ranges) oprot.writeFieldEnd() + if self.total_fragment_instances is not None: + oprot.writeFieldBegin('total_fragment_instances', TType.I64, 3) + oprot.writeI64(self.total_fragment_instances) + oprot.writeFieldEnd() + if self.num_completed_fragment_instances is not None: + oprot.writeFieldBegin('num_completed_fragment_instances', TType.I64, 4) + oprot.writeI64(self.num_completed_fragment_instances) + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -539,12 +572,15 @@ def __ne__(self, other): (6, TType.STRUCT, 'estimated_stats', [TExecStats, None], None, ), # 6 (7, TType.LIST, 'exec_stats', (TType.STRUCT, [TExecStats, None], False), None, ), # 7 (8, TType.BOOL, 'is_broadcast', None, None, ), # 8 + (9, TType.I32, 'num_hosts', None, None, ), # 9 ) all_structs.append(TExecProgress) TExecProgress.thrift_spec = ( None, # 0 (1, TType.I64, 'total_scan_ranges', None, None, ), # 1 (2, TType.I64, 'num_completed_scan_ranges', None, None, ), # 2 + (3, TType.I64, 'total_fragment_instances', None, None, ), # 3 + (4, TType.I64, 'num_completed_fragment_instances', None, None, ), # 4 ) all_structs.append(TExecSummary) TExecSummary.thrift_spec = ( diff --git a/impala/_thrift_gen/ImpalaService/ImpalaHiveServer2Service-remote b/impala/_thrift_gen/ImpalaService/ImpalaHiveServer2Service-remote index 4801daceb..2cdb8c37a 100755 --- a/impala/_thrift_gen/ImpalaService/ImpalaHiveServer2Service-remote +++ b/impala/_thrift_gen/ImpalaService/ImpalaHiveServer2Service-remote @@ -26,6 +26,8 @@ if len(sys.argv) <= 1 or sys.argv[1] == '--help': print('Functions:') print(' TGetExecSummaryResp GetExecSummary(TGetExecSummaryReq req)') print(' TGetRuntimeProfileResp GetRuntimeProfile(TGetRuntimeProfileReq req)') + print(' TPingImpalaHS2ServiceResp PingImpalaHS2Service(TPingImpalaHS2ServiceReq req)') + print(' TCloseImpalaOperationResp CloseImpalaOperation(TCloseImpalaOperationReq req)') print(' TOpenSessionResp OpenSession(TOpenSessionReq req)') print(' TCloseSessionResp CloseSession(TCloseSessionReq req)') print(' TGetInfoResp GetInfo(TGetInfoReq req)') @@ -137,6 +139,18 @@ elif cmd == 'GetRuntimeProfile': sys.exit(1) pp.pprint(client.GetRuntimeProfile(eval(args[0]),)) +elif cmd == 'PingImpalaHS2Service': + if len(args) != 1: + print('PingImpalaHS2Service requires 1 args') + sys.exit(1) + pp.pprint(client.PingImpalaHS2Service(eval(args[0]),)) + +elif cmd == 'CloseImpalaOperation': + if len(args) != 1: + print('CloseImpalaOperation requires 1 args') + sys.exit(1) + pp.pprint(client.CloseImpalaOperation(eval(args[0]),)) + elif cmd == 'OpenSession': if len(args) != 1: print('OpenSession requires 1 args') diff --git a/impala/_thrift_gen/ImpalaService/ImpalaHiveServer2Service.py b/impala/_thrift_gen/ImpalaService/ImpalaHiveServer2Service.py index 7f9a585f1..0b8e734fd 100644 --- a/impala/_thrift_gen/ImpalaService/ImpalaHiveServer2Service.py +++ b/impala/_thrift_gen/ImpalaService/ImpalaHiveServer2Service.py @@ -35,6 +35,22 @@ def GetRuntimeProfile(self, req): """ pass + def PingImpalaHS2Service(self, req): + """ + Parameters: + - req + + """ + pass + + def CloseImpalaOperation(self, req): + """ + Parameters: + - req + + """ + pass + class Client(impala._thrift_gen.TCLIService.TCLIService.Client, Iface): def __init__(self, iprot, oprot=None): @@ -104,12 +120,78 @@ def recv_GetRuntimeProfile(self): return result.success raise TApplicationException(TApplicationException.MISSING_RESULT, "GetRuntimeProfile failed: unknown result") + def PingImpalaHS2Service(self, req): + """ + Parameters: + - req + + """ + self.send_PingImpalaHS2Service(req) + return self.recv_PingImpalaHS2Service() + + def send_PingImpalaHS2Service(self, req): + self._oprot.writeMessageBegin('PingImpalaHS2Service', TMessageType.CALL, self._seqid) + args = PingImpalaHS2Service_args() + args.req = req + args.write(self._oprot) + self._oprot.writeMessageEnd() + self._oprot.trans.flush() + + def recv_PingImpalaHS2Service(self): + iprot = self._iprot + (fname, mtype, rseqid) = iprot.readMessageBegin() + if mtype == TMessageType.EXCEPTION: + x = TApplicationException() + x.read(iprot) + iprot.readMessageEnd() + raise x + result = PingImpalaHS2Service_result() + result.read(iprot) + iprot.readMessageEnd() + if result.success is not None: + return result.success + raise TApplicationException(TApplicationException.MISSING_RESULT, "PingImpalaHS2Service failed: unknown result") + + def CloseImpalaOperation(self, req): + """ + Parameters: + - req + + """ + self.send_CloseImpalaOperation(req) + return self.recv_CloseImpalaOperation() + + def send_CloseImpalaOperation(self, req): + self._oprot.writeMessageBegin('CloseImpalaOperation', TMessageType.CALL, self._seqid) + args = CloseImpalaOperation_args() + args.req = req + args.write(self._oprot) + self._oprot.writeMessageEnd() + self._oprot.trans.flush() + + def recv_CloseImpalaOperation(self): + iprot = self._iprot + (fname, mtype, rseqid) = iprot.readMessageBegin() + if mtype == TMessageType.EXCEPTION: + x = TApplicationException() + x.read(iprot) + iprot.readMessageEnd() + raise x + result = CloseImpalaOperation_result() + result.read(iprot) + iprot.readMessageEnd() + if result.success is not None: + return result.success + raise TApplicationException(TApplicationException.MISSING_RESULT, "CloseImpalaOperation failed: unknown result") + class Processor(impala._thrift_gen.TCLIService.TCLIService.Processor, Iface, TProcessor): def __init__(self, handler): impala._thrift_gen.TCLIService.TCLIService.Processor.__init__(self, handler) self._processMap["GetExecSummary"] = Processor.process_GetExecSummary self._processMap["GetRuntimeProfile"] = Processor.process_GetRuntimeProfile + self._processMap["PingImpalaHS2Service"] = Processor.process_PingImpalaHS2Service + self._processMap["CloseImpalaOperation"] = Processor.process_CloseImpalaOperation self._on_message_begin = None def on_message_begin(self, func): @@ -178,6 +260,52 @@ def process_GetRuntimeProfile(self, seqid, iprot, oprot): oprot.writeMessageEnd() oprot.trans.flush() + def process_PingImpalaHS2Service(self, seqid, iprot, oprot): + args = PingImpalaHS2Service_args() + args.read(iprot) + iprot.readMessageEnd() + result = PingImpalaHS2Service_result() + try: + result.success = self._handler.PingImpalaHS2Service(args.req) + msg_type = TMessageType.REPLY + except TTransport.TTransportException: + raise + except TApplicationException as ex: + logging.exception('TApplication exception in handler') + msg_type = TMessageType.EXCEPTION + result = ex + except Exception: + logging.exception('Unexpected exception in handler') + msg_type = TMessageType.EXCEPTION + result = TApplicationException(TApplicationException.INTERNAL_ERROR, 'Internal error') + oprot.writeMessageBegin("PingImpalaHS2Service", msg_type, seqid) + result.write(oprot) + oprot.writeMessageEnd() + oprot.trans.flush() + + def process_CloseImpalaOperation(self, seqid, iprot, oprot): + args = CloseImpalaOperation_args() + args.read(iprot) + iprot.readMessageEnd() + result = CloseImpalaOperation_result() + try: + result.success = self._handler.CloseImpalaOperation(args.req) + msg_type = TMessageType.REPLY + except TTransport.TTransportException: + raise + except TApplicationException as ex: + logging.exception('TApplication exception in handler') + msg_type = TMessageType.EXCEPTION + result = ex + except Exception: + logging.exception('Unexpected exception in handler') + msg_type = TMessageType.EXCEPTION + result = TApplicationException(TApplicationException.INTERNAL_ERROR, 'Internal error') + oprot.writeMessageBegin("CloseImpalaOperation", msg_type, seqid) + result.write(oprot) + oprot.writeMessageEnd() + oprot.trans.flush() + # HELPER FUNCTIONS AND STRUCTURES @@ -429,5 +557,255 @@ def __ne__(self, other): GetRuntimeProfile_result.thrift_spec = ( (0, TType.STRUCT, 'success', [TGetRuntimeProfileResp, None], None, ), # 0 ) + + +class PingImpalaHS2Service_args(object): + """ + Attributes: + - req + + """ + + + def __init__(self, req=None,): + self.req = req + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRUCT: + self.req = TPingImpalaHS2ServiceReq() + self.req.read(iprot) + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('PingImpalaHS2Service_args') + if self.req is not None: + oprot.writeFieldBegin('req', TType.STRUCT, 1) + self.req.write(oprot) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) +all_structs.append(PingImpalaHS2Service_args) +PingImpalaHS2Service_args.thrift_spec = ( + None, # 0 + (1, TType.STRUCT, 'req', [TPingImpalaHS2ServiceReq, None], None, ), # 1 +) + + +class PingImpalaHS2Service_result(object): + """ + Attributes: + - success + + """ + + + def __init__(self, success=None,): + self.success = success + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 0: + if ftype == TType.STRUCT: + self.success = TPingImpalaHS2ServiceResp() + self.success.read(iprot) + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('PingImpalaHS2Service_result') + if self.success is not None: + oprot.writeFieldBegin('success', TType.STRUCT, 0) + self.success.write(oprot) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) +all_structs.append(PingImpalaHS2Service_result) +PingImpalaHS2Service_result.thrift_spec = ( + (0, TType.STRUCT, 'success', [TPingImpalaHS2ServiceResp, None], None, ), # 0 +) + + +class CloseImpalaOperation_args(object): + """ + Attributes: + - req + + """ + + + def __init__(self, req=None,): + self.req = req + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRUCT: + self.req = TCloseImpalaOperationReq() + self.req.read(iprot) + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('CloseImpalaOperation_args') + if self.req is not None: + oprot.writeFieldBegin('req', TType.STRUCT, 1) + self.req.write(oprot) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) +all_structs.append(CloseImpalaOperation_args) +CloseImpalaOperation_args.thrift_spec = ( + None, # 0 + (1, TType.STRUCT, 'req', [TCloseImpalaOperationReq, None], None, ), # 1 +) + + +class CloseImpalaOperation_result(object): + """ + Attributes: + - success + + """ + + + def __init__(self, success=None,): + self.success = success + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 0: + if ftype == TType.STRUCT: + self.success = TCloseImpalaOperationResp() + self.success.read(iprot) + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('CloseImpalaOperation_result') + if self.success is not None: + oprot.writeFieldBegin('success', TType.STRUCT, 0) + self.success.write(oprot) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) +all_structs.append(CloseImpalaOperation_result) +CloseImpalaOperation_result.thrift_spec = ( + (0, TType.STRUCT, 'success', [TCloseImpalaOperationResp, None], None, ), # 0 +) fix_spec(all_structs) del all_structs diff --git a/impala/_thrift_gen/ImpalaService/ImpalaService-remote b/impala/_thrift_gen/ImpalaService/ImpalaService-remote index f5c8215d0..b46370164 100755 --- a/impala/_thrift_gen/ImpalaService/ImpalaService-remote +++ b/impala/_thrift_gen/ImpalaService/ImpalaService-remote @@ -28,7 +28,7 @@ if len(sys.argv) <= 1 or sys.argv[1] == '--help': print(' TStatus ResetCatalog()') print(' TStatus ResetTable(TResetTableReq request)') print(' string GetRuntimeProfile(QueryHandle query_id)') - print(' TInsertResult CloseInsert(QueryHandle handle)') + print(' TDmlResult CloseInsert(QueryHandle handle)') print(' TPingImpalaServiceResp PingImpalaService()') print(' TExecSummary GetExecSummary(QueryHandle handle)') print(' QueryHandle query(Query query)') diff --git a/impala/_thrift_gen/ImpalaService/ImpalaService.py b/impala/_thrift_gen/ImpalaService/ImpalaService.py index 7eeb79c21..c5aeb56a1 100644 --- a/impala/_thrift_gen/ImpalaService/ImpalaService.py +++ b/impala/_thrift_gen/ImpalaService/ImpalaService.py @@ -1101,7 +1101,7 @@ def read(self, iprot): break if fid == 0: if ftype == TType.STRUCT: - self.success = TInsertResult() + self.success = TDmlResult() self.success.read(iprot) else: iprot.skip(ftype) @@ -1155,7 +1155,7 @@ def __ne__(self, other): return not (self == other) all_structs.append(CloseInsert_result) CloseInsert_result.thrift_spec = ( - (0, TType.STRUCT, 'success', [TInsertResult, None], None, ), # 0 + (0, TType.STRUCT, 'success', [TDmlResult, None], None, ), # 0 (1, TType.STRUCT, 'error', [impala._thrift_gen.beeswax.ttypes.QueryNotFoundException, None], None, ), # 1 (2, TType.STRUCT, 'error2', [impala._thrift_gen.beeswax.ttypes.BeeswaxException, None], None, ), # 2 ) diff --git a/impala/_thrift_gen/ImpalaService/ttypes.py b/impala/_thrift_gen/ImpalaService/ttypes.py index a39c839a0..ed46f5010 100644 --- a/impala/_thrift_gen/ImpalaService/ttypes.py +++ b/impala/_thrift_gen/ImpalaService/ttypes.py @@ -100,6 +100,114 @@ class TImpalaQueryOptions(object): NUM_REMOTE_EXECUTOR_CANDIDATES = 75 NUM_ROWS_PRODUCED_LIMIT = 76 PLANNER_TESTCASE_MODE = 77 + DEFAULT_FILE_FORMAT = 78 + PARQUET_TIMESTAMP_TYPE = 79 + PARQUET_READ_PAGE_INDEX = 80 + PARQUET_WRITE_PAGE_INDEX = 81 + PARQUET_PAGE_ROW_COUNT_LIMIT = 82 + DISABLE_HDFS_NUM_ROWS_ESTIMATE = 83 + DEFAULT_HINTS_INSERT_STATEMENT = 84 + SPOOL_QUERY_RESULTS = 85 + DEFAULT_TRANSACTIONAL_TYPE = 86 + STATEMENT_EXPRESSION_LIMIT = 87 + MAX_STATEMENT_LENGTH_BYTES = 88 + DISABLE_DATA_CACHE = 89 + MAX_RESULT_SPOOLING_MEM = 90 + MAX_SPILLED_RESULT_SPOOLING_MEM = 91 + DISABLE_HBASE_NUM_ROWS_ESTIMATE = 92 + FETCH_ROWS_TIMEOUT_MS = 93 + NOW_STRING = 94 + PARQUET_OBJECT_STORE_SPLIT_SIZE = 95 + MEM_LIMIT_EXECUTORS = 96 + BROADCAST_BYTES_LIMIT = 97 + PREAGG_BYTES_LIMIT = 98 + ENABLE_CNF_REWRITES = 99 + MAX_CNF_EXPRS = 100 + KUDU_SNAPSHOT_READ_TIMESTAMP_MICROS = 101 + RETRY_FAILED_QUERIES = 102 + ENABLED_RUNTIME_FILTER_TYPES = 103 + ASYNC_CODEGEN = 104 + ENABLE_DISTINCT_SEMI_JOIN_OPTIMIZATION = 105 + SORT_RUN_BYTES_LIMIT = 106 + MAX_FS_WRITERS = 107 + REFRESH_UPDATED_HMS_PARTITIONS = 108 + SPOOL_ALL_RESULTS_FOR_RETRIES = 109 + RUNTIME_FILTER_ERROR_RATE = 110 + USE_LOCAL_TZ_FOR_UNIX_TIMESTAMP_CONVERSIONS = 111 + CONVERT_LEGACY_HIVE_PARQUET_UTC_TIMESTAMPS = 112 + ENABLE_OUTER_JOIN_TO_INNER_TRANSFORMATION = 113 + TARGETED_KUDU_SCAN_RANGE_LENGTH = 114 + REPORT_SKEW_LIMIT = 115 + OPTIMIZE_SIMPLE_LIMIT = 116 + USE_DOP_FOR_COSTING = 117 + BROADCAST_TO_PARTITION_FACTOR = 118 + JOIN_ROWS_PRODUCED_LIMIT = 119 + UTF8_MODE = 120 + ANALYTIC_RANK_PUSHDOWN_THRESHOLD = 121 + MINMAX_FILTER_THRESHOLD = 122 + MINMAX_FILTERING_LEVEL = 123 + COMPUTE_COLUMN_MINMAX_STATS = 124 + SHOW_COLUMN_MINMAX_STATS = 125 + DEFAULT_NDV_SCALE = 126 + KUDU_REPLICA_SELECTION = 127 + DELETE_STATS_IN_TRUNCATE = 128 + PARQUET_BLOOM_FILTERING = 129 + MINMAX_FILTER_SORTED_COLUMNS = 130 + MINMAX_FILTER_FAST_CODE_PATH = 131 + ENABLE_KUDU_TRANSACTION = 132 + MINMAX_FILTER_PARTITION_COLUMNS = 133 + PARQUET_BLOOM_FILTER_WRITE = 134 + ORC_READ_STATISTICS = 135 + ENABLE_ASYNC_DDL_EXECUTION = 136 + ENABLE_ASYNC_LOAD_DATA_EXECUTION = 137 + PARQUET_LATE_MATERIALIZATION_THRESHOLD = 138 + PARQUET_DICTIONARY_RUNTIME_FILTER_ENTRY_LIMIT = 139 + ABORT_JAVA_UDF_ON_EXCEPTION = 140 + ORC_ASYNC_READ = 141 + RUNTIME_IN_LIST_FILTER_ENTRY_LIMIT = 142 + ENABLE_REPLAN = 143 + TEST_REPLAN = 144 + LOCK_MAX_WAIT_TIME_S = 145 + ORC_SCHEMA_RESOLUTION = 146 + EXPAND_COMPLEX_TYPES = 147 + FALLBACK_DB_FOR_FUNCTIONS = 148 + DISABLE_CODEGEN_CACHE = 149 + CODEGEN_CACHE_MODE = 150 + STRINGIFY_MAP_KEYS = 151 + ENABLE_TRIVIAL_QUERY_FOR_ADMISSION = 152 + COMPUTE_PROCESSING_COST = 153 + PROCESSING_COST_MIN_THREADS = 154 + JOIN_SELECTIVITY_CORRELATION_FACTOR = 155 + MAX_FRAGMENT_INSTANCES_PER_NODE = 156 + MAX_SORT_RUN_SIZE = 157 + ALLOW_UNSAFE_CASTS = 158 + NUM_THREADS_FOR_TABLE_MIGRATION = 159 + DISABLE_OPTIMIZED_ICEBERG_V2_READ = 160 + VALUES_STMT_AVOID_LOSSY_CHAR_PADDING = 161 + LARGE_AGG_MEM_THRESHOLD = 162 + AGG_MEM_CORRELATION_FACTOR = 163 + MEM_LIMIT_COORDINATORS = 164 + ICEBERG_PREDICATE_PUSHDOWN_SUBSETTING = 165 + HDFS_SCANNER_NON_RESERVED_BYTES = 166 + CODEGEN_OPT_LEVEL = 167 + KUDU_TABLE_RESERVE_SECONDS = 168 + CONVERT_KUDU_UTC_TIMESTAMPS = 169 + DISABLE_KUDU_LOCAL_TIMESTAMP_BLOOM_FILTER = 170 + RUNTIME_FILTER_CARDINALITY_REDUCTION_SCALE = 171 + MAX_NUM_FILTERS_AGGREGATED_PER_HOST = 172 + QUERY_CPU_COUNT_DIVISOR = 173 + ENABLE_TUPLE_CACHE = 174 + ICEBERG_DISABLE_COUNT_STAR_OPTIMIZATION = 175 + RUNTIME_FILTER_IDS_TO_SKIP = 176 + SLOT_COUNT_STRATEGY = 177 + CLEAN_DBCP_DS_CACHE = 178 + USE_NULL_SLOTS_CACHE = 179 + WRITE_KUDU_UTC_TIMESTAMPS = 180 + DISABLE_OPTIMIZED_JSON_COUNT_STAR = 181 + LONG_POLLING_TIME_MS = 182 + ENABLE_TUPLE_CACHE_VERIFICATION = 183 + ENABLE_TUPLE_ANALYSIS_IN_AGGREGATE = 184 + ESTIMATE_DUPLICATE_IN_PREAGG = 185 _VALUES_TO_NAMES = { 0: "ABORT_ON_ERROR", @@ -180,6 +288,114 @@ class TImpalaQueryOptions(object): 75: "NUM_REMOTE_EXECUTOR_CANDIDATES", 76: "NUM_ROWS_PRODUCED_LIMIT", 77: "PLANNER_TESTCASE_MODE", + 78: "DEFAULT_FILE_FORMAT", + 79: "PARQUET_TIMESTAMP_TYPE", + 80: "PARQUET_READ_PAGE_INDEX", + 81: "PARQUET_WRITE_PAGE_INDEX", + 82: "PARQUET_PAGE_ROW_COUNT_LIMIT", + 83: "DISABLE_HDFS_NUM_ROWS_ESTIMATE", + 84: "DEFAULT_HINTS_INSERT_STATEMENT", + 85: "SPOOL_QUERY_RESULTS", + 86: "DEFAULT_TRANSACTIONAL_TYPE", + 87: "STATEMENT_EXPRESSION_LIMIT", + 88: "MAX_STATEMENT_LENGTH_BYTES", + 89: "DISABLE_DATA_CACHE", + 90: "MAX_RESULT_SPOOLING_MEM", + 91: "MAX_SPILLED_RESULT_SPOOLING_MEM", + 92: "DISABLE_HBASE_NUM_ROWS_ESTIMATE", + 93: "FETCH_ROWS_TIMEOUT_MS", + 94: "NOW_STRING", + 95: "PARQUET_OBJECT_STORE_SPLIT_SIZE", + 96: "MEM_LIMIT_EXECUTORS", + 97: "BROADCAST_BYTES_LIMIT", + 98: "PREAGG_BYTES_LIMIT", + 99: "ENABLE_CNF_REWRITES", + 100: "MAX_CNF_EXPRS", + 101: "KUDU_SNAPSHOT_READ_TIMESTAMP_MICROS", + 102: "RETRY_FAILED_QUERIES", + 103: "ENABLED_RUNTIME_FILTER_TYPES", + 104: "ASYNC_CODEGEN", + 105: "ENABLE_DISTINCT_SEMI_JOIN_OPTIMIZATION", + 106: "SORT_RUN_BYTES_LIMIT", + 107: "MAX_FS_WRITERS", + 108: "REFRESH_UPDATED_HMS_PARTITIONS", + 109: "SPOOL_ALL_RESULTS_FOR_RETRIES", + 110: "RUNTIME_FILTER_ERROR_RATE", + 111: "USE_LOCAL_TZ_FOR_UNIX_TIMESTAMP_CONVERSIONS", + 112: "CONVERT_LEGACY_HIVE_PARQUET_UTC_TIMESTAMPS", + 113: "ENABLE_OUTER_JOIN_TO_INNER_TRANSFORMATION", + 114: "TARGETED_KUDU_SCAN_RANGE_LENGTH", + 115: "REPORT_SKEW_LIMIT", + 116: "OPTIMIZE_SIMPLE_LIMIT", + 117: "USE_DOP_FOR_COSTING", + 118: "BROADCAST_TO_PARTITION_FACTOR", + 119: "JOIN_ROWS_PRODUCED_LIMIT", + 120: "UTF8_MODE", + 121: "ANALYTIC_RANK_PUSHDOWN_THRESHOLD", + 122: "MINMAX_FILTER_THRESHOLD", + 123: "MINMAX_FILTERING_LEVEL", + 124: "COMPUTE_COLUMN_MINMAX_STATS", + 125: "SHOW_COLUMN_MINMAX_STATS", + 126: "DEFAULT_NDV_SCALE", + 127: "KUDU_REPLICA_SELECTION", + 128: "DELETE_STATS_IN_TRUNCATE", + 129: "PARQUET_BLOOM_FILTERING", + 130: "MINMAX_FILTER_SORTED_COLUMNS", + 131: "MINMAX_FILTER_FAST_CODE_PATH", + 132: "ENABLE_KUDU_TRANSACTION", + 133: "MINMAX_FILTER_PARTITION_COLUMNS", + 134: "PARQUET_BLOOM_FILTER_WRITE", + 135: "ORC_READ_STATISTICS", + 136: "ENABLE_ASYNC_DDL_EXECUTION", + 137: "ENABLE_ASYNC_LOAD_DATA_EXECUTION", + 138: "PARQUET_LATE_MATERIALIZATION_THRESHOLD", + 139: "PARQUET_DICTIONARY_RUNTIME_FILTER_ENTRY_LIMIT", + 140: "ABORT_JAVA_UDF_ON_EXCEPTION", + 141: "ORC_ASYNC_READ", + 142: "RUNTIME_IN_LIST_FILTER_ENTRY_LIMIT", + 143: "ENABLE_REPLAN", + 144: "TEST_REPLAN", + 145: "LOCK_MAX_WAIT_TIME_S", + 146: "ORC_SCHEMA_RESOLUTION", + 147: "EXPAND_COMPLEX_TYPES", + 148: "FALLBACK_DB_FOR_FUNCTIONS", + 149: "DISABLE_CODEGEN_CACHE", + 150: "CODEGEN_CACHE_MODE", + 151: "STRINGIFY_MAP_KEYS", + 152: "ENABLE_TRIVIAL_QUERY_FOR_ADMISSION", + 153: "COMPUTE_PROCESSING_COST", + 154: "PROCESSING_COST_MIN_THREADS", + 155: "JOIN_SELECTIVITY_CORRELATION_FACTOR", + 156: "MAX_FRAGMENT_INSTANCES_PER_NODE", + 157: "MAX_SORT_RUN_SIZE", + 158: "ALLOW_UNSAFE_CASTS", + 159: "NUM_THREADS_FOR_TABLE_MIGRATION", + 160: "DISABLE_OPTIMIZED_ICEBERG_V2_READ", + 161: "VALUES_STMT_AVOID_LOSSY_CHAR_PADDING", + 162: "LARGE_AGG_MEM_THRESHOLD", + 163: "AGG_MEM_CORRELATION_FACTOR", + 164: "MEM_LIMIT_COORDINATORS", + 165: "ICEBERG_PREDICATE_PUSHDOWN_SUBSETTING", + 166: "HDFS_SCANNER_NON_RESERVED_BYTES", + 167: "CODEGEN_OPT_LEVEL", + 168: "KUDU_TABLE_RESERVE_SECONDS", + 169: "CONVERT_KUDU_UTC_TIMESTAMPS", + 170: "DISABLE_KUDU_LOCAL_TIMESTAMP_BLOOM_FILTER", + 171: "RUNTIME_FILTER_CARDINALITY_REDUCTION_SCALE", + 172: "MAX_NUM_FILTERS_AGGREGATED_PER_HOST", + 173: "QUERY_CPU_COUNT_DIVISOR", + 174: "ENABLE_TUPLE_CACHE", + 175: "ICEBERG_DISABLE_COUNT_STAR_OPTIMIZATION", + 176: "RUNTIME_FILTER_IDS_TO_SKIP", + 177: "SLOT_COUNT_STRATEGY", + 178: "CLEAN_DBCP_DS_CACHE", + 179: "USE_NULL_SLOTS_CACHE", + 180: "WRITE_KUDU_UTC_TIMESTAMPS", + 181: "DISABLE_OPTIMIZED_JSON_COUNT_STAR", + 182: "LONG_POLLING_TIME_MS", + 183: "ENABLE_TUPLE_CACHE_VERIFICATION", + 184: "ENABLE_TUPLE_ANALYSIS_IN_AGGREGATE", + 185: "ESTIMATE_DUPLICATE_IN_PREAGG", } _NAMES_TO_VALUES = { @@ -261,20 +477,130 @@ class TImpalaQueryOptions(object): "NUM_REMOTE_EXECUTOR_CANDIDATES": 75, "NUM_ROWS_PRODUCED_LIMIT": 76, "PLANNER_TESTCASE_MODE": 77, + "DEFAULT_FILE_FORMAT": 78, + "PARQUET_TIMESTAMP_TYPE": 79, + "PARQUET_READ_PAGE_INDEX": 80, + "PARQUET_WRITE_PAGE_INDEX": 81, + "PARQUET_PAGE_ROW_COUNT_LIMIT": 82, + "DISABLE_HDFS_NUM_ROWS_ESTIMATE": 83, + "DEFAULT_HINTS_INSERT_STATEMENT": 84, + "SPOOL_QUERY_RESULTS": 85, + "DEFAULT_TRANSACTIONAL_TYPE": 86, + "STATEMENT_EXPRESSION_LIMIT": 87, + "MAX_STATEMENT_LENGTH_BYTES": 88, + "DISABLE_DATA_CACHE": 89, + "MAX_RESULT_SPOOLING_MEM": 90, + "MAX_SPILLED_RESULT_SPOOLING_MEM": 91, + "DISABLE_HBASE_NUM_ROWS_ESTIMATE": 92, + "FETCH_ROWS_TIMEOUT_MS": 93, + "NOW_STRING": 94, + "PARQUET_OBJECT_STORE_SPLIT_SIZE": 95, + "MEM_LIMIT_EXECUTORS": 96, + "BROADCAST_BYTES_LIMIT": 97, + "PREAGG_BYTES_LIMIT": 98, + "ENABLE_CNF_REWRITES": 99, + "MAX_CNF_EXPRS": 100, + "KUDU_SNAPSHOT_READ_TIMESTAMP_MICROS": 101, + "RETRY_FAILED_QUERIES": 102, + "ENABLED_RUNTIME_FILTER_TYPES": 103, + "ASYNC_CODEGEN": 104, + "ENABLE_DISTINCT_SEMI_JOIN_OPTIMIZATION": 105, + "SORT_RUN_BYTES_LIMIT": 106, + "MAX_FS_WRITERS": 107, + "REFRESH_UPDATED_HMS_PARTITIONS": 108, + "SPOOL_ALL_RESULTS_FOR_RETRIES": 109, + "RUNTIME_FILTER_ERROR_RATE": 110, + "USE_LOCAL_TZ_FOR_UNIX_TIMESTAMP_CONVERSIONS": 111, + "CONVERT_LEGACY_HIVE_PARQUET_UTC_TIMESTAMPS": 112, + "ENABLE_OUTER_JOIN_TO_INNER_TRANSFORMATION": 113, + "TARGETED_KUDU_SCAN_RANGE_LENGTH": 114, + "REPORT_SKEW_LIMIT": 115, + "OPTIMIZE_SIMPLE_LIMIT": 116, + "USE_DOP_FOR_COSTING": 117, + "BROADCAST_TO_PARTITION_FACTOR": 118, + "JOIN_ROWS_PRODUCED_LIMIT": 119, + "UTF8_MODE": 120, + "ANALYTIC_RANK_PUSHDOWN_THRESHOLD": 121, + "MINMAX_FILTER_THRESHOLD": 122, + "MINMAX_FILTERING_LEVEL": 123, + "COMPUTE_COLUMN_MINMAX_STATS": 124, + "SHOW_COLUMN_MINMAX_STATS": 125, + "DEFAULT_NDV_SCALE": 126, + "KUDU_REPLICA_SELECTION": 127, + "DELETE_STATS_IN_TRUNCATE": 128, + "PARQUET_BLOOM_FILTERING": 129, + "MINMAX_FILTER_SORTED_COLUMNS": 130, + "MINMAX_FILTER_FAST_CODE_PATH": 131, + "ENABLE_KUDU_TRANSACTION": 132, + "MINMAX_FILTER_PARTITION_COLUMNS": 133, + "PARQUET_BLOOM_FILTER_WRITE": 134, + "ORC_READ_STATISTICS": 135, + "ENABLE_ASYNC_DDL_EXECUTION": 136, + "ENABLE_ASYNC_LOAD_DATA_EXECUTION": 137, + "PARQUET_LATE_MATERIALIZATION_THRESHOLD": 138, + "PARQUET_DICTIONARY_RUNTIME_FILTER_ENTRY_LIMIT": 139, + "ABORT_JAVA_UDF_ON_EXCEPTION": 140, + "ORC_ASYNC_READ": 141, + "RUNTIME_IN_LIST_FILTER_ENTRY_LIMIT": 142, + "ENABLE_REPLAN": 143, + "TEST_REPLAN": 144, + "LOCK_MAX_WAIT_TIME_S": 145, + "ORC_SCHEMA_RESOLUTION": 146, + "EXPAND_COMPLEX_TYPES": 147, + "FALLBACK_DB_FOR_FUNCTIONS": 148, + "DISABLE_CODEGEN_CACHE": 149, + "CODEGEN_CACHE_MODE": 150, + "STRINGIFY_MAP_KEYS": 151, + "ENABLE_TRIVIAL_QUERY_FOR_ADMISSION": 152, + "COMPUTE_PROCESSING_COST": 153, + "PROCESSING_COST_MIN_THREADS": 154, + "JOIN_SELECTIVITY_CORRELATION_FACTOR": 155, + "MAX_FRAGMENT_INSTANCES_PER_NODE": 156, + "MAX_SORT_RUN_SIZE": 157, + "ALLOW_UNSAFE_CASTS": 158, + "NUM_THREADS_FOR_TABLE_MIGRATION": 159, + "DISABLE_OPTIMIZED_ICEBERG_V2_READ": 160, + "VALUES_STMT_AVOID_LOSSY_CHAR_PADDING": 161, + "LARGE_AGG_MEM_THRESHOLD": 162, + "AGG_MEM_CORRELATION_FACTOR": 163, + "MEM_LIMIT_COORDINATORS": 164, + "ICEBERG_PREDICATE_PUSHDOWN_SUBSETTING": 165, + "HDFS_SCANNER_NON_RESERVED_BYTES": 166, + "CODEGEN_OPT_LEVEL": 167, + "KUDU_TABLE_RESERVE_SECONDS": 168, + "CONVERT_KUDU_UTC_TIMESTAMPS": 169, + "DISABLE_KUDU_LOCAL_TIMESTAMP_BLOOM_FILTER": 170, + "RUNTIME_FILTER_CARDINALITY_REDUCTION_SCALE": 171, + "MAX_NUM_FILTERS_AGGREGATED_PER_HOST": 172, + "QUERY_CPU_COUNT_DIVISOR": 173, + "ENABLE_TUPLE_CACHE": 174, + "ICEBERG_DISABLE_COUNT_STAR_OPTIMIZATION": 175, + "RUNTIME_FILTER_IDS_TO_SKIP": 176, + "SLOT_COUNT_STRATEGY": 177, + "CLEAN_DBCP_DS_CACHE": 178, + "USE_NULL_SLOTS_CACHE": 179, + "WRITE_KUDU_UTC_TIMESTAMPS": 180, + "DISABLE_OPTIMIZED_JSON_COUNT_STAR": 181, + "LONG_POLLING_TIME_MS": 182, + "ENABLE_TUPLE_CACHE_VERIFICATION": 183, + "ENABLE_TUPLE_ANALYSIS_IN_AGGREGATE": 184, + "ESTIMATE_DUPLICATE_IN_PREAGG": 185, } -class TInsertResult(object): +class TDmlResult(object): """ Attributes: - rows_modified + - rows_deleted - num_row_errors """ - def __init__(self, rows_modified=None, num_row_errors=None,): + def __init__(self, rows_modified=None, rows_deleted=None, num_row_errors=None,): self.rows_modified = rows_modified + self.rows_deleted = rows_deleted self.num_row_errors = num_row_errors def read(self, iprot): @@ -297,6 +623,17 @@ def read(self, iprot): iprot.readMapEnd() else: iprot.skip(ftype) + elif fid == 3: + if ftype == TType.MAP: + self.rows_deleted = {} + (_ktype8, _vtype9, _size7) = iprot.readMapBegin() + for _i11 in range(_size7): + _key12 = iprot.readString() + _val13 = iprot.readI64() + self.rows_deleted[_key12] = _val13 + iprot.readMapEnd() + else: + iprot.skip(ftype) elif fid == 2: if ftype == TType.I64: self.num_row_errors = iprot.readI64() @@ -311,19 +648,27 @@ def write(self, oprot): if oprot._fast_encode is not None and self.thrift_spec is not None: oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) return - oprot.writeStructBegin('TInsertResult') + oprot.writeStructBegin('TDmlResult') if self.rows_modified is not None: oprot.writeFieldBegin('rows_modified', TType.MAP, 1) oprot.writeMapBegin(TType.STRING, TType.I64, len(self.rows_modified)) - for kiter7, viter8 in self.rows_modified.items(): - oprot.writeString(kiter7) - oprot.writeI64(viter8) + for kiter14, viter15 in self.rows_modified.items(): + oprot.writeString(kiter14) + oprot.writeI64(viter15) oprot.writeMapEnd() oprot.writeFieldEnd() if self.num_row_errors is not None: oprot.writeFieldBegin('num_row_errors', TType.I64, 2) oprot.writeI64(self.num_row_errors) oprot.writeFieldEnd() + if self.rows_deleted is not None: + oprot.writeFieldBegin('rows_deleted', TType.MAP, 3) + oprot.writeMapBegin(TType.STRING, TType.I64, len(self.rows_deleted)) + for kiter16, viter17 in self.rows_deleted.items(): + oprot.writeString(kiter16) + oprot.writeI64(viter17) + oprot.writeMapEnd() + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -484,18 +829,305 @@ def __ne__(self, other): return not (self == other) +class TPingImpalaHS2ServiceReq(object): + """ + Attributes: + - sessionHandle + + """ + + + def __init__(self, sessionHandle=None,): + self.sessionHandle = sessionHandle + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRUCT: + self.sessionHandle = impala._thrift_gen.TCLIService.ttypes.TSessionHandle() + self.sessionHandle.read(iprot) + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('TPingImpalaHS2ServiceReq') + if self.sessionHandle is not None: + oprot.writeFieldBegin('sessionHandle', TType.STRUCT, 1) + self.sessionHandle.write(oprot) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + if self.sessionHandle is None: + raise TProtocolException(message='Required field sessionHandle is unset!') + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + +class TPingImpalaHS2ServiceResp(object): + """ + Attributes: + - status + - version + - webserver_address + - timestamp + + """ + + + def __init__(self, status=None, version=None, webserver_address=None, timestamp=None,): + self.status = status + self.version = version + self.webserver_address = webserver_address + self.timestamp = timestamp + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRUCT: + self.status = impala._thrift_gen.TCLIService.ttypes.TStatus() + self.status.read(iprot) + else: + iprot.skip(ftype) + elif fid == 2: + if ftype == TType.STRING: + self.version = iprot.readString() + else: + iprot.skip(ftype) + elif fid == 3: + if ftype == TType.STRING: + self.webserver_address = iprot.readString() + else: + iprot.skip(ftype) + elif fid == 4: + if ftype == TType.I64: + self.timestamp = iprot.readI64() + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('TPingImpalaHS2ServiceResp') + if self.status is not None: + oprot.writeFieldBegin('status', TType.STRUCT, 1) + self.status.write(oprot) + oprot.writeFieldEnd() + if self.version is not None: + oprot.writeFieldBegin('version', TType.STRING, 2) + oprot.writeString(self.version) + oprot.writeFieldEnd() + if self.webserver_address is not None: + oprot.writeFieldBegin('webserver_address', TType.STRING, 3) + oprot.writeString(self.webserver_address) + oprot.writeFieldEnd() + if self.timestamp is not None: + oprot.writeFieldBegin('timestamp', TType.I64, 4) + oprot.writeI64(self.timestamp) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + if self.status is None: + raise TProtocolException(message='Required field status is unset!') + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + +class TCloseImpalaOperationReq(object): + """ + Attributes: + - operationHandle + + """ + + + def __init__(self, operationHandle=None,): + self.operationHandle = operationHandle + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRUCT: + self.operationHandle = impala._thrift_gen.TCLIService.ttypes.TOperationHandle() + self.operationHandle.read(iprot) + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('TCloseImpalaOperationReq') + if self.operationHandle is not None: + oprot.writeFieldBegin('operationHandle', TType.STRUCT, 1) + self.operationHandle.write(oprot) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + if self.operationHandle is None: + raise TProtocolException(message='Required field operationHandle is unset!') + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + +class TCloseImpalaOperationResp(object): + """ + Attributes: + - status + - dml_result + + """ + + + def __init__(self, status=None, dml_result=None,): + self.status = status + self.dml_result = dml_result + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRUCT: + self.status = impala._thrift_gen.TCLIService.ttypes.TStatus() + self.status.read(iprot) + else: + iprot.skip(ftype) + elif fid == 2: + if ftype == TType.STRUCT: + self.dml_result = TDmlResult() + self.dml_result.read(iprot) + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('TCloseImpalaOperationResp') + if self.status is not None: + oprot.writeFieldBegin('status', TType.STRUCT, 1) + self.status.write(oprot) + oprot.writeFieldEnd() + if self.dml_result is not None: + oprot.writeFieldBegin('dml_result', TType.STRUCT, 2) + self.dml_result.write(oprot) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + if self.status is None: + raise TProtocolException(message='Required field status is unset!') + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + class TGetExecSummaryReq(object): """ Attributes: - operationHandle - sessionHandle + - include_query_attempts """ - def __init__(self, operationHandle=None, sessionHandle=None,): + def __init__(self, operationHandle=None, sessionHandle=None, include_query_attempts=False,): self.operationHandle = operationHandle self.sessionHandle = sessionHandle + self.include_query_attempts = include_query_attempts def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -518,6 +1150,11 @@ def read(self, iprot): self.sessionHandle.read(iprot) else: iprot.skip(ftype) + elif fid == 3: + if ftype == TType.BOOL: + self.include_query_attempts = iprot.readBool() + else: + iprot.skip(ftype) else: iprot.skip(ftype) iprot.readFieldEnd() @@ -536,6 +1173,10 @@ def write(self, oprot): oprot.writeFieldBegin('sessionHandle', TType.STRUCT, 2) self.sessionHandle.write(oprot) oprot.writeFieldEnd() + if self.include_query_attempts is not None: + oprot.writeFieldBegin('include_query_attempts', TType.BOOL, 3) + oprot.writeBool(self.include_query_attempts) + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -559,13 +1200,15 @@ class TGetExecSummaryResp(object): Attributes: - status - summary + - failed_summaries """ - def __init__(self, status=None, summary=None,): + def __init__(self, status=None, summary=None, failed_summaries=None,): self.status = status self.summary = summary + self.failed_summaries = failed_summaries def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -588,6 +1231,17 @@ def read(self, iprot): self.summary.read(iprot) else: iprot.skip(ftype) + elif fid == 3: + if ftype == TType.LIST: + self.failed_summaries = [] + (_etype21, _size18) = iprot.readListBegin() + for _i22 in range(_size18): + _elem23 = impala._thrift_gen.ExecStats.ttypes.TExecSummary() + _elem23.read(iprot) + self.failed_summaries.append(_elem23) + iprot.readListEnd() + else: + iprot.skip(ftype) else: iprot.skip(ftype) iprot.readFieldEnd() @@ -606,6 +1260,13 @@ def write(self, oprot): oprot.writeFieldBegin('summary', TType.STRUCT, 2) self.summary.write(oprot) oprot.writeFieldEnd() + if self.failed_summaries is not None: + oprot.writeFieldBegin('failed_summaries', TType.LIST, 3) + oprot.writeListBegin(TType.STRUCT, len(self.failed_summaries)) + for iter24 in self.failed_summaries: + iter24.write(oprot) + oprot.writeListEnd() + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -632,14 +1293,16 @@ class TGetRuntimeProfileReq(object): - operationHandle - sessionHandle - format + - include_query_attempts """ - def __init__(self, operationHandle=None, sessionHandle=None, format=0,): + def __init__(self, operationHandle=None, sessionHandle=None, format=0, include_query_attempts=False,): self.operationHandle = operationHandle self.sessionHandle = sessionHandle self.format = format + self.include_query_attempts = include_query_attempts def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -667,6 +1330,11 @@ def read(self, iprot): self.format = iprot.readI32() else: iprot.skip(ftype) + elif fid == 4: + if ftype == TType.BOOL: + self.include_query_attempts = iprot.readBool() + else: + iprot.skip(ftype) else: iprot.skip(ftype) iprot.readFieldEnd() @@ -689,6 +1357,10 @@ def write(self, oprot): oprot.writeFieldBegin('format', TType.I32, 3) oprot.writeI32(self.format) oprot.writeFieldEnd() + if self.include_query_attempts is not None: + oprot.writeFieldBegin('include_query_attempts', TType.BOOL, 4) + oprot.writeBool(self.include_query_attempts) + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -713,14 +1385,18 @@ class TGetRuntimeProfileResp(object): - status - profile - thrift_profile + - failed_profiles + - failed_thrift_profiles """ - def __init__(self, status=None, profile=None, thrift_profile=None,): + def __init__(self, status=None, profile=None, thrift_profile=None, failed_profiles=None, failed_thrift_profiles=None,): self.status = status self.profile = profile self.thrift_profile = thrift_profile + self.failed_profiles = failed_profiles + self.failed_thrift_profiles = failed_thrift_profiles def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -748,6 +1424,27 @@ def read(self, iprot): self.thrift_profile.read(iprot) else: iprot.skip(ftype) + elif fid == 4: + if ftype == TType.LIST: + self.failed_profiles = [] + (_etype28, _size25) = iprot.readListBegin() + for _i29 in range(_size25): + _elem30 = iprot.readString() + self.failed_profiles.append(_elem30) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 5: + if ftype == TType.LIST: + self.failed_thrift_profiles = [] + (_etype34, _size31) = iprot.readListBegin() + for _i35 in range(_size31): + _elem36 = impala._thrift_gen.RuntimeProfile.ttypes.TRuntimeProfileTree() + _elem36.read(iprot) + self.failed_thrift_profiles.append(_elem36) + iprot.readListEnd() + else: + iprot.skip(ftype) else: iprot.skip(ftype) iprot.readFieldEnd() @@ -770,6 +1467,20 @@ def write(self, oprot): oprot.writeFieldBegin('thrift_profile', TType.STRUCT, 3) self.thrift_profile.write(oprot) oprot.writeFieldEnd() + if self.failed_profiles is not None: + oprot.writeFieldBegin('failed_profiles', TType.LIST, 4) + oprot.writeListBegin(TType.STRING, len(self.failed_profiles)) + for iter37 in self.failed_profiles: + oprot.writeString(iter37) + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.failed_thrift_profiles is not None: + oprot.writeFieldBegin('failed_thrift_profiles', TType.LIST, 5) + oprot.writeListBegin(TType.STRUCT, len(self.failed_thrift_profiles)) + for iter38 in self.failed_thrift_profiles: + iter38.write(oprot) + oprot.writeListEnd() + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -788,11 +1499,12 @@ def __eq__(self, other): def __ne__(self, other): return not (self == other) -all_structs.append(TInsertResult) -TInsertResult.thrift_spec = ( +all_structs.append(TDmlResult) +TDmlResult.thrift_spec = ( None, # 0 (1, TType.MAP, 'rows_modified', (TType.STRING, None, TType.I64, None, False), None, ), # 1 (2, TType.I64, 'num_row_errors', None, None, ), # 2 + (3, TType.MAP, 'rows_deleted', (TType.STRING, None, TType.I64, None, False), None, ), # 3 ) all_structs.append(TPingImpalaServiceResp) TPingImpalaServiceResp.thrift_spec = ( @@ -806,17 +1518,43 @@ def __ne__(self, other): (1, TType.STRING, 'db_name', None, None, ), # 1 (2, TType.STRING, 'table_name', None, None, ), # 2 ) +all_structs.append(TPingImpalaHS2ServiceReq) +TPingImpalaHS2ServiceReq.thrift_spec = ( + None, # 0 + (1, TType.STRUCT, 'sessionHandle', [impala._thrift_gen.TCLIService.ttypes.TSessionHandle, None], None, ), # 1 +) +all_structs.append(TPingImpalaHS2ServiceResp) +TPingImpalaHS2ServiceResp.thrift_spec = ( + None, # 0 + (1, TType.STRUCT, 'status', [impala._thrift_gen.TCLIService.ttypes.TStatus, None], None, ), # 1 + (2, TType.STRING, 'version', None, None, ), # 2 + (3, TType.STRING, 'webserver_address', None, None, ), # 3 + (4, TType.I64, 'timestamp', None, None, ), # 4 +) +all_structs.append(TCloseImpalaOperationReq) +TCloseImpalaOperationReq.thrift_spec = ( + None, # 0 + (1, TType.STRUCT, 'operationHandle', [impala._thrift_gen.TCLIService.ttypes.TOperationHandle, None], None, ), # 1 +) +all_structs.append(TCloseImpalaOperationResp) +TCloseImpalaOperationResp.thrift_spec = ( + None, # 0 + (1, TType.STRUCT, 'status', [impala._thrift_gen.TCLIService.ttypes.TStatus, None], None, ), # 1 + (2, TType.STRUCT, 'dml_result', [TDmlResult, None], None, ), # 2 +) all_structs.append(TGetExecSummaryReq) TGetExecSummaryReq.thrift_spec = ( None, # 0 (1, TType.STRUCT, 'operationHandle', [impala._thrift_gen.TCLIService.ttypes.TOperationHandle, None], None, ), # 1 (2, TType.STRUCT, 'sessionHandle', [impala._thrift_gen.TCLIService.ttypes.TSessionHandle, None], None, ), # 2 + (3, TType.BOOL, 'include_query_attempts', None, False, ), # 3 ) all_structs.append(TGetExecSummaryResp) TGetExecSummaryResp.thrift_spec = ( None, # 0 (1, TType.STRUCT, 'status', [impala._thrift_gen.TCLIService.ttypes.TStatus, None], None, ), # 1 (2, TType.STRUCT, 'summary', [impala._thrift_gen.ExecStats.ttypes.TExecSummary, None], None, ), # 2 + (3, TType.LIST, 'failed_summaries', (TType.STRUCT, [impala._thrift_gen.ExecStats.ttypes.TExecSummary, None], False), None, ), # 3 ) all_structs.append(TGetRuntimeProfileReq) TGetRuntimeProfileReq.thrift_spec = ( @@ -824,6 +1562,7 @@ def __ne__(self, other): (1, TType.STRUCT, 'operationHandle', [impala._thrift_gen.TCLIService.ttypes.TOperationHandle, None], None, ), # 1 (2, TType.STRUCT, 'sessionHandle', [impala._thrift_gen.TCLIService.ttypes.TSessionHandle, None], None, ), # 2 (3, TType.I32, 'format', None, 0, ), # 3 + (4, TType.BOOL, 'include_query_attempts', None, False, ), # 4 ) all_structs.append(TGetRuntimeProfileResp) TGetRuntimeProfileResp.thrift_spec = ( @@ -831,6 +1570,8 @@ def __ne__(self, other): (1, TType.STRUCT, 'status', [impala._thrift_gen.TCLIService.ttypes.TStatus, None], None, ), # 1 (2, TType.STRING, 'profile', None, None, ), # 2 (3, TType.STRUCT, 'thrift_profile', [impala._thrift_gen.RuntimeProfile.ttypes.TRuntimeProfileTree, None], None, ), # 3 + (4, TType.LIST, 'failed_profiles', (TType.STRING, None, False), None, ), # 4 + (5, TType.LIST, 'failed_thrift_profiles', (TType.STRUCT, [impala._thrift_gen.RuntimeProfile.ttypes.TRuntimeProfileTree, None], False), None, ), # 5 ) fix_spec(all_structs) del all_structs diff --git a/impala/_thrift_gen/RuntimeProfile/ttypes.py b/impala/_thrift_gen/RuntimeProfile/ttypes.py index ecfaff820..8ed4c6819 100644 --- a/impala/_thrift_gen/RuntimeProfile/ttypes.py +++ b/impala/_thrift_gen/RuntimeProfile/ttypes.py @@ -22,17 +22,20 @@ class TRuntimeProfileFormat(object): STRING = 0 BASE64 = 1 THRIFT = 2 + JSON = 3 _VALUES_TO_NAMES = { 0: "STRING", 1: "BASE64", 2: "THRIFT", + 3: "JSON", } _NAMES_TO_VALUES = { "STRING": 0, "BASE64": 1, "THRIFT": 2, + "JSON": 3, } @@ -121,6 +124,120 @@ def __ne__(self, other): return not (self == other) +class TAggCounter(object): + """ + Attributes: + - name + - unit + - has_value + - values + + """ + + + def __init__(self, name=None, unit=None, has_value=None, values=None,): + self.name = name + self.unit = unit + self.has_value = has_value + self.values = values + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRING: + self.name = iprot.readString() + else: + iprot.skip(ftype) + elif fid == 2: + if ftype == TType.I32: + self.unit = iprot.readI32() + else: + iprot.skip(ftype) + elif fid == 3: + if ftype == TType.LIST: + self.has_value = [] + (_etype3, _size0) = iprot.readListBegin() + for _i4 in range(_size0): + _elem5 = iprot.readBool() + self.has_value.append(_elem5) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 4: + if ftype == TType.LIST: + self.values = [] + (_etype9, _size6) = iprot.readListBegin() + for _i10 in range(_size6): + _elem11 = iprot.readI64() + self.values.append(_elem11) + iprot.readListEnd() + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('TAggCounter') + if self.name is not None: + oprot.writeFieldBegin('name', TType.STRING, 1) + oprot.writeString(self.name) + oprot.writeFieldEnd() + if self.unit is not None: + oprot.writeFieldBegin('unit', TType.I32, 2) + oprot.writeI32(self.unit) + oprot.writeFieldEnd() + if self.has_value is not None: + oprot.writeFieldBegin('has_value', TType.LIST, 3) + oprot.writeListBegin(TType.BOOL, len(self.has_value)) + for iter12 in self.has_value: + oprot.writeBool(iter12) + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.values is not None: + oprot.writeFieldBegin('values', TType.LIST, 4) + oprot.writeListBegin(TType.I64, len(self.values)) + for iter13 in self.values: + oprot.writeI64(iter13) + oprot.writeListEnd() + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + if self.name is None: + raise TProtocolException(message='Required field name is unset!') + if self.unit is None: + raise TProtocolException(message='Required field unit is unset!') + if self.has_value is None: + raise TProtocolException(message='Required field has_value is unset!') + if self.values is None: + raise TProtocolException(message='Required field values is unset!') + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + class TEventSequence(object): """ Attributes: @@ -153,20 +270,20 @@ def read(self, iprot): elif fid == 2: if ftype == TType.LIST: self.timestamps = [] - (_etype3, _size0) = iprot.readListBegin() - for _i4 in range(_size0): - _elem5 = iprot.readI64() - self.timestamps.append(_elem5) + (_etype17, _size14) = iprot.readListBegin() + for _i18 in range(_size14): + _elem19 = iprot.readI64() + self.timestamps.append(_elem19) iprot.readListEnd() else: iprot.skip(ftype) elif fid == 3: if ftype == TType.LIST: self.labels = [] - (_etype9, _size6) = iprot.readListBegin() - for _i10 in range(_size6): - _elem11 = iprot.readString() - self.labels.append(_elem11) + (_etype23, _size20) = iprot.readListBegin() + for _i24 in range(_size20): + _elem25 = iprot.readString() + self.labels.append(_elem25) iprot.readListEnd() else: iprot.skip(ftype) @@ -187,15 +304,15 @@ def write(self, oprot): if self.timestamps is not None: oprot.writeFieldBegin('timestamps', TType.LIST, 2) oprot.writeListBegin(TType.I64, len(self.timestamps)) - for iter12 in self.timestamps: - oprot.writeI64(iter12) + for iter26 in self.timestamps: + oprot.writeI64(iter26) oprot.writeListEnd() oprot.writeFieldEnd() if self.labels is not None: oprot.writeFieldBegin('labels', TType.LIST, 3) oprot.writeListBegin(TType.STRING, len(self.labels)) - for iter13 in self.labels: - oprot.writeString(iter13) + for iter27 in self.labels: + oprot.writeString(iter27) oprot.writeListEnd() oprot.writeFieldEnd() oprot.writeFieldStop() @@ -222,24 +339,424 @@ def __ne__(self, other): return not (self == other) +class TAggEventSequence(object): + """ + Attributes: + - name + - label_dict + - label_idxs + - timestamps + + """ + + + def __init__(self, name=None, label_dict=None, label_idxs=None, timestamps=None,): + self.name = name + self.label_dict = label_dict + self.label_idxs = label_idxs + self.timestamps = timestamps + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRING: + self.name = iprot.readString() + else: + iprot.skip(ftype) + elif fid == 2: + if ftype == TType.LIST: + self.label_dict = [] + (_etype31, _size28) = iprot.readListBegin() + for _i32 in range(_size28): + _elem33 = iprot.readString() + self.label_dict.append(_elem33) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 3: + if ftype == TType.LIST: + self.label_idxs = [] + (_etype37, _size34) = iprot.readListBegin() + for _i38 in range(_size34): + _elem39 = [] + (_etype43, _size40) = iprot.readListBegin() + for _i44 in range(_size40): + _elem45 = iprot.readI32() + _elem39.append(_elem45) + iprot.readListEnd() + self.label_idxs.append(_elem39) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 4: + if ftype == TType.LIST: + self.timestamps = [] + (_etype49, _size46) = iprot.readListBegin() + for _i50 in range(_size46): + _elem51 = [] + (_etype55, _size52) = iprot.readListBegin() + for _i56 in range(_size52): + _elem57 = iprot.readI64() + _elem51.append(_elem57) + iprot.readListEnd() + self.timestamps.append(_elem51) + iprot.readListEnd() + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('TAggEventSequence') + if self.name is not None: + oprot.writeFieldBegin('name', TType.STRING, 1) + oprot.writeString(self.name) + oprot.writeFieldEnd() + if self.label_dict is not None: + oprot.writeFieldBegin('label_dict', TType.LIST, 2) + oprot.writeListBegin(TType.STRING, len(self.label_dict)) + for iter58 in self.label_dict: + oprot.writeString(iter58) + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.label_idxs is not None: + oprot.writeFieldBegin('label_idxs', TType.LIST, 3) + oprot.writeListBegin(TType.LIST, len(self.label_idxs)) + for iter59 in self.label_idxs: + oprot.writeListBegin(TType.I32, len(iter59)) + for iter60 in iter59: + oprot.writeI32(iter60) + oprot.writeListEnd() + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.timestamps is not None: + oprot.writeFieldBegin('timestamps', TType.LIST, 4) + oprot.writeListBegin(TType.LIST, len(self.timestamps)) + for iter61 in self.timestamps: + oprot.writeListBegin(TType.I64, len(iter61)) + for iter62 in iter61: + oprot.writeI64(iter62) + oprot.writeListEnd() + oprot.writeListEnd() + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + if self.name is None: + raise TProtocolException(message='Required field name is unset!') + if self.label_dict is None: + raise TProtocolException(message='Required field label_dict is unset!') + if self.label_idxs is None: + raise TProtocolException(message='Required field label_idxs is unset!') + if self.timestamps is None: + raise TProtocolException(message='Required field timestamps is unset!') + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + class TTimeSeriesCounter(object): """ Attributes: - name - unit - - period_ms - - values - - start_index + - period_ms + - values + - start_index + + """ + + + def __init__(self, name=None, unit=None, period_ms=None, values=None, start_index=None,): + self.name = name + self.unit = unit + self.period_ms = period_ms + self.values = values + self.start_index = start_index + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRING: + self.name = iprot.readString() + else: + iprot.skip(ftype) + elif fid == 2: + if ftype == TType.I32: + self.unit = iprot.readI32() + else: + iprot.skip(ftype) + elif fid == 3: + if ftype == TType.I32: + self.period_ms = iprot.readI32() + else: + iprot.skip(ftype) + elif fid == 4: + if ftype == TType.LIST: + self.values = [] + (_etype66, _size63) = iprot.readListBegin() + for _i67 in range(_size63): + _elem68 = iprot.readI64() + self.values.append(_elem68) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 5: + if ftype == TType.I64: + self.start_index = iprot.readI64() + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('TTimeSeriesCounter') + if self.name is not None: + oprot.writeFieldBegin('name', TType.STRING, 1) + oprot.writeString(self.name) + oprot.writeFieldEnd() + if self.unit is not None: + oprot.writeFieldBegin('unit', TType.I32, 2) + oprot.writeI32(self.unit) + oprot.writeFieldEnd() + if self.period_ms is not None: + oprot.writeFieldBegin('period_ms', TType.I32, 3) + oprot.writeI32(self.period_ms) + oprot.writeFieldEnd() + if self.values is not None: + oprot.writeFieldBegin('values', TType.LIST, 4) + oprot.writeListBegin(TType.I64, len(self.values)) + for iter69 in self.values: + oprot.writeI64(iter69) + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.start_index is not None: + oprot.writeFieldBegin('start_index', TType.I64, 5) + oprot.writeI64(self.start_index) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + if self.name is None: + raise TProtocolException(message='Required field name is unset!') + if self.unit is None: + raise TProtocolException(message='Required field unit is unset!') + if self.period_ms is None: + raise TProtocolException(message='Required field period_ms is unset!') + if self.values is None: + raise TProtocolException(message='Required field values is unset!') + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + +class TAggTimeSeriesCounter(object): + """ + Attributes: + - name + - unit + - period_ms + - values + - start_index + + """ + + + def __init__(self, name=None, unit=None, period_ms=None, values=None, start_index=None,): + self.name = name + self.unit = unit + self.period_ms = period_ms + self.values = values + self.start_index = start_index + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRING: + self.name = iprot.readString() + else: + iprot.skip(ftype) + elif fid == 2: + if ftype == TType.I32: + self.unit = iprot.readI32() + else: + iprot.skip(ftype) + elif fid == 3: + if ftype == TType.LIST: + self.period_ms = [] + (_etype73, _size70) = iprot.readListBegin() + for _i74 in range(_size70): + _elem75 = iprot.readI32() + self.period_ms.append(_elem75) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 4: + if ftype == TType.LIST: + self.values = [] + (_etype79, _size76) = iprot.readListBegin() + for _i80 in range(_size76): + _elem81 = [] + (_etype85, _size82) = iprot.readListBegin() + for _i86 in range(_size82): + _elem87 = iprot.readI64() + _elem81.append(_elem87) + iprot.readListEnd() + self.values.append(_elem81) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 5: + if ftype == TType.LIST: + self.start_index = [] + (_etype91, _size88) = iprot.readListBegin() + for _i92 in range(_size88): + _elem93 = iprot.readI64() + self.start_index.append(_elem93) + iprot.readListEnd() + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('TAggTimeSeriesCounter') + if self.name is not None: + oprot.writeFieldBegin('name', TType.STRING, 1) + oprot.writeString(self.name) + oprot.writeFieldEnd() + if self.unit is not None: + oprot.writeFieldBegin('unit', TType.I32, 2) + oprot.writeI32(self.unit) + oprot.writeFieldEnd() + if self.period_ms is not None: + oprot.writeFieldBegin('period_ms', TType.LIST, 3) + oprot.writeListBegin(TType.I32, len(self.period_ms)) + for iter94 in self.period_ms: + oprot.writeI32(iter94) + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.values is not None: + oprot.writeFieldBegin('values', TType.LIST, 4) + oprot.writeListBegin(TType.LIST, len(self.values)) + for iter95 in self.values: + oprot.writeListBegin(TType.I64, len(iter95)) + for iter96 in iter95: + oprot.writeI64(iter96) + oprot.writeListEnd() + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.start_index is not None: + oprot.writeFieldBegin('start_index', TType.LIST, 5) + oprot.writeListBegin(TType.I64, len(self.start_index)) + for iter97 in self.start_index: + oprot.writeI64(iter97) + oprot.writeListEnd() + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + if self.name is None: + raise TProtocolException(message='Required field name is unset!') + if self.unit is None: + raise TProtocolException(message='Required field unit is unset!') + if self.period_ms is None: + raise TProtocolException(message='Required field period_ms is unset!') + if self.values is None: + raise TProtocolException(message='Required field values is unset!') + if self.start_index is None: + raise TProtocolException(message='Required field start_index is unset!') + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + +class TSummaryStatsCounter(object): + """ + Attributes: + - name + - unit + - sum + - total_num_values + - min_value + - max_value """ - def __init__(self, name=None, unit=None, period_ms=None, values=None, start_index=None,): + def __init__(self, name=None, unit=None, sum=None, total_num_values=None, min_value=None, max_value=None,): self.name = name self.unit = unit - self.period_ms = period_ms - self.values = values - self.start_index = start_index + self.sum = sum + self.total_num_values = total_num_values + self.min_value = min_value + self.max_value = max_value def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -261,23 +778,23 @@ def read(self, iprot): else: iprot.skip(ftype) elif fid == 3: - if ftype == TType.I32: - self.period_ms = iprot.readI32() + if ftype == TType.I64: + self.sum = iprot.readI64() else: iprot.skip(ftype) elif fid == 4: - if ftype == TType.LIST: - self.values = [] - (_etype17, _size14) = iprot.readListBegin() - for _i18 in range(_size14): - _elem19 = iprot.readI64() - self.values.append(_elem19) - iprot.readListEnd() + if ftype == TType.I64: + self.total_num_values = iprot.readI64() else: iprot.skip(ftype) elif fid == 5: if ftype == TType.I64: - self.start_index = iprot.readI64() + self.min_value = iprot.readI64() + else: + iprot.skip(ftype) + elif fid == 6: + if ftype == TType.I64: + self.max_value = iprot.readI64() else: iprot.skip(ftype) else: @@ -289,7 +806,7 @@ def write(self, oprot): if oprot._fast_encode is not None and self.thrift_spec is not None: oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) return - oprot.writeStructBegin('TTimeSeriesCounter') + oprot.writeStructBegin('TSummaryStatsCounter') if self.name is not None: oprot.writeFieldBegin('name', TType.STRING, 1) oprot.writeString(self.name) @@ -298,20 +815,21 @@ def write(self, oprot): oprot.writeFieldBegin('unit', TType.I32, 2) oprot.writeI32(self.unit) oprot.writeFieldEnd() - if self.period_ms is not None: - oprot.writeFieldBegin('period_ms', TType.I32, 3) - oprot.writeI32(self.period_ms) + if self.sum is not None: + oprot.writeFieldBegin('sum', TType.I64, 3) + oprot.writeI64(self.sum) oprot.writeFieldEnd() - if self.values is not None: - oprot.writeFieldBegin('values', TType.LIST, 4) - oprot.writeListBegin(TType.I64, len(self.values)) - for iter20 in self.values: - oprot.writeI64(iter20) - oprot.writeListEnd() + if self.total_num_values is not None: + oprot.writeFieldBegin('total_num_values', TType.I64, 4) + oprot.writeI64(self.total_num_values) oprot.writeFieldEnd() - if self.start_index is not None: - oprot.writeFieldBegin('start_index', TType.I64, 5) - oprot.writeI64(self.start_index) + if self.min_value is not None: + oprot.writeFieldBegin('min_value', TType.I64, 5) + oprot.writeI64(self.min_value) + oprot.writeFieldEnd() + if self.max_value is not None: + oprot.writeFieldBegin('max_value', TType.I64, 6) + oprot.writeI64(self.max_value) oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -321,10 +839,14 @@ def validate(self): raise TProtocolException(message='Required field name is unset!') if self.unit is None: raise TProtocolException(message='Required field unit is unset!') - if self.period_ms is None: - raise TProtocolException(message='Required field period_ms is unset!') - if self.values is None: - raise TProtocolException(message='Required field values is unset!') + if self.sum is None: + raise TProtocolException(message='Required field sum is unset!') + if self.total_num_values is None: + raise TProtocolException(message='Required field total_num_values is unset!') + if self.min_value is None: + raise TProtocolException(message='Required field min_value is unset!') + if self.max_value is None: + raise TProtocolException(message='Required field max_value is unset!') return def __repr__(self): @@ -339,11 +861,12 @@ def __ne__(self, other): return not (self == other) -class TSummaryStatsCounter(object): +class TAggSummaryStatsCounter(object): """ Attributes: - name - unit + - has_value - sum - total_num_values - min_value @@ -352,9 +875,10 @@ class TSummaryStatsCounter(object): """ - def __init__(self, name=None, unit=None, sum=None, total_num_values=None, min_value=None, max_value=None,): + def __init__(self, name=None, unit=None, has_value=None, sum=None, total_num_values=None, min_value=None, max_value=None,): self.name = name self.unit = unit + self.has_value = has_value self.sum = sum self.total_num_values = total_num_values self.min_value = min_value @@ -380,23 +904,53 @@ def read(self, iprot): else: iprot.skip(ftype) elif fid == 3: - if ftype == TType.I64: - self.sum = iprot.readI64() + if ftype == TType.LIST: + self.has_value = [] + (_etype101, _size98) = iprot.readListBegin() + for _i102 in range(_size98): + _elem103 = iprot.readBool() + self.has_value.append(_elem103) + iprot.readListEnd() else: iprot.skip(ftype) elif fid == 4: - if ftype == TType.I64: - self.total_num_values = iprot.readI64() + if ftype == TType.LIST: + self.sum = [] + (_etype107, _size104) = iprot.readListBegin() + for _i108 in range(_size104): + _elem109 = iprot.readI64() + self.sum.append(_elem109) + iprot.readListEnd() else: iprot.skip(ftype) elif fid == 5: - if ftype == TType.I64: - self.min_value = iprot.readI64() + if ftype == TType.LIST: + self.total_num_values = [] + (_etype113, _size110) = iprot.readListBegin() + for _i114 in range(_size110): + _elem115 = iprot.readI64() + self.total_num_values.append(_elem115) + iprot.readListEnd() else: iprot.skip(ftype) elif fid == 6: - if ftype == TType.I64: - self.max_value = iprot.readI64() + if ftype == TType.LIST: + self.min_value = [] + (_etype119, _size116) = iprot.readListBegin() + for _i120 in range(_size116): + _elem121 = iprot.readI64() + self.min_value.append(_elem121) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 7: + if ftype == TType.LIST: + self.max_value = [] + (_etype125, _size122) = iprot.readListBegin() + for _i126 in range(_size122): + _elem127 = iprot.readI64() + self.max_value.append(_elem127) + iprot.readListEnd() else: iprot.skip(ftype) else: @@ -408,7 +962,7 @@ def write(self, oprot): if oprot._fast_encode is not None and self.thrift_spec is not None: oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) return - oprot.writeStructBegin('TSummaryStatsCounter') + oprot.writeStructBegin('TAggSummaryStatsCounter') if self.name is not None: oprot.writeFieldBegin('name', TType.STRING, 1) oprot.writeString(self.name) @@ -417,21 +971,40 @@ def write(self, oprot): oprot.writeFieldBegin('unit', TType.I32, 2) oprot.writeI32(self.unit) oprot.writeFieldEnd() + if self.has_value is not None: + oprot.writeFieldBegin('has_value', TType.LIST, 3) + oprot.writeListBegin(TType.BOOL, len(self.has_value)) + for iter128 in self.has_value: + oprot.writeBool(iter128) + oprot.writeListEnd() + oprot.writeFieldEnd() if self.sum is not None: - oprot.writeFieldBegin('sum', TType.I64, 3) - oprot.writeI64(self.sum) + oprot.writeFieldBegin('sum', TType.LIST, 4) + oprot.writeListBegin(TType.I64, len(self.sum)) + for iter129 in self.sum: + oprot.writeI64(iter129) + oprot.writeListEnd() oprot.writeFieldEnd() if self.total_num_values is not None: - oprot.writeFieldBegin('total_num_values', TType.I64, 4) - oprot.writeI64(self.total_num_values) + oprot.writeFieldBegin('total_num_values', TType.LIST, 5) + oprot.writeListBegin(TType.I64, len(self.total_num_values)) + for iter130 in self.total_num_values: + oprot.writeI64(iter130) + oprot.writeListEnd() oprot.writeFieldEnd() if self.min_value is not None: - oprot.writeFieldBegin('min_value', TType.I64, 5) - oprot.writeI64(self.min_value) + oprot.writeFieldBegin('min_value', TType.LIST, 6) + oprot.writeListBegin(TType.I64, len(self.min_value)) + for iter131 in self.min_value: + oprot.writeI64(iter131) + oprot.writeListEnd() oprot.writeFieldEnd() if self.max_value is not None: - oprot.writeFieldBegin('max_value', TType.I64, 6) - oprot.writeI64(self.max_value) + oprot.writeFieldBegin('max_value', TType.LIST, 7) + oprot.writeListBegin(TType.I64, len(self.max_value)) + for iter132 in self.max_value: + oprot.writeI64(iter132) + oprot.writeListEnd() oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -441,6 +1014,8 @@ def validate(self): raise TProtocolException(message='Required field name is unset!') if self.unit is None: raise TProtocolException(message='Required field unit is unset!') + if self.has_value is None: + raise TProtocolException(message='Required field has_value is unset!') if self.sum is None: raise TProtocolException(message='Required field sum is unset!') if self.total_num_values is None: @@ -531,6 +1106,201 @@ def __ne__(self, other): return not (self == other) +class TAggregatedRuntimeProfileNode(object): + """ + Attributes: + - num_instances + - input_profiles + - counters + - info_strings + - summary_stats_counters + - event_sequences + - time_series_counters + + """ + + + def __init__(self, num_instances=None, input_profiles=None, counters=None, info_strings=None, summary_stats_counters=None, event_sequences=None, time_series_counters=None,): + self.num_instances = num_instances + self.input_profiles = input_profiles + self.counters = counters + self.info_strings = info_strings + self.summary_stats_counters = summary_stats_counters + self.event_sequences = event_sequences + self.time_series_counters = time_series_counters + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.I32: + self.num_instances = iprot.readI32() + else: + iprot.skip(ftype) + elif fid == 2: + if ftype == TType.LIST: + self.input_profiles = [] + (_etype136, _size133) = iprot.readListBegin() + for _i137 in range(_size133): + _elem138 = iprot.readString() + self.input_profiles.append(_elem138) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 3: + if ftype == TType.LIST: + self.counters = [] + (_etype142, _size139) = iprot.readListBegin() + for _i143 in range(_size139): + _elem144 = TAggCounter() + _elem144.read(iprot) + self.counters.append(_elem144) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 4: + if ftype == TType.MAP: + self.info_strings = {} + (_ktype146, _vtype147, _size145) = iprot.readMapBegin() + for _i149 in range(_size145): + _key150 = iprot.readString() + _val151 = {} + (_ktype153, _vtype154, _size152) = iprot.readMapBegin() + for _i156 in range(_size152): + _key157 = iprot.readString() + _val158 = [] + (_etype162, _size159) = iprot.readListBegin() + for _i163 in range(_size159): + _elem164 = iprot.readI32() + _val158.append(_elem164) + iprot.readListEnd() + _val151[_key157] = _val158 + iprot.readMapEnd() + self.info_strings[_key150] = _val151 + iprot.readMapEnd() + else: + iprot.skip(ftype) + elif fid == 5: + if ftype == TType.LIST: + self.summary_stats_counters = [] + (_etype168, _size165) = iprot.readListBegin() + for _i169 in range(_size165): + _elem170 = TAggSummaryStatsCounter() + _elem170.read(iprot) + self.summary_stats_counters.append(_elem170) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 6: + if ftype == TType.LIST: + self.event_sequences = [] + (_etype174, _size171) = iprot.readListBegin() + for _i175 in range(_size171): + _elem176 = TAggEventSequence() + _elem176.read(iprot) + self.event_sequences.append(_elem176) + iprot.readListEnd() + else: + iprot.skip(ftype) + elif fid == 7: + if ftype == TType.LIST: + self.time_series_counters = [] + (_etype180, _size177) = iprot.readListBegin() + for _i181 in range(_size177): + _elem182 = TAggTimeSeriesCounter() + _elem182.read(iprot) + self.time_series_counters.append(_elem182) + iprot.readListEnd() + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('TAggregatedRuntimeProfileNode') + if self.num_instances is not None: + oprot.writeFieldBegin('num_instances', TType.I32, 1) + oprot.writeI32(self.num_instances) + oprot.writeFieldEnd() + if self.input_profiles is not None: + oprot.writeFieldBegin('input_profiles', TType.LIST, 2) + oprot.writeListBegin(TType.STRING, len(self.input_profiles)) + for iter183 in self.input_profiles: + oprot.writeString(iter183) + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.counters is not None: + oprot.writeFieldBegin('counters', TType.LIST, 3) + oprot.writeListBegin(TType.STRUCT, len(self.counters)) + for iter184 in self.counters: + iter184.write(oprot) + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.info_strings is not None: + oprot.writeFieldBegin('info_strings', TType.MAP, 4) + oprot.writeMapBegin(TType.STRING, TType.MAP, len(self.info_strings)) + for kiter185, viter186 in self.info_strings.items(): + oprot.writeString(kiter185) + oprot.writeMapBegin(TType.STRING, TType.LIST, len(viter186)) + for kiter187, viter188 in viter186.items(): + oprot.writeString(kiter187) + oprot.writeListBegin(TType.I32, len(viter188)) + for iter189 in viter188: + oprot.writeI32(iter189) + oprot.writeListEnd() + oprot.writeMapEnd() + oprot.writeMapEnd() + oprot.writeFieldEnd() + if self.summary_stats_counters is not None: + oprot.writeFieldBegin('summary_stats_counters', TType.LIST, 5) + oprot.writeListBegin(TType.STRUCT, len(self.summary_stats_counters)) + for iter190 in self.summary_stats_counters: + iter190.write(oprot) + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.event_sequences is not None: + oprot.writeFieldBegin('event_sequences', TType.LIST, 6) + oprot.writeListBegin(TType.STRUCT, len(self.event_sequences)) + for iter191 in self.event_sequences: + iter191.write(oprot) + oprot.writeListEnd() + oprot.writeFieldEnd() + if self.time_series_counters is not None: + oprot.writeFieldBegin('time_series_counters', TType.LIST, 7) + oprot.writeListBegin(TType.STRUCT, len(self.time_series_counters)) + for iter192 in self.time_series_counters: + iter192.write(oprot) + oprot.writeListEnd() + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + class TRuntimeProfileNode(object): """ Attributes: @@ -546,11 +1316,12 @@ class TRuntimeProfileNode(object): - time_series_counters - summary_stats_counters - node_metadata + - aggregated """ - def __init__(self, name=None, num_children=None, counters=None, metadata=None, indent=None, info_strings=None, info_strings_display_order=None, child_counters_map=None, event_sequences=None, time_series_counters=None, summary_stats_counters=None, node_metadata=None,): + def __init__(self, name=None, num_children=None, counters=None, metadata=None, indent=None, info_strings=None, info_strings_display_order=None, child_counters_map=None, event_sequences=None, time_series_counters=None, summary_stats_counters=None, node_metadata=None, aggregated=None,): self.name = name self.num_children = num_children self.counters = counters @@ -563,6 +1334,7 @@ def __init__(self, name=None, num_children=None, counters=None, metadata=None, i self.time_series_counters = time_series_counters self.summary_stats_counters = summary_stats_counters self.node_metadata = node_metadata + self.aggregated = aggregated def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -586,11 +1358,11 @@ def read(self, iprot): elif fid == 3: if ftype == TType.LIST: self.counters = [] - (_etype24, _size21) = iprot.readListBegin() - for _i25 in range(_size21): - _elem26 = TCounter() - _elem26.read(iprot) - self.counters.append(_elem26) + (_etype196, _size193) = iprot.readListBegin() + for _i197 in range(_size193): + _elem198 = TCounter() + _elem198.read(iprot) + self.counters.append(_elem198) iprot.readListEnd() else: iprot.skip(ftype) @@ -607,70 +1379,70 @@ def read(self, iprot): elif fid == 6: if ftype == TType.MAP: self.info_strings = {} - (_ktype28, _vtype29, _size27) = iprot.readMapBegin() - for _i31 in range(_size27): - _key32 = iprot.readString() - _val33 = iprot.readString() - self.info_strings[_key32] = _val33 + (_ktype200, _vtype201, _size199) = iprot.readMapBegin() + for _i203 in range(_size199): + _key204 = iprot.readString() + _val205 = iprot.readString() + self.info_strings[_key204] = _val205 iprot.readMapEnd() else: iprot.skip(ftype) elif fid == 7: if ftype == TType.LIST: self.info_strings_display_order = [] - (_etype37, _size34) = iprot.readListBegin() - for _i38 in range(_size34): - _elem39 = iprot.readString() - self.info_strings_display_order.append(_elem39) + (_etype209, _size206) = iprot.readListBegin() + for _i210 in range(_size206): + _elem211 = iprot.readString() + self.info_strings_display_order.append(_elem211) iprot.readListEnd() else: iprot.skip(ftype) elif fid == 8: if ftype == TType.MAP: self.child_counters_map = {} - (_ktype41, _vtype42, _size40) = iprot.readMapBegin() - for _i44 in range(_size40): - _key45 = iprot.readString() - _val46 = set() - (_etype50, _size47) = iprot.readSetBegin() - for _i51 in range(_size47): - _elem52 = iprot.readString() - _val46.add(_elem52) + (_ktype213, _vtype214, _size212) = iprot.readMapBegin() + for _i216 in range(_size212): + _key217 = iprot.readString() + _val218 = set() + (_etype222, _size219) = iprot.readSetBegin() + for _i223 in range(_size219): + _elem224 = iprot.readString() + _val218.add(_elem224) iprot.readSetEnd() - self.child_counters_map[_key45] = _val46 + self.child_counters_map[_key217] = _val218 iprot.readMapEnd() else: iprot.skip(ftype) elif fid == 9: if ftype == TType.LIST: self.event_sequences = [] - (_etype56, _size53) = iprot.readListBegin() - for _i57 in range(_size53): - _elem58 = TEventSequence() - _elem58.read(iprot) - self.event_sequences.append(_elem58) + (_etype228, _size225) = iprot.readListBegin() + for _i229 in range(_size225): + _elem230 = TEventSequence() + _elem230.read(iprot) + self.event_sequences.append(_elem230) iprot.readListEnd() else: iprot.skip(ftype) elif fid == 10: if ftype == TType.LIST: self.time_series_counters = [] - (_etype62, _size59) = iprot.readListBegin() - for _i63 in range(_size59): - _elem64 = TTimeSeriesCounter() - _elem64.read(iprot) - self.time_series_counters.append(_elem64) + (_etype234, _size231) = iprot.readListBegin() + for _i235 in range(_size231): + _elem236 = TTimeSeriesCounter() + _elem236.read(iprot) + self.time_series_counters.append(_elem236) iprot.readListEnd() else: iprot.skip(ftype) elif fid == 11: if ftype == TType.LIST: self.summary_stats_counters = [] - (_etype68, _size65) = iprot.readListBegin() - for _i69 in range(_size65): - _elem70 = TSummaryStatsCounter() - _elem70.read(iprot) - self.summary_stats_counters.append(_elem70) + (_etype240, _size237) = iprot.readListBegin() + for _i241 in range(_size237): + _elem242 = TSummaryStatsCounter() + _elem242.read(iprot) + self.summary_stats_counters.append(_elem242) iprot.readListEnd() else: iprot.skip(ftype) @@ -680,6 +1452,12 @@ def read(self, iprot): self.node_metadata.read(iprot) else: iprot.skip(ftype) + elif fid == 13: + if ftype == TType.STRUCT: + self.aggregated = TAggregatedRuntimeProfileNode() + self.aggregated.read(iprot) + else: + iprot.skip(ftype) else: iprot.skip(ftype) iprot.readFieldEnd() @@ -701,8 +1479,8 @@ def write(self, oprot): if self.counters is not None: oprot.writeFieldBegin('counters', TType.LIST, 3) oprot.writeListBegin(TType.STRUCT, len(self.counters)) - for iter71 in self.counters: - iter71.write(oprot) + for iter243 in self.counters: + iter243.write(oprot) oprot.writeListEnd() oprot.writeFieldEnd() if self.metadata is not None: @@ -716,54 +1494,58 @@ def write(self, oprot): if self.info_strings is not None: oprot.writeFieldBegin('info_strings', TType.MAP, 6) oprot.writeMapBegin(TType.STRING, TType.STRING, len(self.info_strings)) - for kiter72, viter73 in self.info_strings.items(): - oprot.writeString(kiter72) - oprot.writeString(viter73) + for kiter244, viter245 in self.info_strings.items(): + oprot.writeString(kiter244) + oprot.writeString(viter245) oprot.writeMapEnd() oprot.writeFieldEnd() if self.info_strings_display_order is not None: oprot.writeFieldBegin('info_strings_display_order', TType.LIST, 7) oprot.writeListBegin(TType.STRING, len(self.info_strings_display_order)) - for iter74 in self.info_strings_display_order: - oprot.writeString(iter74) + for iter246 in self.info_strings_display_order: + oprot.writeString(iter246) oprot.writeListEnd() oprot.writeFieldEnd() if self.child_counters_map is not None: oprot.writeFieldBegin('child_counters_map', TType.MAP, 8) oprot.writeMapBegin(TType.STRING, TType.SET, len(self.child_counters_map)) - for kiter75, viter76 in self.child_counters_map.items(): - oprot.writeString(kiter75) - oprot.writeSetBegin(TType.STRING, len(viter76)) - for iter77 in viter76: - oprot.writeString(iter77) + for kiter247, viter248 in self.child_counters_map.items(): + oprot.writeString(kiter247) + oprot.writeSetBegin(TType.STRING, len(viter248)) + for iter249 in viter248: + oprot.writeString(iter249) oprot.writeSetEnd() oprot.writeMapEnd() oprot.writeFieldEnd() if self.event_sequences is not None: oprot.writeFieldBegin('event_sequences', TType.LIST, 9) oprot.writeListBegin(TType.STRUCT, len(self.event_sequences)) - for iter78 in self.event_sequences: - iter78.write(oprot) + for iter250 in self.event_sequences: + iter250.write(oprot) oprot.writeListEnd() oprot.writeFieldEnd() if self.time_series_counters is not None: oprot.writeFieldBegin('time_series_counters', TType.LIST, 10) oprot.writeListBegin(TType.STRUCT, len(self.time_series_counters)) - for iter79 in self.time_series_counters: - iter79.write(oprot) + for iter251 in self.time_series_counters: + iter251.write(oprot) oprot.writeListEnd() oprot.writeFieldEnd() if self.summary_stats_counters is not None: oprot.writeFieldBegin('summary_stats_counters', TType.LIST, 11) oprot.writeListBegin(TType.STRUCT, len(self.summary_stats_counters)) - for iter80 in self.summary_stats_counters: - iter80.write(oprot) + for iter252 in self.summary_stats_counters: + iter252.write(oprot) oprot.writeListEnd() oprot.writeFieldEnd() if self.node_metadata is not None: oprot.writeFieldBegin('node_metadata', TType.STRUCT, 12) self.node_metadata.write(oprot) oprot.writeFieldEnd() + if self.aggregated is not None: + oprot.writeFieldBegin('aggregated', TType.STRUCT, 13) + self.aggregated.write(oprot) + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -803,13 +1585,15 @@ class TRuntimeProfileTree(object): Attributes: - nodes - exec_summary + - profile_version """ - def __init__(self, nodes=None, exec_summary=None,): + def __init__(self, nodes=None, exec_summary=None, profile_version=None,): self.nodes = nodes self.exec_summary = exec_summary + self.profile_version = profile_version def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -823,11 +1607,11 @@ def read(self, iprot): if fid == 1: if ftype == TType.LIST: self.nodes = [] - (_etype84, _size81) = iprot.readListBegin() - for _i85 in range(_size81): - _elem86 = TRuntimeProfileNode() - _elem86.read(iprot) - self.nodes.append(_elem86) + (_etype256, _size253) = iprot.readListBegin() + for _i257 in range(_size253): + _elem258 = TRuntimeProfileNode() + _elem258.read(iprot) + self.nodes.append(_elem258) iprot.readListEnd() else: iprot.skip(ftype) @@ -837,6 +1621,11 @@ def read(self, iprot): self.exec_summary.read(iprot) else: iprot.skip(ftype) + elif fid == 3: + if ftype == TType.I32: + self.profile_version = iprot.readI32() + else: + iprot.skip(ftype) else: iprot.skip(ftype) iprot.readFieldEnd() @@ -850,14 +1639,18 @@ def write(self, oprot): if self.nodes is not None: oprot.writeFieldBegin('nodes', TType.LIST, 1) oprot.writeListBegin(TType.STRUCT, len(self.nodes)) - for iter87 in self.nodes: - iter87.write(oprot) + for iter259 in self.nodes: + iter259.write(oprot) oprot.writeListEnd() oprot.writeFieldEnd() if self.exec_summary is not None: oprot.writeFieldBegin('exec_summary', TType.STRUCT, 2) self.exec_summary.write(oprot) oprot.writeFieldEnd() + if self.profile_version is not None: + oprot.writeFieldBegin('profile_version', TType.I32, 3) + oprot.writeI32(self.profile_version) + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -903,11 +1696,11 @@ def read(self, iprot): if fid == 1: if ftype == TType.LIST: self.profile_trees = [] - (_etype91, _size88) = iprot.readListBegin() - for _i92 in range(_size88): - _elem93 = TRuntimeProfileTree() - _elem93.read(iprot) - self.profile_trees.append(_elem93) + (_etype263, _size260) = iprot.readListBegin() + for _i264 in range(_size260): + _elem265 = TRuntimeProfileTree() + _elem265.read(iprot) + self.profile_trees.append(_elem265) iprot.readListEnd() else: iprot.skip(ftype) @@ -930,8 +1723,8 @@ def write(self, oprot): if self.profile_trees is not None: oprot.writeFieldBegin('profile_trees', TType.LIST, 1) oprot.writeListBegin(TType.STRUCT, len(self.profile_trees)) - for iter94 in self.profile_trees: - iter94.write(oprot) + for iter266 in self.profile_trees: + iter266.write(oprot) oprot.writeListEnd() oprot.writeFieldEnd() if self.host_profile is not None: @@ -963,6 +1756,14 @@ def __ne__(self, other): (2, TType.I32, 'unit', None, None, ), # 2 (3, TType.I64, 'value', None, None, ), # 3 ) +all_structs.append(TAggCounter) +TAggCounter.thrift_spec = ( + None, # 0 + (1, TType.STRING, 'name', None, None, ), # 1 + (2, TType.I32, 'unit', None, None, ), # 2 + (3, TType.LIST, 'has_value', (TType.BOOL, None, False), None, ), # 3 + (4, TType.LIST, 'values', (TType.I64, None, False), None, ), # 4 +) all_structs.append(TEventSequence) TEventSequence.thrift_spec = ( None, # 0 @@ -970,6 +1771,14 @@ def __ne__(self, other): (2, TType.LIST, 'timestamps', (TType.I64, None, False), None, ), # 2 (3, TType.LIST, 'labels', (TType.STRING, None, False), None, ), # 3 ) +all_structs.append(TAggEventSequence) +TAggEventSequence.thrift_spec = ( + None, # 0 + (1, TType.STRING, 'name', None, None, ), # 1 + (2, TType.LIST, 'label_dict', (TType.STRING, None, False), None, ), # 2 + (3, TType.LIST, 'label_idxs', (TType.LIST, (TType.I32, None, False), False), None, ), # 3 + (4, TType.LIST, 'timestamps', (TType.LIST, (TType.I64, None, False), False), None, ), # 4 +) all_structs.append(TTimeSeriesCounter) TTimeSeriesCounter.thrift_spec = ( None, # 0 @@ -979,6 +1788,15 @@ def __ne__(self, other): (4, TType.LIST, 'values', (TType.I64, None, False), None, ), # 4 (5, TType.I64, 'start_index', None, None, ), # 5 ) +all_structs.append(TAggTimeSeriesCounter) +TAggTimeSeriesCounter.thrift_spec = ( + None, # 0 + (1, TType.STRING, 'name', None, None, ), # 1 + (2, TType.I32, 'unit', None, None, ), # 2 + (3, TType.LIST, 'period_ms', (TType.I32, None, False), None, ), # 3 + (4, TType.LIST, 'values', (TType.LIST, (TType.I64, None, False), False), None, ), # 4 + (5, TType.LIST, 'start_index', (TType.I64, None, False), None, ), # 5 +) all_structs.append(TSummaryStatsCounter) TSummaryStatsCounter.thrift_spec = ( None, # 0 @@ -989,12 +1807,34 @@ def __ne__(self, other): (5, TType.I64, 'min_value', None, None, ), # 5 (6, TType.I64, 'max_value', None, None, ), # 6 ) +all_structs.append(TAggSummaryStatsCounter) +TAggSummaryStatsCounter.thrift_spec = ( + None, # 0 + (1, TType.STRING, 'name', None, None, ), # 1 + (2, TType.I32, 'unit', None, None, ), # 2 + (3, TType.LIST, 'has_value', (TType.BOOL, None, False), None, ), # 3 + (4, TType.LIST, 'sum', (TType.I64, None, False), None, ), # 4 + (5, TType.LIST, 'total_num_values', (TType.I64, None, False), None, ), # 5 + (6, TType.LIST, 'min_value', (TType.I64, None, False), None, ), # 6 + (7, TType.LIST, 'max_value', (TType.I64, None, False), None, ), # 7 +) all_structs.append(TRuntimeProfileNodeMetadata) TRuntimeProfileNodeMetadata.thrift_spec = ( None, # 0 (1, TType.I32, 'plan_node_id', None, None, ), # 1 (2, TType.I32, 'data_sink_id', None, None, ), # 2 ) +all_structs.append(TAggregatedRuntimeProfileNode) +TAggregatedRuntimeProfileNode.thrift_spec = ( + None, # 0 + (1, TType.I32, 'num_instances', None, None, ), # 1 + (2, TType.LIST, 'input_profiles', (TType.STRING, None, False), None, ), # 2 + (3, TType.LIST, 'counters', (TType.STRUCT, [TAggCounter, None], False), None, ), # 3 + (4, TType.MAP, 'info_strings', (TType.STRING, None, TType.MAP, (TType.STRING, None, TType.LIST, (TType.I32, None, False), False), False), None, ), # 4 + (5, TType.LIST, 'summary_stats_counters', (TType.STRUCT, [TAggSummaryStatsCounter, None], False), None, ), # 5 + (6, TType.LIST, 'event_sequences', (TType.STRUCT, [TAggEventSequence, None], False), None, ), # 6 + (7, TType.LIST, 'time_series_counters', (TType.STRUCT, [TAggTimeSeriesCounter, None], False), None, ), # 7 +) all_structs.append(TRuntimeProfileNode) TRuntimeProfileNode.thrift_spec = ( None, # 0 @@ -1010,12 +1850,14 @@ def __ne__(self, other): (10, TType.LIST, 'time_series_counters', (TType.STRUCT, [TTimeSeriesCounter, None], False), None, ), # 10 (11, TType.LIST, 'summary_stats_counters', (TType.STRUCT, [TSummaryStatsCounter, None], False), None, ), # 11 (12, TType.STRUCT, 'node_metadata', [TRuntimeProfileNodeMetadata, None], None, ), # 12 + (13, TType.STRUCT, 'aggregated', [TAggregatedRuntimeProfileNode, None], None, ), # 13 ) all_structs.append(TRuntimeProfileTree) TRuntimeProfileTree.thrift_spec = ( None, # 0 (1, TType.LIST, 'nodes', (TType.STRUCT, [TRuntimeProfileNode, None], False), None, ), # 1 (2, TType.STRUCT, 'exec_summary', [impala._thrift_gen.ExecStats.ttypes.TExecSummary, None], None, ), # 2 + (3, TType.I32, 'profile_version', None, None, ), # 3 ) all_structs.append(TRuntimeProfileForest) TRuntimeProfileForest.thrift_spec = ( diff --git a/impala/_thrift_gen/Types/ttypes.py b/impala/_thrift_gen/Types/ttypes.py index 69143fb9c..57cb20700 100644 --- a/impala/_thrift_gen/Types/ttypes.py +++ b/impala/_thrift_gen/Types/ttypes.py @@ -108,6 +108,9 @@ class TStmtType(object): SET = 5 ADMIN_FN = 6 TESTCASE = 7 + CONVERT = 8 + UNKNOWN = 9 + KILL = 10 _VALUES_TO_NAMES = { 0: "QUERY", @@ -118,6 +121,9 @@ class TStmtType(object): 5: "SET", 6: "ADMIN_FN", 7: "TESTCASE", + 8: "CONVERT", + 9: "UNKNOWN", + 10: "KILL", } _NAMES_TO_VALUES = { @@ -129,6 +135,33 @@ class TStmtType(object): "SET": 5, "ADMIN_FN": 6, "TESTCASE": 7, + "CONVERT": 8, + "UNKNOWN": 9, + "KILL": 10, + } + + +class TIcebergOperation(object): + INSERT = 0 + DELETE = 1 + UPDATE = 2 + OPTIMIZE = 3 + MERGE = 4 + + _VALUES_TO_NAMES = { + 0: "INSERT", + 1: "DELETE", + 2: "UPDATE", + 3: "OPTIMIZE", + 4: "MERGE", + } + + _NAMES_TO_VALUES = { + "INSERT": 0, + "DELETE": 1, + "UPDATE": 2, + "OPTIMIZE": 3, + "MERGE": 4, } @@ -225,6 +258,21 @@ class TFunctionBinaryType(object): } +class TSortingOrder(object): + LEXICAL = 0 + ZORDER = 1 + + _VALUES_TO_NAMES = { + 0: "LEXICAL", + 1: "ZORDER", + } + + _NAMES_TO_VALUES = { + "LEXICAL": 0, + "ZORDER": 1, + } + + class TScalarType(object): """ Attributes: @@ -322,13 +370,15 @@ class TStructField(object): Attributes: - name - comment + - field_id """ - def __init__(self, name=None, comment=None,): + def __init__(self, name=None, comment=None, field_id=None,): self.name = name self.comment = comment + self.field_id = field_id def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -349,6 +399,11 @@ def read(self, iprot): self.comment = iprot.readString() else: iprot.skip(ftype) + elif fid == 3: + if ftype == TType.I32: + self.field_id = iprot.readI32() + else: + iprot.skip(ftype) else: iprot.skip(ftype) iprot.readFieldEnd() @@ -367,6 +422,10 @@ def write(self, oprot): oprot.writeFieldBegin('comment', TType.STRING, 2) oprot.writeString(self.comment) oprot.writeFieldEnd() + if self.field_id is not None: + oprot.writeFieldBegin('field_id', TType.I32, 3) + oprot.writeI32(self.field_id) + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -549,13 +608,15 @@ class TNetworkAddress(object): Attributes: - hostname - port + - uds_address """ - def __init__(self, hostname=None, port=None,): + def __init__(self, hostname=None, port=None, uds_address=None,): self.hostname = hostname self.port = port + self.uds_address = uds_address def read(self, iprot): if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: @@ -576,6 +637,11 @@ def read(self, iprot): self.port = iprot.readI32() else: iprot.skip(ftype) + elif fid == 3: + if ftype == TType.STRING: + self.uds_address = iprot.readString() + else: + iprot.skip(ftype) else: iprot.skip(ftype) iprot.readFieldEnd() @@ -594,6 +660,10 @@ def write(self, oprot): oprot.writeFieldBegin('port', TType.I32, 2) oprot.writeI32(self.port) oprot.writeFieldEnd() + if self.uds_address is not None: + oprot.writeFieldBegin('uds_address', TType.STRING, 3) + oprot.writeString(self.uds_address) + oprot.writeFieldEnd() oprot.writeFieldStop() oprot.writeStructEnd() @@ -616,6 +686,74 @@ def __ne__(self, other): return not (self == other) +class TAddressesList(object): + """ + Attributes: + - addresses + + """ + + + def __init__(self, addresses=None,): + self.addresses = addresses + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.LIST: + self.addresses = [] + (_etype17, _size14) = iprot.readListBegin() + for _i18 in range(_size14): + _elem19 = TNetworkAddress() + _elem19.read(iprot) + self.addresses.append(_elem19) + iprot.readListEnd() + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('TAddressesList') + if self.addresses is not None: + oprot.writeFieldBegin('addresses', TType.LIST, 1) + oprot.writeListBegin(TType.STRUCT, len(self.addresses)) + for iter20 in self.addresses: + iter20.write(oprot) + oprot.writeListEnd() + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + if self.addresses is None: + raise TProtocolException(message='Required field addresses is unset!') + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + class TUniqueId(object): """ Attributes: @@ -1060,11 +1198,11 @@ def read(self, iprot): elif fid == 3: if ftype == TType.LIST: self.arg_types = [] - (_etype17, _size14) = iprot.readListBegin() - for _i18 in range(_size14): - _elem19 = TColumnType() - _elem19.read(iprot) - self.arg_types.append(_elem19) + (_etype24, _size21) = iprot.readListBegin() + for _i25 in range(_size21): + _elem26 = TColumnType() + _elem26.read(iprot) + self.arg_types.append(_elem26) iprot.readListEnd() else: iprot.skip(ftype) @@ -1137,8 +1275,8 @@ def write(self, oprot): if self.arg_types is not None: oprot.writeFieldBegin('arg_types', TType.LIST, 3) oprot.writeListBegin(TType.STRUCT, len(self.arg_types)) - for iter20 in self.arg_types: - iter20.write(oprot) + for iter27 in self.arg_types: + iter27.write(oprot) oprot.writeListEnd() oprot.writeFieldEnd() if self.ret_type is not None: @@ -1208,6 +1346,7 @@ def __ne__(self, other): None, # 0 (1, TType.STRING, 'name', None, None, ), # 1 (2, TType.STRING, 'comment', None, None, ), # 2 + (3, TType.I32, 'field_id', None, None, ), # 3 ) all_structs.append(TTypeNode) TTypeNode.thrift_spec = ( @@ -1226,6 +1365,12 @@ def __ne__(self, other): None, # 0 (1, TType.STRING, 'hostname', None, None, ), # 1 (2, TType.I32, 'port', None, None, ), # 2 + (3, TType.STRING, 'uds_address', None, None, ), # 3 +) +all_structs.append(TAddressesList) +TAddressesList.thrift_spec = ( + None, # 0 + (1, TType.LIST, 'addresses', (TType.STRUCT, [TNetworkAddress, None], False), None, ), # 1 ) all_structs.append(TUniqueId) TUniqueId.thrift_spec = ( diff --git a/impala/_version.py b/impala/_version.py deleted file mode 100644 index 112081a60..000000000 --- a/impala/_version.py +++ /dev/null @@ -1,520 +0,0 @@ - -# This file helps to compute a version number in source trees obtained from -# git-archive tarball (such as those provided by githubs download-from-tag -# feature). Distribution tarballs (built by setup.py sdist) and build -# directories (produced by setup.py build) will contain a much shorter file -# that just contains the computed version number. - -# This file is released into the public domain. Generated by -# versioneer-0.17 (https://github.com/warner/python-versioneer) - -"""Git implementation of _version.py.""" - -import errno -import os -import re -import subprocess -import sys - - -def get_keywords(): - """Get the keywords needed to look up the version information.""" - # these strings will be replaced by git during git-archive. - # setup.py/versioneer.py will grep for the variable names, so they must - # each be defined on a line of their own. _version.py will just call - # get_keywords(). - git_refnames = "$Format:%d$" - git_full = "$Format:%H$" - git_date = "$Format:%ci$" - keywords = {"refnames": git_refnames, "full": git_full, "date": git_date} - return keywords - - -class VersioneerConfig: - """Container for Versioneer configuration parameters.""" - - -def get_config(): - """Create, populate and return the VersioneerConfig() object.""" - # these strings are filled in when 'setup.py versioneer' creates - # _version.py - cfg = VersioneerConfig() - cfg.VCS = "git" - cfg.style = "pep440" - cfg.tag_prefix = "" - cfg.parentdir_prefix = "impala-" - cfg.versionfile_source = "impala/_version.py" - cfg.verbose = False - return cfg - - -class NotThisMethod(Exception): - """Exception raised if a method is not valid for the current scenario.""" - - -LONG_VERSION_PY = {} -HANDLERS = {} - - -def register_vcs_handler(vcs, method): # decorator - """Decorator to mark a method as the handler for a particular VCS.""" - def decorate(f): - """Store f in HANDLERS[vcs][method].""" - if vcs not in HANDLERS: - HANDLERS[vcs] = {} - HANDLERS[vcs][method] = f - return f - return decorate - - -def run_command(commands, args, cwd=None, verbose=False, hide_stderr=False, - env=None): - """Call the given command(s).""" - assert isinstance(commands, list) - p = None - for c in commands: - try: - dispcmd = str([c] + args) - # remember shell=False, so use git.cmd on windows, not just git - p = subprocess.Popen([c] + args, cwd=cwd, env=env, - stdout=subprocess.PIPE, - stderr=(subprocess.PIPE if hide_stderr - else None)) - break - except EnvironmentError: - e = sys.exc_info()[1] - if e.errno == errno.ENOENT: - continue - if verbose: - print("unable to run %s" % dispcmd) - print(e) - return None, None - else: - if verbose: - print("unable to find command, tried %s" % (commands,)) - return None, None - stdout = p.communicate()[0].strip() - if sys.version_info[0] >= 3: - stdout = stdout.decode() - if p.returncode != 0: - if verbose: - print("unable to run %s (error)" % dispcmd) - print("stdout was %s" % stdout) - return None, p.returncode - return stdout, p.returncode - - -def versions_from_parentdir(parentdir_prefix, root, verbose): - """Try to determine the version from the parent directory name. - - Source tarballs conventionally unpack into a directory that includes both - the project name and a version string. We will also support searching up - two directory levels for an appropriately named parent directory - """ - rootdirs = [] - - for i in range(3): - dirname = os.path.basename(root) - if dirname.startswith(parentdir_prefix): - return {"version": dirname[len(parentdir_prefix):], - "full-revisionid": None, - "dirty": False, "error": None, "date": None} - else: - rootdirs.append(root) - root = os.path.dirname(root) # up a level - - if verbose: - print("Tried directories %s but none started with prefix %s" % - (str(rootdirs), parentdir_prefix)) - raise NotThisMethod("rootdir doesn't start with parentdir_prefix") - - -@register_vcs_handler("git", "get_keywords") -def git_get_keywords(versionfile_abs): - """Extract version information from the given file.""" - # the code embedded in _version.py can just fetch the value of these - # keywords. When used from setup.py, we don't want to import _version.py, - # so we do it with a regexp instead. This function is not used from - # _version.py. - keywords = {} - try: - f = open(versionfile_abs, "r") - for line in f.readlines(): - if line.strip().startswith("git_refnames ="): - mo = re.search(r'=\s*"(.*)"', line) - if mo: - keywords["refnames"] = mo.group(1) - if line.strip().startswith("git_full ="): - mo = re.search(r'=\s*"(.*)"', line) - if mo: - keywords["full"] = mo.group(1) - if line.strip().startswith("git_date ="): - mo = re.search(r'=\s*"(.*)"', line) - if mo: - keywords["date"] = mo.group(1) - f.close() - except EnvironmentError: - pass - return keywords - - -@register_vcs_handler("git", "keywords") -def git_versions_from_keywords(keywords, tag_prefix, verbose): - """Get version information from git keywords.""" - if not keywords: - raise NotThisMethod("no keywords at all, weird") - date = keywords.get("date") - if date is not None: - # git-2.2.0 added "%cI", which expands to an ISO-8601 -compliant - # datestamp. However we prefer "%ci" (which expands to an "ISO-8601 - # -like" string, which we must then edit to make compliant), because - # it's been around since git-1.5.3, and it's too difficult to - # discover which version we're using, or to work around using an - # older one. - date = date.strip().replace(" ", "T", 1).replace(" ", "", 1) - refnames = keywords["refnames"].strip() - if refnames.startswith("$Format"): - if verbose: - print("keywords are unexpanded, not using") - raise NotThisMethod("unexpanded keywords, not a git-archive tarball") - refs = set([r.strip() for r in refnames.strip("()").split(",")]) - # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of - # just "foo-1.0". If we see a "tag: " prefix, prefer those. - TAG = "tag: " - tags = set([r[len(TAG):] for r in refs if r.startswith(TAG)]) - if not tags: - # Either we're using git < 1.8.3, or there really are no tags. We use - # a heuristic: assume all version tags have a digit. The old git %d - # expansion behaves like git log --decorate=short and strips out the - # refs/heads/ and refs/tags/ prefixes that would let us distinguish - # between branches and tags. By ignoring refnames without digits, we - # filter out many common branch names like "release" and - # "stabilization", as well as "HEAD" and "master". - tags = set([r for r in refs if re.search(r'\d', r)]) - if verbose: - print("discarding '%s', no digits" % ",".join(refs - tags)) - if verbose: - print("likely tags: %s" % ",".join(sorted(tags))) - for ref in sorted(tags): - # sorting will prefer e.g. "2.0" over "2.0rc1" - if ref.startswith(tag_prefix): - r = ref[len(tag_prefix):] - if verbose: - print("picking %s" % r) - return {"version": r, - "full-revisionid": keywords["full"].strip(), - "dirty": False, "error": None, - "date": date} - # no suitable tags, so version is "0+unknown", but full hex is still there - if verbose: - print("no suitable tags, using unknown + full revision id") - return {"version": "0+unknown", - "full-revisionid": keywords["full"].strip(), - "dirty": False, "error": "no suitable tags", "date": None} - - -@register_vcs_handler("git", "pieces_from_vcs") -def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command): - """Get version from 'git describe' in the root of the source tree. - - This only gets called if the git-archive 'subst' keywords were *not* - expanded, and _version.py hasn't already been rewritten with a short - version string, meaning we're inside a checked out source tree. - """ - GITS = ["git"] - if sys.platform == "win32": - GITS = ["git.cmd", "git.exe"] - - out, rc = run_command(GITS, ["rev-parse", "--git-dir"], cwd=root, - hide_stderr=True) - if rc != 0: - if verbose: - print("Directory %s not under git control" % root) - raise NotThisMethod("'git rev-parse --git-dir' returned error") - - # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty] - # if there isn't one, this yields HEX[-dirty] (no NUM) - describe_out, rc = run_command(GITS, ["describe", "--tags", "--dirty", - "--always", "--long", - "--match", "%s*" % tag_prefix], - cwd=root) - # --long was added in git-1.5.5 - if describe_out is None: - raise NotThisMethod("'git describe' failed") - describe_out = describe_out.strip() - full_out, rc = run_command(GITS, ["rev-parse", "HEAD"], cwd=root) - if full_out is None: - raise NotThisMethod("'git rev-parse' failed") - full_out = full_out.strip() - - pieces = {} - pieces["long"] = full_out - pieces["short"] = full_out[:7] # maybe improved later - pieces["error"] = None - - # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty] - # TAG might have hyphens. - git_describe = describe_out - - # look for -dirty suffix - dirty = git_describe.endswith("-dirty") - pieces["dirty"] = dirty - if dirty: - git_describe = git_describe[:git_describe.rindex("-dirty")] - - # now we have TAG-NUM-gHEX or HEX - - if "-" in git_describe: - # TAG-NUM-gHEX - mo = re.search(r'^(.+)-(\d+)-g([0-9a-f]+)$', git_describe) - if not mo: - # unparseable. Maybe git-describe is misbehaving? - pieces["error"] = ("unable to parse git-describe output: '%s'" - % describe_out) - return pieces - - # tag - full_tag = mo.group(1) - if not full_tag.startswith(tag_prefix): - if verbose: - fmt = "tag '%s' doesn't start with prefix '%s'" - print(fmt % (full_tag, tag_prefix)) - pieces["error"] = ("tag '%s' doesn't start with prefix '%s'" - % (full_tag, tag_prefix)) - return pieces - pieces["closest-tag"] = full_tag[len(tag_prefix):] - - # distance: number of commits since tag - pieces["distance"] = int(mo.group(2)) - - # commit: short hex revision ID - pieces["short"] = mo.group(3) - - else: - # HEX: no tags - pieces["closest-tag"] = None - count_out, rc = run_command(GITS, ["rev-list", "HEAD", "--count"], - cwd=root) - pieces["distance"] = int(count_out) # total number of commits - - # commit date: see ISO-8601 comment in git_versions_from_keywords() - date = run_command(GITS, ["show", "-s", "--format=%ci", "HEAD"], - cwd=root)[0].strip() - pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1) - - return pieces - - -def plus_or_dot(pieces): - """Return a + if we don't already have one, else return a .""" - if "+" in pieces.get("closest-tag", ""): - return "." - return "+" - - -def render_pep440(pieces): - """Build up version string, with post-release "local version identifier". - - Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you - get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty - - Exceptions: - 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty] - """ - if pieces["closest-tag"]: - rendered = pieces["closest-tag"] - if pieces["distance"] or pieces["dirty"]: - rendered += plus_or_dot(pieces) - rendered += "%d.g%s" % (pieces["distance"], pieces["short"]) - if pieces["dirty"]: - rendered += ".dirty" - else: - # exception #1 - rendered = "0+untagged.%d.g%s" % (pieces["distance"], - pieces["short"]) - if pieces["dirty"]: - rendered += ".dirty" - return rendered - - -def render_pep440_pre(pieces): - """TAG[.post.devDISTANCE] -- No -dirty. - - Exceptions: - 1: no tags. 0.post.devDISTANCE - """ - if pieces["closest-tag"]: - rendered = pieces["closest-tag"] - if pieces["distance"]: - rendered += ".post.dev%d" % pieces["distance"] - else: - # exception #1 - rendered = "0.post.dev%d" % pieces["distance"] - return rendered - - -def render_pep440_post(pieces): - """TAG[.postDISTANCE[.dev0]+gHEX] . - - The ".dev0" means dirty. Note that .dev0 sorts backwards - (a dirty tree will appear "older" than the corresponding clean one), - but you shouldn't be releasing software with -dirty anyways. - - Exceptions: - 1: no tags. 0.postDISTANCE[.dev0] - """ - if pieces["closest-tag"]: - rendered = pieces["closest-tag"] - if pieces["distance"] or pieces["dirty"]: - rendered += ".post%d" % pieces["distance"] - if pieces["dirty"]: - rendered += ".dev0" - rendered += plus_or_dot(pieces) - rendered += "g%s" % pieces["short"] - else: - # exception #1 - rendered = "0.post%d" % pieces["distance"] - if pieces["dirty"]: - rendered += ".dev0" - rendered += "+g%s" % pieces["short"] - return rendered - - -def render_pep440_old(pieces): - """TAG[.postDISTANCE[.dev0]] . - - The ".dev0" means dirty. - - Eexceptions: - 1: no tags. 0.postDISTANCE[.dev0] - """ - if pieces["closest-tag"]: - rendered = pieces["closest-tag"] - if pieces["distance"] or pieces["dirty"]: - rendered += ".post%d" % pieces["distance"] - if pieces["dirty"]: - rendered += ".dev0" - else: - # exception #1 - rendered = "0.post%d" % pieces["distance"] - if pieces["dirty"]: - rendered += ".dev0" - return rendered - - -def render_git_describe(pieces): - """TAG[-DISTANCE-gHEX][-dirty]. - - Like 'git describe --tags --dirty --always'. - - Exceptions: - 1: no tags. HEX[-dirty] (note: no 'g' prefix) - """ - if pieces["closest-tag"]: - rendered = pieces["closest-tag"] - if pieces["distance"]: - rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) - else: - # exception #1 - rendered = pieces["short"] - if pieces["dirty"]: - rendered += "-dirty" - return rendered - - -def render_git_describe_long(pieces): - """TAG-DISTANCE-gHEX[-dirty]. - - Like 'git describe --tags --dirty --always -long'. - The distance/hash is unconditional. - - Exceptions: - 1: no tags. HEX[-dirty] (note: no 'g' prefix) - """ - if pieces["closest-tag"]: - rendered = pieces["closest-tag"] - rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) - else: - # exception #1 - rendered = pieces["short"] - if pieces["dirty"]: - rendered += "-dirty" - return rendered - - -def render(pieces, style): - """Render the given version pieces into the requested style.""" - if pieces["error"]: - return {"version": "unknown", - "full-revisionid": pieces.get("long"), - "dirty": None, - "error": pieces["error"], - "date": None} - - if not style or style == "default": - style = "pep440" # the default - - if style == "pep440": - rendered = render_pep440(pieces) - elif style == "pep440-pre": - rendered = render_pep440_pre(pieces) - elif style == "pep440-post": - rendered = render_pep440_post(pieces) - elif style == "pep440-old": - rendered = render_pep440_old(pieces) - elif style == "git-describe": - rendered = render_git_describe(pieces) - elif style == "git-describe-long": - rendered = render_git_describe_long(pieces) - else: - raise ValueError("unknown style '%s'" % style) - - return {"version": rendered, "full-revisionid": pieces["long"], - "dirty": pieces["dirty"], "error": None, - "date": pieces.get("date")} - - -def get_versions(): - """Get version information or return default if unable to do so.""" - # I am in _version.py, which lives at ROOT/VERSIONFILE_SOURCE. If we have - # __file__, we can work backwards from there to the root. Some - # py2exe/bbfreeze/non-CPython implementations don't do __file__, in which - # case we can only use expanded keywords. - - cfg = get_config() - verbose = cfg.verbose - - try: - return git_versions_from_keywords(get_keywords(), cfg.tag_prefix, - verbose) - except NotThisMethod: - pass - - try: - root = os.path.realpath(__file__) - # versionfile_source is the relative path from the top of the source - # tree (where the .git directory might live) to this file. Invert - # this to find the root from __file__. - for i in cfg.versionfile_source.split('/'): - root = os.path.dirname(root) - except NameError: - return {"version": "0+unknown", "full-revisionid": None, - "dirty": None, - "error": "unable to find root of source tree", - "date": None} - - try: - pieces = git_pieces_from_vcs(cfg.tag_prefix, root, verbose) - return render(pieces, cfg.style) - except NotThisMethod: - pass - - try: - if cfg.parentdir_prefix: - return versions_from_parentdir(cfg.parentdir_prefix, root, verbose) - except NotThisMethod: - pass - - return {"version": "0+unknown", "full-revisionid": None, - "dirty": None, - "error": "unable to compute version", "date": None} diff --git a/impala/compat.py b/impala/compat.py deleted file mode 100644 index 961455ebd..000000000 --- a/impala/compat.py +++ /dev/null @@ -1,37 +0,0 @@ -# Copyright 2015 Cloudera Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# pylint: disable=unused-import,wrong-import-position - -from __future__ import absolute_import - -import six - -if six.PY3: - def lzip(*x): - return list(zip(*x)) - - from decimal import Decimal -elif six.PY2: - lzip = zip - - try: - from cdecimal import Decimal - except ImportError: - from decimal import Decimal # noqa - -try: - _xrange = xrange -except NameError: - _xrange = range # python3 compatibilty diff --git a/impala/dbapi.py b/impala/dbapi.py index c7b37b0f9..4e22ffa02 100644 --- a/impala/dbapi.py +++ b/impala/dbapi.py @@ -14,9 +14,6 @@ """Implements the Python DB API 2.0 (PEP 249) for Impala""" -from __future__ import absolute_import - -import six import time import datetime @@ -25,7 +22,7 @@ OperationalError, ProgrammingError, IntegrityError, DataError, NotSupportedError) from impala.util import ( - warn_deprecate, warn_protocol_param, warn_unused_jwt, warn_nontls_jwt) + warn_deprecate, warn_protocol_param, warn_nontls_jwt) import impala.hiveserver2 as hs2 @@ -44,7 +41,8 @@ def connect(host='localhost', port=21050, database=None, timeout=None, ldap_user=None, ldap_password=None, use_kerberos=None, protocol=None, krb_host=None, use_http_transport=False, http_path='', auth_cookie_names=None, http_cookie_names=None, - retries=3, jwt=None, user_agent=None): + retries=3, jwt=None, user_agent=None, + get_user_custom_headers_func=None, verify_cert=False): """Get a connection to HiveServer2 (HS2). These options are largely compatible with the impala-shell command line @@ -65,9 +63,9 @@ def connect(host='localhost', port=21050, database=None, timeout=None, use_ssl : bool, optional Enable SSL. ca_cert : str, optional - Local path to the the third-party CA certificate. If SSL is enabled but - the certificate is not specified, the server certificate will not be - validated. + Local path to the the third-party CA certificate. If set, the server certificate + will be verified using the provided CA cert. If SSL is enabled but + the certificate is not specified, see 'verify_cert' for behavior. auth_mechanism : {'NOSASL', 'PLAIN', 'GSSAPI', 'LDAP', 'JWT'} Specify the authentication mechanism. `'NOSASL'` for unsecured Impala. `'PLAIN'` for unsecured Hive (because Hive requires the SASL @@ -91,11 +89,11 @@ def connect(host='localhost', port=21050, database=None, timeout=None, name only, a str value can be specified instead of a list. If a cookie with one of these names is returned in an http response by the server or an intermediate proxy then it will be included in each subsequent request for - the same connection. + the same connection. If set to wildcard ('*'), all cookies in an http response + will be preserved. By default 'http_cookie_names' is set to '*'. Used only when `use_http_transport` is True. - By default 'http_cookie_names' is set to the list of HTTP cookie names used by - Impala and Hive. The names of authentication cookies are expected to end with - ".auth" string, for example, "impala.auth" for Impala authentication cookies. + The names of authentication cookies are expected to end with ".auth" string, for + example, "impala.auth" for Impala authentication cookies. If 'http_cookie_names' is explicitly set to a not None empty value ([], or ''), Impyla won't attempt to do cookie based authentication or session management. Currently cookie retention is supported for GSSAPI/LDAP/SASL/NOSASL/JWT over http. @@ -105,6 +103,14 @@ def connect(host='localhost', port=21050, database=None, timeout=None, 'Python/ImpylaHttpClient' is used use_ldap : bool, optional Specify `auth_mechanism='LDAP'` instead. + get_user_custom_headers_func : function, optional + Used to add custom headers to the http messages when using hs2-http protocol. + This is a function returning a list of tuples, each tuple contains a key-value + pair. This allows duplicate headers to be set. + verify_cert : bool, optional + Whether to verify the server's TLS certificate when using SSL using the systems's + CA certificates. Ignored if 'ca_cert' is provided, in which case the certificate + will be verified using the provided CA cert. .. deprecated:: 0.18.0 auth_cookie_names : list of str or str, optional @@ -163,10 +169,13 @@ def connect(host='localhost', port=21050, database=None, timeout=None, raise NotSupportedError('JWT authentication is only supported for HTTP transport') if not use_ssl: warn_nontls_jwt() + if user is not None or ldap_user is not None: + raise NotSupportedError("'user' argument cannot be specified with '{0}' authentication".format(auth_mechanism)) + if password is not None or ldap_password is not None: + raise NotSupportedError("'password' argument cannot be specified with '{0}' authentication".format(auth_mechanism)) else: if jwt is not None: - warn_unused_jwt() - + raise NotSupportedError("'jwt' argument cannot be specified with '{0}' authentication".format(auth_mechanism)) if ldap_user is not None: warn_deprecate('ldap_user', 'user') @@ -188,8 +197,8 @@ def connect(host='localhost', port=21050, database=None, timeout=None, warn_deprecate('auth_cookie_names', 'http_cookie_names') http_cookie_names = auth_cookie_names elif http_cookie_names is None: - # Set default value as the list of HTTP cookie names used by Impala and Hive. - http_cookie_names = ['impala.auth', 'impala.session.id', 'hive.server2.auth'] + # Preserve all cookies. + http_cookie_names = '*' service = hs2.connect(host=host, port=port, timeout=timeout, use_ssl=use_ssl, @@ -200,7 +209,9 @@ def connect(host='localhost', port=21050, database=None, timeout=None, http_path=http_path, http_cookie_names=http_cookie_names, retries=retries, - jwt=jwt, user_agent=user_agent) + jwt=jwt, user_agent=user_agent, + get_user_custom_headers_func=get_user_custom_headers_func, + verify_cert=verify_cert) return hs2.HiveServer2Connection(service, default_db=database) @@ -245,6 +256,5 @@ def TimeFromTicks(ticks): def TimestampFromTicks(ticks): return Timestamp(*time.localtime(ticks)[:6]) -if six.PY3: - buffer = memoryview +buffer = memoryview Binary = buffer diff --git a/impala/exec_summary.py b/impala/exec_summary.py new file mode 100755 index 000000000..96f06ced4 --- /dev/null +++ b/impala/exec_summary.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from impala._thrift_gen.ExecStats.ttypes import TExecStats + + +def build_exec_summary_table(summary, idx, indent_level, new_indent_level, output, + is_prettyprint=True, separate_prefix_column=False): + """Direct translation of Coordinator::PrintExecSummary() to recursively build a list + of rows of summary statistics, one per exec node + + summary: the TExecSummary object that contains all the summary data + + idx: the index of the node to print + + indent_level: the number of spaces to print before writing the node's label, to give + the appearance of a tree. The 0th child of a node has the same indent_level as its + parent. All other children have an indent_level of one greater than their parent. + + new_indent_level: If true, this indent level is different from the previous row's. + + output: the list of rows into which to append the rows produced for this node and its + children. + + is_prettyprint: Optional. If True, print time, units, and bytes columns in pretty + printed format. + + separate_prefix_column: Optional. If True, the prefix and operator name will be + returned as separate column. Otherwise, prefix and operater name will be concatenated + into single column. + + Returns the index of the next exec node in summary.exec_nodes that should be + processed, used internally to this method only. + """ + if not summary.nodes: + # Summary nodes is empty or None. Nothing to build. + return + assert idx < len(summary.nodes), ( + "Index ({0}) must be less than exec summary count ({1})").format( + idx, len(summary.nodes)) + + attrs = ["latency_ns", "cpu_time_ns", "cardinality", "memory_used"] + + # Initialise aggregate and maximum stats + agg_stats, max_stats = TExecStats(), TExecStats() + for attr in attrs: + setattr(agg_stats, attr, 0) + setattr(max_stats, attr, 0) + + node = summary.nodes[idx] + instances = 0 + if node.exec_stats: + # exec_stats is not None or an empty list. + instances = len(node.exec_stats) + for stats in node.exec_stats: + for attr in attrs: + val = getattr(stats, attr) + if val is not None: + setattr(agg_stats, attr, getattr(agg_stats, attr) + val) + setattr(max_stats, attr, max(getattr(max_stats, attr), val)) + avg_time = agg_stats.latency_ns / instances + else: + avg_time = 0 + + is_sink = node.node_id == -1 + # If the node is a broadcast-receiving exchange node, the cardinality of rows produced + # is the max over all instances (which should all have received the same number of + # rows). Otherwise, the cardinality is the sum over all instances which process + # disjoint partitions. + if is_sink: + cardinality = -1 + elif node.is_broadcast: + cardinality = max_stats.cardinality + else: + cardinality = agg_stats.cardinality + + est_stats = node.estimated_stats + label_prefix = "" + if indent_level > 0: + label_prefix = "|" + label_prefix += " |" * (indent_level - 1) + if new_indent_level: + label_prefix += "--" + else: + label_prefix += " " + + def prettyprint(val, units, divisor): + for unit in units: + if val < divisor: + if unit == units[0]: + return "%d%s" % (val, unit) + else: + return "%3.2f%s" % (val, unit) + val /= divisor + + def prettyprint_bytes(byte_val): + return prettyprint(byte_val, [' B', ' KB', ' MB', ' GB', ' TB'], 1024.0) + + def prettyprint_units(unit_val): + return prettyprint(unit_val, ["", "K", "M", "B"], 1000.0) + + def prettyprint_time(time_val): + return prettyprint(time_val, ["ns", "us", "ms", "s"], 1000.0) + + latency = max_stats.latency_ns + cardinality_est = est_stats.cardinality + memory_used = max_stats.memory_used + memory_est = est_stats.memory_used + if (is_prettyprint): + avg_time = prettyprint_time(avg_time) + latency = prettyprint_time(latency) + cardinality = "" if is_sink else prettyprint_units(cardinality) + cardinality_est = "" if is_sink else prettyprint_units(cardinality_est) + memory_used = prettyprint_bytes(memory_used) + memory_est = prettyprint_bytes(memory_est) + + row = list() + if separate_prefix_column: + row.append(label_prefix) + row.append(node.label) + else: + row.append(label_prefix + node.label) + + row.extend([ + node.num_hosts, + instances, + avg_time, + latency, + cardinality, + cardinality_est, + memory_used, + memory_est, + node.label_detail]) + + output.append(row) + try: + sender_idx = summary.exch_to_sender_map[idx] + # This is an exchange node or a join node with a separate builder, so the source + # is a fragment root, and should be printed next. + sender_indent_level = indent_level + node.num_children + sender_new_indent_level = node.num_children > 0 + build_exec_summary_table(summary, sender_idx, sender_indent_level, + sender_new_indent_level, output, is_prettyprint, + separate_prefix_column) + except (KeyError, TypeError): + # Fall through if idx not in map, or if exch_to_sender_map itself is not set + pass + + idx += 1 + if node.num_children > 0: + first_child_output = [] + idx = build_exec_summary_table(summary, idx, indent_level, False, first_child_output, + is_prettyprint, separate_prefix_column) + for _ in range(1, node.num_children): + # All other children are indented + idx = build_exec_summary_table(summary, idx, indent_level + 1, True, output, + is_prettyprint, separate_prefix_column) + output += first_child_output + return idx diff --git a/impala/hiveserver2.py b/impala/hiveserver2.py index 73f4555ff..7704e7c2a 100644 --- a/impala/hiveserver2.py +++ b/impala/hiveserver2.py @@ -12,23 +12,21 @@ # See the License for the specific language governing permissions and # limitations under the License. -from __future__ import absolute_import - import getpass import re import socket import datetime +from decimal import Decimal import operator -import six import sys import time from bitarray import bitarray -from six.moves import range from thrift.transport.TTransport import TTransportException from thrift.Thrift import TApplicationException from thrift.protocol.TBinaryProtocol import TBinaryProtocolAccelerated +from impala._thrift_gen.ExecStats.ttypes import TExecStats from impala._thrift_gen.TCLIService.ttypes import ( TOpenSessionReq, TFetchResultsReq, TCloseSessionReq, TExecuteStatementReq, TGetInfoReq, TGetInfoType, TTypeId, @@ -37,13 +35,13 @@ TGetOperationStatusReq, TOperationState, TCancelOperationReq, TCloseOperationReq, TGetLogReq, TProtocolVersion) from impala._thrift_gen.ImpalaService.ImpalaHiveServer2Service import ( - TGetRuntimeProfileReq, TGetExecSummaryReq) + TGetRuntimeProfileReq, TGetExecSummaryReq, TCloseImpalaOperationReq) from impala._thrift_api import ( get_socket, get_http_transport, get_transport, ThriftClient) from impala._thrift_gen.RuntimeProfile.ttypes import TRuntimeProfileFormat -from impala.compat import (Decimal, _xrange as xrange) from impala.error import (NotSupportedError, OperationalError, ProgrammingError, HiveServer2Error, HttpError) +from impala.exec_summary import build_exec_summary_table from impala.interface import Connection, Cursor, _bind_parameters from impala.util import get_logger_and_init_null @@ -84,7 +82,8 @@ def rollback(self): raise NotSupportedError def cursor(self, user=None, configuration=None, convert_types=True, - dictify=False, fetch_error=True): + dictify=False, fetch_error=True, close_finished_queries=True, + convert_strings_to_unicode=True): """Get a cursor from the HiveServer2 (HS2) connection. Parameters @@ -96,6 +95,12 @@ def cursor(self, user=None, configuration=None, convert_types=True, When `False`, timestamps and decimal values will not be converted to Python `datetime` and `Decimal` values. (These conversions are expensive.) Only applies when using HS2 protocol versions > 6. + convert_strings_to_unicode : bool, optional + When `True`, the following types, which are transmitted as strings + in HS2 protocol, will be converted to unicode: STRING, LIST, MAP, + STRUCT, UNIONTYPE, NULL, VARCHAR, CHAR, TIMESTAMP, DECIMAL, DATE. + When `False`, conversion will occur only for types expected by + convert_types in python3: TIMESTAMP, DECIMAL, DATE. dictify : bool, optional When `True` cursor will return key value pairs instead of rows. fetch_error : bool, optional @@ -112,6 +117,20 @@ def cursor(self, user=None, configuration=None, convert_types=True, handle remains valid and impyla will raise an exception with a message of "Operation is in ERROR_STATE". The Default option is `True`. + close_finished_queries : bool, optional + If True, queries are closed after: + - queries with results set: all rows are returned with fetch + - DDL/DML: execution is finished + If False, then the query will be only closed when: + - execute() is called again on the cursor with a new query + - close() is called on the cursor + - the cursor's destructor is called + Property 'rowcount' will not be available in the 'False' case for DML + statements. + Before closing the query GetLog() is called as this will be no longer + possible after closing. + The Default option is `True`. + Returns ------- @@ -136,7 +155,9 @@ def cursor(self, user=None, configuration=None, convert_types=True, cursor_class = HiveServer2DictCursor if dictify else HiveServer2Cursor cursor = cursor_class(session, convert_types=convert_types, - fetch_error=fetch_error) + fetch_error=fetch_error, + close_finished_queries=close_finished_queries, + convert_strings_to_unicode=convert_strings_to_unicode) if self.default_db is not None: log.info('Using database %s as default', self.default_db) @@ -153,16 +174,20 @@ class HiveServer2Cursor(Cursor): # HiveServer2Cursor objects are associated with a Session # they are instantiated with alive session_handles - def __init__(self, session, convert_types=True, fetch_error=True): + def __init__(self, session, convert_types=True, fetch_error=True, close_finished_queries=True, + convert_strings_to_unicode=True): self.session = session self.convert_types = convert_types + self.convert_strings_to_unicode = convert_strings_to_unicode self.fetch_error = fetch_error + self.close_finished_queries = close_finished_queries self._last_operation = None self._last_operation_string = None self._last_operation_active = False self._last_operation_finished = False + self._last_operation_log = None self._buffersize = None self._buffer = Batch() # zero-length @@ -203,9 +228,12 @@ def rowcount(self): @property def rowcounts(self): - # Work around to get the number of rows modified for Inserts/Update/Delte statements + # Work around to get the number of rows modified for Inserts/Update/Delete statements + # Todo: For the non-Kudu case, this function could use self._rowcount without fetching + # and parsing the profile. This wouldn't be enough for Kudu as NumRowErrors is not + # included in DmlResult. modifiedRows, errorRows = -1, -1 - if self._last_operation_active: + if self._last_operation is not None: logList = self.get_profile().split('\n') resultDict = {} subs = ['NumModifiedRows', 'NumRowErrors'] @@ -289,7 +317,7 @@ def close(self): # If there was an error when closing last operation then # raise exception if exc_info: - six.reraise(*exc_info) + raise exc_info[1].with_traceback(exc_info[2]) def cancel_operation(self, reset_state=True): if self._last_operation_active: @@ -299,8 +327,8 @@ def cancel_operation(self, reset_state=True): self._reset_state() def close_operation(self): - if self._last_operation_active: - log.info('Closing active operation') + if self._last_operation is not None: + log.debug('Closing operation') self._reset_state() def _reset_state(self): @@ -314,11 +342,32 @@ def _reset_state(self): self._last_operation_finished = False self._last_operation_string = None self._last_operation = None + self._last_operation_log = None + self._rowcount = -1 + + def _set_rowcount_from_close_result(self, close_result): + if not hasattr(close_result, 'dml_result') or not close_result.dml_result: + return + rows_modified_per_partition = close_result.dml_result.rows_modified + self._rowcount = 0 + for _, val in rows_modified_per_partition.items(): + self._rowcount += val + + def _close_finished_operation(self): + # Save the log as it can't be accessed after closing the query. + self._last_operation_log = self.get_log() + self._last_operation_active = False + close_result = self._last_operation.close() + log.debug('Query closed') + # Set rowcount for DMLs. + self._set_rowcount_from_close_result(close_result) def execute(self, operation, parameters=None, configuration=None): """Synchronously execute a SQL query. Blocks until results are available. + For DMLs/DDLs if close_finished_queries is true then the query is + closed once finished. Parameters ---------- @@ -342,6 +391,10 @@ def execute(self, operation, parameters=None, configuration=None): log.debug('Waiting for query to finish') self._wait_to_finish() # make execute synchronous log.debug('Query finished') + if not self.has_result_set and self.close_finished_queries: + # Close query if no results need to be fetched. + self._close_finished_operation() + def execute_async(self, operation, parameters=None, configuration=None): """Asynchronously execute a SQL query. @@ -388,7 +441,7 @@ def op(): self._execute_async(op) def _debug_log_state(self): - if self._last_operation_active: + if self._last_operation is not None: handle = self._last_operation.handle else: handle = None @@ -416,6 +469,7 @@ def _wait_to_finish(self): return loop_start = time.time() while True: + start_rpc_time = time.time() req = TGetOperationStatusReq(operationHandle=self._last_operation.handle) resp = self._last_operation._rpc('GetOperationStatus', req, True) self._last_operation.update_has_result_set(resp) @@ -435,7 +489,13 @@ def _wait_to_finish(self): if not self._op_state_is_executing(operation_state): self._last_operation_finished = True break - time.sleep(self._get_sleep_interval(loop_start)) + rpc_time = time.time() - start_rpc_time + sleep_time = self._get_sleep_interval(loop_start) + # Subtract RPC time from the total sleep time. If query option + # long_polling_time_ms is set then Impala will sleep in GetOperationStatus + # meaning that impyla may not need to sleep at all (IMPALA-13294). + if rpc_time < sleep_time: + time.sleep(sleep_time - rpc_time) def status(self): if self._last_operation is None: @@ -478,11 +538,18 @@ def _get_sleep_interval(self, start_time): def executemany(self, operation, seq_of_parameters, configuration=None): # PEP 249 log.debug('Attempting to execute %s queries', len(seq_of_parameters)) + rowcount = -1 for parameters in seq_of_parameters: self.execute(operation, parameters, configuration) if self.has_result_set: raise ProgrammingError("Operations that have result sets are " "not allowed with executemany.") + if self._rowcount != -1: + if rowcount == -1: + rowcount = self._rowcount + else: + rowcount += self._rowcount + self._rowcount = rowcount def fetchone(self): # PEP 249 @@ -518,7 +585,8 @@ def fetchcbatch(self): batch = (self._last_operation.fetch( self.description, self.buffersize, - convert_types=self.convert_types)) + convert_types=self.convert_types, + convert_strings_to_unicode=self.convert_strings_to_unicode)) if len(batch) == 0: return None return batch @@ -568,9 +636,12 @@ def fetchcolumnar(self): batch = (self._last_operation.fetch( self.description, self.buffersize, - convert_types=self.convert_types)) + convert_types=self.convert_types, + convert_strings_to_unicode=self.convert_strings_to_unicode)) if len(batch) == 0: - break + if not batch.expect_more_rows: + break + continue batches.append(batch) return batches @@ -591,9 +662,12 @@ def next(self): def __next__(self): self._ensure_buffer_is_filled() log.debug('__next__: popping row out of buffer') + self._rowcount += 1 return self._buffer.pop() def _ensure_buffer_is_filled(self): + if self._rowcount == -1: + self._rowcount = 0 while True: if not self.has_result_set: raise ProgrammingError( @@ -604,12 +678,19 @@ def _ensure_buffer_is_filled(self): log.debug('_ensure_buffer_is_filled: buffer empty and op is active ' '=> fetching more data') self._buffer = self._last_operation.fetch(self.description, - self.buffersize, - convert_types=self.convert_types) + self.buffersize, + convert_types=self.convert_types, + convert_strings_to_unicode=self.convert_strings_to_unicode) if len(self._buffer) > 0: return if not self._buffer.expect_more_rows: log.debug('_ensure_buffer_is_filled: no more data to fetch') + if self.close_finished_queries: + # Close query as it no longer has rows. + # TODO: this could be done earlier after calling fetch - not sure + # if this would bring enough benefits to worth complicating + # the state machine + self._close_finished_operation() raise StopIteration # If we didn't get rows, but more are expected, need to iterate again. else: @@ -619,7 +700,9 @@ def _ensure_buffer_is_filled(self): def _pop_from_buffer(self, size): self._ensure_buffer_is_filled() log.debug('pop_from_buffer: popping row out of buffer') - return self._buffer.pop_many(size) + elements = self._buffer.pop_many(size) + self._rowcount += len(elements) + return elements def ping(self): """Checks connection to server by requesting some info.""" @@ -629,6 +712,9 @@ def ping(self): def get_log(self): if self._last_operation is None: raise ProgrammingError("Operation state is not available") + if self._last_operation_log is not None: + # Return the log saved before closing the query. + return self._last_operation_log return self._last_operation.get_log() def get_profile(self, profile_format=TRuntimeProfileFormat.STRING): @@ -641,8 +727,9 @@ def get_summary(self): def build_summary_table(self, summary, output, idx=0, is_fragment_root=False, indent_level=0): - return build_summary_table(summary, idx, is_fragment_root, - indent_level, output) + return build_exec_summary_table( + summary, idx, indent_level, is_fragment_root, output, is_prettyprint=True, + separate_prefix_column=False) def get_databases(self): def op(): @@ -829,7 +916,7 @@ def connect(host, port, timeout=None, use_ssl=False, ca_cert=None, user=None, password=None, kerberos_service_name='impala', auth_mechanism=None, krb_host=None, use_http_transport=False, http_path='', http_cookie_names=None, retries=3, jwt=None, - user_agent=None): + user_agent=None, get_user_custom_headers_func=None, verify_cert=False): log.debug('Connecting to HiveServer2 %s:%s with %s authentication ' 'mechanism', host, port, auth_mechanism) @@ -839,21 +926,19 @@ def connect(host, port, timeout=None, use_ssl=False, ca_cert=None, kerberos_host = host if use_http_transport: - # TODO(#362): Add server authentication with thrift 0.12. - if ca_cert: - raise NotSupportedError("Server authentication is not supported " + - "with HTTP endpoints") - - transport = get_http_transport(host, port, http_path=http_path, - use_ssl=use_ssl, ca_cert=ca_cert, - auth_mechanism=auth_mechanism, - user=user, password=password, - kerberos_host=kerberos_host, - kerberos_service_name=kerberos_service_name, - http_cookie_names=http_cookie_names, - jwt=jwt, user_agent=user_agent) + transport = get_http_transport( + host, port, http_path=http_path, + use_ssl=use_ssl, ca_cert=ca_cert, + auth_mechanism=auth_mechanism, + user=user, password=password, + kerberos_host=kerberos_host, + kerberos_service_name=kerberos_service_name, + http_cookie_names=http_cookie_names, + jwt=jwt, user_agent=user_agent, + get_user_custom_headers_func=get_user_custom_headers_func, + verify_cert=verify_cert) else: - sock = get_socket(host, port, use_ssl, ca_cert) + sock = get_socket(host, port, use_ssl, ca_cert, verify_cert) if timeout is not None: timeout = timeout * 1000. # TSocket expects millis @@ -938,7 +1023,7 @@ def pop_to_preallocated_list(self, output_list, count, offset=0, stride=1): count = min(count, self.rows_left) start_pos = self.num_rows - self.rows_left self.rows_left -= count - for pos in xrange(start_pos, start_pos + count): + for pos in range(start_pos, start_pos + count): output_list[offset] = None if self.nulls[pos] else self.values[pos] offset += stride return count @@ -946,17 +1031,27 @@ def pop_to_preallocated_list(self, output_list, count, offset=0, stride=1): class CBatch(Batch): - def __init__(self, trowset, expect_more_rows, schema, convert_types=True): + def __init__(self, trowset, expect_more_rows, schema, convert_types=True, + convert_strings_to_unicode=True): self.expect_more_rows = expect_more_rows self.schema = schema - tcols = [_TTypeId_to_TColumnValue_getters[schema[i][1]](col) - for (i, col) in enumerate(trowset.columns)] - num_cols = len(tcols) - num_rows = len(tcols[0].values) + if trowset: + tcols = [_TTypeId_to_TColumnValue_getters[schema[i][1]](col) + for (i, col) in enumerate(trowset.columns)] + num_cols = len(tcols) + num_rows = len(tcols[0].values) + else: + # No results returned with STILL_EXECUTING_STATUS + tcols = [] + num_cols = 0 + num_rows = 0 self.remaining_rows = num_rows log.debug('CBatch: input TRowSet num_cols=%s num_rows=%s tcols=%s', num_cols, num_rows, tcols) + + HS2_STRING_TYPES = ["STRING", "LIST", "MAP", "STRUCT", "UNIONTYPE", "NULL", "VARCHAR", "CHAR", "TIMESTAMP", "DECIMAL", "DATE"] + CONVERTED_TYPES=["TIMESTAMP", "DECIMAL", "DATE"] self.columns = [] for j in range(num_cols): @@ -974,14 +1069,31 @@ def __init__(self, trowset, expect_more_rows, schema, convert_types=True): # STRING columns are read as binary and decoded here to be able to handle # non-valid utf-8 strings in Python 3. - if six.PY3: - self._convert_strings_to_unicode(type_, is_null, values) + + if convert_strings_to_unicode: + self._convert_strings_to_unicode(type_, is_null, values, types=HS2_STRING_TYPES) + elif convert_types: + self._convert_strings_to_unicode(type_, is_null, values, types=CONVERTED_TYPES) if convert_types: values = self._convert_values(type_, is_null, values) self.columns.append(Column(type_, values, is_null)) + def _convert_strings_to_unicode(self, type_, is_null, values, types): + if type_ in types: + for i in range(len(values)): + if is_null[i]: + values[i] = None + continue + try: + # Do similar handling of non-valid UTF-8 strings as Thriftpy2: + # https://github.com/Thriftpy/thriftpy2/blob/8e218b3fd89c597c2e83d129efecfe4d280bdd89/thriftpy2/protocol/binary.py#L241 + # If decoding fails then keep the original bytearray. + values[i] = values[i].decode("UTF-8") + except UnicodeDecodeError: + pass + def _convert_values(self, type_, is_null, values): # pylint: disable=consider-using-enumerate if type_ == 'TIMESTAMP': @@ -996,20 +1108,6 @@ def _convert_values(self, type_, is_null, values): values[i] = (None if is_null[i] else _parse_date(values[i])) return values - def _convert_strings_to_unicode(self, type_, is_null, values): - if type_ in ["STRING", "LIST", "MAP", "STRUCT", "UNIONTYPE", "DECIMAL", "DATE", "TIMESTAMP", "NULL", "VARCHAR", "CHAR"]: - for i in range(len(values)): - if is_null[i]: - values[i] = None - continue - try: - # Do similar handling of non-valid UTF-8 strings as Thriftpy2: - # https://github.com/Thriftpy/thriftpy2/blob/8e218b3fd89c597c2e83d129efecfe4d280bdd89/thriftpy2/protocol/binary.py#L241 - # If decoding fails then keep the original bytearray. - values[i] = values[i].decode("UTF-8") - except UnicodeDecodeError: - pass - def __len__(self): return self.remaining_rows @@ -1036,7 +1134,7 @@ def pop_many(self, row_count): assert row_count == rows_returned # Split 'dataset' to 'col_count' sized sublists and create tuples from them. return [tuple(dataset[i * col_count: (i + 1) * col_count]) - for i in xrange(row_count)] + for i in range(row_count)] class RBatch(Batch): @@ -1045,6 +1143,9 @@ def __init__(self, trowset, expect_more_rows, schema): self.expect_more_rows = expect_more_rows self.schema = schema self.rows = [] + # Can be None with STILL_EXECUTING_STATUS + if not trowset: + return for trow in trowset.rows: row = [] for (i, col_val) in enumerate(trow.colVals): @@ -1078,41 +1179,40 @@ def __init__(self, client, retries=3): self.client = client self.retries = retries - def _rpc(self, func_name, request, retry_on_http_error=False): + def _rpc(self, func_name, request, safe_to_retry=False): self._log_request(func_name, request) - response = self._execute(func_name, request, retry_on_http_error) + response = self._execute(func_name, request, safe_to_retry) self._log_response(func_name, response) err_if_rpc_not_ok(response) return response - def _execute(self, func_name, request, retry_on_http_error=False): + def _execute(self, func_name, request, safe_to_retry=False): # pylint: disable=protected-access # get the thrift transport transport = self.client._iprot.trans tries_left = self.retries - last_http_exception = None + last_exception = None + open_finished = False while tries_left > 0: try: log.debug('Attempting to open transport (tries_left=%s)', tries_left) open_transport(transport) + open_finished = True log.debug('Transport opened') func = getattr(self.client, func_name) return func(request) - except socket.error: - log.exception('Failed to open transport (tries_left=%s)', - tries_left) - last_http_exception = None - except TTransportException: - log.exception('Failed to open transport (tries_left=%s)', - tries_left) - last_http_exception = None + except (socket.error, TTransportException) as e: + if open_finished and not safe_to_retry: raise e + msg = "RPC failed" if open_finished else "Failed to open transport" + log.exception('%s (tries_left=%s)', msg, tries_left) + last_exception = e except HttpError as h: - if not retry_on_http_error: + if not safe_to_retry: log.debug('Caught HttpError %s %s in %s which is not retryable', h, str(h.body or ''), func_name) raise - last_http_exception = h + last_exception = h if tries_left > 1: retry_secs = None retry_after = h.http_headers.get('Retry-After', None) @@ -1137,15 +1237,16 @@ def _execute(self, func_name, request, retry_on_http_error=False): raise log.debug('Closing transport (tries_left=%s)', tries_left) transport.close() + open_finished = False tries_left -= 1 - if last_http_exception is not None: - raise last_http_exception + if last_exception: + raise last_exception raise HiveServer2Error('Failed after retrying {0} times' .format(self.retries)) - def _operation(self, kind, request, retry_on_http_error=False): - resp = self._rpc(kind, request, retry_on_http_error) + def _operation(self, kind, request, safe_to_retry=False): + resp = self._rpc(kind, request, safe_to_retry) return self._get_operation(resp.operationHandle) def _log_request(self, kind, request): @@ -1190,7 +1291,8 @@ def open_session(self, user, configuration=None): resp = self._rpc('OpenSession', req, True) return HS2Session(self, resp.sessionHandle, resp.configuration, - resp.serverProtocolVersion) + resp.serverProtocolVersion, + retries=self.retries) class HS2Session(ThriftRPC): @@ -1220,7 +1322,7 @@ def execute(self, statement, configuration=None, run_async=False): statement=statement, confOverlay=configuration, runAsync=run_async) - # Do not try to retry http requests. + # Do not attempt to retry requests. # Read queries should be idempotent but most dml queries are not. Also retrying # query execution from client could be expensive and so likely makes sense to do # it if server is also aware of the retries. @@ -1291,7 +1393,8 @@ def ping(self): return True def _get_operation(self, handle): - return Operation(self, handle) + return Operation(self, handle, + retries=self.retries) class Operation(ThriftRPC): @@ -1346,7 +1449,7 @@ def get_log(self, max_rows=1024, orientation=TFetchOrientation.FETCH_NEXT): resp = self._rpc('FetchResults', req, False) schema = [('Log', 'STRING', None, None, None, None, None)] log = self._wrap_results(resp.results, resp.hasMoreRows, schema, - convert_types=True) + convert_types=True, convert_strings_to_unicode=True) log = '\n'.join(l[0] for l in log) return log @@ -1356,10 +1459,21 @@ def cancel(self): return self._rpc('CancelOperation', req, True) def close(self): - req = TCloseOperationReq(operationHandle=self.handle) - # CloseOperation rpc is not idempotent for dml and we're not sure + # Try Impala specific CloseImpalaOperation() as it also returns the number of + # modified rows for DML statements. + # If it doesn't exist (Hive, old Impala) fallback to regular HS2 CloseOperation() + # The RPCs are not retried as CloseOperation rpc is not idempotent for dml and we're not sure # here if this is dml or not. - return self._rpc('CloseOperation', req, False) + # TODO: we know in many cases that the query can't be a DML. Not sure if it worth putting + # effort into retrying close() + try: + req = TCloseImpalaOperationReq(operationHandle=self.handle) + return self._rpc('CloseImpalaOperation', req, False) + except TApplicationException as e: + if not e.type == TApplicationException.UNKNOWN_METHOD: + raise + req = TCloseOperationReq(operationHandle=self.handle) + return self._rpc('CloseOperation', req, False) def get_profile(self, profile_format=TRuntimeProfileFormat.STRING): req = TGetRuntimeProfileReq(operationHandle=self.handle, @@ -1380,7 +1494,7 @@ def get_summary(self): def fetch(self, schema=None, max_rows=1024, orientation=TFetchOrientation.FETCH_NEXT, - convert_types=True): + convert_types=True, convert_strings_to_unicode=True): if not self.has_result_set: log.debug('fetch_results: has_result_set=False') return None @@ -1396,15 +1510,18 @@ def fetch(self, schema=None, max_rows=1024, # results are kept around for retry to be successful. resp = self._rpc('FetchResults', req, False) return self._wrap_results(resp.results, resp.hasMoreRows, schema, - convert_types=convert_types) + convert_types=convert_types, + convert_strings_to_unicode=convert_strings_to_unicode) - def _wrap_results(self, results, expect_more_rows, schema, convert_types=True): + def _wrap_results(self, results, expect_more_rows, schema, convert_types=True, + convert_strings_to_unicode=True): if self.is_columnar: log.debug('fetch_results: constructing CBatch') - return CBatch(results, expect_more_rows, schema, convert_types=convert_types) + return CBatch(results, expect_more_rows, schema, convert_types=convert_types, + convert_strings_to_unicode=convert_strings_to_unicode) else: log.debug('fetch_results: constructing RBatch') - # TODO: RBatch ignores 'convert_types' + # TODO: RBatch ignores 'convert_types' and 'convert_strings_to_unicode' return RBatch(results, expect_more_rows, schema) @property @@ -1438,122 +1555,3 @@ def get_result_schema(self): log.debug('get_result_schema: schema=%s', schema) return schema - - -def build_summary_table(summary, idx, is_fragment_root, indent_level, output): - """Direct translation of Coordinator::PrintExecSummary() to recursively - build a list of rows of summary statistics, one per exec node - - summary: the TExecSummary object that contains all the summary data - - idx: the index of the node to print - - is_fragment_root: true if the node to print is the root of a fragment (and - therefore feeds into an exchange) - - indent_level: the number of spaces to print before writing the node's - label, to give the appearance of a tree. The 0th child of a node has the - same indent_level as its parent. All other children have an indent_level - of one greater than their parent. - - output: the list of rows into which to append the rows produced for this - node and its children. - - Returns the index of the next exec node in summary.exec_nodes that should - be processed, used internally to this method only. - """ - # pylint: disable=too-many-locals - - attrs = ["latency_ns", "cpu_time_ns", "cardinality", "memory_used"] - - # Initialise aggregate and maximum stats - agg_stats, max_stats = TExecStats(), TExecStats() - for attr in attrs: - setattr(agg_stats, attr, 0) - setattr(max_stats, attr, 0) - - node = summary.nodes[idx] - for stats in node.exec_stats: - for attr in attrs: - val = getattr(stats, attr) - if val is not None: - setattr(agg_stats, attr, getattr(agg_stats, attr) + val) - setattr(max_stats, attr, max(getattr(max_stats, attr), val)) - - if len(node.exec_stats) > 0: - avg_time = agg_stats.latency_ns / len(node.exec_stats) - else: - avg_time = 0 - - # If the node is a broadcast-receiving exchange node, the cardinality of - # rows produced is the max over all instances (which should all have - # received the same number of rows). Otherwise, the cardinality is the sum - # over all instances which process disjoint partitions. - if node.is_broadcast and is_fragment_root: - cardinality = max_stats.cardinality - else: - cardinality = agg_stats.cardinality - - est_stats = node.estimated_stats - label_prefix = "" - if indent_level > 0: - label_prefix = "|" - if is_fragment_root: - label_prefix += " " * indent_level - else: - label_prefix += "--" * indent_level - - def prettyprint(val, units, divisor): - for unit in units: - if val < divisor: - if unit == units[0]: - return "%d%s" % (val, unit) - else: - return "%3.2f%s" % (val, unit) - val /= divisor - - def prettyprint_bytes(byte_val): - return prettyprint( - byte_val, [' B', ' KB', ' MB', ' GB', ' TB'], 1024.0) - - def prettyprint_units(unit_val): - return prettyprint(unit_val, ["", "K", "M", "B"], 1000.0) - - def prettyprint_time(time_val): - return prettyprint(time_val, ["ns", "us", "ms", "s"], 1000.0) - - row = [label_prefix + node.label, - len(node.exec_stats), - prettyprint_time(avg_time), - prettyprint_time(max_stats.latency_ns), - prettyprint_units(cardinality), - prettyprint_units(est_stats.cardinality), - prettyprint_bytes(max_stats.memory_used), - prettyprint_bytes(est_stats.memory_used), - node.label_detail] - - output.append(row) - try: - sender_idx = summary.exch_to_sender_map[idx] - # This is an exchange node, so the sender is a fragment root, and - # should be printed next. - build_summary_table(summary, sender_idx, True, indent_level, output) - except (KeyError, TypeError): - # Fall through if idx not in map, or if exch_to_sender_map itself is - # not set - pass - - idx += 1 - if node.num_children > 0: - first_child_output = [] - idx = build_summary_table(summary, idx, False, indent_level, - first_child_output) - # pylint: disable=unused-variable - # TODO: is child_idx supposed to be unused? See #120 - for child_idx in range(1, node.num_children): - # All other children are indented (we only have 0, 1 or 2 children - # for every exec node at the moment) - idx = build_summary_table(summary, idx, False, indent_level + 1, - output) - output += first_child_output - return idx diff --git a/impala/interface.py b/impala/interface.py index 2efb8fbc7..028dce481 100644 --- a/impala/interface.py +++ b/impala/interface.py @@ -12,12 +12,8 @@ # See the License for the specific language governing permissions and # limitations under the License. -from __future__ import absolute_import - import datetime import re -import six -from six import reraise from impala.util import _escape from impala.error import ( # pylint: disable=unused-import @@ -66,7 +62,7 @@ def __enter__(self): def __exit__(self, exc_type, exc_val, exc_tb): self.close() if exc_type is not None: - reraise(exc_type, exc_val, exc_tb) + raise exc_val.with_traceback(exc_tb) # optional DB API addition to make the errors attributes of Connection Error = Error @@ -176,7 +172,7 @@ def __enter__(self): def __exit__(self, exc_type, exc_val, exc_tb): self.close() if exc_type is not None: - reraise(exc_type, exc_val, exc_tb) + raise exc_val.with_traceback(exc_tb) def _replace_numeric_markers(operation, string_parameters, paramstyle): @@ -244,7 +240,7 @@ def _bind_parameters_list(operation, parameters, paramstyle): for value in parameters: if value is None: string_parameters.append('NULL') - elif isinstance(value, six.string_types): + elif isinstance(value, str): string_parameters.append("'" + _escape(value) + "'") elif isinstance(value, datetime.datetime): string_parameters.append("'" + str(value) + "'") @@ -259,10 +255,10 @@ def _bind_parameters_list(operation, parameters, paramstyle): def _bind_parameters_dict(operation, parameters): string_parameters = {} - for (name, value) in six.iteritems(parameters): + for (name, value) in parameters.items(): if value is None: string_parameters[name] = 'NULL' - elif isinstance(value, six.string_types): + elif isinstance(value, str): string_parameters[name] = "'" + _escape(value) + "'" elif isinstance(value, datetime.date): string_parameters[name] = "'{0}'".format(value) @@ -270,7 +266,7 @@ def _bind_parameters_dict(operation, parameters): string_parameters[name] = str(value) # replace named parameters by their pyformat equivalents - operation = re.sub(":([^\d\W]\w*)", "%(\g<1>)s", operation) + operation = re.sub(r":([^\d\W]\w*)", r"%(\g<1>)s", operation) # replace pyformat parameters return operation % string_parameters diff --git a/impala/sqlalchemy.py b/impala/sqlalchemy.py index 003eb61fa..15feeded3 100644 --- a/impala/sqlalchemy.py +++ b/impala/sqlalchemy.py @@ -14,8 +14,6 @@ # some inspiration from Dropbox's PyHive -from __future__ import absolute_import - import re from sqlalchemy.dialects import registry @@ -24,8 +22,9 @@ from sqlalchemy.sql.compiler import (DDLCompiler, GenericTypeCompiler, IdentifierPreparer) from sqlalchemy.types import (BOOLEAN, SMALLINT, BIGINT, TIMESTAMP, FLOAT, - DECIMAL, Integer, Float, String, VARCHAR, + DECIMAL, Integer, Float, String, CHAR, VARCHAR, DATE) +from sqlalchemy import text registry.register('impala', 'impala.sqlalchemy', 'ImpalaDialect') @@ -170,6 +169,7 @@ def __init__(self, dialect): 'DOUBLE': DOUBLE, 'STRING': STRING, 'DECIMAL': DECIMAL, + 'CHAR': CHAR, 'VARCHAR': VARCHAR, 'DATE': DATE } @@ -200,13 +200,24 @@ class ImpalaDialect(DefaultDialect): ddl_compiler = ImpalaDDLCompiler type_compiler = ImpalaTypeCompiler execution_ctx_cls = ImpalaExecutionContext + # disable supports_statement_cache explicitly to avoid warning in sqlalchemy. + # TODO: it is not clear whether it would be safe to enable, needs more research + supports_statement_cache = False + # sqlalchemy dbapi() is deprecated in favor of import_dbapi(). + # Keeping both to remain as compatible as possible. @classmethod def dbapi(cls): # pylint: disable=method-hidden import impala.dbapi return impala.dbapi + @classmethod + def import_dbapi(cls): + # pylint: disable=method-hidden + import impala.dbapi + return impala.dbapi + def create_connect_args(self, url): kwargs = { 'host': url.host, @@ -222,12 +233,12 @@ def initialize(self, connection): self.default_schema_name = connection.connection.default_db def _get_server_version_info(self, connection): - raw = connection.execute('select version()').scalar() + raw = connection.execute(text('select version()')).scalar() v = raw.split()[2] - m = re.match('.*?(\d{1,3})\.(\d{1,3})\.(\d{1,3}).*', v) + m = re.match(r'.*?(\d{1,3})\.(\d{1,3})\.(\d{1,3}).*', v) return tuple([int(x) for x in m.group(1, 2, 3) if x is not None]) - def has_table(self, connection, table_name, schema=None): + def has_table(self, connection, table_name, schema=None, **kw): tables = self.get_table_names(connection, schema) if table_name in tables: return True @@ -241,12 +252,17 @@ def get_table_names(self, connection, schema=None, **kw): query += ' IN %s' % escaped_schema tables = [ tup[1] if len(tup) > 1 else tup[0] - for tup in connection.execute(query).fetchall() + for tup in connection.execute(text(query)).fetchall() ] return tables + def get_view_names(self, connection, schema=None, **kw): + # Impala doesn't distinguish between tables and view when calling + # SHOW TABLES. So return a blank list. + return [] + def get_schema_names(self, connection, **kw): - rp = connection.execute("SHOW SCHEMAS") + rp = connection.execute(text("SHOW SCHEMAS")) return [r[0] for r in rp] def get_columns(self, connection, table_name, schema=None, **kwargs): @@ -255,15 +271,16 @@ def get_columns(self, connection, table_name, schema=None, **kwargs): if schema is not None: name = '%s.%s' % (schema, name) query = 'SELECT * FROM %s LIMIT 0' % name - cursor = connection.execute(query) + cursor = connection.execute(text(query)) schema = cursor.cursor.description # We need to fetch the empty results otherwise these queries remain in # flight cursor.fetchall() column_info = [] + # `select * from table` results in 'tablename.columnname', so strip off 'tablename.' for col in schema: column_info.append({ - 'name': col[0], + 'name': col[0].split('.')[-1].strip() if '.' in col[0] else col[0], 'type': _impala_type_to_sqlalchemy_type[col[1]], 'nullable': True, 'autoincrement': False}) @@ -293,3 +310,13 @@ class Impala4Dialect(ImpalaDialect): name = 'impala4' driver = 'impala4' type_compiler = Impala4TypeCompiler + + # disable supports_statement_cache explicitly to avoid warning in sqlalchemy. + # TODO: it is not clear whether it would be safe to enable, needs more research + supports_statement_cache = False + + @classmethod + def import_dbapi(cls): + # pylint: disable=method-hidden + import impala.dbapi + return impala.dbapi diff --git a/impala/tests/_dbapi20_tests.py b/impala/tests/_dbapi20_tests.py index 7537d8558..f72db4c8f 100644 --- a/impala/tests/_dbapi20_tests.py +++ b/impala/tests/_dbapi20_tests.py @@ -11,17 +11,12 @@ -- Ian Bicking ''' -from __future__ import absolute_import - __rcs_id__ = '$Id: dbapi20.py,v 1.11 2005/01/02 02:41:01 zenzen Exp $' __version__ = '$Revision: 1.12 $'[11:-2] __author__ = 'Stuart Bishop ' import time import sys -from six.moves import range - -from impala.tests.compat import unittest # Revision 1.12 2009/02/06 03:35:11 kf7xm @@ -75,9 +70,10 @@ # nothing # - Fix bugs in test_setoutputsize_basic and test_setinputsizes # + +import unittest + def str2bytes(sval): - if sys.version_info < (3,0) and isinstance(sval, str): - sval = sval.decode("latin1") return sval.encode("latin1") class DatabaseAPI20Test(unittest.TestCase): diff --git a/impala/tests/compat.py b/impala/tests/compat.py deleted file mode 100644 index ad75042fb..000000000 --- a/impala/tests/compat.py +++ /dev/null @@ -1,23 +0,0 @@ -# Copyright 2015 Cloudera Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# pylint: disable=unused-import,import-error - -import sys - - -if sys.version_info[:2] <= (2, 6): - import unittest2 as unittest -else: - import unittest # noqa diff --git a/impala/tests/conftest.py b/impala/tests/conftest.py index e5e19c3ea..ea72d3121 100644 --- a/impala/tests/conftest.py +++ b/impala/tests/conftest.py @@ -12,9 +12,9 @@ # See the License for the specific language governing permissions and # limitations under the License. -from __future__ import absolute_import - import logging +from logging import config +import sys from pytest import fixture, skip @@ -35,6 +35,7 @@ def pytest_addoption(parser): parser.addoption('--log-debug', action='store_true', default=False, help='Enable DEBUG logging') +ENV = ImpylaTestEnv() def pytest_configure(config): # if both --log-debug and --log-info are set, the DEBUG takes precedence @@ -47,6 +48,12 @@ def pytest_configure(config): root_logger.setLevel(logging.INFO) root_logger.addHandler(logging.StreamHandler()) config.addinivalue_line("markers", "connect") + config.addinivalue_line("markers", "params_neg: marks tests that verify invalid parameters are not allowed") + config.addinivalue_line("markers", "ssl: marks tests that require SSL") + config.addinivalue_line("markers", "jwt_auth: mark tests that require JWT authentication") + suite_name = "%spy%d.%d" % (ENV.auth_mech , sys.version_info[0], sys.version_info[1]) + config.inicfg['junit_suite_name'] = suite_name + def pytest_runtest_setup(item): if (getattr(item.obj, 'connect', None) and @@ -56,8 +63,6 @@ def pytest_runtest_setup(item): # testing fixtures - -ENV = ImpylaTestEnv() hive = ENV.auth_mech == 'PLAIN' @@ -108,7 +113,22 @@ def con(host, port, auth_mech, tmp_db): @fixture(scope='session') +def session_cur(con): + # session level cursor usable in session/module level fixtures + cur = con.cursor() + yield cur + cur.close() + + +@fixture(scope='function') def cur(con): cur = con.cursor() yield cur cur.close() + + +@fixture(scope='session') +def cur_no_string_conv(con): + cur = con.cursor(convert_types=True, convert_strings_to_unicode=False) + yield cur + cur.close() diff --git a/impala/tests/test_data_types.py b/impala/tests/test_data_types.py index 936970bfa..ab50a82b5 100644 --- a/impala/tests/test_data_types.py +++ b/impala/tests/test_data_types.py @@ -16,19 +16,20 @@ import datetime import pytest from pytest import fixture +from decimal import Decimal @fixture(scope='module') -def decimal_table(cur): +def decimal_table(session_cur): table_name = 'tmp_decimal_table' ddl = """CREATE TABLE {0} ( f1 decimal(10, 2), f2 decimal(7, 5), f3 decimal(38, 17))""".format(table_name) - cur.execute(ddl) + session_cur.execute(ddl) try: yield table_name finally: - cur.execute("DROP TABLE {0}".format(table_name)) + session_cur.execute("DROP TABLE {0}".format(table_name)) @pytest.mark.connect @@ -49,48 +50,99 @@ def test_cursor_description_precision_scale(cur, decimal_table): for (exp, obs) in zip(expected, observed): assert exp == obs + @fixture(scope='module') -def date_table(cur): - table_name = 'tmp_date_table' - ddl = """CREATE TABLE {0} (d date)""".format(table_name) - cur.execute(ddl) +def decimal_table2(session_cur): + table_name = 'tmp_decimal_table2' + ddl = """CREATE TABLE {0} (val decimal(18, 9))""".format(table_name) + session_cur.execute(ddl) + session_cur.execute('''insert into {0} + values (cast(123456789.123456789 as decimal(18, 9))), + (cast(-123456789.123456789 as decimal(18, 9))), + (cast(0.000000001 as decimal(18, 9))), + (cast(-0.000000001 as decimal(18, 9))), + (cast(999999999.999999999 as decimal(18, 9))), + (cast(-999999999.999999999 as decimal(18, 9))), + (NULL)'''.format(table_name)) try: yield table_name finally: - cur.execute("DROP TABLE {0}".format(table_name)) + session_cur.execute("DROP TABLE {0}".format(table_name)) -@pytest.mark.connect -def test_date_basic(cur, date_table): - """Insert and read back a couple of data values in a wide range.""" - cur.execute('''insert into {0} - values (date "0001-01-01"), (date "1999-9-9")'''.format(date_table)) - cur.execute('select d from {0} order by d'.format(date_table)) +def common_test_decimal(cur, decimal_table): + """Read back a few decimal values in a wide range.""" + cur.execute('select val from {0} order by val'.format(decimal_table)) results = cur.fetchall() - assert results == [(datetime.date(1, 1, 1),), (datetime.date(1999, 9, 9),)] + assert results == [(Decimal('-999999999.999999999'),), + (Decimal('-123456789.123456789'),), + (Decimal('-0.000000001'),), + (Decimal('0.000000001'),), + (Decimal('123456789.123456789'),), + (Decimal('999999999.999999999'),), + (None,)] + + +@pytest.mark.connect +def test_decimal_basic(cur, decimal_table2): + common_test_decimal(cur, decimal_table2) + + +@pytest.mark.connect +def test_decimal_no_string_conv(cur_no_string_conv, decimal_table2): + common_test_decimal(cur_no_string_conv, decimal_table2) @fixture(scope='module') -def timestamp_table(cur): - table_name = 'tmp_timestamp_table' - ddl = """CREATE TABLE {0} (ts timestamp)""".format(table_name) - cur.execute(ddl) +def date_table(session_cur): + table_name = 'tmp_date_table' + ddl = """CREATE TABLE {0} (d date)""".format(table_name) + session_cur.execute(ddl) + session_cur.execute('''insert into {0} + values (date "0001-01-01"), (date "1999-9-9")'''.format(table_name)) try: yield table_name finally: - cur.execute("DROP TABLE {0}".format(table_name)) + session_cur.execute("DROP TABLE {0}".format(table_name)) + + +def common_test_date(cur, date_table): + """Read back a couple of data values in a wide range.""" + cur.execute('select d from {0} order by d'.format(date_table)) + results = cur.fetchall() + assert results == [(datetime.date(1, 1, 1),), (datetime.date(1999, 9, 9),)] @pytest.mark.connect -def test_timestamp_basic(cur, timestamp_table): - """Insert and read back a few timestamp values in a wide range.""" - cur.execute('''insert into {0} +def test_date_basic(cur, date_table): + common_test_date(cur, date_table) + + +@pytest.mark.connect +def test_date_no_string_conv(cur_no_string_conv, date_table): + common_test_date(cur_no_string_conv, date_table) + + +@fixture(scope='module') +def timestamp_table(session_cur): + table_name = 'tmp_timestamp_table' + ddl = """CREATE TABLE {0} (ts timestamp)""".format(table_name) + session_cur.execute(ddl) + session_cur.execute('''insert into {0} values (cast("1400-01-01 00:00:00" as timestamp)), (cast("2014-06-23 13:30:51" as timestamp)), (cast("2014-06-23 13:30:51.123" as timestamp)), (cast("2014-06-23 13:30:51.123456" as timestamp)), (cast("2014-06-23 13:30:51.123456789" as timestamp)), - (cast("9999-12-31 23:59:59" as timestamp))'''.format(timestamp_table)) + (cast("9999-12-31 23:59:59" as timestamp))'''.format(table_name)) + try: + yield table_name + finally: + session_cur.execute("DROP TABLE {0}".format(table_name)) + + +def common_test_timestamp(cur, timestamp_table): + """Read back a few timestamp values in a wide range.""" cur.execute('select ts from {0} order by ts'.format(timestamp_table)) results = cur.fetchall() assert results == [(datetime.datetime(1400, 1, 1, 0, 0),), @@ -101,6 +153,16 @@ def test_timestamp_basic(cur, timestamp_table): (datetime.datetime(9999, 12, 31, 23, 59, 59),)] +@pytest.mark.connect +def test_timestamp_basic(cur, timestamp_table): + common_test_timestamp(cur, timestamp_table) + + +@pytest.mark.connect +def test_timestamp_no_string_conv(cur_no_string_conv, timestamp_table): + common_test_timestamp(cur_no_string_conv, timestamp_table) + + @pytest.mark.connect def test_utf8_strings(cur): """Use STRING/VARCHAR/CHAR values with multi byte unicode code points in a query.""" @@ -120,3 +182,19 @@ def test_utf8_strings(cur): result = cur.fetchone()[0] assert result == b"\xaa" assert result.decode("UTF-8", "replace") == u"�" + + +@pytest.mark.connect +def test_string_conv(cur): + cur.execute('select "Test string"') + result = cur.fetchone() + is_unicode = isinstance(result[0], str) + + +@pytest.mark.connect +def test_string_no_string_conv(cur_no_string_conv): + cur = cur_no_string_conv + cur.execute('select "Test string"') + result = cur.fetchone() + + assert isinstance(result[0], bytes) diff --git a/impala/tests/test_dbapi_compliance.py b/impala/tests/test_dbapi_compliance.py index 2f2fb4888..baa92aee8 100644 --- a/impala/tests/test_dbapi_compliance.py +++ b/impala/tests/test_dbapi_compliance.py @@ -19,8 +19,6 @@ """ -from __future__ import absolute_import, print_function - import pytest import impala.dbapi diff --git a/impala/tests/test_dbapi_connect.py b/impala/tests/test_dbapi_connect.py index dd4578dc7..67c9be7fd 100644 --- a/impala/tests/test_dbapi_connect.py +++ b/impala/tests/test_dbapi_connect.py @@ -12,42 +12,53 @@ # See the License for the specific language governing permissions and # limitations under the License. -from __future__ import absolute_import, print_function - -import sys +import ssl +import time from impala.error import NotSupportedError -if sys.version_info[:2] <= (2, 6): - import unittest2 as unittest -else: - import unittest +import unittest import pytest -from impala.dbapi import connect +from impala.dbapi import connect, AUTH_MECHANISMS from impala.util import _random_id from impala.tests.util import ImpylaTestEnv, SocketTracker +from thrift.transport.TTransport import TTransportException ENV = ImpylaTestEnv() -DEFAULT_AUTH = True +DEFAULT_AUTH = ENV.auth_mech != "NOSASL" DEFAULT_AUTH_ERROR = "Non-default authorization method" +# TODO: write tests for this - it needs Hive restart with hive.server2.authentication=NOSASL +HIVE_NO_SASL_DISABLED = True + +# TODO: does this make any sense? Impala doesn't support plain auth +IMPALA_PLAIN_DISABLED = True + +# TODO: write LDAP tests - these are currently in Java in Impala. +# note that from client perspective, LDAP auth is the same as PLAIN, which is +# covered in Hive tests +LDAP_DISABLED = ENV.auth_mech != "LDAP" +LDAP_DISABLED_ERROR = "LDAP authentication disabled" + # JWT tests require the Impala coordinator to have the following startup flags: # --jwt_token_auth=true --jwt_validate_signature=false --jwt_allow_without_tls=true -JWT_DISABLED = True +JWT_DISABLED = ENV.auth_mech != "JWT" JWT_DISABLED_ERROR = "JWT authentication disabled" - # SSL can be enabled in Impala dev environment with the following commands: # export IMPALA_SSL_CERT_DIR=$IMPALA_HOME/be/src/testutil # export IMPALA_SSL_ARGS="--ssl_client_ca_certificate=$IMPALA_SSL_CERT_DIR/server-cert.pem --ssl_server_certificate=$IMPALA_SSL_CERT_DIR/server-cert.pem --ssl_private_key=$IMPALA_SSL_CERT_DIR/server-key.pem --hostname=localhost" # bin/start-impala-cluster.py --impalad_args="$IMPALA_SSL_ARGS" --catalogd_args="$IMPALA_SSL_ARGS" --state_store_args="$IMPALA_SSL_ARGS" # export IMPYLA_SSL_CERT=$IMPALA_SSL_CERT_DIR/server-cert.pem +# export IMPYLA_SSL_WRONG_CERT=$IMPALA_SSL_CERT_DIR/incorrect-commonname-cert.pem SSL_DISABLED = ENV.ssl_cert == "" SSL_DISABLED_ERROR = "No ssl certificate set." +# Table loading / DDLs can take several seconds in Impala. Use a larger timeout to avoid flaky tests. +TIMEOUT_S = 10 class ImpalaConnectionTests(unittest.TestCase): @@ -62,19 +73,11 @@ def tearDown(self): self.connection.close() def _execute_queries(self, con): - ddl = """ - CREATE TABLE {0} ( - f1 INT, - f2 INT) - """.format(self.tablename) - try: - cur = con.cursor() - cur.execute(ddl) - con.commit() - cur.execute('DROP TABLE {0}'.format(self.tablename)) - con.commit() - except: - raise + cur = con.cursor() + cur.execute("SELECT 1 + 1") + assert cur.fetchall() == [(2,)] + cur.execute("SELECT 2 + 2") + assert cur.fetchall() == [(4,)] def _execute_query_get_username(self, con): query = "select user()" @@ -94,30 +97,30 @@ def _execute_query_get_username(self, con): return username def test_impala_nosasl_connect(self): - self.connection = connect(ENV.host, ENV.port, timeout=5) + self.connection = connect(ENV.host, ENV.port, timeout=TIMEOUT_S) self._execute_queries(self.connection) @pytest.mark.skipif(ENV.skip_hive_tests, reason="Skipping hive tests") def test_hive_plain_connect(self): self.connection = connect(ENV.host, ENV.hive_port, auth_mechanism="PLAIN", - timeout=5, + timeout=TIMEOUT_S, user=ENV.hive_user, password="cloudera") self._execute_queries(self.connection) - @pytest.mark.skipif(DEFAULT_AUTH, reason=DEFAULT_AUTH_ERROR) + @pytest.mark.skipif(IMPALA_PLAIN_DISABLED, reason=DEFAULT_AUTH_ERROR) def test_impala_plain_connect(self): self.connection = connect(ENV.host, ENV.port, auth_mechanism="PLAIN", - timeout=5, + timeout=TIMEOUT_S, user=ENV.hive_user, password="cloudera") self._execute_queries(self.connection) - @pytest.mark.skipif(DEFAULT_AUTH, reason=DEFAULT_AUTH_ERROR) + @pytest.mark.skipif(LDAP_DISABLED, reason=LDAP_DISABLED_ERROR) def test_impala_ldap_connect_user_agent(self): self.connection = connect(ENV.host, ENV.port, auth_mechanism="LDAP", - timeout=5, + timeout=TIMEOUT_S, user=ENV.hive_user, password="cloudera", http_path="http-path", @@ -126,21 +129,22 @@ def test_impala_ldap_connect_user_agent(self): user_agent="cloudera/impyla") self._execute_queries(self.connection) - @pytest.mark.skipif(DEFAULT_AUTH, reason=DEFAULT_AUTH_ERROR) + @pytest.mark.skipif(HIVE_NO_SASL_DISABLED, reason="Skipping hive tests") def test_hive_nosasl_connect(self): - self.connection = connect(ENV.host, ENV.hive_port, timeout=5) + self.connection = connect(ENV.host, ENV.hive_port, timeout=TIMEOUT_S) self._execute_queries(self.connection) + @pytest.mark.params_neg def test_bad_auth(self): """Test some simple error messages""" try: connect(ENV.host, ENV.port, auth_mechanism="foo") - assert False, "should have got exception" + assert False, "'connect' method should have thrown an exception but did not" except NotSupportedError as e: assert 'Unsupported authentication mechanism: FOO' in str(e) - @pytest.mark.skipif(JWT_DISABLED, reason=JWT_DISABLED_ERROR) + @pytest.mark.jwt_auth def test_jwt_auth(self): """Test for connecting via the auth_mechanism=JWT""" # This is a JWT generated via jwt.io's online generator @@ -160,42 +164,214 @@ def test_jwt_auth(self): jwt = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwidXNlcm5hbWUiOiJpbXB5bGFqd3R0ZXN0IiwiaWF0IjoxNTE2MjM5MDIyfQ.Uvq2TZ9nRtM-JgFEd_fL2Z0kFcflx0Tr5cbJR4yySyc" self.connection = connect(ENV.host, ENV.http_port, use_http_transport=True, http_path="cliservice", auth_mechanism="JWT", - timeout=5, jwt=jwt) + timeout=TIMEOUT_S, jwt=jwt) username = self._execute_query_get_username(self.connection) assert(username == "impylajwttest") print("Username: {0}".format(username)) self._execute_queries(self.connection) - @pytest.mark.skipif(JWT_DISABLED, reason=JWT_DISABLED_ERROR) - def test_jwt_auth_negative(self): - """Test negative cases for connecting via the auth_mechanism=JWT""" - # Case 1: Connect without specifying JWT + @pytest.mark.params_neg + def test_jwt_auth_missing_jwt(self): + """Test for the expected error when JWT auth is used and a JWT is not provided""" try: - connect(ENV.host, ENV.http_port, use_http_transport=True, - http_path="cliservice", auth_mechanism="JWT", - timeout=5) + connect(use_http_transport=True, auth_mechanism="JWT") + assert False, "'connect' method should have thrown an exception but did not" except NotSupportedError as e: assert "JWT authentication requires specifying the 'jwt' argument" in str(e) - # Case 2: Connect with JWT to non-HTTP + @pytest.mark.params_neg + def test_jwt_auth_invalid_transport(self): + """Test for the expected error when JWT auth is used without the HTTP transport""" try: - connect(ENV.host, ENV.http_port, http_path="cliservice", auth_mechanism="JWT", - timeout=5, jwt="dummy.jwt.arg") + connect(auth_mechanism="JWT", jwt="dummy.jwt.arg") + assert False, "'connect' method should have thrown an exception but did not" except NotSupportedError as e: assert "JWT authentication is only supported for HTTP transport" in str(e) + @pytest.mark.params_neg + def test_non_jwt_auth_with_jwt(self): + """Test for the expected error when authentication is anything other than JWT and the 'jwt' parameter is specified""" + def run_test(auth_mechanism): + try: + connect(auth_mechanism=auth_mechanism, jwt="dummy.jwt.arg") + assert False, "'connect' method should have thrown an exception but did not" + except NotSupportedError as e: + assert "'jwt' argument cannot be specified with '{0}' authentication".format(auth_mechanism) in str(e) + + for auth_mech in AUTH_MECHANISMS: + if auth_mech is not "JWT": + run_test(auth_mech) + + @pytest.mark.params_neg + def test_jwt_auth_with_user(self): + """Test for the expected error when authentication is JWT and the 'user' parameter is specified""" + try: + connect(auth_mechanism="JWT", jwt="dummy.jwt.arg", use_http_transport=True, user="any_user") + assert False, "'connect' method should have thrown an exception but did not" + except NotSupportedError as e: + assert "'user' argument cannot be specified with 'JWT' authentication" in str(e) + + @pytest.mark.params_neg + def test_jwt_auth_with_ldap_user(self): + """Test for the expected error when authentication is JWT and the 'ldap_user' parameter is specified""" + try: + connect(auth_mechanism="JWT", jwt="dummy.jwt.arg", use_http_transport=True, ldap_user="any_user") + assert False, "'connect' method should have thrown an exception but did not" + except NotSupportedError as e: + assert "'user' argument cannot be specified with 'JWT' authentication" in str(e) + + @pytest.mark.params_neg + def test_jwt_auth_with_password(self): + """Test for the expected error when authentication is JWT and the 'password' parameter is specified""" + try: + connect(auth_mechanism="JWT", jwt="dummy.jwt.arg", use_http_transport=True, password="any_password") + assert False, "'connect' method should have thrown an exception but did not" + except NotSupportedError as e: + assert "'password' argument cannot be specified with 'JWT' authentication" in str(e) + + @pytest.mark.params_neg + def test_jwt_auth_with_ldap_password(self): + """Test for the expected error when authentication is JWT and the 'ldap_password' parameter is specified""" + try: + connect(auth_mechanism="JWT", jwt="dummy.jwt.arg", use_http_transport=True, ldap_password="any_password") + assert False, "'connect' method should have thrown an exception but did not" + except NotSupportedError as e: + assert "'password' argument cannot be specified with 'JWT' authentication" in str(e) + @pytest.mark.skipif(SSL_DISABLED, reason=SSL_DISABLED_ERROR) + @pytest.mark.ssl def test_ssl_connection_no_cert(self): - self.connection = connect(ENV.host, ENV.port, timeout=5, use_ssl=True) + self.connection = connect(ENV.host, ENV.port, timeout=TIMEOUT_S, use_ssl=True) self._execute_queries(self.connection) @pytest.mark.skipif(SSL_DISABLED, reason=SSL_DISABLED_ERROR) + @pytest.mark.ssl def test_ssl_connection_with_cert(self): self.connection = connect( - ENV.host, ENV.port, use_ssl=True, timeout=5, ca_cert=ENV.ssl_cert) + ENV.host, ENV.port, use_ssl=True, timeout=TIMEOUT_S, ca_cert=ENV.ssl_cert) + self._execute_queries(self.connection) + + @pytest.mark.skipif(SSL_DISABLED, reason=SSL_DISABLED_ERROR) + @pytest.mark.ssl + def test_ssl_connection_wrong_cert(self): + try: + connect( + ENV.host, ENV.port, use_ssl=True, timeout=TIMEOUT_S, ca_cert=ENV.ssl_wrong_cert) + assert False, "'connect' method should have thrown an exception but did not" + except TTransportException as e: + # The message is not too informative, verification error is swallowed by thrift. + assert "CERTIFICATE_VERIFY_FAILED" in str(e.inner) + + @pytest.mark.skipif(SSL_DISABLED, reason=SSL_DISABLED_ERROR) + @pytest.mark.ssl + def test_ssl_connection_default_certs(self): + try: + # With verify_cert=True, the system's CA certificates will be used to verify + # the server certificate. Since the server certificate is self-signed and not + # in the system CA store, verification should fail. + # TODO: writing positive test would be nice but is more difficult + connect( + ENV.host, ENV.port, use_ssl=True, timeout=TIMEOUT_S, verify_cert=True) + assert False, "'connect' method should have thrown an exception but did not" + except TTransportException as e: + # The message is not too informative, verification error is swallowed by thrift. + assert "CERTIFICATE_VERIFY_FAILED" in str(e.inner) + + @pytest.mark.skipif(SSL_DISABLED, reason=SSL_DISABLED_ERROR) + @pytest.mark.ssl + def test_https_connection_nocert(self): + self.connection = connect(ENV.host, ENV.http_port, use_http_transport=True, + http_path="cliservice", use_ssl=True, timeout=TIMEOUT_S) self._execute_queries(self.connection) + @pytest.mark.skipif(SSL_DISABLED, reason=SSL_DISABLED_ERROR) + @pytest.mark.ssl + def test_https_connection_with_cert(self): + self.connection = connect(ENV.host, ENV.http_port, use_http_transport=True, + http_path="cliservice", use_ssl=True, timeout=TIMEOUT_S, + ca_cert=ENV.ssl_cert) + self._execute_queries(self.connection) + + @pytest.mark.skipif(SSL_DISABLED, reason=SSL_DISABLED_ERROR) + @pytest.mark.ssl + def test_https_connection_wrong_cert(self): + try: + connection = connect(ENV.host, ENV.http_port, use_http_transport=True, + http_path="cliservice", use_ssl=True, timeout=TIMEOUT_S, + ca_cert=ENV.ssl_wrong_cert) + connection.cursor() + assert False, "'cursor' method should have thrown an exception but did not" + except Exception as e: + # TODO: replace with ssl.SSLCertVerificationError when dropping Python 2.7 + assert "CERTIFICATE_VERIFY_FAILED" in str(e) + + @pytest.mark.skipif(SSL_DISABLED, reason=SSL_DISABLED_ERROR) + @pytest.mark.ssl + def test_https_connection_default_certs(self): + try: + # With verify_cert=True, the system's CA certificates will be used to verify + # the server certificate. Since the server certificate is self-signed and not + # in the system CA store, verification should fail. + # TODO: writing positive test would be nice but is more difficult + connection = connect(ENV.host, ENV.http_port, use_http_transport=True, + http_path="cliservice", use_ssl=True, timeout=TIMEOUT_S, + verify_cert=True) + connection.cursor() + assert False, "'cursor' method should have thrown an exception but did not" + except Exception as e: + # TODO: replace with ssl.SSLCertVerificationError when dropping Python 2.7 + assert "CERTIFICATE_VERIFY_FAILED" in str(e) + + def test_retry_dml(self): + """Regression test for #549. + Checks the INSERT statements are not retried when the client disconnects due to timeout + during execute().""" + # Connection with higher timeout to avoid errors. + self.connection = connect(ENV.host, ENV.port, timeout=TIMEOUT_S) + create = "CREATE TABLE {0} (f1 INT)".format(self.tablename) + refresh = "REFRESH {0}".format(self.tablename) + insert = "INSERT INTO TABLE {0} SELECT 1".format(self.tablename) + select = "SELECT * FROM {0}".format(self.tablename) + drop = "DROP TABLE {0}".format(self.tablename) + reliable_cur = self.connection.cursor() + reliable_cur.execute(create) + reliable_cur.execute(refresh) # Load table metadata to make subsequent operations faster. + successful_inserts = 0 + NUM_INSERTS = 3 + for i in range(NUM_INSERTS): + # Connect with low timeout to trigger failed RPCs. + LOW_TIMEOUT_S = 1 + low_timeout_connection = connect(ENV.host, ENV.port, timeout=LOW_TIMEOUT_S) + low_timeout_cur = low_timeout_connection.cursor() + try: + # Use IMPALAD_LOAD_TABLES_DELAY>LOW_TIMEOUT_S to trigger timeout. + # IMPALAD_LOAD_TABLES_DELAY was added in Impala 4.4.0 (IMPALA-12493), so with + # older Impala servers the test is not expected to work. + DELAY_S = LOW_TIMEOUT_S + 1 + configuration = {"debug_action": "IMPALAD_LOAD_TABLES_DELAY:SLEEP@{}".format(1000 * DELAY_S)} + low_timeout_cur.execute(insert, configuration=configuration) + successful_inserts += 1 + except TTransportException: + # Sleep until the insert is expected to be finished and close the session. + SLEEP_S = DELAY_S - LOW_TIMEOUT_S + 2 + time.sleep(SLEEP_S) + # Reopen the transport to allow closing the session correctly. + low_timeout_cur.session.client._iprot.trans.close() + low_timeout_cur.session.client._iprot.trans.open() + low_timeout_cur.close() + low_timeout_connection.close() + assert successful_inserts == 0 + # Use the reliable connection for result checking and cleanup. + reliable_cur.execute(select) + result = reliable_cur.fetchall() + # All inserts must have actually finished after client timeout. + assert len(result) == NUM_INSERTS + reliable_cur.execute(drop) + # If all inserts are successful then probably the Impala cluster is too fast. + assert successful_inserts < NUM_INSERTS + + class ImpalaSocketTests(unittest.TestCase): def run_a_query(self): diff --git a/impala/tests/test_hive_dict_cursor.py b/impala/tests/test_hive_dict_cursor.py index b29032e77..0bb49ac01 100644 --- a/impala/tests/test_hive_dict_cursor.py +++ b/impala/tests/test_hive_dict_cursor.py @@ -15,7 +15,7 @@ from pytest import fixture -@fixture(scope='session') +@fixture(scope='function') def cur2(con): cur = con.cursor(dictify=True) yield cur diff --git a/impala/tests/test_hs2_fault_injection.py b/impala/tests/test_hs2_fault_injection.py index e192b443f..98cba9dd5 100644 --- a/impala/tests/test_hs2_fault_injection.py +++ b/impala/tests/test_hs2_fault_injection.py @@ -13,8 +13,6 @@ # limitations under the License. import logging -import six - from thrift.protocol.TBinaryProtocol import TBinaryProtocolAccelerated # noinspection PyProtectedMember from impala._thrift_gen.ImpalaService import ImpalaHiveServer2Service @@ -105,37 +103,38 @@ def _read(self, sz): class TestHS2FaultInjection(object): """Class for testing the http fault injection in various rpcs used by Impyla""" - def setup(self): - url = 'http://%s:%s/%s' % (ENV.host, ENV.http_port, "cliservice") + def setup_method(self): + host = "[%s]" % ENV.host if ":" in ENV.host else ENV.host + url = 'http://%s:%s/%s' % (host, ENV.http_port, "cliservice") self.transport = FaultInjectingHttpClient(url) self.configuration = {'idle_session_timeout': '30'} - def teardown(self): + def teardown_method(self): self.transport.disable_fault() - def connect(self): + def connect(self, retries=3): self.transport.open() protocol = TBinaryProtocolAccelerated(self.transport) service = ThriftClient(protocol) - service = HS2Service(service, retries=3) + service = HS2Service(service, retries=retries) return hs2.HiveServer2Connection(service, default_db=None) - def __expect_msg_retry(self, impala_rpc_name): + def __expect_msg_retry(self, impala_rpc_name, retries): """Returns expected log message for rpcs which can be retried""" - return ("Caught HttpError HTTP code 502: Injected Fault in {0} (tries_left=3)". - format(impala_rpc_name)) + return ("Caught HttpError HTTP code 502: Injected Fault in {0} (tries_left={1})". + format(impala_rpc_name, retries)) - def __expect_msg_retry_with_extra(self, impala_rpc_name): + def __expect_msg_retry_with_extra(self, impala_rpc_name, retries): """Returns expected log message for rpcs which can be retried and where the http message has a message body""" - return ("Caught HttpError HTTP code 503: Injected Fault EXTRA in {0} (tries_left=3)". - format(impala_rpc_name)) + return ("Caught HttpError HTTP code 503: Injected Fault EXTRA in {0} (tries_left={1})". + format(impala_rpc_name, retries)) - def __expect_msg_retry_with_retry_after(self, impala_rpc_name): + def __expect_msg_retry_with_retry_after(self, impala_rpc_name, retries): """Returns expected log message for rpcs which can be retried and where the http message has a body and a Retry-After header that can be correctly decoded""" - return ("Caught HttpError HTTP code 503: Injected Fault EXTRA in {0} (tries_left=3), retry after 1 secs". - format(impala_rpc_name)) + return ("Caught HttpError HTTP code 503: Injected Fault EXTRA in {0} (tries_left={1}), retry after 1 secs". + format(impala_rpc_name, retries)) def __expect_msg_retry_with_retry_after_sleep(self): """Returns expected log message for the sleep which uses a value @@ -146,12 +145,12 @@ def __expect_msg_retry_after_default_sleep(self): """Returns expected log message for the default sleep time of 1 second""" return ("sleeping for 1 second before retrying") - def __expect_msg_retry_with_retry_after_no_extra(self, impala_rpc_name): + def __expect_msg_retry_with_retry_after_no_extra(self, impala_rpc_name, retries): """Returns expected log message for rpcs which can be retried and the http message has a Retry-After header that can be correctly decoded""" - return ("Caught HttpError HTTP code 503: Injected Fault in {0} (tries_left=3), retry after 1 secs". - format(impala_rpc_name)) - + return ("Caught HttpError HTTP code 503: Injected Fault in {0} (tries_left={1}), retry after 1 secs". + format(impala_rpc_name, retries)) + def __expect_msg_no_retry(self, impala_rpc_name): """Returns expected log message for rpcs which can not be retried""" return ("Caught HttpError HTTP code 502: Injected Fault in {0} which is not retryable". @@ -163,11 +162,11 @@ def test_connect(self, caplog): Retries results in a successful connection.""" caplog.set_level(logging.DEBUG) self.transport.enable_fault(502, "Injected Fault", 0.2) - con = self.connect() + con = self.connect(retries=4) cur = con.cursor(configuration=self.configuration) cur.close() con.close() - assert self.__expect_msg_retry("OpenSession") in caplog.text + assert self.__expect_msg_retry("OpenSession", 4) in caplog.text def test_connect_proxy(self, caplog): """Tests fault injection in cursor() call. @@ -176,11 +175,11 @@ def test_connect_proxy(self, caplog): Retries results in a successful connection.""" caplog.set_level(logging.DEBUG) self.transport.enable_fault(503, "Injected Fault", 0.20, 'EXTRA') - con = self.connect() + con = self.connect(retries=4) cur = con.cursor(configuration=self.configuration) cur.close() con.close() - assert self.__expect_msg_retry_with_extra("OpenSession") in caplog.text + assert self.__expect_msg_retry_with_extra("OpenSession", 4) in caplog.text assert self.__expect_msg_retry_after_default_sleep() in caplog.text def test_connect_proxy_no_retry(self, caplog): @@ -191,11 +190,11 @@ def test_connect_proxy_no_retry(self, caplog): caplog.set_level(logging.DEBUG) self.transport.enable_fault(503, "Injected Fault", 0.20, 'EXTRA', {"header1": "value1"}) - con = self.connect() + con = self.connect(retries=4) cur = con.cursor(configuration=self.configuration) cur.close() con.close() - assert self.__expect_msg_retry_with_extra("OpenSession") in caplog.text + assert self.__expect_msg_retry_with_extra("OpenSession", 4) in caplog.text assert self.__expect_msg_retry_after_default_sleep() in caplog.text def test_connect_proxy_bad_retry(self, caplog): @@ -207,11 +206,11 @@ def test_connect_proxy_bad_retry(self, caplog): self.transport.enable_fault(503, "Injected Fault", 0.20, 'EXTRA', {"header1": "value1", "Retry-After": "junk"}) - con = self.connect() + con = self.connect(retries=4) cur = con.cursor(configuration=self.configuration) cur.close() con.close() - assert self.__expect_msg_retry_with_extra("OpenSession") in caplog.text + assert self.__expect_msg_retry_with_extra("OpenSession", 4) in caplog.text assert self.__expect_msg_retry_after_default_sleep() in caplog.text def test_connect_proxy_retry(self, caplog): @@ -222,11 +221,11 @@ def test_connect_proxy_retry(self, caplog): self.transport.enable_fault(503, "Injected Fault", 0.20, 'EXTRA', {"header1": "value1", "Retry-After": "1"}) - con = self.connect() + con = self.connect(retries=4) cur = con.cursor(configuration=self.configuration) cur.close() con.close() - assert self.__expect_msg_retry_with_retry_after("OpenSession") in caplog.text + assert self.__expect_msg_retry_with_retry_after("OpenSession", 4) in caplog.text assert self.__expect_msg_retry_with_retry_after_sleep() in caplog.text def test_connect_proxy_retry_no_body(self, caplog): @@ -237,11 +236,11 @@ def test_connect_proxy_retry_no_body(self, caplog): self.transport.enable_fault(503, "Injected Fault", 0.20, None, {"header1": "value1", "Retry-After": "1"}) - con = self.connect() + con = self.connect(retries=4) cur = con.cursor(configuration=self.configuration) cur.close() con.close() - assert self.__expect_msg_retry_with_retry_after_no_extra("OpenSession") in caplog.text + assert self.__expect_msg_retry_with_retry_after_no_extra("OpenSession", 4) in caplog.text def test_execute_query(self, caplog): """Tests fault injection in execute(). @@ -265,7 +264,7 @@ def test_execute_query(self, caplog): def test_get_query_state(self, caplog): """Tests fault injection in fetchall(). GetOperationStatus rpc fails but is retried successfully.""" - con = self.connect() + con = self.connect(retries=4) cur = con.cursor(configuration=self.configuration) caplog.set_level(logging.DEBUG) cur.execute_async('select 1', {}) @@ -273,12 +272,12 @@ def test_get_query_state(self, caplog): cur.fetchall() cur.close() con.close() - assert self.__expect_msg_retry("GetOperationStatus") in caplog.text + assert self.__expect_msg_retry("GetOperationStatus", 4) in caplog.text def test_get_result_set_metadata(self, caplog): """Tests fault injection in fetchcbatch(). GetResultSetMetadata rpc fails and is retried succesfully.""" - con = self.connect() + con = self.connect(retries=4) cur = con.cursor(configuration=self.configuration) caplog.set_level(logging.DEBUG) cur.execute('select 1', {}) @@ -286,7 +285,7 @@ def test_get_result_set_metadata(self, caplog): cur.fetchcbatch() cur.close() con.close() - assert self.__expect_msg_retry("GetResultSetMetadata") in caplog.text + assert self.__expect_msg_retry("GetResultSetMetadata", 4) in caplog.text def test_fetch_results(self, caplog): """Tests fault injection in fetchcbatch(). @@ -322,13 +321,13 @@ def test_close_operation(self, caplog): self.transport.disable_fault() cur.close() con.close() - assert self.__expect_msg_no_retry("CloseOperation") in caplog.text + assert self.__expect_msg_no_retry("CloseImpalaOperation") in caplog.text def test_get_runtime_profile_summary(self, caplog): """Tests fault injection in get_profile(), get_summary(), and get_log(). GetRuntimeProfile, GetExecSummary and GetLog rpcs fail due to fault, but succeed after retries""" - con = self.connect() + con = self.connect(retries=4) cur = con.cursor(configuration=self.configuration) caplog.set_level(logging.DEBUG) cur.execute('select 1', {}) @@ -343,6 +342,6 @@ def test_get_runtime_profile_summary(self, caplog): self.transport.disable_fault() cur.close() con.close() - assert self.__expect_msg_retry("GetRuntimeProfile") in caplog.text - assert self.__expect_msg_retry("GetExecSummary") in caplog.text - assert self.__expect_msg_retry("GetLog") in caplog.text + assert self.__expect_msg_retry("GetRuntimeProfile", 4) in caplog.text + assert self.__expect_msg_retry("GetExecSummary", 4) in caplog.text + assert self.__expect_msg_retry("GetLog", 4) in caplog.text diff --git a/impala/tests/test_http_connect.py b/impala/tests/test_http_connect.py index 496b1ac0d..35d712df2 100644 --- a/impala/tests/test_http_connect.py +++ b/impala/tests/test_http_connect.py @@ -17,16 +17,26 @@ from contextlib import closing import pytest -import six -from six.moves import SimpleHTTPServer -from six.moves import http_client -from six.moves import socketserver +import requests +from http import server as SimpleHTTPServer +import http.client as http_client +import socketserver from impala.error import HttpError -from impala.tests.util import ImpylaTestEnv +from impala.tests.util import ImpylaTestEnv, is_ipv6_only_host ENV = ImpylaTestEnv() +IS_IPV6_ONLY_HOST = is_ipv6_only_host(ENV.host, ENV.port) +LOCAL_HOST = "::1" if IS_IPV6_ONLY_HOST else "127.0.0.1" + +# socketserver.TCPServer cannot listen both on ipv4 and ipv6. Listen to ipv6 +# if the hs2-http server has only ipv6 address. +class IPv4or6TcpServer(socketserver.TCPServer): + address_family = socket.AF_INET6 if IS_IPV6_ONLY_HOST else socket.AF_INET + def __init__(self, host_port, req_handler): + socketserver.TCPServer.__init__(self, host_port, req_handler) + @pytest.fixture def http_503_server(): class RequestHandler503(SimpleHTTPServer.SimpleHTTPRequestHandler): @@ -37,15 +47,8 @@ def do_POST(self): # Ensure that only one 'Host' header is contained in the request before responding. request_headers = None host_hdr_count = 0 - if six.PY2: - # The unfortunately named self.headers here is an instance of mimetools.Message that - # contains the request headers. - request_headers = self.headers.headers - host_hdr_count = sum([header.startswith('Host:') for header in request_headers]) - if six.PY3: - # In Python3 self.Headers is an HTTPMessage. - request_headers = self.headers - host_hdr_count = sum([header[0] == 'Host' for header in request_headers.items()]) + request_headers = self.headers + host_hdr_count = sum([header[0] == 'Host' for header in request_headers.items()]) assert host_hdr_count == 1, "need single 'Host:' header in %s" % request_headers # Respond with 503. @@ -55,9 +58,9 @@ def do_POST(self): class TestHTTPServer503(object): def __init__(self): - self.HOST = "localhost" + self.HOST = LOCAL_HOST self.PORT = get_unused_port() - self.httpd = socketserver.TCPServer((self.HOST, self.PORT), RequestHandler503) + self.httpd = IPv4or6TcpServer((self.HOST, self.PORT), RequestHandler503) self.http_server_thread = threading.Thread(target=self.httpd.serve_forever) self.http_server_thread.start() @@ -66,16 +69,75 @@ def __init__(self): yield server # Cleanup after test. - if server.httpd is not None: - server.httpd.shutdown() - if server.http_server_thread is not None: - server.http_server_thread.join() + shutdown_server(server) + + +@pytest.fixture +def http_proxy_server(): + """A fixture that creates a reverse http proxy.""" + + class RequestHandlerProxy(SimpleHTTPServer.SimpleHTTPRequestHandler): + """A custom http handler that acts as a reverse http proxy. This proxy will forward + http messages to Impala, and copy the responses back to the client. In addition, it + will save the outgoing http message headers in a class variable so that they can be + accessed by test code.""" + + # This class variable is used to store the most recently seen outgoing http + # message headers. + saved_headers=None + + def __init__(self, request, client_address, server): + SimpleHTTPServer.SimpleHTTPRequestHandler.__init__(self, request, client_address, + server) + + def do_POST(self): + # Read the body of the incoming http post message. + data_string = self.rfile.read(int(self.headers['Content-Length'])) + # Save the http headers from the message in a class variable. + RequestHandlerProxy.saved_headers = self.headers._headers + # Forward the http post message to Impala and get a response message. + host = "[%s]" % ENV.host if ":" in ENV.host else ENV.host + response = requests.post( + url="http://{0}:{1}/cliservice".format(host, ENV.http_port), + headers=self.headers, data=data_string) + # Send the response message back to the client. + self.send_response(code=response.status_code) + # Send the http headers. + # In python3 response.headers is a CaseInsensitiveDict + # In python2 response.headers is a dict + for key, value in response.headers.items(): + self.send_header(keyword=key, value=value) + self.end_headers() + # Send the message body. + self.wfile.write(response.content) + + + + class TestHTTPServerProxy(object): + def __init__(self, clazz): + self.clazz = clazz + self.HOST = LOCAL_HOST + self.PORT = get_unused_port() + self.httpd = IPv4or6TcpServer((self.HOST, self.PORT), clazz) + self.http_server_thread = threading.Thread(target=self.httpd.serve_forever) + self.http_server_thread.start() + + def get_headers(self): + """Return the most recently seen outgoing http message headers.""" + return self.clazz.saved_headers + + server = TestHTTPServerProxy(RequestHandlerProxy) + yield server + + # Cleanup after test. + shutdown_server(server) + from impala.dbapi import connect class TestHttpConnect(object): def test_simple_connect(self): - con = connect("localhost", ENV.http_port, use_http_transport=True, http_path="cliservice") + con = connect(ENV.host, ENV.http_port, use_http_transport=True, http_path="cliservice") cur = con.cursor() cur.execute('select 1') rows = cur.fetchall() @@ -84,7 +146,7 @@ def test_simple_connect(self): def test_http_interactions(self, http_503_server): """Test interactions with the http server when using hs2-http protocol. Check that there is an HttpError exception when the server returns a 503 error.""" - con = connect("localhost", http_503_server.PORT, use_http_transport=True) + con = connect(ENV.host, http_503_server.PORT, use_http_transport=True) try: con.cursor() assert False, "Should have got exception" @@ -93,6 +155,51 @@ def test_http_interactions(self, http_503_server): assert e.code == http_client.SERVICE_UNAVAILABLE assert e.body.decode("utf-8") == "extra text" + def test_duplicate_headers(self, http_proxy_server): + """Test that we can use 'connect' with the get_user_custom_headers_func parameter + to add duplicate http message headers to outgoing messages.""" + con = connect(ENV.host, http_proxy_server.PORT, use_http_transport=True, + get_user_custom_headers_func=get_user_custom_headers_func) + cur = con.cursor() + cur.execute('select 1') + rows = cur.fetchall() + assert rows == [(1,)] + + # Get the outgoing message headers from the last outgoing http message. + headers = http_proxy_server.get_headers() + # For sanity test the count of a few simple expected headers. + assert count_tuples_with_key(headers, "Host") == 1 + assert count_tuples_with_key(headers, "User-Agent") == 1 + # Check that the custom headers are present. + assert count_tuples_with_key(headers, "key1") == 2 + assert count_tuples_with_key(headers, "key2") == 1 + assert count_tuples_with_key(headers, "key3") == 0 + + def test_basic_auth_headers(self, http_proxy_server): + con = connect( + ENV.host, + http_proxy_server.PORT, + use_http_transport=True, + user="thisisaratherlongusername", + password="very!long!passwordthatcreatesalongbasic64encoding", + auth_mechanism="PLAIN" + ) + cur = con.cursor() + cur.execute('select 1') + rows = cur.fetchall() + assert rows == [(1,)] + + headers = http_proxy_server.get_headers() + assert ('Authorization', "Basic dGhpc2lzYXJhdGhlcmxvbmd1c2VybmFtZTp2ZXJ5IWxvbmchcGFzc3dvcmR0aGF0Y3JlYXRlc2Fsb25nYmFzaWM2NGVuY29kaW5n") in headers + +def get_user_custom_headers_func(): + """Insert some custom http headers, including a duplicate.""" + headers = [] + headers.append(('key1', 'value1')) + headers.append(('key1', 'value2')) + headers.append(('key2', 'value3')) + return headers + def get_unused_port(): """ Find an unused port http://stackoverflow.com/questions/1365265 """ @@ -100,3 +207,24 @@ def get_unused_port(): s.bind(('', 0)) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) return s.getsockname()[1] + +def shutdown_server(server): + """Helper method to shutdown a http server.""" + if server.httpd is not None: + server.httpd.shutdown() + if server.http_server_thread is not None: + server.http_server_thread.join() + +def count_tuples_with_key(tuple_list, key_to_count): + """Counts the number of tuples in a list that have a specific key. + Args: + tuple_list: A list of key-value tuples. + key_to_count: The key to count occurrences of. + Returns: + The number of tuples with the specified key. + """ + count = 0 + for key, _ in tuple_list: + if key == key_to_count: + count += 1 + return count diff --git a/impala/tests/test_impala.py b/impala/tests/test_impala.py index baddcedb8..450c71c76 100644 --- a/impala/tests/test_impala.py +++ b/impala/tests/test_impala.py @@ -14,28 +14,27 @@ import sys import pytest -from impala.compat import _xrange as xrange from pytest import fixture BIGGER_TABLE_NUM_ROWS = 100 @fixture(scope='module') -def bigger_table(cur): +def bigger_table(session_cur): table_name = 'tmp_bigger_table' ddl = """CREATE TABLE {0} (s string) STORED AS PARQUET""".format(table_name) - cur.execute(ddl) + session_cur.execute(ddl) dml = """INSERT INTO {0} VALUES {1}""".format(table_name, - ",".join(["('row{0}')".format(i) for i in xrange(BIGGER_TABLE_NUM_ROWS)])) + ",".join(["('row{0}')".format(i) for i in range(BIGGER_TABLE_NUM_ROWS)])) # Disable codegen and expr rewrites so query runs faster. - cur.execute("set disable_codegen=1") - cur.execute("set enable_expr_rewrites=0") - cur.execute(dml) + session_cur.execute("set disable_codegen=1") + session_cur.execute("set enable_expr_rewrites=0") + session_cur.execute(dml) try: yield table_name finally: - cur.execute("DROP TABLE {0}".format(table_name)) + session_cur.execute("DROP TABLE {0}".format(table_name)) def test_has_more_rows(cur, bigger_table): @@ -50,5 +49,146 @@ def test_has_more_rows(cur, bigger_table): cur.execute("""select * from {0} where s != cast(sleep(2) as string)""".format(bigger_table)) - expected_rows = [("row{0}".format(i),) for i in xrange(BIGGER_TABLE_NUM_ROWS)] + expected_rows = [("row{0}".format(i),) for i in range(BIGGER_TABLE_NUM_ROWS)] assert sorted(cur.fetchall()) == sorted(expected_rows) + + +def test_has_more_rows_fetchcolumnar(cur, bigger_table): + """Test that impyla correctly handles empty row batches returned with the + hasMoreRows flag when using columnar batching.""" + # Set the fetch timeout very low and add sleeps so that Impala will return + # empty batches. Run on a single node with a single thread to make as predictable + # as possible. + cur.execute("set fetch_rows_timeout_ms=1") + cur.execute("set num_nodes=1") + cur.execute("set mt_dop=1") + cur.execute("""select * + from {0} + where s != cast(sleep(2) as string)""".format(bigger_table)) + expected_rows = [("row{0}".format(i),) for i in range(BIGGER_TABLE_NUM_ROWS)] + assert sorted(cur.fetchcolumnar()[0]) == sorted(expected_rows) + + +@fixture(scope='function') +def empty_table(cur): + table_name = 'tmp_empty_table' + ddl = """CREATE TABLE {0} (i int)""".format(table_name) + cur.execute(ddl) + try: + yield table_name + finally: + cur.execute("DROP TABLE {0}".format(table_name)) + +def test_dml_rowcount(cur, empty_table): + """Test that impyla correctly sets rowcount for insert statements.""" + dml = """INSERT INTO {0} + VALUES (0)""".format(empty_table) + cur.execute(dml) + assert cur.rowcount == 1 + +def test_row_count_in_empty_result(cur, empty_table): + """Test that impyla correctly sets rowcount when 0 rows are returned. + This case is missing from dbapi2 compliance tests. + """ + query = """SELECT * FROM {0}""".format(empty_table) + cur.execute(query) + cur.fetchall() + assert cur.rowcount == 0 + +def test_get_log(cur, empty_table): + """Test that impyla can return the result of get_log after the query + is closed. + """ + query = """SELECT * FROM {0}""".format(empty_table) + for mt_dop in ['0', '2']: + cur.execute(query, configuration={'mt_dop': mt_dop}) + cur.fetchall() + validate_log(cur) + cur.close_operation() + +def validate_log(cur): + # The query should be closed at this point. + assert not cur._last_operation_active + log = cur.get_log() + assert "100% Complete" in log + # Also check that summary and runtime profile are available + summary = cur.get_summary() + assert summary is not None + for node in summary.nodes: + assert hasattr(node, 'node_id') + assert hasattr(node, 'fragment_idx') + assert hasattr(node, 'label') + assert hasattr(node, 'label_detail') + assert hasattr(node, 'num_children') + assert hasattr(node, 'estimated_stats') + assert hasattr(node, 'exec_stats') + assert hasattr(node, 'is_broadcast') + assert hasattr(node, 'num_hosts') + assert node.num_hosts > 0 + assert len(node.exec_stats) >= node.num_hosts + profile = cur.get_profile() + assert profile is not None + +def test_build_summary_table(tmp_db, cur, empty_table): + """Test build_exec_summary function of impyla. + """ + tmp_db_lower = tmp_db.lower() + # Assert column Operator, #Host, #Inst, #Rows, Est. #Rows, Est. Peak Mem, and Detail. + # Skip column Avg Time, Max Time, and Peak Mem. + + def skip_cols(row): + assert len(row) == 10, row + output = list(row) + del output[7] + del output[4] + del output[3] + return output + + def validate_summary_table(table, expected): + for i in range(0, len(expected)): + row = skip_cols(table[i]) + assert expected[i] == row, 'Expect {0} but found {1}'.format( + str(expected[i]), str(row)) + + query = """SELECT * FROM {0} a INNER JOIN {1} b ON (a.i = b.i)""".format( + empty_table, empty_table) + cur.execute(query) + cur.fetchall() + summary = cur.get_summary() + output_dop_0 = list() + cur.build_summary_table(summary, output_dop_0) + assert len(output_dop_0) == 8, output_dop_0 + expected_dop_0 = [ + ['F02:ROOT', 1, 1, '', '', '4.00 MB', ''], + ['04:EXCHANGE', 1, 1, '0', '0', '16.00 KB', 'UNPARTITIONED'], + ['F00:EXCHANGE SENDER', 1, 1, '', '', '64.00 KB', ''], + ['02:HASH JOIN', 1, 1, '0', '0', '1.94 MB', 'INNER JOIN, BROADCAST'], + ['|--03:EXCHANGE', 1, 1, '0', '0', '16.00 KB', 'BROADCAST'], + ['| F01:EXCHANGE SENDER', 1, 1, '', '', '32.00 KB', ''], + ['| 01:SCAN HDFS', 1, 1, '0', '0', '0 B', + '{0}.{1} b'.format(tmp_db_lower, empty_table)], + ['00:SCAN HDFS', 1, 1, '0', '0', '0 B', + '{0}.{1} a'.format(tmp_db_lower, empty_table)], + ] + validate_summary_table(output_dop_0, expected_dop_0) + + cur.execute(query, configuration={'mt_dop': '2'}) + cur.fetchall() + summary = cur.get_summary() + output_dop_2 = list() + cur.build_summary_table(summary, output_dop_2) + assert len(output_dop_2) == 9, output_dop_2 + expected_dop_2 = [ + ['F02:ROOT', 1, 1, '', '', '4.00 MB', ''], + ['04:EXCHANGE', 1, 1, '0', '0', '16.00 KB', 'UNPARTITIONED'], + ['F00:EXCHANGE SENDER', 1, 1, '', '', '64.00 KB', ''], + ['02:HASH JOIN', 1, 1, '0', '0', '0 B', 'INNER JOIN, BROADCAST'], + ['|--F03:JOIN BUILD', 1, 1, '', '', '3.88 MB', ''], + ['| 03:EXCHANGE', 1, 1, '0', '0', '16.00 KB', 'BROADCAST'], + ['| F01:EXCHANGE SENDER', 1, 1, '', '', '32.00 KB', ''], + ['| 01:SCAN HDFS', 1, 1, '0', '0', '0 B', + '{0}.{1} b'.format(tmp_db_lower, empty_table)], + ['00:SCAN HDFS', 1, 1, '0', '0', '0 B', + '{0}.{1} a'.format(tmp_db_lower, empty_table)], + ] + validate_summary_table(output_dop_2, expected_dop_2) diff --git a/impala/tests/test_query_parameters.py b/impala/tests/test_query_parameters.py index 9fefbc611..0e9286beb 100644 --- a/impala/tests/test_query_parameters.py +++ b/impala/tests/test_query_parameters.py @@ -14,8 +14,6 @@ # Additional tests specific for query parameters # -from __future__ import absolute_import - from pytest import raises from impala.interface import _bind_parameters diff --git a/impala/tests/test_sqlalchemy.py b/impala/tests/test_sqlalchemy.py index c2fea15f3..b7a2c9229 100644 --- a/impala/tests/test_sqlalchemy.py +++ b/impala/tests/test_sqlalchemy.py @@ -12,20 +12,19 @@ # See the License for the specific language governing permissions and # limitations under the License. -from __future__ import absolute_import - from sqlalchemy.engine import create_engine -from sqlalchemy import Table, Column, select +from sqlalchemy import Table, Column, select, insert, text from sqlalchemy.schema import MetaData, CreateTable from impala.sqlalchemy import STRING, INT, DOUBLE, TINYINT, DATE, VARCHAR from impala.tests.util import ImpylaTestEnv +import pandas as pd TEST_ENV = ImpylaTestEnv() -def create_partitioned_test_table(engine): - metadata = MetaData(engine) +def create_partitioned_test_table(): + metadata = MetaData() # TODO: add other types to this table (e.g., functional.all_types) return Table("mytable", metadata, @@ -42,8 +41,8 @@ def create_partitioned_test_table(engine): 'transactional_properties': 'insert_only' }) -def create_simple_test_table(engine): - metadata = MetaData(engine) +def create_simple_test_table(): + metadata = MetaData() return Table("mytable", metadata, Column('col1', STRING), @@ -53,11 +52,12 @@ def create_simple_test_table(engine): ) def create_test_engine(diealect): - return create_engine('{0}://{1}:{2}'.format(diealect, TEST_ENV.host, TEST_ENV.port)) + host = "[%s]" % TEST_ENV.host if ":" in TEST_ENV.host else TEST_ENV.host + return create_engine('{0}://{1}:{2}'.format(diealect, host, TEST_ENV.port)) def test_sqlalchemy_impala_compilation(): engine = create_test_engine("impala") - observed = CreateTable(create_partitioned_test_table(engine), bind=engine) + observed = CreateTable(create_partitioned_test_table()).compile(engine) # The DATE column type of 'col5' will be replaced with TIMESTAMP. expected = ('\nCREATE TABLE mytable (\n\tcol1 STRING, \n\tcol2 TINYINT, ' '\n\tcol3 INT, \n\tcol4 DOUBLE, \n\tcol5 TIMESTAMP, \n\tcol6 VARCHAR(10)\n)' @@ -69,7 +69,7 @@ def test_sqlalchemy_impala_compilation(): def test_sqlalchemy_impala4_compilation(): engine = create_test_engine("impala4") - observed = CreateTable(create_partitioned_test_table(engine), bind=engine) + observed = CreateTable(create_partitioned_test_table()).compile(engine) # The DATE column type of 'col5' will be left as is. expected = ('\nCREATE TABLE mytable (\n\tcol1 STRING, \n\tcol2 TINYINT, ' '\n\tcol3 INT, \n\tcol4 DOUBLE, \n\tcol5 DATE, \n\tcol6 VARCHAR(10)\n)' @@ -80,26 +80,45 @@ def test_sqlalchemy_impala4_compilation(): def test_sqlalchemy_multiinsert(): engine = create_test_engine("impala4") - table = create_simple_test_table(engine) + table = create_simple_test_table() # TODO: Creating a non partitioned table as I am not sure about how to insert to # a partitioned table in SQL alchemy - create_table_stmt = CreateTable(table, bind=engine) + create_table_stmt = CreateTable(table) data = [ {"col1": "a", "col2": 1, "col3": 1, "col4": 1.0}, {"col1": "b", "col2": 2, "col3": 3, "col4": 2.0} ] - insert_stmt = table.insert(data) + insert_stmt = insert(table).values(data).compile(engine) expected_insert = 'INSERT INTO mytable (col1, col2, col3, col4) VALUES '\ '(%(col1_m0)s, %(col2_m0)s, %(col3_m0)s, %(col4_m0)s), '\ '(%(col1_m1)s, %(col2_m1)s, %(col3_m1)s, %(col4_m1)s)' assert expected_insert == str(insert_stmt) - engine.execute(create_table_stmt) - try: - engine.execute(insert_stmt) - result = engine.execute(select(table.c).order_by(table.c.col1)).fetchall() - expected_result = [('a', 1, 1, 1.0), ('b', 2, 3, 2.0)] - assert expected_result == result - finally: - table.drop() + with engine.connect() as conn: + conn.execute(create_table_stmt) + try: + conn.execute(insert_stmt) + result = conn.execute(select(table.c).order_by(table.c.col1)).fetchall() + expected_result = [('a', 1, 1, 1.0), ('b', 2, 3, 2.0)] + assert expected_result == result + finally: + table.drop(conn) + +def test_pandas_dataframe_to_sql(): + engine = create_test_engine("impala") + # Creating a sample dataframe to push to the DB. + df = pd.DataFrame([[1, 2, 3], [4, 5, 6], [7, 8, 9]], columns=['a', 'b', 'c']) + + with engine.connect() as conn: + try: + df.to_sql('test_table', conn, if_exists='replace', index=False) + table = pd.read_sql('DESCRIBE test_table', conn) + columns = table['name'].tolist() + assert ['a', 'b', 'c'] == columns + + finally: + try: + conn.execute(text('DROP TABLE IF EXISTS test_table')) + except Exception: + pass \ No newline at end of file diff --git a/impala/tests/test_thrift_api.py b/impala/tests/test_thrift_api.py new file mode 100644 index 000000000..3463e5bc6 --- /dev/null +++ b/impala/tests/test_thrift_api.py @@ -0,0 +1,24 @@ +import os + +import pytest + +from impala._thrift_api import ImpalaHttpClient + + +@pytest.fixture() +def proxy_env(): + reset_value = os.environ.get("HTTPS_PROXY") + os.environ["HTTPS_PROXY"] = "https://foo:%3F%40%3D@localhost" + yield "proxy_env" + if reset_value is None: + del os.environ["HTTPS_PROXY"] + else: + os.environ["HTTPS_PROXY"] = reset_value + + +class TestHttpTransport(object): + def test_proxy_auth_header(self, proxy_env): + client = ImpalaHttpClient( + uri_or_host="https://localhost:443/cliservice", + ) + assert client.proxy_auth == "Basic Zm9vOj9APQ==" diff --git a/impala/tests/test_util.py b/impala/tests/test_util.py index a3fab1a0b..8e5773161 100644 --- a/impala/tests/test_util.py +++ b/impala/tests/test_util.py @@ -12,18 +12,14 @@ # See the License for the specific language governing permissions and # limitations under the License. -from __future__ import absolute_import - -import sys from datetime import datetime, timedelta +import http.client as http_client -if sys.version_info[:2] <= (2, 6): - import unittest2 as unittest -else: - import unittest +import unittest import pytest -from impala.util import cookie_matches_path, get_cookie_expiry, get_all_matching_cookies +from impala.util import (cookie_matches_path, get_cookie_expiry, get_all_cookies, + get_all_matching_cookies, get_basic_credentials_for_request_headers) class ImpalaUtilTests(unittest.TestCase): @@ -77,14 +73,78 @@ def test_get_cookie_expiry(self): now = datetime.now() assert now + days2k <= get_cookie_expiry({'max-age': '172800000'}) <= now + days2k + sec + def test_get_matching_cookies(self): + cookies = get_all_cookies('/path', {}) + assert not cookies + + headers = make_cookie_headers([ + ('c_cookie', 'c_value'), + ('b_cookie', 'b_value'), + ('a_cookie', 'a_value') + ]) + cookies = csort(get_all_cookies('/path', headers)) + assert len(cookies) == 3 + assert cookies[0].key == 'a_cookie' and cookies[0].value == 'a_value' + assert cookies[1].key == 'b_cookie' and cookies[1].value == 'b_value' + assert cookies[2].key == 'c_cookie' and cookies[2].value == 'c_value' + + headers = make_cookie_headers([ + ('c_cookie', 'c_value;Path=/'), + ('b_cookie', 'b_value;Path=/path'), + ('a_cookie', 'a_value;Path=/') + ]) + cookies = csort(get_all_cookies('/path', headers)) + assert len(cookies) == 3 + assert cookies[0].key == 'a_cookie' and cookies[0].value == 'a_value' + assert cookies[1].key == 'b_cookie' and cookies[1].value == 'b_value' + assert cookies[2].key == 'c_cookie' and cookies[2].value == 'c_value' + + headers = make_cookie_headers([ + ('c_cookie', 'c_value;Path=/'), + ('B_cookie', 'B_value;Path=/path'), + ('a_cookie', 'a_value;Path=/path/path2') + ]) + cookies = csort(get_all_cookies('/path', headers)) + assert len(cookies) == 2 + assert cookies[0].key == 'B_cookie' and cookies[0].value == 'B_value' + assert cookies[1].key == 'c_cookie' and cookies[1].value == 'c_value' + + headers = make_cookie_headers([ + ('c_cookie', 'c_value;Path=/'), + ('b_cookie', 'b_value;Path=/path'), + ('a_cookie', 'a_value;Path=/path') + ]) + cookies = csort(get_all_cookies('/path/path1', headers)) + assert len(cookies) == 3 + assert cookies[0].key == 'a_cookie' and cookies[0].value == 'a_value' + assert cookies[1].key == 'b_cookie' and cookies[1].value == 'b_value' + assert cookies[2].key == 'c_cookie' and cookies[2].value == 'c_value' + + headers = make_cookie_headers([ + ('b_cookie', 'b_value;Path=/path1'), + ('a_cookie', 'a_value;Path=/path2') + ]) + cookies = get_all_cookies('/path', headers) + assert not cookies + + headers = make_cookie_headers([ + ('c_cookie', 'c_value;Path=/'), + ('b_cookie', 'b_value;Path=/path1'), + ('a_cookie', 'a_value;Path=/path2') + ]) + cookies = get_all_cookies('/path', headers) + assert len(cookies) == 1 + assert cookies[0].key == 'c_cookie' and cookies[0].value == 'c_value' def test_get_all_matching_cookies(self): cookies = get_all_matching_cookies(['a', 'b'], '/path', {}) assert not cookies - headers = {'Set-Cookie': '''c_cookie=c_value - b_cookie=b_value - a_cookie=a_value'''} + headers = make_cookie_headers([ + ('c_cookie', 'c_value'), + ('b_cookie', 'b_value'), + ('a_cookie', 'a_value') + ]) cookies = get_all_matching_cookies(['a_cookie', 'b_cookie'], '/path', headers) assert len(cookies) == 2 assert cookies[0].key == 'a_cookie' and cookies[0].value == 'a_value' @@ -94,17 +154,21 @@ def test_get_all_matching_cookies(self): assert cookies[0].key == 'b_cookie' and cookies[0].value == 'b_value' assert cookies[1].key == 'a_cookie' and cookies[1].value == 'a_value' - headers = {'Set-Cookie': '''c_cookie=c_value;Path=/ - b_cookie=b_value;Path=/path - a_cookie=a_value;Path=/'''} + headers = make_cookie_headers([ + ('c_cookie', 'c_value;Path=/'), + ('b_cookie', 'b_value;Path=/path'), + ('a_cookie', 'a_value;Path=/') + ]) cookies = get_all_matching_cookies(['a_cookie', 'b_cookie'], '/path', headers) assert len(cookies) == 2 assert cookies[0].key == 'a_cookie' and cookies[0].value == 'a_value' assert cookies[1].key == 'b_cookie' and cookies[1].value == 'b_value' - headers = {'Set-Cookie': '''c_cookie=c_value;Path=/ - b_cookie=b_value;Path=/path - a_cookie=a_value;Path=/path/path2'''} + headers = make_cookie_headers([ + ('c_cookie', 'c_value;Path=/'), + ('b_cookie', 'b_value;Path=/path'), + ('a_cookie', 'a_value;Path=/path/path2') + ]) cookies = get_all_matching_cookies(['a_cookie', 'b_cookie'], '/path', headers) assert len(cookies) == 1 assert cookies[0].key == 'b_cookie' and cookies[0].value == 'b_value' @@ -112,25 +176,62 @@ def test_get_all_matching_cookies(self): assert len(cookies) == 1 assert cookies[0].key == 'b_cookie' and cookies[0].value == 'b_value' - headers = {'Set-Cookie': '''c_cookie=c_value;Path=/ - b_cookie=b_value;Path=/path - a_cookie=a_value;Path=/path'''} + headers = make_cookie_headers([ + ('c_cookie', 'c_value;Path=/'), + ('b_cookie', 'b_value;Path=/path'), + ('a_cookie', 'a_value;Path=/path') + ]) cookies = get_all_matching_cookies(['a_cookie', 'b_cookie'], '/path/path1', headers) assert len(cookies) == 2 assert cookies[0].key == 'a_cookie' and cookies[0].value == 'a_value' assert cookies[1].key == 'b_cookie' and cookies[1].value == 'b_value' - headers = {'Set-Cookie': '''c_cookie=c_value;Path=/ - b_cookie=b_value;Path=/path1 - a_cookie=a_value;Path=/path2'''} + headers = make_cookie_headers([ + ('c_cookie', 'c_value;Path=/'), + ('b_cookie', 'b_value;Path=/path1'), + ('a_cookie', 'a_value;Path=/path2') + ]) cookies = get_all_matching_cookies(['a_cookie', 'b_cookie'], '/path', headers) assert not cookies - headers = {'Set-Cookie': '''c_cookie=c_value;Path=/ - b_cookie=b_value;Path=/path1 - a_cookie=a_value;Path=/path2'''} + headers = make_cookie_headers([ + ('c_cookie', 'c_value;Path=/'), + ('b_cookie', 'b_value;Path=/path1'), + ('a_cookie', 'a_value;Path=/path2') + ]) cookies = get_all_matching_cookies(['a_cookie', 'b_cookie', 'c_cookie'], '/path', headers) assert len(cookies) == 1 - assert cookies[0].key == 'c_cookie' and cookies[0].value == 'c_value' \ No newline at end of file + assert cookies[0].key == 'c_cookie' and cookies[0].value == 'c_value' + + def test_get_basic_credentials_for_request_headers(self): + assert get_basic_credentials_for_request_headers( + user="foo", + password="bar" + ) == "Zm9vOmJhcg==" + assert get_basic_credentials_for_request_headers( + user="thisisaratherlongusername", + password="withanotherverylongpasswordresultinginanencodinglongerthan76chars" + ) == "dGhpc2lzYXJhdGhlcmxvbmd1c2VybmFtZTp3aXRoYW5vdGhlcnZlcnlsb25ncGFzc3dvcmRyZXN1bHRpbmdpbmFuZW5jb2Rpbmdsb25nZXJ0aGFuNzZjaGFycw==" + assert get_basic_credentials_for_request_headers( + user="?", + password="?" + ) == "Pzo/" + + +def make_cookie_headers(cookie_vals): + """Make an HTTPMessage containing Set-Cookie headers.""" + # The HTTPMessage is an email.message.Message, and the + # Set-Cookie values appear as duplicate headers. + headers = http_client.HTTPMessage() + for pair in cookie_vals: + name = pair[0] + value = pair[1] + headers.add_header('Set-Cookie', name + "=" + value) + return headers + +def csort(cookies): + """Sort list of Morsels as header order is not guaranteed.""" + cookies.sort(key=lambda c: c.key) + return cookies diff --git a/impala/tests/util.py b/impala/tests/util.py index 60eb891ad..b7947135a 100644 --- a/impala/tests/util.py +++ b/impala/tests/util.py @@ -14,7 +14,6 @@ import os import sys -import six import socket @@ -28,6 +27,17 @@ def get_env_var(name, coercer, default): sys.stderr.write("{0} not set; using {1!r}\n".format(name, default)) return default +def is_ipv6_only_host(host, port): + has_ipv6 = False + for addr in socket.getaddrinfo(host, port, socket.AF_UNSPEC, + socket.SOCK_STREAM, socket.IPPROTO_TCP): + (family, _, _, _, _) = addr + if family == socket.AF_INET: + return False # found ipv4 + elif family == socket.AF_INET6: + has_ipv6 = True + return has_ipv6 + class ImpylaTestEnv(object): @@ -65,10 +75,11 @@ def __init__(self, host=None, port=None, hive_port=None, auth_mech=None, 'NOSASL') self.ssl_cert = get_env_var('IMPYLA_SSL_CERT', identity, "") + self.ssl_wrong_cert = get_env_var('IMPYLA_SSL_WRONG_CERT', identity, "") def __repr__(self): kvs = ['{0}={1}'.format(k, v) - for (k, v) in six.iteritems(self.__dict__)] + for (k, v) in self.__dict__.items()] return 'ImpylaTestEnv(\n {0})'.format(',\n '.join(kvs)) class SocketTracker(object): diff --git a/impala/thrift/ErrorCodes.thrift b/impala/thrift/ErrorCodes.thrift index 6b4cf1178..ff122c943 100644 --- a/impala/thrift/ErrorCodes.thrift +++ b/impala/thrift/ErrorCodes.thrift @@ -158,7 +158,37 @@ enum TErrorCode { EXEC_TIME_LIMIT_EXCEEDED = 128, CPU_LIMIT_EXCEEDED = 129, SCAN_BYTES_LIMIT_EXCEEDED = 130, - ROWS_PRODUCED_LIMIT_EXCEEDED = 131 + ROWS_PRODUCED_LIMIT_EXCEEDED = 131, + EXPR_REWRITE_RESULT_LIMIT_EXCEEDED = 132, + UNRESPONSIVE_BACKEND = 133, + PARQUET_DATE_OUT_OF_RANGE = 134, + DISCONNECTED_SESSION_CLOSED = 135, + UNAUTHORIZED_SESSION_USER = 136, + ZSTD_ERROR = 137, + LZ4_BLOCK_DECOMPRESS_DECOMPRESS_SIZE_INCORRECT = 138, + LZ4_BLOCK_DECOMPRESS_INVALID_INPUT_LENGTH = 139, + LZ4_BLOCK_DECOMPRESS_INVALID_COMPRESSED_LENGTH = 140, + LZ4_DECOMPRESS_SAFE_FAILED = 141, + LZ4_COMPRESS_DEFAULT_FAILED = 142, + MAX_STATEMENT_LENGTH_EXCEEDED = 143, + AVRO_INVALID_DATE = 144, + ORC_TIMESTAMP_OUT_OF_RANGE = 145, + ORC_DATE_OUT_OF_RANGE = 146, + ORC_NESTED_TYPE_MISMATCH = 147, + ORC_TYPE_NOT_ROOT_AT_STRUCT = 148, + NAAJ_OUT_OF_MEMORY = 149, + INVALID_QUERY_HANDLE = 150, + JOIN_ROWS_PRODUCED_LIMIT_EXCEEDED = 151, + LOCAL_DISK_FAULTY = 152, + JWKS_PARSE_ERROR = 153, + JWT_VERIFY_FAILED = 154, + PARQUET_ROWS_SKIPPING = 155, + QUERY_OPTION_PARSE_FAILED = 156, + CATALOG_INCOMPATIBLE_PROTOCOL = 157, + STATESTORE_INCOMPATIBLE_PROTOCOL = 158, + JDBC_CONFIGURATION_ERROR = 159, + TUPLE_CACHE_INCONSISTENCY = 160, + OAUTH_VERIFY_FAILED = 161 } const list TErrorMessage = [ // OK @@ -196,7 +226,7 @@ const list TErrorMessage = [ // PARQUET_MISSING_PRECISION "File '$0' column '$1' does not have the decimal precision set.", // PARQUET_WRONG_PRECISION - "File '$0' column '$1' has a precision that does not match the table metadata precision. File metadata precision: $2, table metadata precision: $3.", + "File '$0' column '$1' has a precision that does not match the table metadata precision. File metadata precision: $2, table metadata precision: $3.", // PARQUET_BAD_CONVERTED_TYPE "File '$0' column '$1' does not have converted type set to DECIMAL", // PARQUET_INCOMPATIBLE_DECIMAL @@ -282,7 +312,7 @@ const list TErrorMessage = [ // STALE_METADATA_FILE_TOO_SHORT "Metadata for file '$0' appears stale. Try running \"refresh $1\" to reload the file metadata.", // PARQUET_BAD_VERSION_NUMBER - "File '$0' has an invalid version number: $1\nThis could be due to stale metadata. Try running \"refresh $2\".", + "File '$0' has an invalid Parquet version number: $1.\nPlease check that it is a valid Parquet file. This error can also occur due to stale metadata. If you believe this is a valid Parquet file, try running \"refresh $2\".", // SCANNER_INCOMPLETE_READ "Tried to read $0 bytes but could only read $1 bytes. This may indicate data file corruption. (file $2, byte offset: $3)", // SCANNER_INVALID_READ @@ -310,7 +340,7 @@ const list TErrorMessage = [ // IMPALA_KUDU_TYPE_MISSING "Impala type $0 is not available in Kudu.", // KUDU_NOT_SUPPORTED_ON_OS - "Kudu is not supported on this operating system.", + "Not in use.", // KUDU_NOT_ENABLED "Kudu features are disabled by the startup flag --disable_kudu.", // PARTITIONED_HASH_JOIN_REPARTITION_FAILS @@ -364,13 +394,13 @@ const list TErrorMessage = [ // PARQUET_TIMESTAMP_OUT_OF_RANGE "Parquet file '$0' column '$1' contains an out of range timestamp. The valid date range is 1400-01-01..9999-12-31.", // SCRATCH_ALLOCATION_FAILED - "Could not create files in any configured scratch directories (--scratch_dirs=$0) on backend '$1'. $2 of scratch is currently in use by this Impala Daemon ($3 by this query). See logs for previous errors that may have prevented creating or writing scratch files.", + "Could not create files in any configured scratch directories (--scratch_dirs=$0) on backend '$1'. $2 of scratch is currently in use by this Impala Daemon ($3 by this query). See logs for previous errors that may have prevented creating or writing scratch files. The following directories were at capacity: $4", // SCRATCH_READ_TRUNCATED "Error reading $0 bytes from scratch file '$1' on backend $2 at offset $3: could only read $4 bytes", // KUDU_TIMESTAMP_OUT_OF_RANGE "Kudu table '$0' column '$1' contains an out of range timestamp. The valid date range is 1400-01-01..9999-12-31.", // MAX_ROW_SIZE - "Row of size $0 could not be materialized in plan node with id $1. Increase the max_row_size query option (currently $2) to process larger rows.", + "Row of size $0 could not be materialized by $1. Increase the max_row_size query option (currently $2) to process larger rows.", // IR_VERIFY_FAILED "Failed to verify generated IR function $0, see log for more details.", // MINIMUM_RESERVATION_UNAVAILABLE @@ -378,7 +408,7 @@ const list TErrorMessage = [ // ADMISSION_REJECTED "Rejected query from pool $0: $1", // ADMISSION_TIMED_OUT - "Admission for query exceeded timeout $0ms in pool $1. Queued reason: $2", + "Admission for query exceeded timeout $0ms in pool $1. Queued reason: $2 Additional Details: $3", // THREAD_CREATION_FAILED "Failed to create thread $0 in category $1: $2", // DISK_IO_ERROR @@ -424,5 +454,65 @@ const list TErrorMessage = [ // SCAN_BYTES_LIMIT_EXCEEDED "Query $0 terminated due to scan bytes limit of $1", // ROWS_PRODUCED_LIMIT_EXCEEDED - "Query $0 terminated due to rows produced limit of $1. Unset or increase NUM_ROWS_PRODUCED_LIMIT query option to produce more rows." -] + "Query $0 terminated due to rows produced limit of $1. Unset or increase NUM_ROWS_PRODUCED_LIMIT query option to produce more rows.", + // EXPR_REWRITE_RESULT_LIMIT_EXCEEDED + "Expression rewrite rejected due to result size ($0) exceeding the limit ($1).", + // UNRESPONSIVE_BACKEND + "Query $0 cancelled due to unresponsive backend: $1 has not sent a report in $2ms (max allowed lag is $3ms)", + // PARQUET_DATE_OUT_OF_RANGE + "Parquet file '$0' column '$1' contains an out of range date. The valid date range is 0001-01-01..9999-12-31.", + // DISCONNECTED_SESSION_CLOSED + "Session closed because it has no active connections", + // UNAUTHORIZED_SESSION_USER + "The user authorized on the connection '$0' does not match the session username '$1'", + // ZSTD_ERROR + "$0 failed with error: $1", + // LZ4_BLOCK_DECOMPRESS_DECOMPRESS_SIZE_INCORRECT + "LZ4Block: Decompressed size is not correct.", + // LZ4_BLOCK_DECOMPRESS_INVALID_INPUT_LENGTH + "LZ4Block: Invalid input length.", + // LZ4_BLOCK_DECOMPRESS_INVALID_COMPRESSED_LENGTH + "LZ4Block: Invalid compressed length. Data is likely corrupt.", + // LZ4_DECOMPRESS_SAFE_FAILED + "LZ4: LZ4_decompress_safe failed", + // LZ4_COMPRESS_DEFAULT_FAILED + "LZ4: LZ4_compress_default failed", + // MAX_STATEMENT_LENGTH_EXCEEDED + "Statement length of $0 bytes exceeds the maximum statement length ($1 bytes)", + // AVRO_INVALID_DATE + "Avro file '$0' is corrupt: out of range date value $1 at offset $2. The valid date range is -719162..2932896 (0001-01-01..9999-12-31).", + // ORC_TIMESTAMP_OUT_OF_RANGE + "ORC file '$0' column '$1' contains an out of range timestamp. The valid date range is 1400-01-01..9999-12-31.", + // ORC_DATE_OUT_OF_RANGE + "ORC file '$0' column '$1' contains an out of range date. The valid date range is 0001-01-01..9999-12-31.", + // ORC_NESTED_TYPE_MISMATCH + "File '$0' has an incompatible ORC schema for column '$1', Column type: $2, ORC schema: $3", + // ORC_TYPE_NOT_ROOT_AT_STRUCT + "Root of the $0 type returned by the ORC lib is not STRUCT: $1. Either there are bugs in the ORC lib or ORC file '$2' is corrupt.", + // NAAJ_OUT_OF_MEMORY + "Unable to perform Null-Aware Anti-Join. Could not get enough reservation to fit all rows with NULLs from the build side in memory. Memory required for $0 rows was $1. $2/$3 of the join's reservation was available for the rows.", + // INVALID_QUERY_HANDLE + "Invalid or unknown query handle: $0.", + // JOIN_ROWS_PRODUCED_LIMIT_EXCEEDED + "Query $0 terminated due to join rows produced exceeds the limit of $1 at node with id $2. Unset or increase JOIN_ROWS_PRODUCED_LIMIT query option to produce more rows.", + // LOCAL_DISK_FAULTY + "Query execution failure caused by local disk IO fatal error on backend: $0.", + // JWKS_PARSE_ERROR + "Error parsing JWKS: $0.", + // JWT_VERIFY_FAILED + "Error verifying JWT Token: $0.", + // PARQUET_ROWS_SKIPPING + "Couldn't skip rows in column '$0' in file '$1'.", + // QUERY_OPTION_PARSE_FAILED + "Failed to parse query option '$0': $1", + // CATALOG_INCOMPATIBLE_PROTOCOL + "Client has incompatible protocol version V$0 conflicting with catalogd's version V$1", + // STATESTORE_INCOMPATIBLE_PROTOCOL + "Subscriber '$0' has incompatible protocol version V$1 conflicting with statestored's version V$2", + // JDBC_CONFIGURATION_ERROR + "Error in JDBC table configuration: $0.", + // TUPLE_CACHE_INCONSISTENCY + "Inconsistent tuple cache found: $0.", + // OAUTH_VERIFY_FAILED + "Error verifying OAuth Token: $0." +] \ No newline at end of file diff --git a/impala/thrift/ExecStats.thrift b/impala/thrift/ExecStats.thrift index 8861e6dd3..882094622 100644 --- a/impala/thrift/ExecStats.thrift +++ b/impala/thrift/ExecStats.thrift @@ -48,7 +48,6 @@ struct TExecStats { // Total CPU time spent across all threads. For operators that have an async // component (e.g. multi-threaded) this will be >= latency_ns. - // TODO-MT: remove this or latency_ns 2: optional i64 cpu_time_ns // Number of rows returned. @@ -76,12 +75,18 @@ struct TPlanNodeExecSummary { // If true, this is an exchange node that is the receiver of a broadcast. 8: optional bool is_broadcast + + // The number of hosts. It cannot be inferred from exec_stats, since the length of the + // list can be greater when mt_dop > 0. + 9: optional i32 num_hosts } // Progress counters for an in-flight query. struct TExecProgress { 1: optional i64 total_scan_ranges 2: optional i64 num_completed_scan_ranges + 3: optional i64 total_fragment_instances; + 4: optional i64 num_completed_fragment_instances; } // Execution summary of an entire query. @@ -95,8 +100,9 @@ struct TExecSummary { // Flattened execution summary of the plan tree. 3: optional list nodes - // For each exch node in 'nodes', contains the index to the root node of the sending - // fragment for this exch. Both the key and value are indices into 'nodes'. + // For each node in 'nodes' that consumes input from the root of a different fragment, + // i.e. an exchange or join node with a separate build, contains the index to the root + // node of the source fragment. Both the key and value are indices into 'nodes'. 4: optional map exch_to_sender_map // List of errors that were encountered during execution. This can be non-empty diff --git a/impala/thrift/ImpalaService.thrift b/impala/thrift/ImpalaService.thrift index 0302621cc..9249d5f2c 100644 --- a/impala/thrift/ImpalaService.thrift +++ b/impala/thrift/ImpalaService.thrift @@ -90,15 +90,28 @@ enum TImpalaQueryOptions { // invalid, the option is ignored. // // 2. Global actions - // ":@@@...", - // global labels are marked in the code with DEBUG_ACTION*() macros. + // "::...::@@@...", + // Used with the DebugAction() call, the action will be performed if the label and + // optional arguments all match. The arguments can be used to make the debug action + // context dependent, for example to only fail rpcs when a particular hostname matches. + // Note that some debug actions must be specified as a query option while others must + // be passed in with the startup flag. // Available global actions: // - SLEEP@ will sleep for the 'ms' milliseconds. // - JITTER@[@] will sleep for a random amount of time between 0 // and 'ms' milliseconds with the given probability. If is omitted, // it is 1.0. - // - FAIL[@] returns an INTERNAL_ERROR status with the given - // probability. If is omitted, it is 1.0. + // - FAIL[@][@] returns an INTERNAL_ERROR status with the given + // probability and error. If is omitted, it is 1.0. If 'error' is + // omitted, a generic error of the form: 'Debug Action: