From d6ce59b192190dd2aef54695e7fe195c70c65944 Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Mon, 22 Jan 2024 22:25:28 -0800 Subject: [PATCH 01/17] Adding `expansion.bzl` for Fully Expanding Environment Variables The built-in functionality (exposed Skylark methods on `ctx`) for expanding environment variables leaves a lot of implementation to do in `bzl` files. This PR adds in that missing functionality for easy reuse with a new `expansion` struct. `expansion` has various methods for expanding environment variables with a flexible API. The existing APIs handle only one piece at a time: * `ctx.expand_location()` only handles `$(location ...)` (and similar) expansion. It does not handle any "make variable" expansion (no expansion from `TemplateVariableInfo` or other providers via toolchains). * `ctx.expand_make_variables()` only handles make variables. If it encounters `$(location ...)` (or similar), it [errors out with no means of recovery](https://github.com/bazelbuild/bazel/blob/addf41bce1f26e8bc4bd0b09cb2fbffbb0446f25/src/main/java/com/google/devtools/build/lib/analysis/ConfigurationMakeVariableContext.java#L136). This method is also marked as deprecated, with `ctx.var` listed as the preferred API. * `ctx.var` is a simple dictionary, which contains resolved mappings based on toolchains. However, being a simple data structure (not a function) leaves recursive functionality and/or integration with `ctx.expand_location()` to be implemented by hand (or in this PR). Many internal systems make use of functionality that fully resolves both make variables and `$(location ...)` (and does so recursively). This is done with `ruleContext.getExpander().withDataLocations()` ([example](https://github.com/bazelbuild/bazel/blob/addf41bce1f26e8bc4bd0b09cb2fbffbb0446f25/src/main/java/com/google/devtools/build/lib/analysis/RunfilesSupport.java#L522)). However, this is never exposed to skylark. This means that built-in rules will have the `env` attribute fully expanded, but custom rules cannot (easily). The above mentioned methods have their own (somewhat duplicated) implementations ([`ctx.expand_location()`](https://github.com/bazelbuild/bazel/blob/addf41bce1f26e8bc4bd0b09cb2fbffbb0446f25/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkRuleContext.java#L1011), [`ctx.expand_make_variables()`](https://github.com/bazelbuild/bazel/blob/addf41bce1f26e8bc4bd0b09cb2fbffbb0446f25/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkRuleContext.java#L964), [`ctx.var`](https://github.com/bazelbuild/bazel/blob/addf41bce1f26e8bc4bd0b09cb2fbffbb0446f25/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkRuleContext.java#L823)). Note the identical `ConfigurationMakeVariableContext` initialization [here](https://github.com/bazelbuild/bazel/blob/addf41bce1f26e8bc4bd0b09cb2fbffbb0446f25/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkRuleContext.java#L949) and [here](https://github.com/bazelbuild/bazel/blob/36fa60b1805faa7da2c4b5330b4b186740f5f00d/src/main/java/com/google/devtools/build/lib/analysis/RuleContext.java#L978) -- identical except for the use of `additionalSubstitutionsMap`, which could have been delegated (similar to the impl of `LocationTemplateContext`). Also note the separate/duplicated parsing implementations in [`LocationTemplateContext`](https://github.com/bazelbuild/bazel/blob/addf41bce1f26e8bc4bd0b09cb2fbffbb0446f25/src/main/java/com/google/devtools/build/lib/analysis/stringtemplate/TemplateExpander.java#L77) and in [`LocationExpander`](https://github.com/bazelbuild/bazel/blob/36fa60b1805faa7da2c4b5330b4b186740f5f00d/src/main/java/com/google/devtools/build/lib/analysis/LocationExpander.java#L179). This PR tries to avoid more duplicate implementations (which can't really happen anyway; as this is in external Skylark, not Java / Bazel runtime). These new methods make use of `ctx.expand_location()` and `ctx.var` to allow fully recursive variable expansion that incorporates both inputs (toolchains and `location`). The methods avoid copies/string mutations/extra loops when necessary to ensure that the functions can run quickly. The methods' API allows for manual/direct data structures (lists/dicts) as input, or for pulling values directly from `ctx`. `tests/expansion_tests.bzl` added to test all added methods. This PR is being submitted here so that it can be reused (instead of copied in many repos). It is also preferable to add functionality here in Skylib, instead of (or in addition to) any changes in the [Bazel repo](https://github.com/bazelbuild/bazel) so that it can be more easily pulled into projects with a pinned Bazel version. Please feel free to leave comments or suggestion on ways to improve this PR, or let me know if you have any questions. Thanks! --- lib/expansion.bzl | 338 ++++++++++++++++++ tests/BUILD | 3 + tests/expansion_tests.bzl | 697 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 1038 insertions(+) create mode 100644 lib/expansion.bzl create mode 100644 tests/expansion_tests.bzl diff --git a/lib/expansion.bzl b/lib/expansion.bzl new file mode 100644 index 00000000..47caae9e --- /dev/null +++ b/lib/expansion.bzl @@ -0,0 +1,338 @@ + +"""Skylib module containing functions that aid in environment variable expansion.""" + +def _valid_char_for_env_var_name(char): + return char.isalnum() or char == "_" + +def _key_to_be_expanded(str_with_key, key, start_of_key_index): + # Check that the string at index is prefixed with an odd number of `$`. + # Odd number means that the last `$` is not escaped. + dollar_sign_count = 0 + for index in range(start_of_key_index, -1, -1): + if str_with_key[index] != "$": + break + dollar_sign_count += 1 + + # Check that the key is correctly matched. + # Specifically, check the key isn't matching to another key (substring). + key_mismatch = False + if key[-1] not in (")", "}"): + end_of_key_index = start_of_key_index + len(key) + key_mismatch = ( + (end_of_key_index < len(str_with_key)) and + _valid_char_for_env_var_name(str_with_key[end_of_key_index]) + ) + + return (not key_mismatch) and (dollar_sign_count % 2) == 1 + +def _expand_key_in_str(key, val, unexpanded_str): + key_len = len(key) + val_len = len(val) + searched_index = 0 + expanded_str = unexpanded_str + # Max iterations at the length of the str; will likely break out earlier. + for _ in range(len(expanded_str)): + used_key_index = expanded_str.find(key, searched_index) + if used_key_index < 0: + break + if _key_to_be_expanded(expanded_str, key, used_key_index): + # Only replace this one instance that we have verified (count = 1). + # Avoid extra string splicing, if possible. + if searched_index == 0: + expanded_str = expanded_str.replace(key, val, 1) + else: + expanded_str = ( + expanded_str[0:searched_index-1] + + expanded_str[searched_index:].replace(key, val, 1) + ) + searched_index += val_len + else: + searched_index += key_len + return expanded_str + +def _expand_all_keys_in_str_from_dict(replacement_dict, unexpanded_str): + # Manually expand variables based on the var dict. + # Do not use `ctx.expand_make_variables()` as it will error out if any variable expands to + # `$(location )` (or similar) instead of leaving it untouched. + expanded_val = unexpanded_str + for avail_key, corresponding_val in replacement_dict.items(): + if expanded_val.find(avail_key) < 0: + continue + considered_key_formats = ("${}", "${{{}}}", "$({})") + formatted_keys = [key_format.format(avail_key) for key_format in considered_key_formats] + # Skip self-references (e.g. {"VAR": "$(VAR)"}) + # This may happen (and is ok) for the `env` attribute, where keys can be reused to be + # expanded by the resolved dict. + if corresponding_val in formatted_keys: + continue + # Expand each format style of this key, if it exists. + for formatted_key in formatted_keys: + expanded_val = _expand_key_in_str(formatted_key, corresponding_val, expanded_val) + return expanded_val + +def _expand_tc_all_keys_in_str(resolved_replacement_dict, env_replacement_dict, unexpanded_str): + if unexpanded_str.find("$") < 0: + return unexpanded_str + + expanded_val = unexpanded_str + prev_val = expanded_val + # Max iterations at the length of the str; will likely break out earlier. + for _ in range(len(expanded_val)): + # Expand values first from the `env` attribute, then by the toolchain resolved values. + expanded_val = _expand_all_keys_in_str_from_dict(env_replacement_dict, expanded_val) + expanded_val = _expand_all_keys_in_str_from_dict(resolved_replacement_dict, expanded_val) + + # Break out early if nothing changed in this iteration. + if prev_val == expanded_val: + break + prev_val = expanded_val + + return expanded_val + +def _expand_tc_and_loc_all_keys_in_str( + expand_location, + resolved_replacement_dict, + env_replacement_dict, + unexpanded_str): + if unexpanded_str.find("$") < 0: + return unexpanded_str + + expanded_val = unexpanded_str + prev_val = expanded_val + # Max iterations at the length of the str; will likely break out earlier. + for _ in range(len(expanded_val)): + # First let's try the safe `location` (et al) expansion logic. + # `$VAR`, `$(VAR)`, and `${VAR}` will be left untouched. + expanded_val = expand_location(expanded_val) + + # Break early if nothing left to expand. + if expanded_val.find("$") < 0: + break + + # Expand values first from the `env` attribute, then by the toolchain resolved values. + expanded_val = _expand_all_keys_in_str_from_dict(env_replacement_dict, expanded_val) + expanded_val = _expand_all_keys_in_str_from_dict(resolved_replacement_dict, expanded_val) + + # Break out early if nothing changed in this iteration. + if prev_val == expanded_val: + break + prev_val = expanded_val + + return expanded_val + +def _expand_with_manual_dict(resolution_dict, source_env_dict): + """ + Recursively expands all values in `source_env_dict` using the given lookup data. + + All keys of `source_env_dict` are returned in the resultant dict with values expanded by + lookups via `resolution_dict` dict. + This function does not modify any of the given parameters. + + Args: + resolution_dict: (Required) A dictionary with resolved key/value pairs to be used for + lookup when resolving values. This may come from toolchains (via + `ctx.var`) or other sources. + source_env_dict: (Required) The source for all desired expansions. All key/value pairs + will appear within the returned dictionary, with all values fully + expanded by lookups in `resolution_dict`. + + Returns: + A new dict with all key/values from `source_env_dict`, where all values have been recursively + expanded. + """ + expanded_envs = {} + for env_key, unexpanded_val in source_env_dict.items(): + expanded_envs[env_key] = ( + _expand_tc_all_keys_in_str( + resolution_dict, + source_env_dict, + unexpanded_val + ) + ) + return expanded_envs + +def _expand_with_manual_dict_and_location(expand_location, resolution_dict, source_env_dict): + """ + Recursively expands all values in `source_env_dict` using the given logic / lookup data. + + All keys of `source_env_dict` are returned in the resultant dict with values expanded by + location expansion logic via `expand_location` and by lookups via `resolution_dict` dict. + This function does not modify any of the given parameters. + + Args: + expand_location: (Required) A function that takes in a string and properly replaces + `$(location ...)` (and similar) with the corresponding values. This + likely should correspond to `ctx.expand_location()`. + resolution_dict: (Required) A dictionary with resolved key/value pairs to be used for + lookup when resolving values. This may come from toolchains (via + `ctx.var`) or other sources. + source_env_dict: (Required) The source for all desired expansions. All key/value pairs + will appear within the returned dictionary, with all values fully + expanded by the logic expansion logic of `expand_location` and by + lookup in `resolution_dict`. + + Returns: + A new dict with all key/values from `source_env_dict`, where all values have been recursively + expanded. + """ + expanded_envs = {} + for env_key, unexpanded_val in source_env_dict.items(): + expanded_envs[env_key] = ( + _expand_tc_and_loc_all_keys_in_str( + expand_location, + resolution_dict, + source_env_dict, + unexpanded_val) + ) + return expanded_envs + +def _expand_with_toolchains(ctx, source_env_dict, additional_lookup_dict = None): + """ + Recursively expands all values in `source_env_dict` using the given lookup data. + + All keys of `source_env_dict` are returned in the resultant dict with values expanded by + lookups via `ctx.var` dict (unioned with optional `additional_lookup_dict` parameter). + This function does not modify any of the given parameters. + + Args: + ctx: (Required) The bazel context object. This is used to access the `ctx.var` member + for use as the "resolution dict". This makes use of providers from toolchains for + environment variable expansion. + source_env_dict: (Required) The source for all desired expansions. All key/value pairs + will appear within the returned dictionary, with all values fully + expanded by lookups in `resolution_dict`. + additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for + variable expansion. + + Returns: + A new dict with all key/values from `source_env_dict`, where all values have been recursively + expanded. + """ + resolution_dict = ctx.var + if additional_lookup_dict: + resolution_dict = resolution_dict | additional_lookup_dict + return _expand_with_manual_dict(resolution_dict, source_env_dict) + +def _expand_with_toolchains_and_location( + ctx, + deps, + source_env_dict, + additional_lookup_dict = None): + """ + Recursively expands all values in `source_env_dict` using the `ctx` logic / lookup data. + + All keys of `source_env_dict` are returned in the resultant dict with values expanded by + location expansion logic via `ctx.expand_location` and by lookups via `ctx.var` dict (unioned + with optional `additional_lookup_dict` parameter). + This function does not modify any of the given parameters. + + Args: + ctx: (Required) The bazel context object. This is used to access the `ctx.var` member + for use as the "resolution dict". This makes use of providers from toolchains for + environment variable expansion. This object is also used for the + `ctx.expand_location` method to handle `$(location ...)` (and similar) expansion + logic. + deps: (Required) The set of targets used with `ctx.expand_location` for expanding + `$(location ...)` (and similar) expressions. + source_env_dict: (Required) The source for all desired expansions. All key/value pairs + will appear within the returned dictionary, with all values fully + expanded by the logic expansion logic of `expand_location` and by + lookup in `resolution_dict`. + additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for + variable expansion. + + Returns: + A new dict with all key/values from `source_env_dict`, where all values have been recursively + expanded. + """ + def _simpler_expand_location(input_str): + return ctx.expand_location(input_str, deps) + resolution_dict = ctx.var + if additional_lookup_dict: + resolution_dict = resolution_dict | additional_lookup_dict + return _expand_with_manual_dict_and_location( + _simpler_expand_location, + resolution_dict, + source_env_dict + ) + +def _expand_with_toolchains_attr(ctx, env_attr_name = "env", additional_lookup_dict = None): + """ + Recursively expands all values in "env" attr dict using the `ctx` lookup data. + + All keys of `env` attribute are returned in the resultant dict with values expanded by + lookups via `ctx.var` dict (unioned with optional `additional_lookup_dict` parameter). + The attribute used can be changed (instead of `env`) via the optional `env_attr_name` paramter. + This function does not modify any of the given parameters. + + Args: + ctx: (Required) The bazel context object. This is used to access the `ctx.var` member + for use as the "resolution dict". This makes use of providers from toolchains for + environment variable expansion. This object is also used to retrieve various + necessary attributes via `ctx.attr.`. + env_attr_name: (Optional) The name of the attribute that is used as the source for all + desired expansions. All key/value pairs will appear within the returned + dictionary, with all values fully expanded by lookups in `resolution_dict`. + Default value is "env". + additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for + variable expansion. + + Returns: + A new dict with all key/values from source attribute (default "env" attribute), where all + values have been recursively expanded. + """ + return _expand_with_toolchains( + ctx, + getattr(ctx.attr, env_attr_name), + additional_lookup_dict = additional_lookup_dict + ) + +def _expand_with_toolchains_and_location_attr( + ctx, + deps_attr_name = "deps", + env_attr_name = "env", + additional_lookup_dict = None): + """ + Recursively expands all values in "env" attr dict using the `ctx` logic / lookup data. + + All keys of `env` attribute are returned in the resultant dict with values expanded by + location expansion logic via `ctx.expand_location` and by lookups via `ctx.var` dict (unioned + with optional `additional_lookup_dict` parameter). + This function does not modify any of the given parameters. + + Args: + ctx: (Required) The bazel context object. This is used to access the `ctx.var` member + for use as the "resolution dict". This makes use of providers from toolchains for + environment variable expansion. This object is also used for the + `ctx.expand_location` method to handle `$(location ...)` (and similar) expansion + logic. This object is also used to retrieve various necessary attributes via + `ctx.attr.`. Default value is "deps". + deps_attr_name: (Optional) The name of the attribute which contains the set of targets used + with `ctx.expand_location` for expanding `$(location ...)` (and similar) + expressions. + env_attr_name: (Optional) The name of the attribute that is used as the source for all + desired expansions. All key/value pairs will appear within the returned + dictionary, with all values fully expanded by lookups in `resolution_dict`. + Default value is "env". + additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for + variable expansion. + + Returns: + A new dict with all key/values from source attribute (default "env" attribute), where all + values have been recursively expanded. + """ + return _expand_with_toolchains_and_location( + ctx, + getattr(ctx.attr, deps_attr_name), + getattr(ctx.attr, env_attr_name), + additional_lookup_dict = additional_lookup_dict + ) + +expansion = struct( + expand_with_manual_dict = _expand_with_manual_dict, + expand_with_manual_dict_and_location = _expand_with_manual_dict_and_location, + expand_with_toolchains = _expand_with_toolchains, + expand_with_toolchains_attr = _expand_with_toolchains_attr, + expand_with_toolchains_and_location = _expand_with_toolchains_and_location, + expand_with_toolchains_and_location_attr = _expand_with_toolchains_and_location_attr, +) diff --git a/tests/BUILD b/tests/BUILD index 7f056d2b..f1b1ae58 100644 --- a/tests/BUILD +++ b/tests/BUILD @@ -3,6 +3,7 @@ load(":build_test_tests.bzl", "build_test_test_suite") load(":collections_tests.bzl", "collections_test_suite") load(":common_settings_tests.bzl", "common_settings_test_suite") load(":dicts_tests.bzl", "dicts_test_suite") +load(":expansion_tests.bzl", "expansion_test_suite") load(":new_sets_tests.bzl", "new_sets_test_suite") load(":partial_tests.bzl", "partial_test_suite") load(":paths_tests.bzl", "paths_test_suite") @@ -29,6 +30,8 @@ common_settings_test_suite() dicts_test_suite() +expansion_test_suite() + new_sets_test_suite() partial_test_suite() diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl new file mode 100644 index 00000000..b570b8cb --- /dev/null +++ b/tests/expansion_tests.bzl @@ -0,0 +1,697 @@ + +"""Unit tests for expansion.bzl.""" + +load("//lib:expansion.bzl", "expansion") +load("//lib:unittest.bzl", "asserts", "unittest") + +# String constants + +_TEST_DEP_TARGET_NAME = "expansion_tests__dummy" + +_MOCK_LOCATION_PATH_OF_DUMMY = "location/path/of/dummy" +_MOCK_EXECPATH_PATH_OF_DUMMY = "execpath/path/of/dummy" +_MOCK_ROOTPATH_PATH_OF_DUMMY = "rootpath/path/of/dummy" +_MOCK_RLOCATIONPATH_PATH_OF_DUMMY = "rlocationpath/path/of/dummy" + +_GENRULE_LOCATION_PATH_OF_DUMMY = "bazel-out/k8-fastbuild/bin/tests/dummy.txt" +_GENRULE_EXECPATH_PATH_OF_DUMMY = "bazel-out/k8-fastbuild/bin/tests/dummy.txt" +_GENRULE_ROOTPATH_PATH_OF_DUMMY = "tests/dummy.txt" +_GENRULE_RLOCATIONPATH_PATH_OF_DUMMY = "_main/tests/dummy.txt" + +# Test input dicts + +_ENV_DICT = { + "SIMPLE_VAL": "hello_world", + "ESCAPED_SIMPLE_VAL": "$$SIMPLE_VAL", + "LOCATION_VAL": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "EXECPATH_VAL": "$(execpath :" + _TEST_DEP_TARGET_NAME + ")", + "ROOTPATH_VAL": "$(rootpath :" + _TEST_DEP_TARGET_NAME + ")", + "RLOCATIONPATH_VAL": "$(rlocationpath :" + _TEST_DEP_TARGET_NAME + ")", + "TOOLCHAIN_ENV_VAR_RAW": "$TOOLCHAIN_ENV_VAR", + "TOOLCHAIN_ENV_VAR_PAREN": "$(TOOLCHAIN_ENV_VAR)", + "TOOLCHAIN_ENV_VAR_CURLY": "${TOOLCHAIN_ENV_VAR}", + "TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": "$TOOLCHAIN_TO_LOCATION_ENV_VAR", + "TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": "$(TOOLCHAIN_TO_LOCATION_ENV_VAR)", + "TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": "${TOOLCHAIN_TO_LOCATION_ENV_VAR}", + "INDIRECT_SIMPLE_VAL_RAW": "$SIMPLE_VAL", + "INDIRECT_SIMPLE_VAL_PAREN": "$(SIMPLE_VAL)", + "INDIRECT_SIMPLE_VAL_CURLY": "${SIMPLE_VAL}", + "INDIRECT_ESCAPED_SIMPLE_VAL_RAW": "$ESCAPED_SIMPLE_VAL", + "INDIRECT_ESCAPED_SIMPLE_VAL_PAREN": "$(ESCAPED_SIMPLE_VAL)", + "INDIRECT_ESCAPED_SIMPLE_VAL_CURLY": "${ESCAPED_SIMPLE_VAL}", + "INDIRECT_LOCATION_VAL_RAW": "$LOCATION_VAL", + "INDIRECT_LOCATION_VAL_PAREN": "$(LOCATION_VAL)", + "INDIRECT_LOCATION_VAL_CURLY": "${LOCATION_VAL}", + "INDIRECT_EXECPATH_VAL_RAW": "$EXECPATH_VAL", + "INDIRECT_EXECPATH_VAL_PAREN": "$(EXECPATH_VAL)", + "INDIRECT_EXECPATH_VAL_CURLY": "${EXECPATH_VAL}", + "INDIRECT_ROOTPATH_VAL_RAW": "$ROOTPATH_VAL", + "INDIRECT_ROOTPATH_VAL_PAREN": "$(ROOTPATH_VAL)", + "INDIRECT_ROOTPATH_VAL_CURLY": "${ROOTPATH_VAL}", + "INDIRECT_RLOCATIONPATH_VAL_RAW": "$RLOCATIONPATH_VAL", + "INDIRECT_RLOCATIONPATH_VAL_PAREN": "$(RLOCATIONPATH_VAL)", + "INDIRECT_RLOCATIONPATH_VAL_CURLY": "${RLOCATIONPATH_VAL}", + "INDIRECT_TOOLCHAIN_ENV_VAR_RAW": "$TOOLCHAIN_ENV_VAR_RAW", + "INDIRECT_TOOLCHAIN_ENV_VAR_PAREN": "$(TOOLCHAIN_ENV_VAR_RAW)", + "INDIRECT_TOOLCHAIN_ENV_VAR_CURLY": "${TOOLCHAIN_ENV_VAR_RAW}", + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": "$TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW", + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": "$(TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW)", + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": "${TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW}", + "MULTI_INDIRECT_RAW": ( + "$INDIRECT_SIMPLE_VAL_RAW-$INDIRECT_ESCAPED_SIMPLE_VAL_RAW-" + + "$INDIRECT_LOCATION_VAL_RAW-$INDIRECT_RLOCATIONPATH_VAL_RAW-" + + "$INDIRECT_TOOLCHAIN_ENV_VAR_RAW-$INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW" + ), + "MULTI_INDIRECT_PAREN": ( + "$(INDIRECT_SIMPLE_VAL_RAW)-$(INDIRECT_ESCAPED_SIMPLE_VAL_RAW)-" + + "$(INDIRECT_LOCATION_VAL_RAW)-$(INDIRECT_RLOCATIONPATH_VAL_RAW)-" + + "$(INDIRECT_TOOLCHAIN_ENV_VAR_RAW)-$(INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW)" + ), + "MULTI_INDIRECT_CURLY": ( + "${INDIRECT_SIMPLE_VAL_RAW}-${INDIRECT_ESCAPED_SIMPLE_VAL_RAW}-" + + "${INDIRECT_LOCATION_VAL_RAW}-${INDIRECT_RLOCATIONPATH_VAL_RAW}-" + + "${INDIRECT_TOOLCHAIN_ENV_VAR_RAW}-${INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW}" + ), + "UNRECOGNIZED_VAR": "$NOPE", + "UNRECOGNIZED_FUNC": "$(nope :" + _TEST_DEP_TARGET_NAME + ")", +} + +_TOOLCHAIN_DICT = { + "TOOLCHAIN_ENV_VAR": "flag_value", + "TOOLCHAIN_TO_LOCATION_ENV_VAR": "$(location :" + _TEST_DEP_TARGET_NAME + ")", +} + +# Test expected output dicts + +_EXPECTED_RESOLVED_DICT_NO_LOCATION = { + "SIMPLE_VAL": "hello_world", + "ESCAPED_SIMPLE_VAL": "$$SIMPLE_VAL", + "LOCATION_VAL": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "EXECPATH_VAL": "$(execpath :" + _TEST_DEP_TARGET_NAME + ")", + "ROOTPATH_VAL": "$(rootpath :" + _TEST_DEP_TARGET_NAME + ")", + "RLOCATIONPATH_VAL": "$(rlocationpath :" + _TEST_DEP_TARGET_NAME + ")", + "TOOLCHAIN_ENV_VAR_RAW": "flag_value", + "TOOLCHAIN_ENV_VAR_PAREN": "flag_value", + "TOOLCHAIN_ENV_VAR_CURLY": "flag_value", + "TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_SIMPLE_VAL_RAW": "hello_world", + "INDIRECT_SIMPLE_VAL_PAREN": "hello_world", + "INDIRECT_SIMPLE_VAL_CURLY": "hello_world", + "INDIRECT_ESCAPED_SIMPLE_VAL_RAW": "$$SIMPLE_VAL", + "INDIRECT_ESCAPED_SIMPLE_VAL_PAREN": "$$SIMPLE_VAL", + "INDIRECT_ESCAPED_SIMPLE_VAL_CURLY": "$$SIMPLE_VAL", + "INDIRECT_LOCATION_VAL_RAW": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_LOCATION_VAL_PAREN": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_LOCATION_VAL_CURLY": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_EXECPATH_VAL_RAW": "$(execpath :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_EXECPATH_VAL_PAREN": "$(execpath :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_EXECPATH_VAL_CURLY": "$(execpath :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_ROOTPATH_VAL_RAW": "$(rootpath :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_ROOTPATH_VAL_PAREN": "$(rootpath :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_ROOTPATH_VAL_CURLY": "$(rootpath :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_RLOCATIONPATH_VAL_RAW": "$(rlocationpath :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_RLOCATIONPATH_VAL_PAREN": "$(rlocationpath :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_RLOCATIONPATH_VAL_CURLY": "$(rlocationpath :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_TOOLCHAIN_ENV_VAR_RAW": "flag_value", + "INDIRECT_TOOLCHAIN_ENV_VAR_PAREN": "flag_value", + "INDIRECT_TOOLCHAIN_ENV_VAR_CURLY": "flag_value", + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "MULTI_INDIRECT_RAW": ( + "hello_world-$$SIMPLE_VAL-$(location :" + + _TEST_DEP_TARGET_NAME + + ")-$(rlocationpath :" + + _TEST_DEP_TARGET_NAME + + ")-flag_value-$(location :" + + _TEST_DEP_TARGET_NAME + + ")" + ), + "MULTI_INDIRECT_PAREN": ( + "hello_world-$$SIMPLE_VAL-$(location :" + + _TEST_DEP_TARGET_NAME + + ")-$(rlocationpath :" + + _TEST_DEP_TARGET_NAME + + ")-flag_value-$(location :" + + _TEST_DEP_TARGET_NAME + + ")" + ), + "MULTI_INDIRECT_CURLY": ( + "hello_world-$$SIMPLE_VAL-$(location :" + + _TEST_DEP_TARGET_NAME + + ")-$(rlocationpath :" + + _TEST_DEP_TARGET_NAME + + ")-flag_value-$(location :" + + _TEST_DEP_TARGET_NAME + + ")" + ), + "UNRECOGNIZED_VAR": "$NOPE", + "UNRECOGNIZED_FUNC": "$(nope :" + _TEST_DEP_TARGET_NAME + ")", +} + +_EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { + "LOCATION_VAL": _MOCK_LOCATION_PATH_OF_DUMMY, + "EXECPATH_VAL": _MOCK_EXECPATH_PATH_OF_DUMMY, + "ROOTPATH_VAL": _MOCK_ROOTPATH_PATH_OF_DUMMY, + "RLOCATIONPATH_VAL": _MOCK_RLOCATIONPATH_PATH_OF_DUMMY, + "TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": _MOCK_LOCATION_PATH_OF_DUMMY, + "TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": _MOCK_LOCATION_PATH_OF_DUMMY, + "TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": _MOCK_LOCATION_PATH_OF_DUMMY, + "INDIRECT_LOCATION_VAL_RAW": _MOCK_LOCATION_PATH_OF_DUMMY, + "INDIRECT_LOCATION_VAL_PAREN": _MOCK_LOCATION_PATH_OF_DUMMY, + "INDIRECT_LOCATION_VAL_CURLY": _MOCK_LOCATION_PATH_OF_DUMMY, + "INDIRECT_EXECPATH_VAL_RAW": _MOCK_EXECPATH_PATH_OF_DUMMY, + "INDIRECT_EXECPATH_VAL_PAREN": _MOCK_EXECPATH_PATH_OF_DUMMY, + "INDIRECT_EXECPATH_VAL_CURLY": _MOCK_EXECPATH_PATH_OF_DUMMY, + "INDIRECT_ROOTPATH_VAL_RAW": _MOCK_ROOTPATH_PATH_OF_DUMMY, + "INDIRECT_ROOTPATH_VAL_PAREN": _MOCK_ROOTPATH_PATH_OF_DUMMY, + "INDIRECT_ROOTPATH_VAL_CURLY": _MOCK_ROOTPATH_PATH_OF_DUMMY, + "INDIRECT_RLOCATIONPATH_VAL_RAW": _MOCK_RLOCATIONPATH_PATH_OF_DUMMY, + "INDIRECT_RLOCATIONPATH_VAL_PAREN": _MOCK_RLOCATIONPATH_PATH_OF_DUMMY, + "INDIRECT_RLOCATIONPATH_VAL_CURLY": _MOCK_RLOCATIONPATH_PATH_OF_DUMMY, + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": _MOCK_LOCATION_PATH_OF_DUMMY, + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": _MOCK_LOCATION_PATH_OF_DUMMY, + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": _MOCK_LOCATION_PATH_OF_DUMMY, + "MULTI_INDIRECT_RAW": ( + "hello_world-$$SIMPLE_VAL-" + + _MOCK_LOCATION_PATH_OF_DUMMY + + "-" + + _MOCK_RLOCATIONPATH_PATH_OF_DUMMY + + "-flag_value-" + + _MOCK_LOCATION_PATH_OF_DUMMY + ), + "MULTI_INDIRECT_PAREN": ( + "hello_world-$$SIMPLE_VAL-" + + _MOCK_LOCATION_PATH_OF_DUMMY + + "-" + + _MOCK_RLOCATIONPATH_PATH_OF_DUMMY + + "-flag_value-" + + _MOCK_LOCATION_PATH_OF_DUMMY + ), + "MULTI_INDIRECT_CURLY": ( + "hello_world-$$SIMPLE_VAL-" + + _MOCK_LOCATION_PATH_OF_DUMMY + + "-" + + _MOCK_RLOCATIONPATH_PATH_OF_DUMMY + + "-flag_value-" + + _MOCK_LOCATION_PATH_OF_DUMMY + ), +} + +_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { + "LOCATION_VAL": _GENRULE_LOCATION_PATH_OF_DUMMY, + "EXECPATH_VAL": _GENRULE_EXECPATH_PATH_OF_DUMMY, + "ROOTPATH_VAL": _GENRULE_ROOTPATH_PATH_OF_DUMMY, + "RLOCATIONPATH_VAL": _GENRULE_RLOCATIONPATH_PATH_OF_DUMMY, + "TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": _GENRULE_LOCATION_PATH_OF_DUMMY, + "TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": _GENRULE_LOCATION_PATH_OF_DUMMY, + "TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": _GENRULE_LOCATION_PATH_OF_DUMMY, + "INDIRECT_LOCATION_VAL_RAW": _GENRULE_LOCATION_PATH_OF_DUMMY, + "INDIRECT_LOCATION_VAL_PAREN": _GENRULE_LOCATION_PATH_OF_DUMMY, + "INDIRECT_LOCATION_VAL_CURLY": _GENRULE_LOCATION_PATH_OF_DUMMY, + "INDIRECT_EXECPATH_VAL_RAW": _GENRULE_EXECPATH_PATH_OF_DUMMY, + "INDIRECT_EXECPATH_VAL_PAREN": _GENRULE_EXECPATH_PATH_OF_DUMMY, + "INDIRECT_EXECPATH_VAL_CURLY": _GENRULE_EXECPATH_PATH_OF_DUMMY, + "INDIRECT_ROOTPATH_VAL_RAW": _GENRULE_ROOTPATH_PATH_OF_DUMMY, + "INDIRECT_ROOTPATH_VAL_PAREN": _GENRULE_ROOTPATH_PATH_OF_DUMMY, + "INDIRECT_ROOTPATH_VAL_CURLY": _GENRULE_ROOTPATH_PATH_OF_DUMMY, + "INDIRECT_RLOCATIONPATH_VAL_RAW": _GENRULE_RLOCATIONPATH_PATH_OF_DUMMY, + "INDIRECT_RLOCATIONPATH_VAL_PAREN": _GENRULE_RLOCATIONPATH_PATH_OF_DUMMY, + "INDIRECT_RLOCATIONPATH_VAL_CURLY": _GENRULE_RLOCATIONPATH_PATH_OF_DUMMY, + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": _GENRULE_LOCATION_PATH_OF_DUMMY, + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": _GENRULE_LOCATION_PATH_OF_DUMMY, + "INDIRECT_TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": _GENRULE_LOCATION_PATH_OF_DUMMY, + "MULTI_INDIRECT_RAW": ( + "hello_world-$$SIMPLE_VAL-" + + _GENRULE_LOCATION_PATH_OF_DUMMY + + "-" + + _GENRULE_RLOCATIONPATH_PATH_OF_DUMMY + + "-flag_value-" + + _GENRULE_LOCATION_PATH_OF_DUMMY + ), + "MULTI_INDIRECT_PAREN": ( + "hello_world-$$SIMPLE_VAL-" + + _GENRULE_LOCATION_PATH_OF_DUMMY + + "-" + + _GENRULE_RLOCATIONPATH_PATH_OF_DUMMY + + "-flag_value-" + + _GENRULE_LOCATION_PATH_OF_DUMMY + ), + "MULTI_INDIRECT_CURLY": ( + "hello_world-$$SIMPLE_VAL-" + + _GENRULE_LOCATION_PATH_OF_DUMMY + + "-" + + _GENRULE_RLOCATIONPATH_PATH_OF_DUMMY + + "-flag_value-" + + _GENRULE_LOCATION_PATH_OF_DUMMY + ), +} + +def _test_toolchain_impl(ctx): + _ignore = [ctx] # @unused + return [platform_common.TemplateVariableInfo(_TOOLCHAIN_DICT)] + +_test_toolchain = rule( + implementation = _test_toolchain_impl, +) + +def _mock_expand_location(input_str): + return input_str.replace( + "$(location :" + _TEST_DEP_TARGET_NAME + ")", + _MOCK_LOCATION_PATH_OF_DUMMY + ).replace( + "$(execpath :" + _TEST_DEP_TARGET_NAME + ")", + _MOCK_EXECPATH_PATH_OF_DUMMY + ).replace( + "$(rootpath :" + _TEST_DEP_TARGET_NAME + ")", + _MOCK_ROOTPATH_PATH_OF_DUMMY + ).replace( + "$(rlocationpath :" + _TEST_DEP_TARGET_NAME + ")", + _MOCK_RLOCATIONPATH_PATH_OF_DUMMY + ) + +def _expand_with_manual_dict_test_impl(ctx): + """Test `expansion.expand_with_manual_dict()`""" + env = unittest.begin(ctx) + + env_dict_copy = dict(_ENV_DICT) + toolchain_dict_copy = dict(_TOOLCHAIN_DICT) + + resolved_dict = expansion.expand_with_manual_dict(_TOOLCHAIN_DICT, _ENV_DICT) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_copy, _ENV_DICT) + asserts.equals(env, toolchain_dict_copy, _TOOLCHAIN_DICT) + + # Check that the output has exact same key set as original input. + asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) + + # Check all output resolved values against expected resolved values. + for env_key, _ in _ENV_DICT.items(): + expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] + resolved_val = resolved_dict[env_key] + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_with_manual_dict_test = unittest.make(_expand_with_manual_dict_test_impl) + +def _expand_with_manual_dict_and_location_test_impl(ctx): + """Test `expansion.expand_with_manual_dict_and_location()`""" + env = unittest.begin(ctx) + + env_dict_copy = dict(_ENV_DICT) + toolchain_dict_copy = dict(_TOOLCHAIN_DICT) + + resolved_dict = expansion.expand_with_manual_dict_and_location( + _mock_expand_location, + _TOOLCHAIN_DICT, + _ENV_DICT + ) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_copy, _ENV_DICT) + asserts.equals(env, toolchain_dict_copy, _TOOLCHAIN_DICT) + + # Check that the output has exact same key set as original input. + asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) + + # Check all output resolved values against expected resolved values. + for env_key, _ in _ENV_DICT.items(): + expected_val = _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION[env_key] + resolved_val = resolved_dict[env_key] + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_with_manual_dict_and_location_test = unittest.make( + _expand_with_manual_dict_and_location_test_impl, +) + +def _expand_with_toolchains_test_impl(ctx): + """Test `expansion.expand_with_toolchains()` without extra dict""" + env = unittest.begin(ctx) + + env_dict_copy = dict(_ENV_DICT) + toolchain_dict_copy = dict(env.ctx.var) + + resolved_dict = expansion.expand_with_toolchains(env.ctx, _ENV_DICT) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_copy, _ENV_DICT) + asserts.equals(env, toolchain_dict_copy, env.ctx.var) + + # Check that the output has exact same key set as original input. + asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) + + # Check all output resolved values against expected resolved values. + for env_key, _ in _ENV_DICT.items(): + expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] + resolved_val = resolved_dict[env_key] + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_with_toolchains_test = unittest.make(_expand_with_toolchains_test_impl) + +def _expand_with_toolchains_with_additional_dict_test_impl(ctx): + """Test `expansion.expand_with_toolchains()` with extra dict""" + env = unittest.begin(ctx) + + env_dict_copy = dict(_ENV_DICT) + toolchain_dict_copy = dict(env.ctx.var) + + additional_lookup_dict = { + "NOPE": "naw, it's fine now.", + } + + resolved_dict = expansion.expand_with_toolchains( + env.ctx, + _ENV_DICT, + additional_lookup_dict = additional_lookup_dict + ) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_copy, _ENV_DICT) + asserts.equals(env, toolchain_dict_copy, env.ctx.var) + + # Check that the output has exact same key set as original input. + asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) + + updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { + "UNRECOGNIZED_VAR": "naw, it's fine now.", + } + # Check all output resolved values against expected resolved values. + for env_key, _ in _ENV_DICT.items(): + expected_val = updated_expected_dict[env_key] + resolved_val = resolved_dict[env_key] + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_with_toolchains_with_additional_dict_test = unittest.make( + _expand_with_toolchains_with_additional_dict_test_impl +) + +def _expand_with_toolchains_attr_test_impl(ctx): + """Test `expansion.expand_with_toolchains_attr()` without extra dict""" + env = unittest.begin(ctx) + + env_dict_copy = dict(env.ctx.attr.env) + toolchain_dict_copy = dict(env.ctx.var) + + resolved_dict = expansion.expand_with_toolchains_attr(env.ctx) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_copy, env.ctx.attr.env) + asserts.equals(env, toolchain_dict_copy, env.ctx.var) + + # Check that the output has exact same key set as original input. + asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) + + # Check all output resolved values against expected resolved values. + for env_key, _ in env.ctx.attr.env.items(): + expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] + resolved_val = resolved_dict[env_key] + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_with_toolchains_attr_test = unittest.make( + _expand_with_toolchains_attr_test_impl, + attrs = { + "env": attr.string_dict(), + } +) + +def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): + """Test `expansion.expand_with_toolchains_attr()` with extra dict""" + env = unittest.begin(ctx) + + env_dict_copy = dict(env.ctx.attr.env) + toolchain_dict_copy = dict(env.ctx.var) + + additional_lookup_dict = { + "NOPE": "naw, it's fine now.", + } + + resolved_dict = expansion.expand_with_toolchains_attr( + env.ctx, + additional_lookup_dict = additional_lookup_dict + ) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_copy, env.ctx.attr.env) + asserts.equals(env, toolchain_dict_copy, env.ctx.var) + + # Check that the output has exact same key set as original input. + asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) + + updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { + "UNRECOGNIZED_VAR": "naw, it's fine now.", + } + # Check all output resolved values against expected resolved values. + for env_key, _ in env.ctx.attr.env.items(): + expected_val = updated_expected_dict[env_key] + resolved_val = resolved_dict[env_key] + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_with_toolchains_attr_with_additional_dict_test = unittest.make( + _expand_with_toolchains_attr_with_additional_dict_test_impl, + attrs = { + "env": attr.string_dict(), + } +) + +def _expand_with_toolchains_and_location_test_impl(ctx): + """Test `expansion.expand_with_toolchains_and_location()` without extra dict""" + env = unittest.begin(ctx) + + env_dict_copy = dict(_ENV_DICT) + toolchain_dict_copy = dict(env.ctx.var) + + resolved_dict = expansion.expand_with_toolchains_and_location( + env.ctx, + [ctx.attr.target], + _ENV_DICT + ) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_copy, _ENV_DICT) + asserts.equals(env, toolchain_dict_copy, env.ctx.var) + + # Check that the output has exact same key set as original input. + asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) + + # Check all output resolved values against expected resolved values. + for env_key, _ in _ENV_DICT.items(): + expected_val = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION[env_key] + resolved_val = resolved_dict[env_key] + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_with_toolchains_and_location_test = unittest.make( + _expand_with_toolchains_and_location_test_impl, + attrs = { + "target": attr.label(), + } +) + +def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): + """Test `expansion.expand_with_toolchains_and_location()` with extra dict""" + env = unittest.begin(ctx) + + env_dict_copy = dict(_ENV_DICT) + toolchain_dict_copy = dict(env.ctx.var) + + additional_lookup_dict = { + "NOPE": "naw, it's fine now.", + } + + resolved_dict = expansion.expand_with_toolchains_and_location( + env.ctx, + [ctx.attr.target], + _ENV_DICT, + additional_lookup_dict = additional_lookup_dict + ) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_copy, _ENV_DICT) + asserts.equals(env, toolchain_dict_copy, env.ctx.var) + + # Check that the output has exact same key set as original input. + asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) + + updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { + "UNRECOGNIZED_VAR": "naw, it's fine now.", + } + # Check all output resolved values against expected resolved values. + for env_key, _ in _ENV_DICT.items(): + expected_val = updated_expected_dict[env_key] + resolved_val = resolved_dict[env_key] + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_with_toolchains_and_location_with_additional_dict_test = unittest.make( + _expand_with_toolchains_and_location_with_additional_dict_test_impl, + attrs = { + "target": attr.label(), + } +) + +def _expand_with_toolchains_and_location_attr_test_impl(ctx): + """Test `expansion.expand_with_toolchains_and_location_attr()` without extra dict""" + env = unittest.begin(ctx) + + env_dict_copy = dict(env.ctx.attr.env) + toolchain_dict_copy = dict(env.ctx.var) + + resolved_dict = expansion.expand_with_toolchains_and_location_attr(env.ctx) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_copy, env.ctx.attr.env) + asserts.equals(env, toolchain_dict_copy, env.ctx.var) + + # Check that the output has exact same key set as original input. + asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) + + # Check all output resolved values against expected resolved values. + for env_key, _ in env.ctx.attr.env.items(): + expected_val = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION[env_key] + resolved_val = resolved_dict[env_key] + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_with_toolchains_and_location_attr_test = unittest.make( + _expand_with_toolchains_and_location_attr_test_impl, + attrs = { + "deps": attr.label_list(), + "env": attr.string_dict(), + } +) + +def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx): + """Test `expansion.expand_with_toolchains_and_location_attr()` with extra dict""" + env = unittest.begin(ctx) + + env_dict_copy = dict(env.ctx.attr.env) + toolchain_dict_copy = dict(env.ctx.var) + + additional_lookup_dict = { + "NOPE": "naw, it's fine now.", + } + + resolved_dict = expansion.expand_with_toolchains_and_location_attr( + env.ctx, + additional_lookup_dict = additional_lookup_dict + ) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_copy, env.ctx.attr.env) + asserts.equals(env, toolchain_dict_copy, env.ctx.var) + + # Check that the output has exact same key set as original input. + asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) + + updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { + "UNRECOGNIZED_VAR": "naw, it's fine now.", + } + # Check all output resolved values against expected resolved values. + for env_key, _ in env.ctx.attr.env.items(): + expected_val = updated_expected_dict[env_key] + resolved_val = resolved_dict[env_key] + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_with_toolchains_and_location_attr_with_additional_dict_test = unittest.make( + _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl, + attrs = { + "deps": attr.label_list(), + "env": attr.string_dict(), + } +) + +# buildifier: disable=unnamed-macro +def expansion_test_suite(): + """Creates the test targets and test suite for expansion.bzl tests.""" + + native.genrule( + name = _TEST_DEP_TARGET_NAME, + outs = ["dummy.txt"], + cmd = "touch $@", + ) + + _test_toolchain( + name = "expansion_tests__test_toolchain", + ) + + _expand_with_manual_dict_test( + name = "expansion_tests__expand_with_manual_dict_test", + ) + _expand_with_manual_dict_and_location_test( + name = "expansion_tests__expand_with_manual_dict_and_location_test", + ) + _expand_with_toolchains_test( + name = "expansion_tests__expand_with_toolchains_test", + toolchains = [":expansion_tests__test_toolchain"], + ) + _expand_with_toolchains_with_additional_dict_test( + name = "expansion_tests__expand_with_toolchains_with_additional_dict_test", + toolchains = [":expansion_tests__test_toolchain"], + ) + _expand_with_toolchains_attr_test( + name = "expansion_tests__expand_with_toolchains_attr_test", + env = _ENV_DICT, + toolchains = [":expansion_tests__test_toolchain"], + ) + _expand_with_toolchains_attr_with_additional_dict_test( + name = "expansion_tests__expand_with_toolchains_attr_with_additional_dict_test", + env = _ENV_DICT, + toolchains = [":expansion_tests__test_toolchain"], + ) + _expand_with_toolchains_and_location_test( + name = "expansion_tests__expand_with_toolchains_and_location_test", + target = ":" + _TEST_DEP_TARGET_NAME, + toolchains = [":expansion_tests__test_toolchain"], + ) + _expand_with_toolchains_and_location_with_additional_dict_test( + name = "expansion_tests__expand_with_toolchains_and_location_with_additional_dict_test", + target = ":" + _TEST_DEP_TARGET_NAME, + toolchains = [":expansion_tests__test_toolchain"], + ) + _expand_with_toolchains_and_location_attr_test( + name = "expansion_tests__expand_with_toolchains_and_location_attr_test", + deps = [":" + _TEST_DEP_TARGET_NAME], + env = _ENV_DICT, + toolchains = [":expansion_tests__test_toolchain"], + ) + _expand_with_toolchains_and_location_attr_with_additional_dict_test( + name = "expansion_tests__expand_with_toolchains_and_location_attr_with_additional_dict_test", + deps = [":" + _TEST_DEP_TARGET_NAME], + env = _ENV_DICT, + toolchains = [":expansion_tests__test_toolchain"], + ) + + native.test_suite( + name = "expansion_tests", + tests = [ + ":expansion_tests__expand_with_manual_dict_test", + ":expansion_tests__expand_with_manual_dict_and_location_test", + ":expansion_tests__expand_with_toolchains_test", + ":expansion_tests__expand_with_toolchains_with_additional_dict_test", + ":expansion_tests__expand_with_toolchains_attr_test", + ":expansion_tests__expand_with_toolchains_attr_with_additional_dict_test", + ":expansion_tests__expand_with_toolchains_and_location_test", + ":expansion_tests__expand_with_toolchains_and_location_with_additional_dict_test", + ":expansion_tests__expand_with_toolchains_and_location_attr_test", + ":expansion_tests__expand_with_toolchains_and_location_attr_with_additional_dict_test", + ], + ) From d7245435847b481e0330790a75849398dd604a71 Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Tue, 23 Jan 2024 00:29:52 -0800 Subject: [PATCH 02/17] * Adding `bzl_library` target for `expansion.bzl` in `lib/BUILD`. * Making `buildifier` happy. --- lib/BUILD | 5 +++++ lib/expansion.bzl | 23 +++++++++++++++-------- tests/expansion_tests.bzl | 39 +++++++++++++++++++++------------------ 3 files changed, 41 insertions(+), 26 deletions(-) diff --git a/lib/BUILD b/lib/BUILD index 23280816..7bfc045c 100644 --- a/lib/BUILD +++ b/lib/BUILD @@ -20,6 +20,11 @@ bzl_library( srcs = ["dicts.bzl"], ) +bzl_library( + name = "expansion", + srcs = ["expansion.bzl"], +) + bzl_library( name = "partial", srcs = ["partial.bzl"], diff --git a/lib/expansion.bzl b/lib/expansion.bzl index 47caae9e..674ca0ec 100644 --- a/lib/expansion.bzl +++ b/lib/expansion.bzl @@ -1,4 +1,3 @@ - """Skylib module containing functions that aid in environment variable expansion.""" def _valid_char_for_env_var_name(char): @@ -10,7 +9,7 @@ def _key_to_be_expanded(str_with_key, key, start_of_key_index): dollar_sign_count = 0 for index in range(start_of_key_index, -1, -1): if str_with_key[index] != "$": - break + break dollar_sign_count += 1 # Check that the key is correctly matched. @@ -30,6 +29,7 @@ def _expand_key_in_str(key, val, unexpanded_str): val_len = len(val) searched_index = 0 expanded_str = unexpanded_str + # Max iterations at the length of the str; will likely break out earlier. for _ in range(len(expanded_str)): used_key_index = expanded_str.find(key, searched_index) @@ -42,7 +42,7 @@ def _expand_key_in_str(key, val, unexpanded_str): expanded_str = expanded_str.replace(key, val, 1) else: expanded_str = ( - expanded_str[0:searched_index-1] + + expanded_str[0:searched_index - 1] + expanded_str[searched_index:].replace(key, val, 1) ) searched_index += val_len @@ -60,11 +60,13 @@ def _expand_all_keys_in_str_from_dict(replacement_dict, unexpanded_str): continue considered_key_formats = ("${}", "${{{}}}", "$({})") formatted_keys = [key_format.format(avail_key) for key_format in considered_key_formats] + # Skip self-references (e.g. {"VAR": "$(VAR)"}) # This may happen (and is ok) for the `env` attribute, where keys can be reused to be # expanded by the resolved dict. if corresponding_val in formatted_keys: continue + # Expand each format style of this key, if it exists. for formatted_key in formatted_keys: expanded_val = _expand_key_in_str(formatted_key, corresponding_val, expanded_val) @@ -76,6 +78,7 @@ def _expand_tc_all_keys_in_str(resolved_replacement_dict, env_replacement_dict, expanded_val = unexpanded_str prev_val = expanded_val + # Max iterations at the length of the str; will likely break out earlier. for _ in range(len(expanded_val)): # Expand values first from the `env` attribute, then by the toolchain resolved values. @@ -99,6 +102,7 @@ def _expand_tc_and_loc_all_keys_in_str( expanded_val = unexpanded_str prev_val = expanded_val + # Max iterations at the length of the str; will likely break out earlier. for _ in range(len(expanded_val)): # First let's try the safe `location` (et al) expansion logic. @@ -146,7 +150,7 @@ def _expand_with_manual_dict(resolution_dict, source_env_dict): _expand_tc_all_keys_in_str( resolution_dict, source_env_dict, - unexpanded_val + unexpanded_val, ) ) return expanded_envs @@ -182,7 +186,8 @@ def _expand_with_manual_dict_and_location(expand_location, resolution_dict, sour expand_location, resolution_dict, source_env_dict, - unexpanded_val) + unexpanded_val, + ) ) return expanded_envs @@ -245,15 +250,17 @@ def _expand_with_toolchains_and_location( A new dict with all key/values from `source_env_dict`, where all values have been recursively expanded. """ + def _simpler_expand_location(input_str): return ctx.expand_location(input_str, deps) + resolution_dict = ctx.var if additional_lookup_dict: resolution_dict = resolution_dict | additional_lookup_dict return _expand_with_manual_dict_and_location( _simpler_expand_location, resolution_dict, - source_env_dict + source_env_dict, ) def _expand_with_toolchains_attr(ctx, env_attr_name = "env", additional_lookup_dict = None): @@ -284,7 +291,7 @@ def _expand_with_toolchains_attr(ctx, env_attr_name = "env", additional_lookup_d return _expand_with_toolchains( ctx, getattr(ctx.attr, env_attr_name), - additional_lookup_dict = additional_lookup_dict + additional_lookup_dict = additional_lookup_dict, ) def _expand_with_toolchains_and_location_attr( @@ -325,7 +332,7 @@ def _expand_with_toolchains_and_location_attr( ctx, getattr(ctx.attr, deps_attr_name), getattr(ctx.attr, env_attr_name), - additional_lookup_dict = additional_lookup_dict + additional_lookup_dict = additional_lookup_dict, ) expansion = struct( diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index b570b8cb..b1424c09 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -1,4 +1,3 @@ - """Unit tests for expansion.bzl.""" load("//lib:expansion.bzl", "expansion") @@ -260,16 +259,16 @@ _test_toolchain = rule( def _mock_expand_location(input_str): return input_str.replace( "$(location :" + _TEST_DEP_TARGET_NAME + ")", - _MOCK_LOCATION_PATH_OF_DUMMY + _MOCK_LOCATION_PATH_OF_DUMMY, ).replace( "$(execpath :" + _TEST_DEP_TARGET_NAME + ")", - _MOCK_EXECPATH_PATH_OF_DUMMY + _MOCK_EXECPATH_PATH_OF_DUMMY, ).replace( "$(rootpath :" + _TEST_DEP_TARGET_NAME + ")", - _MOCK_ROOTPATH_PATH_OF_DUMMY + _MOCK_ROOTPATH_PATH_OF_DUMMY, ).replace( "$(rlocationpath :" + _TEST_DEP_TARGET_NAME + ")", - _MOCK_RLOCATIONPATH_PATH_OF_DUMMY + _MOCK_RLOCATIONPATH_PATH_OF_DUMMY, ) def _expand_with_manual_dict_test_impl(ctx): @@ -308,7 +307,7 @@ def _expand_with_manual_dict_and_location_test_impl(ctx): resolved_dict = expansion.expand_with_manual_dict_and_location( _mock_expand_location, _TOOLCHAIN_DICT, - _ENV_DICT + _ENV_DICT, ) # Check that the inputs are not mutated. @@ -370,7 +369,7 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): resolved_dict = expansion.expand_with_toolchains( env.ctx, _ENV_DICT, - additional_lookup_dict = additional_lookup_dict + additional_lookup_dict = additional_lookup_dict, ) # Check that the inputs are not mutated. @@ -383,6 +382,7 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { "UNRECOGNIZED_VAR": "naw, it's fine now.", } + # Check all output resolved values against expected resolved values. for env_key, _ in _ENV_DICT.items(): expected_val = updated_expected_dict[env_key] @@ -392,7 +392,7 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): return unittest.end(env) _expand_with_toolchains_with_additional_dict_test = unittest.make( - _expand_with_toolchains_with_additional_dict_test_impl + _expand_with_toolchains_with_additional_dict_test_impl, ) def _expand_with_toolchains_attr_test_impl(ctx): @@ -423,7 +423,7 @@ _expand_with_toolchains_attr_test = unittest.make( _expand_with_toolchains_attr_test_impl, attrs = { "env": attr.string_dict(), - } + }, ) def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): @@ -439,7 +439,7 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): resolved_dict = expansion.expand_with_toolchains_attr( env.ctx, - additional_lookup_dict = additional_lookup_dict + additional_lookup_dict = additional_lookup_dict, ) # Check that the inputs are not mutated. @@ -452,6 +452,7 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { "UNRECOGNIZED_VAR": "naw, it's fine now.", } + # Check all output resolved values against expected resolved values. for env_key, _ in env.ctx.attr.env.items(): expected_val = updated_expected_dict[env_key] @@ -464,7 +465,7 @@ _expand_with_toolchains_attr_with_additional_dict_test = unittest.make( _expand_with_toolchains_attr_with_additional_dict_test_impl, attrs = { "env": attr.string_dict(), - } + }, ) def _expand_with_toolchains_and_location_test_impl(ctx): @@ -477,7 +478,7 @@ def _expand_with_toolchains_and_location_test_impl(ctx): resolved_dict = expansion.expand_with_toolchains_and_location( env.ctx, [ctx.attr.target], - _ENV_DICT + _ENV_DICT, ) # Check that the inputs are not mutated. @@ -499,7 +500,7 @@ _expand_with_toolchains_and_location_test = unittest.make( _expand_with_toolchains_and_location_test_impl, attrs = { "target": attr.label(), - } + }, ) def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): @@ -517,7 +518,7 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): env.ctx, [ctx.attr.target], _ENV_DICT, - additional_lookup_dict = additional_lookup_dict + additional_lookup_dict = additional_lookup_dict, ) # Check that the inputs are not mutated. @@ -530,6 +531,7 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { "UNRECOGNIZED_VAR": "naw, it's fine now.", } + # Check all output resolved values against expected resolved values. for env_key, _ in _ENV_DICT.items(): expected_val = updated_expected_dict[env_key] @@ -542,7 +544,7 @@ _expand_with_toolchains_and_location_with_additional_dict_test = unittest.make( _expand_with_toolchains_and_location_with_additional_dict_test_impl, attrs = { "target": attr.label(), - } + }, ) def _expand_with_toolchains_and_location_attr_test_impl(ctx): @@ -574,7 +576,7 @@ _expand_with_toolchains_and_location_attr_test = unittest.make( attrs = { "deps": attr.label_list(), "env": attr.string_dict(), - } + }, ) def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx): @@ -590,7 +592,7 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx resolved_dict = expansion.expand_with_toolchains_and_location_attr( env.ctx, - additional_lookup_dict = additional_lookup_dict + additional_lookup_dict = additional_lookup_dict, ) # Check that the inputs are not mutated. @@ -603,6 +605,7 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { "UNRECOGNIZED_VAR": "naw, it's fine now.", } + # Check all output resolved values against expected resolved values. for env_key, _ in env.ctx.attr.env.items(): expected_val = updated_expected_dict[env_key] @@ -616,7 +619,7 @@ _expand_with_toolchains_and_location_attr_with_additional_dict_test = unittest.m attrs = { "deps": attr.label_list(), "env": attr.string_dict(), - } + }, ) # buildifier: disable=unnamed-macro From 587f254bbb42591de832c90e6a4137155ae3c466 Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Tue, 23 Jan 2024 00:41:28 -0800 Subject: [PATCH 03/17] * Replacing platform dependent strings (from `$(location ...)` and similar) for consistent assertions. --- tests/expansion_tests.bzl | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index b1424c09..e62ac77d 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -17,6 +17,10 @@ _GENRULE_EXECPATH_PATH_OF_DUMMY = "bazel-out/k8-fastbuild/bin/tests/dummy.txt" _GENRULE_ROOTPATH_PATH_OF_DUMMY = "tests/dummy.txt" _GENRULE_RLOCATIONPATH_PATH_OF_DUMMY = "_main/tests/dummy.txt" +_LINUX_FASTBUILD_SUBPATH = "k8-fastbuild" +_MAC_FASTBUILD_SUBPATH = "darwin_x86_64-fastbuild" +_WIN_FASTBUILD_SUBPATH = "x64_windows-fastbuild" + # Test input dicts _ENV_DICT = { @@ -271,6 +275,16 @@ def _mock_expand_location(input_str): _MOCK_RLOCATIONPATH_PATH_OF_DUMMY, ) +def fix_platform_dependent_path_for_assertions(platform_dependent_val): + return platform_dependent_val.replace( + _MAC_FASTBUILD_SUBPATH, + _LINUX_FASTBUILD_SUBPATH, + ).replace( + _WIN_FASTBUILD_SUBPATH, + _LINUX_FASTBUILD_SUBPATH, + ) + + def _expand_with_manual_dict_test_impl(ctx): """Test `expansion.expand_with_manual_dict()`""" env = unittest.begin(ctx) @@ -291,6 +305,7 @@ def _expand_with_manual_dict_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] + resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -321,6 +336,7 @@ def _expand_with_manual_dict_and_location_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION[env_key] resolved_val = resolved_dict[env_key] + resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -349,6 +365,7 @@ def _expand_with_toolchains_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] + resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -387,6 +404,7 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] + resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -415,6 +433,7 @@ def _expand_with_toolchains_attr_test_impl(ctx): for env_key, _ in env.ctx.attr.env.items(): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] + resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -457,6 +476,7 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): for env_key, _ in env.ctx.attr.env.items(): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] + resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -492,6 +512,7 @@ def _expand_with_toolchains_and_location_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION[env_key] resolved_val = resolved_dict[env_key] + resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -536,6 +557,7 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] + resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -567,6 +589,7 @@ def _expand_with_toolchains_and_location_attr_test_impl(ctx): for env_key, _ in env.ctx.attr.env.items(): expected_val = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION[env_key] resolved_val = resolved_dict[env_key] + resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -610,6 +633,7 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx for env_key, _ in env.ctx.attr.env.items(): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] + resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) From 416305efb90713f89f4b81b1656058f78874d661 Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Tue, 23 Jan 2024 00:44:25 -0800 Subject: [PATCH 04/17] * Making `buildifier` happy again (removing extra empty line). --- tests/expansion_tests.bzl | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index e62ac77d..e7e91ded 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -284,7 +284,6 @@ def fix_platform_dependent_path_for_assertions(platform_dependent_val): _LINUX_FASTBUILD_SUBPATH, ) - def _expand_with_manual_dict_test_impl(ctx): """Test `expansion.expand_with_manual_dict()`""" env = unittest.begin(ctx) From 789a32f798723f72d9b955d04aa31cfbb48d9007 Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Tue, 23 Jan 2024 00:51:53 -0800 Subject: [PATCH 05/17] * Prefixing fix method in test with `_`. --- tests/expansion_tests.bzl | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index e7e91ded..a0fe4a08 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -275,7 +275,7 @@ def _mock_expand_location(input_str): _MOCK_RLOCATIONPATH_PATH_OF_DUMMY, ) -def fix_platform_dependent_path_for_assertions(platform_dependent_val): +def _fix_platform_dependent_path_for_assertions(platform_dependent_val): return platform_dependent_val.replace( _MAC_FASTBUILD_SUBPATH, _LINUX_FASTBUILD_SUBPATH, @@ -304,7 +304,7 @@ def _expand_with_manual_dict_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] - resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -335,7 +335,7 @@ def _expand_with_manual_dict_and_location_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION[env_key] resolved_val = resolved_dict[env_key] - resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -364,7 +364,7 @@ def _expand_with_toolchains_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] - resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -403,7 +403,7 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] - resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -432,7 +432,7 @@ def _expand_with_toolchains_attr_test_impl(ctx): for env_key, _ in env.ctx.attr.env.items(): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] - resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -475,7 +475,7 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): for env_key, _ in env.ctx.attr.env.items(): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] - resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -511,7 +511,7 @@ def _expand_with_toolchains_and_location_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION[env_key] resolved_val = resolved_dict[env_key] - resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -556,7 +556,7 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): for env_key, _ in _ENV_DICT.items(): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] - resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -588,7 +588,7 @@ def _expand_with_toolchains_and_location_attr_test_impl(ctx): for env_key, _ in env.ctx.attr.env.items(): expected_val = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION[env_key] resolved_val = resolved_dict[env_key] - resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -632,7 +632,7 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx for env_key, _ in env.ctx.attr.env.items(): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] - resolved_val = fix_platform_dependent_path_for_assertions(resolved_val) + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) asserts.equals(env, expected_val, resolved_val) return unittest.end(env) From c38d3e86a9ec492087e1fcfef1944cef78a108e9 Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Tue, 23 Jan 2024 19:25:48 -0800 Subject: [PATCH 06/17] * Reducing some duplication (merging `_expand_tc_all_keys_in_str()` and `_expand_tc_and_loc_all_keys_in_str()` into `_expand_all_keys_in_str()` with None-able arg). * Updating handling / init of optional `additional_lookup_dict` arg. * Cleaning up some doc comments. * Updating tests. * Adding demonstration where `additional_lookup_dict` overrides value from toolchains. * Adding demonstration where recursive expansion involves env dict and toolchain dict back and forth. * Adding a few more section separator comments. --- lib/expansion.bzl | 56 ++++++++++++++------------------------- tests/expansion_tests.bzl | 36 +++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 36 deletions(-) diff --git a/lib/expansion.bzl b/lib/expansion.bzl index 674ca0ec..71d9a63d 100644 --- a/lib/expansion.bzl +++ b/lib/expansion.bzl @@ -72,27 +72,7 @@ def _expand_all_keys_in_str_from_dict(replacement_dict, unexpanded_str): expanded_val = _expand_key_in_str(formatted_key, corresponding_val, expanded_val) return expanded_val -def _expand_tc_all_keys_in_str(resolved_replacement_dict, env_replacement_dict, unexpanded_str): - if unexpanded_str.find("$") < 0: - return unexpanded_str - - expanded_val = unexpanded_str - prev_val = expanded_val - - # Max iterations at the length of the str; will likely break out earlier. - for _ in range(len(expanded_val)): - # Expand values first from the `env` attribute, then by the toolchain resolved values. - expanded_val = _expand_all_keys_in_str_from_dict(env_replacement_dict, expanded_val) - expanded_val = _expand_all_keys_in_str_from_dict(resolved_replacement_dict, expanded_val) - - # Break out early if nothing changed in this iteration. - if prev_val == expanded_val: - break - prev_val = expanded_val - - return expanded_val - -def _expand_tc_and_loc_all_keys_in_str( +def _expand_all_keys_in_str( expand_location, resolved_replacement_dict, env_replacement_dict, @@ -107,7 +87,8 @@ def _expand_tc_and_loc_all_keys_in_str( for _ in range(len(expanded_val)): # First let's try the safe `location` (et al) expansion logic. # `$VAR`, `$(VAR)`, and `${VAR}` will be left untouched. - expanded_val = expand_location(expanded_val) + if expand_location: + expanded_val = expand_location(expanded_val) # Break early if nothing left to expand. if expanded_val.find("$") < 0: @@ -147,7 +128,8 @@ def _expand_with_manual_dict(resolution_dict, source_env_dict): expanded_envs = {} for env_key, unexpanded_val in source_env_dict.items(): expanded_envs[env_key] = ( - _expand_tc_all_keys_in_str( + _expand_all_keys_in_str( + None, # No `expand_location` available resolution_dict, source_env_dict, unexpanded_val, @@ -182,7 +164,7 @@ def _expand_with_manual_dict_and_location(expand_location, resolution_dict, sour expanded_envs = {} for env_key, unexpanded_val in source_env_dict.items(): expanded_envs[env_key] = ( - _expand_tc_and_loc_all_keys_in_str( + _expand_all_keys_in_str( expand_location, resolution_dict, source_env_dict, @@ -197,6 +179,7 @@ def _expand_with_toolchains(ctx, source_env_dict, additional_lookup_dict = None) All keys of `source_env_dict` are returned in the resultant dict with values expanded by lookups via `ctx.var` dict (unioned with optional `additional_lookup_dict` parameter). + Expansion occurs recursively through all given dicts. This function does not modify any of the given parameters. Args: @@ -205,7 +188,7 @@ def _expand_with_toolchains(ctx, source_env_dict, additional_lookup_dict = None) environment variable expansion. source_env_dict: (Required) The source for all desired expansions. All key/value pairs will appear within the returned dictionary, with all values fully - expanded by lookups in `resolution_dict`. + expanded by lookups in `ctx.var` and optional `additional_lookup_dict`. additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for variable expansion. @@ -213,10 +196,11 @@ def _expand_with_toolchains(ctx, source_env_dict, additional_lookup_dict = None) A new dict with all key/values from `source_env_dict`, where all values have been recursively expanded. """ - resolution_dict = ctx.var - if additional_lookup_dict: - resolution_dict = resolution_dict | additional_lookup_dict - return _expand_with_manual_dict(resolution_dict, source_env_dict) + additional_lookup_dict = additional_lookup_dict or {} + return _expand_with_manual_dict( + ctx.var | additional_lookup_dict, + source_env_dict, + ) def _expand_with_toolchains_and_location( ctx, @@ -242,7 +226,7 @@ def _expand_with_toolchains_and_location( source_env_dict: (Required) The source for all desired expansions. All key/value pairs will appear within the returned dictionary, with all values fully expanded by the logic expansion logic of `expand_location` and by - lookup in `resolution_dict`. + lookups in `ctx.var` and optional `additional_lookup_dict`. additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for variable expansion. @@ -254,12 +238,10 @@ def _expand_with_toolchains_and_location( def _simpler_expand_location(input_str): return ctx.expand_location(input_str, deps) - resolution_dict = ctx.var - if additional_lookup_dict: - resolution_dict = resolution_dict | additional_lookup_dict + additional_lookup_dict = additional_lookup_dict or {} return _expand_with_manual_dict_and_location( _simpler_expand_location, - resolution_dict, + ctx.var | additional_lookup_dict, source_env_dict, ) @@ -279,7 +261,8 @@ def _expand_with_toolchains_attr(ctx, env_attr_name = "env", additional_lookup_d necessary attributes via `ctx.attr.`. env_attr_name: (Optional) The name of the attribute that is used as the source for all desired expansions. All key/value pairs will appear within the returned - dictionary, with all values fully expanded by lookups in `resolution_dict`. + dictionary, with all values fully expanded by lookups in `ctx.var` and + optional `additional_lookup_dict`. Default value is "env". additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for variable expansion. @@ -319,7 +302,8 @@ def _expand_with_toolchains_and_location_attr( expressions. env_attr_name: (Optional) The name of the attribute that is used as the source for all desired expansions. All key/value pairs will appear within the returned - dictionary, with all values fully expanded by lookups in `resolution_dict`. + dictionary, with all values fully expanded by lookups in `ctx.var` and + optional `additional_lookup_dict`. Default value is "env". additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for variable expansion. diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index a0fe4a08..56afb250 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -33,9 +33,15 @@ _ENV_DICT = { "TOOLCHAIN_ENV_VAR_RAW": "$TOOLCHAIN_ENV_VAR", "TOOLCHAIN_ENV_VAR_PAREN": "$(TOOLCHAIN_ENV_VAR)", "TOOLCHAIN_ENV_VAR_CURLY": "${TOOLCHAIN_ENV_VAR}", + "TOOLCHAIN_ENV_VAR2_RAW": "$TOOLCHAIN_ENV_VAR2", + "TOOLCHAIN_ENV_VAR2_PAREN": "$(TOOLCHAIN_ENV_VAR2)", + "TOOLCHAIN_ENV_VAR2_CURLY": "${TOOLCHAIN_ENV_VAR2}", "TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": "$TOOLCHAIN_TO_LOCATION_ENV_VAR", "TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": "$(TOOLCHAIN_TO_LOCATION_ENV_VAR)", "TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": "${TOOLCHAIN_TO_LOCATION_ENV_VAR}", + "TOOLCHAIN_TO_ENV_DICT_ENV_VAR_RAW": "$BACK_TO_ENV_DICT_VAR_RAW", + "TOOLCHAIN_TO_ENV_DICT_ENV_VAR_PAREN": "$(BACK_TO_ENV_DICT_VAR_PAREN)", + "TOOLCHAIN_TO_ENV_DICT_ENV_VAR_CURLY": "${BACK_TO_ENV_DICT_VAR_CURLY}", "INDIRECT_SIMPLE_VAL_RAW": "$SIMPLE_VAL", "INDIRECT_SIMPLE_VAL_PAREN": "$(SIMPLE_VAL)", "INDIRECT_SIMPLE_VAL_CURLY": "${SIMPLE_VAL}", @@ -81,7 +87,11 @@ _ENV_DICT = { _TOOLCHAIN_DICT = { "TOOLCHAIN_ENV_VAR": "flag_value", + "TOOLCHAIN_ENV_VAR2": "flag_value_2", "TOOLCHAIN_TO_LOCATION_ENV_VAR": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "BACK_TO_ENV_DICT_VAR_RAW": "$SIMPLE_VAL", + "BACK_TO_ENV_DICT_VAR_PAREN": "$(SIMPLE_VAL)", + "BACK_TO_ENV_DICT_VAR_CURLY": "${SIMPLE_VAL}", } # Test expected output dicts @@ -96,9 +106,15 @@ _EXPECTED_RESOLVED_DICT_NO_LOCATION = { "TOOLCHAIN_ENV_VAR_RAW": "flag_value", "TOOLCHAIN_ENV_VAR_PAREN": "flag_value", "TOOLCHAIN_ENV_VAR_CURLY": "flag_value", + "TOOLCHAIN_ENV_VAR2_RAW": "flag_value_2", + "TOOLCHAIN_ENV_VAR2_PAREN": "flag_value_2", + "TOOLCHAIN_ENV_VAR2_CURLY": "flag_value_2", "TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": "$(location :" + _TEST_DEP_TARGET_NAME + ")", "TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": "$(location :" + _TEST_DEP_TARGET_NAME + ")", "TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "TOOLCHAIN_TO_ENV_DICT_ENV_VAR_RAW": "hello_world", + "TOOLCHAIN_TO_ENV_DICT_ENV_VAR_PAREN": "hello_world", + "TOOLCHAIN_TO_ENV_DICT_ENV_VAR_CURLY": "hello_world", "INDIRECT_SIMPLE_VAL_RAW": "hello_world", "INDIRECT_SIMPLE_VAL_PAREN": "hello_world", "INDIRECT_SIMPLE_VAL_CURLY": "hello_world", @@ -252,6 +268,8 @@ _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCAT ), } +# Test helper functions and rules + def _test_toolchain_impl(ctx): _ignore = [ctx] # @unused return [platform_common.TemplateVariableInfo(_TOOLCHAIN_DICT)] @@ -284,6 +302,8 @@ def _fix_platform_dependent_path_for_assertions(platform_dependent_val): _LINUX_FASTBUILD_SUBPATH, ) +# Test cases + def _expand_with_manual_dict_test_impl(ctx): """Test `expansion.expand_with_manual_dict()`""" env = unittest.begin(ctx) @@ -379,6 +399,7 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): toolchain_dict_copy = dict(env.ctx.var) additional_lookup_dict = { + "TOOLCHAIN_ENV_VAR2": "expanded from additional dict instead", "NOPE": "naw, it's fine now.", } @@ -396,6 +417,9 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { + "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", + "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", + "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", } @@ -452,6 +476,7 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): toolchain_dict_copy = dict(env.ctx.var) additional_lookup_dict = { + "TOOLCHAIN_ENV_VAR2": "expanded from additional dict instead", "NOPE": "naw, it's fine now.", } @@ -468,6 +493,9 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { + "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", + "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", + "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", } @@ -531,6 +559,7 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): toolchain_dict_copy = dict(env.ctx.var) additional_lookup_dict = { + "TOOLCHAIN_ENV_VAR2": "expanded from additional dict instead", "NOPE": "naw, it's fine now.", } @@ -549,6 +578,9 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { + "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", + "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", + "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", } @@ -609,6 +641,7 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx toolchain_dict_copy = dict(env.ctx.var) additional_lookup_dict = { + "TOOLCHAIN_ENV_VAR2": "expanded from additional dict instead", "NOPE": "naw, it's fine now.", } @@ -625,6 +658,9 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { + "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", + "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", + "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", } From a823ff7a6f3f2838d6536201672733c6395a2c0d Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sun, 28 Jan 2024 01:06:46 -0800 Subject: [PATCH 07/17] * Adding validation logic. * Adding `_validate_all_keys_expanded()` to validate there are no expandable keys in a given string. * Will either call `fail()` or return list of failures (depending on `fail_instead_of_return` parameter). * Failure messages will contain the parsed unexpanded variable and the whole string which contains it. * Adding optional `validate_expansion` parameter to allow automatic validation after the expansion process. * Adding exposed `validate_expansions` to allow the client to call the functionality directly. * Adding new tests for validation logic. * Parameterized over many different configurations. * All tests pass in <= 0.1s each. * Pulling out `_CONSIDERED_KEY_FORMATS` into file-scoped constant. * Pulling out "odd count dollar sign" logic into `_odd_count_dollar_sign_repeat()`. * Updating some variable names and adding more comments. --- lib/expansion.bzl | 288 ++++++++++++++++++++++++++++++++++---- tests/expansion_tests.bzl | 251 +++++++++++++++++++++++++++++++++ 2 files changed, 508 insertions(+), 31 deletions(-) diff --git a/lib/expansion.bzl b/lib/expansion.bzl index 71d9a63d..912e5ba3 100644 --- a/lib/expansion.bzl +++ b/lib/expansion.bzl @@ -1,28 +1,170 @@ """Skylib module containing functions that aid in environment variable expansion.""" +_CONSIDERED_KEY_FORMATS = ("${}", "${{{}}}", "$({})") + def _valid_char_for_env_var_name(char): return char.isalnum() or char == "_" -def _key_to_be_expanded(str_with_key, key, start_of_key_index): - # Check that the string at index is prefixed with an odd number of `$`. - # Odd number means that the last `$` is not escaped. +def _find_env_var_name_index_index( + string, + str_len, + search_start, + special_ending_char = None): + for offset in range(str_len - search_start): + index = search_start + offset + char = string[index] + if special_ending_char: + if char == special_ending_char: + return index + elif not _valid_char_for_env_var_name(char): + return index - 1 + return str_len - 1 + +def _odd_count_dollar_sign_repeat(containing_str, end_of_dollar_signs_index): dollar_sign_count = 0 - for index in range(start_of_key_index, -1, -1): - if str_with_key[index] != "$": + for index in range(end_of_dollar_signs_index, -1, -1): + if containing_str[index] != "$": break dollar_sign_count += 1 + return (dollar_sign_count % 2) == 1 + +def _key_to_be_expanded(str_with_key, key, start_of_key_index): + # Check that the string at index is prefixed with an odd number of `$`. + # Odd number means that the last `$` is not escaped. + odd_count = _odd_count_dollar_sign_repeat(str_with_key, start_of_key_index) + + if not odd_count: + return False # Check that the key is correctly matched. # Specifically, check the key isn't matching to another key (substring). key_mismatch = False if key[-1] not in (")", "}"): - end_of_key_index = start_of_key_index + len(key) + index_after_key = start_of_key_index + len(key) key_mismatch = ( - (end_of_key_index < len(str_with_key)) and - _valid_char_for_env_var_name(str_with_key[end_of_key_index]) + (index_after_key < len(str_with_key)) and + _valid_char_for_env_var_name(str_with_key[index_after_key]) ) - return (not key_mismatch) and (dollar_sign_count % 2) == 1 + return not key_mismatch + +def _fail_validation(fail_instead_of_return, found_errors_list, failure_message): + if fail_instead_of_return: + fail(failure_message) + else: + found_errors_list.append(failure_message) + +def _validate_unterminated_expression( + expanded_val, + fail_instead_of_return, + found_errors, + dollar_sign_index, + next_char_after_dollar_sign): + if next_char_after_dollar_sign == "(": + if expanded_val.find(")", dollar_sign_index + 1) < 0: + unterminated_expr = expanded_val[dollar_sign_index:] + _fail_validation( + fail_instead_of_return, + found_errors, + "Unterminated '$(...)' expression ('{}' in '{}').".format(unterminated_expr, expanded_val), + ) + return False + elif next_char_after_dollar_sign == "{": + if expanded_val.find("}", dollar_sign_index + 1) < 0: + unterminated_expr = expanded_val[dollar_sign_index:] + _fail_validation( + fail_instead_of_return, + found_errors, + "Unterminated '${{...}}' expression ('{}' in '{}').".format(unterminated_expr, expanded_val), + ) + return False + return True + +def _validate_unexpanded_expression( + expanded_val, + fail_instead_of_return, + str_len, + found_errors, + dollar_sign_index, + next_char_after_dollar_sign): + # Find special ending char, if wrapped expression. + special_ending_char = None + if next_char_after_dollar_sign == "(": + special_ending_char = ")" + elif next_char_after_dollar_sign == "{": + special_ending_char = "}" + + # Check for unexpanded expressions. + name_end_index = _find_env_var_name_index_index( + expanded_val, + str_len, + dollar_sign_index + 1, + special_ending_char = special_ending_char, + ) + _fail_validation( + fail_instead_of_return, + found_errors, + "Unexpanded expression ('{}' in '{}').".format( + expanded_val[dollar_sign_index:name_end_index + 1], + expanded_val, + ), + ) + +def _validate_all_keys_expanded(expanded_val, fail_instead_of_return): + str_len = len(expanded_val) + str_iter = 0 + found_errors = [] + + # Max iterations at the length of the str; will likely break out earlier. + for _ in range(str_len): + if str_iter >= str_len: + break + next_dollar_sign_index = expanded_val.find("$", str_iter) + if next_dollar_sign_index < 0: + break + str_iter = next_dollar_sign_index + 1 + + # Check for unterminated (non-escaped) ending dollar sign(s). + if next_dollar_sign_index == str_len - 1: + if _odd_count_dollar_sign_repeat(expanded_val, next_dollar_sign_index): + _fail_validation( + fail_instead_of_return, + found_errors, + "Unterminated '$' expression in '{}'.".format(expanded_val), + ) + + # No error if escaped. Still at end of string, break out. + break + + next_char_after_dollar_sign = expanded_val[next_dollar_sign_index + 1] + + # Check for continued dollar signs string (no need to handle yet). + if next_char_after_dollar_sign == "$": + continue + + # Check for escaped dollar signs (which are ok). + if not _odd_count_dollar_sign_repeat(expanded_val, next_dollar_sign_index): + continue + + # Check for unterminated expressions. + if _validate_unterminated_expression( + expanded_val, + fail_instead_of_return, + found_errors, + next_dollar_sign_index, + next_char_after_dollar_sign, + ): + # If not unterminated, it's unexpanded. + _validate_unexpanded_expression( + expanded_val, + fail_instead_of_return, + str_len, + found_errors, + next_dollar_sign_index, + next_char_after_dollar_sign, + ) + + return found_errors def _expand_key_in_str(key, val, unexpanded_str): key_len = len(key) @@ -58,8 +200,7 @@ def _expand_all_keys_in_str_from_dict(replacement_dict, unexpanded_str): for avail_key, corresponding_val in replacement_dict.items(): if expanded_val.find(avail_key) < 0: continue - considered_key_formats = ("${}", "${{{}}}", "$({})") - formatted_keys = [key_format.format(avail_key) for key_format in considered_key_formats] + formatted_keys = [key_format.format(avail_key) for key_format in _CONSIDERED_KEY_FORMATS] # Skip self-references (e.g. {"VAR": "$(VAR)"}) # This may happen (and is ok) for the `env` attribute, where keys can be reused to be @@ -105,7 +246,7 @@ def _expand_all_keys_in_str( return expanded_val -def _expand_with_manual_dict(resolution_dict, source_env_dict): +def _expand_with_manual_dict(resolution_dict, source_env_dict, validate_expansion = False): """ Recursively expands all values in `source_env_dict` using the given lookup data. @@ -120,6 +261,12 @@ def _expand_with_manual_dict(resolution_dict, source_env_dict): source_env_dict: (Required) The source for all desired expansions. All key/value pairs will appear within the returned dictionary, with all values fully expanded by lookups in `resolution_dict`. + validate_expansion: (Optional) If set to True, all expanded strings will be validated to + ensure that no unexpanded (but seemingly expandable) values remain. If + any unexpanded values are found, `fail()` will be called. The + validation logic is the same as + `expansion.validate_expansions_in_dict()`. + Default value is False. Returns: A new dict with all key/values from `source_env_dict`, where all values have been recursively @@ -127,17 +274,22 @@ def _expand_with_manual_dict(resolution_dict, source_env_dict): """ expanded_envs = {} for env_key, unexpanded_val in source_env_dict.items(): - expanded_envs[env_key] = ( - _expand_all_keys_in_str( - None, # No `expand_location` available - resolution_dict, - source_env_dict, - unexpanded_val, - ) + expanded_val = _expand_all_keys_in_str( + None, # No `expand_location` available + resolution_dict, + source_env_dict, + unexpanded_val, ) + if validate_expansion: + _validate_all_keys_expanded(expanded_val, fail_instead_of_return = True) + expanded_envs[env_key] = expanded_val return expanded_envs -def _expand_with_manual_dict_and_location(expand_location, resolution_dict, source_env_dict): +def _expand_with_manual_dict_and_location( + expand_location, + resolution_dict, + source_env_dict, + validate_expansion = False): """ Recursively expands all values in `source_env_dict` using the given logic / lookup data. @@ -156,6 +308,12 @@ def _expand_with_manual_dict_and_location(expand_location, resolution_dict, sour will appear within the returned dictionary, with all values fully expanded by the logic expansion logic of `expand_location` and by lookup in `resolution_dict`. + validate_expansion: (Optional) If set to True, all expanded strings will be validated to + ensure that no unexpanded (but seemingly expandable) values remain. If + any unexpanded values are found, `fail()` will be called. The + validation logic is the same as + `expansion.validate_expansions_in_dict()`. + Default value is False. Returns: A new dict with all key/values from `source_env_dict`, where all values have been recursively @@ -163,17 +321,22 @@ def _expand_with_manual_dict_and_location(expand_location, resolution_dict, sour """ expanded_envs = {} for env_key, unexpanded_val in source_env_dict.items(): - expanded_envs[env_key] = ( - _expand_all_keys_in_str( - expand_location, - resolution_dict, - source_env_dict, - unexpanded_val, - ) + expanded_val = _expand_all_keys_in_str( + expand_location, + resolution_dict, + source_env_dict, + unexpanded_val, ) + if validate_expansion: + _validate_all_keys_expanded(expanded_val, fail_instead_of_return = True) + expanded_envs[env_key] = expanded_val return expanded_envs -def _expand_with_toolchains(ctx, source_env_dict, additional_lookup_dict = None): +def _expand_with_toolchains( + ctx, + source_env_dict, + additional_lookup_dict = None, + validate_expansion = False): """ Recursively expands all values in `source_env_dict` using the given lookup data. @@ -191,6 +354,12 @@ def _expand_with_toolchains(ctx, source_env_dict, additional_lookup_dict = None) expanded by lookups in `ctx.var` and optional `additional_lookup_dict`. additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for variable expansion. + validate_expansion: (Optional) If set to True, all expanded strings will be validated + to ensure that no unexpanded (but seemingly expandable) values + remain. If any unexpanded values are found, `fail()` will be + called. The validation logic is the same as + `expansion.validate_expansions_in_dict()`. + Default value is False. Returns: A new dict with all key/values from `source_env_dict`, where all values have been recursively @@ -200,13 +369,15 @@ def _expand_with_toolchains(ctx, source_env_dict, additional_lookup_dict = None) return _expand_with_manual_dict( ctx.var | additional_lookup_dict, source_env_dict, + validate_expansion = validate_expansion, ) def _expand_with_toolchains_and_location( ctx, deps, source_env_dict, - additional_lookup_dict = None): + additional_lookup_dict = None, + validate_expansion = False): """ Recursively expands all values in `source_env_dict` using the `ctx` logic / lookup data. @@ -229,6 +400,12 @@ def _expand_with_toolchains_and_location( lookups in `ctx.var` and optional `additional_lookup_dict`. additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for variable expansion. + validate_expansion: (Optional) If set to True, all expanded strings will be validated + to ensure that no unexpanded (but seemingly expandable) values + remain. If any unexpanded values are found, `fail()` will be + called. The validation logic is the same as + `expansion.validate_expansions_in_dict()`. + Default value is False. Returns: A new dict with all key/values from `source_env_dict`, where all values have been recursively @@ -243,9 +420,14 @@ def _expand_with_toolchains_and_location( _simpler_expand_location, ctx.var | additional_lookup_dict, source_env_dict, + validate_expansion = validate_expansion, ) -def _expand_with_toolchains_attr(ctx, env_attr_name = "env", additional_lookup_dict = None): +def _expand_with_toolchains_attr( + ctx, + env_attr_name = "env", + additional_lookup_dict = None, + validate_expansion = False): """ Recursively expands all values in "env" attr dict using the `ctx` lookup data. @@ -266,6 +448,12 @@ def _expand_with_toolchains_attr(ctx, env_attr_name = "env", additional_lookup_d Default value is "env". additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for variable expansion. + validate_expansion: (Optional) If set to True, all expanded strings will be validated + to ensure that no unexpanded (but seemingly expandable) values + remain. If any unexpanded values are found, `fail()` will be + called. The validation logic is the same as + `expansion.validate_expansions_in_dict()`. + Default value is False. Returns: A new dict with all key/values from source attribute (default "env" attribute), where all @@ -275,13 +463,15 @@ def _expand_with_toolchains_attr(ctx, env_attr_name = "env", additional_lookup_d ctx, getattr(ctx.attr, env_attr_name), additional_lookup_dict = additional_lookup_dict, + validate_expansion = validate_expansion, ) def _expand_with_toolchains_and_location_attr( ctx, deps_attr_name = "deps", env_attr_name = "env", - additional_lookup_dict = None): + additional_lookup_dict = None, + validate_expansion = False): """ Recursively expands all values in "env" attr dict using the `ctx` logic / lookup data. @@ -307,6 +497,12 @@ def _expand_with_toolchains_and_location_attr( Default value is "env". additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for variable expansion. + validate_expansion: (Optional) If set to True, all expanded strings will be validated + to ensure that no unexpanded (but seemingly expandable) values + remain. If any unexpanded values are found, `fail()` will be + called. The validation logic is the same as + `expansion.validate_expansions_in_dict()`. + Default value is False. Returns: A new dict with all key/values from source attribute (default "env" attribute), where all @@ -317,8 +513,37 @@ def _expand_with_toolchains_and_location_attr( getattr(ctx.attr, deps_attr_name), getattr(ctx.attr, env_attr_name), additional_lookup_dict = additional_lookup_dict, + validate_expansion = validate_expansion, ) +def _validate_expansions(expanded_values, fail_instead_of_return = True): + """ + Validates all given strings to no longer have unexpanded expressions. + + Validates all expanded strings in `expanded_values` to ensure that no unexpanded (but seemingly + expandable) values remain. + Any unterminated or unexpanded expressions of the form `$VAR`, $(VAR)`, or `${VAR}` will result + in an error (with fail message). + + Args: + expanded_values: (Required) List of string values to validate. + fail_instead_of_return: (Optional) If set to True, `fail()` will be called upon first + invalid (unexpanded) value found. If set to False, error + messages will be collected and returned (no failure will + occur); it will be the caller's responsibility to check the + returned list. + Default value is True. + + Returns: + A list with all found invalid (unexpanded) values. Will be an empty list if all values are + completely expanded. This function will not return if there were unexpanded substrings and if + `fail_instead_of_return` is set to True (due to `fail()` being called). + """ + found_errors = [] + for expanded_val in expanded_values: + found_errors += _validate_all_keys_expanded(expanded_val, fail_instead_of_return) + return found_errors + expansion = struct( expand_with_manual_dict = _expand_with_manual_dict, expand_with_manual_dict_and_location = _expand_with_manual_dict_and_location, @@ -326,4 +551,5 @@ expansion = struct( expand_with_toolchains_attr = _expand_with_toolchains_attr, expand_with_toolchains_and_location = _expand_with_toolchains_and_location, expand_with_toolchains_and_location_attr = _expand_with_toolchains_and_location_attr, + validate_expansions = _validate_expansions, ) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index 56afb250..f92de650 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -268,6 +268,112 @@ _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCAT ), } +# Unresolved/unterminated validation test input values and expected values + +_UNRESOLVED_SUBSTRINGS = [ + "$SOME_UNEXPANDED_VAR_RAW", + "$(SOME_UNEXPANDED_VAR_PAREN)", + "${SOME_UNEXPANDED_VAR_CURLY}", +] + +_UNRESOLVED_SUBSTRINGS_FORMAT_STRINGS = [ + "{}", + "prefix-{}", + "prefix {}", + "{}-suffix", + "{} suffix", + "prefix-{}-suffix", + "prefix {} suffix", +] + +_UNRESOLVED_SUBSTRING_EXPECTED_ERROR_MESSAGES = [ + "Unexpanded expression ('{}' in '{}').".format(unresolved_substr, full_string) + for unresolved_substr in _UNRESOLVED_SUBSTRINGS + for full_string in [ + format_str.format(unresolved_substr) + for format_str in _UNRESOLVED_SUBSTRINGS_FORMAT_STRINGS + ] +] + +_TWO_UNRESOLVED_SUBSTRINGS = [ + "{}-{}".format(a, b) + for a in _UNRESOLVED_SUBSTRINGS + for b in _UNRESOLVED_SUBSTRINGS +] + [ + "{} {}".format(a, b) + for a in _UNRESOLVED_SUBSTRINGS + for b in _UNRESOLVED_SUBSTRINGS +] + +_TWO_UNRESOLVED_SUBSTRING_EXPECTED_ERROR_MESSAGES = [ + "Unexpanded expression ('{}' in '{}').".format(unresolved_substr, full_string) + for unresolved_substr in _UNRESOLVED_SUBSTRINGS + for full_string in _TWO_UNRESOLVED_SUBSTRINGS + if unresolved_substr in full_string +] + +_UNRESOLVED_NAMELESS_SUBSTRINGS_FORMAT_STRINGS = [ + "{}-suffix", + "{} suffix", + "prefix-{}-suffix", + "prefix {} suffix", +] + +_UNRESOLVED_NAMELESS_SUBSTRING_EXPECTED_ERROR_MESSAGES = [ + "Unexpanded expression ('{}' in '{}').".format("$", full_string) + for unresolved_substr in _UNRESOLVED_SUBSTRINGS + for full_string in [ + format_str.format("$") + for format_str in _UNRESOLVED_NAMELESS_SUBSTRINGS_FORMAT_STRINGS + ] +] + +_UNTERMINATED_SUBSTRINGS_PAREN = [ + "$(", + "$(VAR", + "$(VAR ", + "$(VAR VAL", +] + +_UNTERMINATED_SUBSTRINGS_CURLY = [ + "${", + "${VAR", + "${VAR ", + "${VAR VAL", +] + +_UNTERMINATED_SUBSTRINGS_FORMAT_STRINGS = [ + "{}", + "prefix-{}", + "prefix {}", +] + +_UNTERMINATED_SUBSTRING_EXPECTED_ERROR_MESSAGES_RAW = [ + "Unterminated '$' expression in '{}'.".format(full_string) + for full_string in [ + format_str.format("$") + for format_str in _UNTERMINATED_SUBSTRINGS_FORMAT_STRINGS + ] +] + +_UNTERMINATED_SUBSTRING_EXPECTED_ERROR_MESSAGES_PAREN = [ + "Unterminated '$(...)' expression ('{}' in '{}').".format(unresolved_substr, full_string) + for unresolved_substr in _UNTERMINATED_SUBSTRINGS_PAREN + for full_string in [ + format_str.format(unresolved_substr) + for format_str in _UNTERMINATED_SUBSTRINGS_FORMAT_STRINGS + ] +] + +_UNTERMINATED_SUBSTRING_EXPECTED_ERROR_MESSAGES_CURLY = [ + "Unterminated '${{...}}' expression ('{}' in '{}').".format(unresolved_substr, full_string) + for unresolved_substr in _UNTERMINATED_SUBSTRINGS_CURLY + for full_string in [ + format_str.format(unresolved_substr) + for format_str in _UNTERMINATED_SUBSTRINGS_FORMAT_STRINGS + ] +] + # Test helper functions and rules def _test_toolchain_impl(ctx): @@ -681,6 +787,114 @@ _expand_with_toolchains_and_location_attr_with_additional_dict_test = unittest.m }, ) +def _validate_expansions_on_fully_resolved_values_test_impl(ctx): + """Test `expansion.validate_expansions()` with fully resolved strings""" + env = unittest.begin(ctx) + + # Remove the values that are meant to be unexpanded in other tests. + _values_with_mocked_location = dict(_EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION) + _values_with_mocked_location.pop("UNRECOGNIZED_VAR") + _values_with_mocked_location.pop("UNRECOGNIZED_FUNC") + + _values_with_genrule_location = dict(_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION) + _values_with_genrule_location.pop("UNRECOGNIZED_VAR") + _values_with_genrule_location.pop("UNRECOGNIZED_FUNC") + + failure_messages = [] + for fail_on_unexpanded in (True, False): + failure_messages += expansion.validate_expansions( + _values_with_mocked_location.values(), + fail_instead_of_return = fail_on_unexpanded, + ) + failure_messages += expansion.validate_expansions( + _values_with_genrule_location.values(), + fail_instead_of_return = fail_on_unexpanded, + ) + + failure_messages += expansion.validate_expansions( + [ + "$$SOME_ESCAPED_VAR", + "prefix-$$SOME_ESCAPED_VAR", + "$$SOME_ESCAPED_VAR-suffix", + "prefix-$$SOME_ESCAPED_VAR-suffix", + "$$", + "prefix-$$", + "$$-suffix", + "prefix-$$-suffix", + ], + fail_instead_of_return = fail_on_unexpanded, + ) + asserts.equals(env, [], failure_messages) + + return unittest.end(env) + +_validate_expansions_on_fully_resolved_values_test = unittest.make( + _validate_expansions_on_fully_resolved_values_test_impl, +) + +def _validate_expansions_on_unresolved_or_unterminated_values_test_impl(ctx): + """ + Test `expansion.validate_expansions()` with unresolved or unterminated values + + Will be used with multiple targets for specific unresolved/unterminated values. + """ + env = unittest.begin(ctx) + + all_error_messages = [] + for bad_substring in env.ctx.attr.bad_substrings: + for string_format in env.ctx.attr.string_formats: + bad_str = string_format.format(bad_substring) + + # Get out error messages (instead of calling `fail()` internally). + # This allows us to parameterize the test in this impl, instead of parameterizing the + # test target definitions. + # Also allows using `unittest` without `analysistest`. + error_messages = expansion.validate_expansions( + [bad_str], + fail_instead_of_return = False, + ) + + expected_error_messages_per_call = ctx.attr.expected_error_messages_per_call + asserts.true( + env, + len(error_messages) == expected_error_messages_per_call, + "Wrong error message size. Expected count: {}.\n Got messages:\n{}".format( + expected_error_messages_per_call, + "\n".join(error_messages), + ), + ) + all_error_messages.extend(error_messages) + + for expected_failure in ctx.attr.expected_failures: + asserts.true( + env, + expected_failure in all_error_messages, + "'{}' not in:\n{}".format(expected_failure, "\n".join(all_error_messages)), + ) + + return unittest.end(env) + +_validate_expansions_on_unresolved_or_unterminated_values_test = unittest.make( + _validate_expansions_on_unresolved_or_unterminated_values_test_impl, + attrs = { + "bad_substrings": attr.string_list( + mandatory = True, + allow_empty = False, + ), + "expected_error_messages_per_call": attr.int( + default = 1, + ), + "expected_failures": attr.string_list( + mandatory = True, + allow_empty = False, + ), + "string_formats": attr.string_list( + mandatory = True, + allow_empty = False, + ), + }, +) + # buildifier: disable=unnamed-macro def expansion_test_suite(): """Creates the test targets and test suite for expansion.bzl tests.""" @@ -741,6 +955,38 @@ def expansion_test_suite(): env = _ENV_DICT, toolchains = [":expansion_tests__test_toolchain"], ) + _validate_expansions_on_fully_resolved_values_test( + name = "expansion_tests__validate_expansions_on_fully_resolved_values_test", + ) + _validate_expansions_on_unresolved_or_unterminated_values_test( + name = "expansion_tests__validate_expansions_on_unresolved_values_test", + bad_substrings = _UNRESOLVED_SUBSTRINGS, + expected_failures = _UNRESOLVED_SUBSTRING_EXPECTED_ERROR_MESSAGES, + string_formats = _UNRESOLVED_SUBSTRINGS_FORMAT_STRINGS, + ) + _validate_expansions_on_unresolved_or_unterminated_values_test( + name = "expansion_tests__validate_expansions_on_two_unresolved_values_test", + bad_substrings = _TWO_UNRESOLVED_SUBSTRINGS, + expected_error_messages_per_call = 2, + expected_failures = _TWO_UNRESOLVED_SUBSTRING_EXPECTED_ERROR_MESSAGES, + string_formats = ["{}"], + ) + _validate_expansions_on_unresolved_or_unterminated_values_test( + name = "expansion_tests__validate_expansions_on_nameless_var_values_test", + bad_substrings = ["$"], + expected_failures = _UNRESOLVED_NAMELESS_SUBSTRING_EXPECTED_ERROR_MESSAGES, + string_formats = _UNRESOLVED_NAMELESS_SUBSTRINGS_FORMAT_STRINGS, + ) + _validate_expansions_on_unresolved_or_unterminated_values_test( + name = "expansion_tests__validate_expansions_on_unterminated_values_test", + bad_substrings = ["$"] + _UNTERMINATED_SUBSTRINGS_PAREN + _UNTERMINATED_SUBSTRINGS_CURLY, + expected_failures = ( + _UNTERMINATED_SUBSTRING_EXPECTED_ERROR_MESSAGES_RAW + + _UNTERMINATED_SUBSTRING_EXPECTED_ERROR_MESSAGES_PAREN + + _UNTERMINATED_SUBSTRING_EXPECTED_ERROR_MESSAGES_CURLY + ), + string_formats = _UNTERMINATED_SUBSTRINGS_FORMAT_STRINGS, + ) native.test_suite( name = "expansion_tests", @@ -755,5 +1001,10 @@ def expansion_test_suite(): ":expansion_tests__expand_with_toolchains_and_location_with_additional_dict_test", ":expansion_tests__expand_with_toolchains_and_location_attr_test", ":expansion_tests__expand_with_toolchains_and_location_attr_with_additional_dict_test", + ":expansion_tests__validate_expansions_on_fully_resolved_values_test", + ":expansion_tests__validate_expansions_on_unresolved_values_test", + ":expansion_tests__validate_expansions_on_two_unresolved_values_test", + ":expansion_tests__validate_expansions_on_nameless_var_values_test", + ":expansion_tests__validate_expansions_on_unterminated_values_test", ], ) From 2ad568fceffe842fb9bb1386c3c3c81fe4b6aec7 Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sun, 28 Jan 2024 08:59:20 -0800 Subject: [PATCH 08/17] * Fixing doc comment for default value. --- lib/expansion.bzl | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/expansion.bzl b/lib/expansion.bzl index 912e5ba3..1736e71f 100644 --- a/lib/expansion.bzl +++ b/lib/expansion.bzl @@ -486,10 +486,11 @@ def _expand_with_toolchains_and_location_attr( environment variable expansion. This object is also used for the `ctx.expand_location` method to handle `$(location ...)` (and similar) expansion logic. This object is also used to retrieve various necessary attributes via - `ctx.attr.`. Default value is "deps". + `ctx.attr.`. deps_attr_name: (Optional) The name of the attribute which contains the set of targets used with `ctx.expand_location` for expanding `$(location ...)` (and similar) expressions. + Default value is "deps". env_attr_name: (Optional) The name of the attribute that is used as the source for all desired expansions. All key/value pairs will appear within the returned dictionary, with all values fully expanded by lookups in `ctx.var` and From 9e4779a3f9890fdc567df42477d12385d13b4331 Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sun, 18 Feb 2024 01:16:33 -0800 Subject: [PATCH 09/17] * Adding doc comments and other updates from PR review for bazel-bats, https://github.com/filmil/bazel-bats/pull/28. --- lib/expansion.bzl | 258 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 246 insertions(+), 12 deletions(-) diff --git a/lib/expansion.bzl b/lib/expansion.bzl index 1736e71f..668d4119 100644 --- a/lib/expansion.bzl +++ b/lib/expansion.bzl @@ -3,15 +3,49 @@ _CONSIDERED_KEY_FORMATS = ("${}", "${{{}}}", "$({})") def _valid_char_for_env_var_name(char): + """ + Determines if the given character could be used as a part of variable name. + + Args: + char: (Required) A string (intended to be length 1) to be checked. + + Returns: + True if the character could be a part of a variable name. False otherwise. + """ return char.isalnum() or char == "_" def _find_env_var_name_index_index( string, str_len, - search_start, + search_start_index, special_ending_char = None): - for offset in range(str_len - search_start): - index = search_start + offset + """ + Searches for the end of a variable name in the given string, starting from the given index. + + Search will start from `search_start_index` and conclude once a character, which cannot be part + of a variable name, is encountered or until the end of the string is reached. + + Args: + string: (Required) The string to search through. + str_len: (Required) The precomputed length of the given `string` parameter. + search_start_index: (Required) The index to start searching from. This is intended to be + somewhere within (the start?) of a variable name. + special_ending_char: (Optional) A special character which will count as the end of the + variable name. This can be used for `$(VAR)`, `${VAR}`, or similar. + This replaces the "valid variable name character" checking, + allowing for other characters to occur before the given special + ending character. + If set to `None`, no special character will be checked for + (only checking for non-variable characters or the end of the + string). + The default value is `None`. + + Returns: + The index (with respect to the start of `string`) of the last character of the variable + name. + """ + for offset in range(str_len - search_start_index): + index = search_start_index + offset char = string[index] if special_ending_char: if char == special_ending_char: @@ -20,20 +54,50 @@ def _find_env_var_name_index_index( return index - 1 return str_len - 1 -def _odd_count_dollar_sign_repeat(containing_str, end_of_dollar_signs_index): +def _even_count_dollar_sign_repeat(containing_str, end_of_dollar_signs_index): + """ + Searches backwards through the given string, counting the contiguous `$` characters. + + An even number of `$` characters is indicative of escaped variables, which should not be + expanded (left as is in a string). + + Args: + containing_str: (Required) The string to search through. + end_of_dollar_signs_index: (Required) The index of the end of the contiguous `$` + characters in `containing_str`. This is the starting + index for the backwards search. + + Returns: + True if the set of contiguous `$` characters has even length. False if the length is odd. + """ dollar_sign_count = 0 for index in range(end_of_dollar_signs_index, -1, -1): if containing_str[index] != "$": break dollar_sign_count += 1 - return (dollar_sign_count % 2) == 1 + return (dollar_sign_count % 2) == 0 def _key_to_be_expanded(str_with_key, key, start_of_key_index): - # Check that the string at index is prefixed with an odd number of `$`. - # Odd number means that the last `$` is not escaped. - odd_count = _odd_count_dollar_sign_repeat(str_with_key, start_of_key_index) + """ + Examines the given string and determines if the given "key" should be expanded. + + The "key" was located within the given string (as a substring). This function + determines whether the key is complete and is to be expanded. + + Args: + str_with_key: (Required) The string that `key` is found within. + key: (Required) The found substring in `str_with_key` which needs to possibly be + expanded. + start_of_key_index: (Required) The index where `key` was found within `str_with_key`. - if not odd_count: + Returns: + True if the found key is complete (not a substring of another potential key) and is not + escaped (even number of preceding `$`). + """ + + # Check that the string at index is prefixed with an even number of `$`. + # An even number means that the last `$` is escaped. + if _even_count_dollar_sign_repeat(str_with_key, start_of_key_index): return False # Check that the key is correctly matched. @@ -49,6 +113,21 @@ def _key_to_be_expanded(str_with_key, key, start_of_key_index): return not key_mismatch def _fail_validation(fail_instead_of_return, found_errors_list, failure_message): + """ + This is called when a failure has occured and handles propagation of a failure message. + + Will either call `fail()` with the given failure message (to hard fail immediately) or append + the given failure message to the given list. + + Args: + fail_instead_of_return: (Required) If set to True, `fail()` will be called (will not + return). If set to False, `found_errors_list` will be appended to + and the function will return normally. + found_errors_list: (Required) In/out list for error messages to be appended into. Will + only be used if `fail_instead_of_return` is False. + failure_message: (Required) Failure message to be either passed to `fail()` or + appended into `found_errors_list`. + """ if fail_instead_of_return: fail(failure_message) else: @@ -60,6 +139,28 @@ def _validate_unterminated_expression( found_errors, dollar_sign_index, next_char_after_dollar_sign): + """ + Checks if given string contains an unterminated expression of the form `$(VAR)` or `${VAR}`. + + If the given variable/expression is of the correct form, and unterminated, an error will be + noted (either by calling `fail()` or by appending it into the given error list). + + Args: + expanded_val: (Required) The string which contains a `$` preceding a variable (to be + expanded). + fail_instead_of_return: (Required) If set to True, `fail()` will be called (will not + return) when an unterminated variable is found. If set to False, + `found_errors` will be appended to and the function will return + normally. + found_errors: (Required) In/out list for error messages to be appended into. Will only be + used if `fail_instead_of_return` is False. + dollar_sign_index: (Required) The index of the `$` at the start of the expression. + next_char_after_dollar_sign: (Required) The character that immediately follows the `$`. + + Returns: + The validaity of the string. + Returns False if the variable was of the form and unterminated. Returns True otherwise. + """ if next_char_after_dollar_sign == "(": if expanded_val.find(")", dollar_sign_index + 1) < 0: unterminated_expr = expanded_val[dollar_sign_index:] @@ -87,6 +188,26 @@ def _validate_unexpanded_expression( found_errors, dollar_sign_index, next_char_after_dollar_sign): + """ + Always generates an error for the given string (containing unexpanded variable). + + The given string contains a variable which unexpanded (and is not escaped), an error will be + noted (either by calling `fail()` or by appending it into the given error list). + + Args: + expanded_val: (Required) The string which contains a `$` preceding a variable (to be + expanded). + fail_instead_of_return: (Required) If set to True, `fail()` will be called (will not + return). If set to False, `found_errors` will be appended to and + the function will return normally. + str_len: (Required) The precomputed length of the given `expanded_val` parameter. + found_errors: (Required) In/out list for error messages to be appended into. Will only be + used if `fail_instead_of_return` is False. + dollar_sign_index: (Required) The index of the `$` at the start of the unexpanded + expression. + next_char_after_dollar_sign: (Required) The character that immediately follows the `$`. + """ + # Find special ending char, if wrapped expression. special_ending_char = None if next_char_after_dollar_sign == "(": @@ -94,7 +215,7 @@ def _validate_unexpanded_expression( elif next_char_after_dollar_sign == "{": special_ending_char = "}" - # Check for unexpanded expressions. + # Find info for unexpanded expression and fail. name_end_index = _find_env_var_name_index_index( expanded_val, str_len, @@ -111,6 +232,27 @@ def _validate_unexpanded_expression( ) def _validate_all_keys_expanded(expanded_val, fail_instead_of_return): + """ + Iterates over the entire given string, searching for any unexpanded variables/expressions. + + If any unexpanded/unterminated variables/expressions are found, an error will be noted (either + by calling `fail()` and hard failing immediately, or by collecting all such found errors and + returning it in a list). + + Args: + expanded_val: (Required) The string to be checked for any potentially unescaped and + unexpanded/unterminated variables/expressions. + fail_instead_of_return: (Required) If set to True, `fail()` will be called (will not + return) when the first error has been found. If set to False, the + function will return normally and return a list of all found + errors. + + Returns: + A list of found errors. Each element in the list is a failure message with details about + the unescaped and unexpanded/unterminated variable/expression. The list will be empty if + no such expressions were found. This function does not return if `fail_instead_of_return` + was set to True (`fail()` will be called). + """ str_len = len(expanded_val) str_iter = 0 found_errors = [] @@ -126,7 +268,7 @@ def _validate_all_keys_expanded(expanded_val, fail_instead_of_return): # Check for unterminated (non-escaped) ending dollar sign(s). if next_dollar_sign_index == str_len - 1: - if _odd_count_dollar_sign_repeat(expanded_val, next_dollar_sign_index): + if not _even_count_dollar_sign_repeat(expanded_val, next_dollar_sign_index): _fail_validation( fail_instead_of_return, found_errors, @@ -143,7 +285,7 @@ def _validate_all_keys_expanded(expanded_val, fail_instead_of_return): continue # Check for escaped dollar signs (which are ok). - if not _odd_count_dollar_sign_repeat(expanded_val, next_dollar_sign_index): + if _even_count_dollar_sign_repeat(expanded_val, next_dollar_sign_index): continue # Check for unterminated expressions. @@ -167,6 +309,29 @@ def _validate_all_keys_expanded(expanded_val, fail_instead_of_return): return found_errors def _expand_key_in_str(key, val, unexpanded_str): + """ + Expand the given key, by replacing it with the given value, in the given string. + + The given `key` may or may not be contained in the given string `unexpanded_str`. + If the given key is found, it will be expanded/replaced by the given `val` string. + The key is given in its full formatted form with preceding `$` (`$VAR`, `$(VAR)`, `${VAR}`, + `$(VAR VAL)`, etc). + The key will not be expanded if it is escaped (an even number of contiguous `$` characters at + the start) or if the found key is a substring of another potential key (e.g. `$VAR` will not be + expanded if the found location is `$VARIABLE`). + The given key will be replaced (as appropriate) for all occurences within the given string. + + Args: + key: (Required) The key to search for (within the given string, `unexpanded_str`) and + replace all occurences of the key with the given replacement value, `val`. + val: (Required) The value to replace all found occurences of the given key, `key`, into + the given string, `unexpanded_str`. + unexpanded_str: (Required) The string to search for `key` and replace with `val`. + + Returns: + A copy of `unexpanded_str` with all occurences of `key` replaced with `val` (as necessary). + The returned string will be `unexpanded_str` (not a copy), if `key` is not found/expanded. + """ key_len = len(key) val_len = len(val) searched_index = 0 @@ -193,6 +358,29 @@ def _expand_key_in_str(key, val, unexpanded_str): return expanded_str def _expand_all_keys_in_str_from_dict(replacement_dict, unexpanded_str): + """ + Uses the given dictionary to replace keys with values in the given string. + + Each key is intended to be a variable name (e.g. `VARIABLE_NAME`), which can be wrapped with + `$`, `$( )`, or `${ }` (to express the given example key as the "formatted key" + `$VARIABLE_NAME`, `$(VARIABLE_NAME)`, or `${VARIABLE_NAME}`). The corresponding value (in the + dict) is to be used as the intended replacement string when any matching formatted key (of the + given variable name key) is found within the given string, `unexpanded_str`. + + Args: + replacement_dict: (Required) The set of key/value pairs to be used for search/replacement + within the given `unexpanded_str` string. + unexpanded_str: (Required) The string to search for the formatted versions of each key + set within `replacement_dict`, where each found occurence will be + expanded/replaced with the associated value. + + Returns: + A copy of `unexpanded_str` with all occurences of each key (when formatted into an + unexpanded variable) within `replacement_dict` replaced with corresponding value (as + necessary). + The returned string will be `unexpanded_str` (not a copy), if no expansion occurs. + """ + # Manually expand variables based on the var dict. # Do not use `ctx.expand_make_variables()` as it will error out if any variable expands to # `$(location )` (or similar) instead of leaving it untouched. @@ -218,6 +406,52 @@ def _expand_all_keys_in_str( resolved_replacement_dict, env_replacement_dict, unexpanded_str): + """ + Uses the given dictionaries to replace keys with values in the given string. + + Each key, in the given dictionaries, is intended to be a variable name (e.g. `VARIABLE_NAME`), + which can be wrapped with `$`, `$( )`, or `${ }` (to express the given example key as the + "formatted key" `$VARIABLE_NAME`, `$(VARIABLE_NAME)`, or `${VARIABLE_NAME}`). The corresponding + value (in the dict) is to be used as the intended replacement string when any matching + formatted key (of the given variable name key) is found within the given string, + `unexpanded_str`. + + Expansion happens iteratively. In each iteration, three steps occur: + 1) If `expand_location` is not `None`, it will be invoked to replace any occurrences of + `$(location ...)` (or similar). Technically, this function can execute any high-priority + expansion logic -- but it is intended for functionality similar to `ctx.expand_location()`. + 2) Each variable name key in `env_replacement_dict` will be searched for (in `unexpanded_str`) + and expanded into the corresponding value within the dict for the given found variable + name. This is intended for the use with the `env` attribute for a given target (but + supports any general "higher priority" dict replacement). + 3) Each variable name key in `resolved_replacement_dict` will be searched for (in + `unexpanded_str`) and expanded into the corresponding value within the dict for the given + found variable name. This is intended for the use with `ctx.var` which contains toolchain + resolved key/values (but supports any general "lower priority" dict replacement). + + Args: + expand_location: (Required) A None-able function used for optional "location" + expansion logic (`$(location ...)` or similar). + resolved_replacement_dict: (Required) A set of key/value pairs to be used for + search/replacement within the given `unexpanded_str` string. + Replacement logic will occur after (lower priority) replacement + for `env_replacement_dict`. + env_replacement_dict: (Required) A set of key/value pairs to be used for + search/replacement within the given `unexpanded_str` string. + Replacement logic will occur before (higher priority) replacement + for `resolved_replacement_dict`. + unexpanded_str: (Required) The string to perform expansion variable upon (optionally + invoke `expand_location`, and search for the formatted versions of each + key set within `env_replacement_dict` and `resolved_replacement_dict`, + where each found occurence will be expanded/replaced with the + associated value). + + Returns: + A copy of `unexpanded_str` with all occurences of each key (when formatted into an + unexpanded variable) within `replacement_dict` replaced with corresponding value (as + necessary). + The returned string will be `unexpanded_str` (not a copy), if no expansion occurs. + """ if unexpanded_str.find("$") < 0: return unexpanded_str From f0cbec4fd8a53399e05447ddaa2eef68e7b1dec2 Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sat, 11 May 2024 15:05:42 -0700 Subject: [PATCH 10/17] * Updating expected rlocation path for changes from merged-in master. --- tests/expansion_tests.bzl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index f92de650..6176fb36 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -15,7 +15,7 @@ _MOCK_RLOCATIONPATH_PATH_OF_DUMMY = "rlocationpath/path/of/dummy" _GENRULE_LOCATION_PATH_OF_DUMMY = "bazel-out/k8-fastbuild/bin/tests/dummy.txt" _GENRULE_EXECPATH_PATH_OF_DUMMY = "bazel-out/k8-fastbuild/bin/tests/dummy.txt" _GENRULE_ROOTPATH_PATH_OF_DUMMY = "tests/dummy.txt" -_GENRULE_RLOCATIONPATH_PATH_OF_DUMMY = "_main/tests/dummy.txt" +_GENRULE_RLOCATIONPATH_PATH_OF_DUMMY = "bazel_skylib/tests/dummy.txt" _LINUX_FASTBUILD_SUBPATH = "k8-fastbuild" _MAC_FASTBUILD_SUBPATH = "darwin_x86_64-fastbuild" From 4fd038818c855dce97ee95028e60331ff8bdc3bb Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sat, 11 May 2024 16:21:54 -0700 Subject: [PATCH 11/17] * Updating expected rlocation value to work with different versions of bazel used in testing. --- tests/expansion_tests.bzl | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index 6176fb36..f2b3fff2 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -15,7 +15,7 @@ _MOCK_RLOCATIONPATH_PATH_OF_DUMMY = "rlocationpath/path/of/dummy" _GENRULE_LOCATION_PATH_OF_DUMMY = "bazel-out/k8-fastbuild/bin/tests/dummy.txt" _GENRULE_EXECPATH_PATH_OF_DUMMY = "bazel-out/k8-fastbuild/bin/tests/dummy.txt" _GENRULE_ROOTPATH_PATH_OF_DUMMY = "tests/dummy.txt" -_GENRULE_RLOCATIONPATH_PATH_OF_DUMMY = "bazel_skylib/tests/dummy.txt" +_GENRULE_RLOCATIONPATH_PATH_OF_DUMMY = "_main/tests/dummy.txt" _LINUX_FASTBUILD_SUBPATH = "k8-fastbuild" _MAC_FASTBUILD_SUBPATH = "darwin_x86_64-fastbuild" @@ -727,6 +727,9 @@ def _expand_with_toolchains_and_location_attr_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") asserts.equals(env, expected_val, resolved_val) return unittest.end(env) From 7e24e95d8d38c9bf75d39b2ec68cd1b6430c95cc Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sat, 11 May 2024 16:30:49 -0700 Subject: [PATCH 12/17] * Applying update to all impl functions. --- tests/expansion_tests.bzl | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index f2b3fff2..ef77c1a4 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -431,6 +431,9 @@ def _expand_with_manual_dict_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -462,6 +465,9 @@ def _expand_with_manual_dict_and_location_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -491,6 +497,9 @@ def _expand_with_toolchains_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -534,6 +543,9 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -563,6 +575,9 @@ def _expand_with_toolchains_attr_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -610,6 +625,9 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -646,6 +664,9 @@ def _expand_with_toolchains_and_location_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -695,6 +716,9 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") asserts.equals(env, expected_val, resolved_val) return unittest.end(env) @@ -778,6 +802,9 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") asserts.equals(env, expected_val, resolved_val) return unittest.end(env) From a2ff961bfee42545ef7d6f66e8214e3cfc79a77e Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sat, 11 May 2024 16:46:36 -0700 Subject: [PATCH 13/17] * Downgrading usage of `dict | dict` in `tests/expansion_tests.bzl` to use `dict(dict).update(dict)` to be backwards compatible with Bazel 5.x. --- tests/expansion_tests.bzl | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index ef77c1a4..281ec3a6 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -170,7 +170,7 @@ _EXPECTED_RESOLVED_DICT_NO_LOCATION = { "UNRECOGNIZED_FUNC": "$(nope :" + _TEST_DEP_TARGET_NAME + ")", } -_EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { +_EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION).update({ "LOCATION_VAL": _MOCK_LOCATION_PATH_OF_DUMMY, "EXECPATH_VAL": _MOCK_EXECPATH_PATH_OF_DUMMY, "ROOTPATH_VAL": _MOCK_ROOTPATH_PATH_OF_DUMMY, @@ -217,9 +217,9 @@ _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCATI "-flag_value-" + _MOCK_LOCATION_PATH_OF_DUMMY ), -} +}) -_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { +_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION).update({ "LOCATION_VAL": _GENRULE_LOCATION_PATH_OF_DUMMY, "EXECPATH_VAL": _GENRULE_EXECPATH_PATH_OF_DUMMY, "ROOTPATH_VAL": _GENRULE_ROOTPATH_PATH_OF_DUMMY, @@ -266,7 +266,7 @@ _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCAT "-flag_value-" + _GENRULE_LOCATION_PATH_OF_DUMMY ), -} +}) # Unresolved/unterminated validation test input values and expected values @@ -531,12 +531,12 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) - updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { + updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION).update({ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - } + }) # Check all output resolved values against expected resolved values. for env_key, _ in _ENV_DICT.items(): @@ -613,12 +613,12 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) - updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { + updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION).update({ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - } + }) # Check all output resolved values against expected resolved values. for env_key, _ in env.ctx.attr.env.items(): @@ -704,12 +704,12 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) - updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { + updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION).update({ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - } + }) # Check all output resolved values against expected resolved values. for env_key, _ in _ENV_DICT.items(): @@ -790,12 +790,12 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx # Check that the output has exact same key set as original input. asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) - updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { + updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION).update({ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - } + }) # Check all output resolved values against expected resolved values. for env_key, _ in env.ctx.attr.env.items(): From aa133b97774027e8ba1c3ff949abe26943d4526b Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sat, 11 May 2024 17:11:27 -0700 Subject: [PATCH 14/17] * Revering previous commit. --- tests/expansion_tests.bzl | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index 281ec3a6..ef77c1a4 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -170,7 +170,7 @@ _EXPECTED_RESOLVED_DICT_NO_LOCATION = { "UNRECOGNIZED_FUNC": "$(nope :" + _TEST_DEP_TARGET_NAME + ")", } -_EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION).update({ +_EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { "LOCATION_VAL": _MOCK_LOCATION_PATH_OF_DUMMY, "EXECPATH_VAL": _MOCK_EXECPATH_PATH_OF_DUMMY, "ROOTPATH_VAL": _MOCK_ROOTPATH_PATH_OF_DUMMY, @@ -217,9 +217,9 @@ _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_L "-flag_value-" + _MOCK_LOCATION_PATH_OF_DUMMY ), -}) +} -_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION).update({ +_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { "LOCATION_VAL": _GENRULE_LOCATION_PATH_OF_DUMMY, "EXECPATH_VAL": _GENRULE_EXECPATH_PATH_OF_DUMMY, "ROOTPATH_VAL": _GENRULE_ROOTPATH_PATH_OF_DUMMY, @@ -266,7 +266,7 @@ _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_ "-flag_value-" + _GENRULE_LOCATION_PATH_OF_DUMMY ), -}) +} # Unresolved/unterminated validation test input values and expected values @@ -531,12 +531,12 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) - updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION).update({ + updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - }) + } # Check all output resolved values against expected resolved values. for env_key, _ in _ENV_DICT.items(): @@ -613,12 +613,12 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) - updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION).update({ + updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - }) + } # Check all output resolved values against expected resolved values. for env_key, _ in env.ctx.attr.env.items(): @@ -704,12 +704,12 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) - updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION).update({ + updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - }) + } # Check all output resolved values against expected resolved values. for env_key, _ in _ENV_DICT.items(): @@ -790,12 +790,12 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx # Check that the output has exact same key set as original input. asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) - updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION).update({ + updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - }) + } # Check all output resolved values against expected resolved values. for env_key, _ in env.ctx.attr.env.items(): From 826b30c1b4b001400fa13c0fb21b1c67dcfde27b Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sat, 11 May 2024 17:55:39 -0700 Subject: [PATCH 15/17] * Fixing formatting. --- tests/expansion_tests.bzl | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index ef77c1a4..8b82db33 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -431,6 +431,7 @@ def _expand_with_manual_dict_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). if "_main" in expected_val and "_main" not in resolved_val: expected_val = expected_val.replace("_main", "bazel_skylib") @@ -465,6 +466,7 @@ def _expand_with_manual_dict_and_location_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). if "_main" in expected_val and "_main" not in resolved_val: expected_val = expected_val.replace("_main", "bazel_skylib") @@ -497,6 +499,7 @@ def _expand_with_toolchains_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). if "_main" in expected_val and "_main" not in resolved_val: expected_val = expected_val.replace("_main", "bazel_skylib") @@ -543,6 +546,7 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). if "_main" in expected_val and "_main" not in resolved_val: expected_val = expected_val.replace("_main", "bazel_skylib") @@ -575,6 +579,7 @@ def _expand_with_toolchains_attr_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). if "_main" in expected_val and "_main" not in resolved_val: expected_val = expected_val.replace("_main", "bazel_skylib") @@ -625,6 +630,7 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). if "_main" in expected_val and "_main" not in resolved_val: expected_val = expected_val.replace("_main", "bazel_skylib") @@ -664,6 +670,7 @@ def _expand_with_toolchains_and_location_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). if "_main" in expected_val and "_main" not in resolved_val: expected_val = expected_val.replace("_main", "bazel_skylib") @@ -716,6 +723,7 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). if "_main" in expected_val and "_main" not in resolved_val: expected_val = expected_val.replace("_main", "bazel_skylib") @@ -751,6 +759,7 @@ def _expand_with_toolchains_and_location_attr_test_impl(ctx): expected_val = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). if "_main" in expected_val and "_main" not in resolved_val: expected_val = expected_val.replace("_main", "bazel_skylib") @@ -802,6 +811,7 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx expected_val = updated_expected_dict[env_key] resolved_val = resolved_dict[env_key] resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + # Replace `expected` with proper value given bazel version (may differ for rlocation path). if "_main" in expected_val and "_main" not in resolved_val: expected_val = expected_val.replace("_main", "bazel_skylib") From e5d911255ba6d86cb6becb71316a98c30deef6cd Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sat, 11 May 2024 18:16:08 -0700 Subject: [PATCH 16/17] * Downgrading usage of `dict | dict` to use `dict(dict, **dict)` to be backwards compatible with Bazel 5.x. * `dict(dict).update(dict)` is not a good replacement as it returns `NoneType`. --- lib/expansion.bzl | 4 ++-- tests/expansion_tests.bzl | 24 ++++++++++++------------ 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/lib/expansion.bzl b/lib/expansion.bzl index 668d4119..6a318916 100644 --- a/lib/expansion.bzl +++ b/lib/expansion.bzl @@ -601,7 +601,7 @@ def _expand_with_toolchains( """ additional_lookup_dict = additional_lookup_dict or {} return _expand_with_manual_dict( - ctx.var | additional_lookup_dict, + dict(ctx.var, **additional_lookup_dict), source_env_dict, validate_expansion = validate_expansion, ) @@ -652,7 +652,7 @@ def _expand_with_toolchains_and_location( additional_lookup_dict = additional_lookup_dict or {} return _expand_with_manual_dict_and_location( _simpler_expand_location, - ctx.var | additional_lookup_dict, + dict(ctx.var, **additional_lookup_dict), source_env_dict, validate_expansion = validate_expansion, ) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index 8b82db33..9f0b33e6 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -170,7 +170,7 @@ _EXPECTED_RESOLVED_DICT_NO_LOCATION = { "UNRECOGNIZED_FUNC": "$(nope :" + _TEST_DEP_TARGET_NAME + ")", } -_EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { +_EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION, **{ "LOCATION_VAL": _MOCK_LOCATION_PATH_OF_DUMMY, "EXECPATH_VAL": _MOCK_EXECPATH_PATH_OF_DUMMY, "ROOTPATH_VAL": _MOCK_ROOTPATH_PATH_OF_DUMMY, @@ -217,9 +217,9 @@ _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCATI "-flag_value-" + _MOCK_LOCATION_PATH_OF_DUMMY ), -} +}) -_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { +_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION, **{ "LOCATION_VAL": _GENRULE_LOCATION_PATH_OF_DUMMY, "EXECPATH_VAL": _GENRULE_EXECPATH_PATH_OF_DUMMY, "ROOTPATH_VAL": _GENRULE_ROOTPATH_PATH_OF_DUMMY, @@ -266,7 +266,7 @@ _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = _EXPECTED_RESOLVED_DICT_NO_LOCAT "-flag_value-" + _GENRULE_LOCATION_PATH_OF_DUMMY ), -} +}) # Unresolved/unterminated validation test input values and expected values @@ -534,12 +534,12 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) - updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { + updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION, **{ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - } + }) # Check all output resolved values against expected resolved values. for env_key, _ in _ENV_DICT.items(): @@ -618,12 +618,12 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) - updated_expected_dict = _EXPECTED_RESOLVED_DICT_NO_LOCATION | { + updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION, **{ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - } + }) # Check all output resolved values against expected resolved values. for env_key, _ in env.ctx.attr.env.items(): @@ -711,12 +711,12 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) - updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { + updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION, **{ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - } + }) # Check all output resolved values against expected resolved values. for env_key, _ in _ENV_DICT.items(): @@ -799,12 +799,12 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx # Check that the output has exact same key set as original input. asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) - updated_expected_dict = _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION | { + updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION, **{ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_CURLY": "expanded from additional dict instead", "UNRECOGNIZED_VAR": "naw, it's fine now.", - } + }) # Check all output resolved values against expected resolved values. for env_key, _ in env.ctx.attr.env.items(): From 5046fb05772c7954d814fa2ec16b2e3981f4cde5 Mon Sep 17 00:00:00 2001 From: Cauhx Milloy Date: Sun, 30 Jun 2024 12:41:03 -0700 Subject: [PATCH 17/17] * Updating `bzl/expansion.bzl`. * Renaming all `expand_with_*` functions to `expand_dict_strings_with_*`. * Explicit that it acts on a dict of strings. * Adding `expand_list_strings_with_manual_dict` and `expand_list_strings_with_manual_dict_and_location` functions. * Similar to the existing `expand_dict_strings_with_manual_dict*` functions, these expand values but from a list of strings instead of dict of strings. * Note that these methods only perform recursion via the "manual dict" (the input datatype, `list`, is not an associative type). * Simpler API for just expanding a few string values (of any attr type, or not necessarily associated with any attr). * Renaming `expand_location` parameter(s) to `wrapped_expand_location`. * This makes it more clear that this is not `ctx.expand_location` directly, but rather a wrapped version of that function. * Adding `wrap_expand_location` method. * Offers a helper function for creating `wrapped_expand_location` (used by many expansion functions) from a given `ctx` and `deps`. * Updating doc string comments. * Updating `tests/bzl/expansion_tests.bzl`. * Renaming all `expand_with_*` tests to `expand_dict_strings_with_*`. * Adding `expand_list_strings_with_*` tests (for new functions). * Adding "no recursion" "expected resolved" dicts for new tests. * Adding `TOOLCHAIN_INDIRECT_ENV_VAR*` values to expand, which show recursion specifically among only toolchain values. --- lib/expansion.bzl | 203 ++++++++++++++++++++------ tests/expansion_tests.bzl | 299 ++++++++++++++++++++++++++++---------- 2 files changed, 380 insertions(+), 122 deletions(-) diff --git a/lib/expansion.bzl b/lib/expansion.bzl index 6a318916..adb6cab8 100644 --- a/lib/expansion.bzl +++ b/lib/expansion.bzl @@ -402,7 +402,7 @@ def _expand_all_keys_in_str_from_dict(replacement_dict, unexpanded_str): return expanded_val def _expand_all_keys_in_str( - expand_location, + wrapped_expand_location, resolved_replacement_dict, env_replacement_dict, unexpanded_str): @@ -417,7 +417,7 @@ def _expand_all_keys_in_str( `unexpanded_str`. Expansion happens iteratively. In each iteration, three steps occur: - 1) If `expand_location` is not `None`, it will be invoked to replace any occurrences of + 1) If `wrapped_expand_location` is not `None`, it will be invoked to replace any occurrences of `$(location ...)` (or similar). Technically, this function can execute any high-priority expansion logic -- but it is intended for functionality similar to `ctx.expand_location()`. 2) Each variable name key in `env_replacement_dict` will be searched for (in `unexpanded_str`) @@ -430,7 +430,7 @@ def _expand_all_keys_in_str( resolved key/values (but supports any general "lower priority" dict replacement). Args: - expand_location: (Required) A None-able function used for optional "location" + wrapped_expand_location: (Required) A None-able function used for optional "location" expansion logic (`$(location ...)` or similar). resolved_replacement_dict: (Required) A set of key/value pairs to be used for search/replacement within the given `unexpanded_str` string. @@ -440,11 +440,11 @@ def _expand_all_keys_in_str( search/replacement within the given `unexpanded_str` string. Replacement logic will occur before (higher priority) replacement for `resolved_replacement_dict`. - unexpanded_str: (Required) The string to perform expansion variable upon (optionally - invoke `expand_location`, and search for the formatted versions of each - key set within `env_replacement_dict` and `resolved_replacement_dict`, - where each found occurence will be expanded/replaced with the - associated value). + unexpanded_str: (Required) The string to perform expansion variable upon. Expansion is + done by optionally invoking `wrapped_expand_location` and search for + the formatted versions of each key set within `env_replacement_dict` + and `resolved_replacement_dict`, where each found occurence will be + expanded/replaced with the associated value). Returns: A copy of `unexpanded_str` with all occurences of each key (when formatted into an @@ -462,8 +462,8 @@ def _expand_all_keys_in_str( for _ in range(len(expanded_val)): # First let's try the safe `location` (et al) expansion logic. # `$VAR`, `$(VAR)`, and `${VAR}` will be left untouched. - if expand_location: - expanded_val = expand_location(expanded_val) + if wrapped_expand_location: + expanded_val = wrapped_expand_location(expanded_val) # Break early if nothing left to expand. if expanded_val.find("$") < 0: @@ -480,7 +480,7 @@ def _expand_all_keys_in_str( return expanded_val -def _expand_with_manual_dict(resolution_dict, source_env_dict, validate_expansion = False): +def _expand_dict_strings_with_manual_dict(resolution_dict, source_env_dict, validate_expansion = False): """ Recursively expands all values in `source_env_dict` using the given lookup data. @@ -509,7 +509,7 @@ def _expand_with_manual_dict(resolution_dict, source_env_dict, validate_expansio expanded_envs = {} for env_key, unexpanded_val in source_env_dict.items(): expanded_val = _expand_all_keys_in_str( - None, # No `expand_location` available + None, # No `wrapped_expand_location` available resolution_dict, source_env_dict, unexpanded_val, @@ -519,8 +519,8 @@ def _expand_with_manual_dict(resolution_dict, source_env_dict, validate_expansio expanded_envs[env_key] = expanded_val return expanded_envs -def _expand_with_manual_dict_and_location( - expand_location, +def _expand_dict_strings_with_manual_dict_and_location( + wrapped_expand_location, resolution_dict, source_env_dict, validate_expansion = False): @@ -528,20 +528,22 @@ def _expand_with_manual_dict_and_location( Recursively expands all values in `source_env_dict` using the given logic / lookup data. All keys of `source_env_dict` are returned in the resultant dict with values expanded by - location expansion logic via `expand_location` and by lookups via `resolution_dict` dict. + location expansion logic via `wrapped_expand_location` and by lookups via `resolution_dict` + dict. This function does not modify any of the given parameters. Args: - expand_location: (Required) A function that takes in a string and properly replaces - `$(location ...)` (and similar) with the corresponding values. This - likely should correspond to `ctx.expand_location()`. + wrapped_expand_location: (Required) A function that takes in a string and properly + replaces `$(location ...)` (and similar) with the corresponding + values. This likely should correspond to + `ctx.expand_location()` via `expansion.wrap_expand_location()`. resolution_dict: (Required) A dictionary with resolved key/value pairs to be used for lookup when resolving values. This may come from toolchains (via `ctx.var`) or other sources. source_env_dict: (Required) The source for all desired expansions. All key/value pairs will appear within the returned dictionary, with all values fully - expanded by the logic expansion logic of `expand_location` and by - lookup in `resolution_dict`. + expanded by the logic expansion logic of `wrapped_expand_location` and + by lookup in `resolution_dict`. validate_expansion: (Optional) If set to True, all expanded strings will be validated to ensure that no unexpanded (but seemingly expandable) values remain. If any unexpanded values are found, `fail()` will be called. The @@ -556,7 +558,7 @@ def _expand_with_manual_dict_and_location( expanded_envs = {} for env_key, unexpanded_val in source_env_dict.items(): expanded_val = _expand_all_keys_in_str( - expand_location, + wrapped_expand_location, resolution_dict, source_env_dict, unexpanded_val, @@ -566,7 +568,7 @@ def _expand_with_manual_dict_and_location( expanded_envs[env_key] = expanded_val return expanded_envs -def _expand_with_toolchains( +def _expand_dict_strings_with_toolchains( ctx, source_env_dict, additional_lookup_dict = None, @@ -600,13 +602,13 @@ def _expand_with_toolchains( expanded. """ additional_lookup_dict = additional_lookup_dict or {} - return _expand_with_manual_dict( + return _expand_dict_strings_with_manual_dict( dict(ctx.var, **additional_lookup_dict), source_env_dict, validate_expansion = validate_expansion, ) -def _expand_with_toolchains_and_location( +def _expand_dict_strings_with_toolchains_and_location( ctx, deps, source_env_dict, @@ -630,7 +632,7 @@ def _expand_with_toolchains_and_location( `$(location ...)` (and similar) expressions. source_env_dict: (Required) The source for all desired expansions. All key/value pairs will appear within the returned dictionary, with all values fully - expanded by the logic expansion logic of `expand_location` and by + expanded by the logic expansion logic of `ctx.expand_location()` and by lookups in `ctx.var` and optional `additional_lookup_dict`. additional_lookup_dict: (Optional) Additional dict to be used with `ctx.var` (union) for variable expansion. @@ -646,18 +648,15 @@ def _expand_with_toolchains_and_location( expanded. """ - def _simpler_expand_location(input_str): - return ctx.expand_location(input_str, deps) - additional_lookup_dict = additional_lookup_dict or {} - return _expand_with_manual_dict_and_location( - _simpler_expand_location, + return _expand_dict_strings_with_manual_dict_and_location( + _wrap_expand_location(ctx, deps), dict(ctx.var, **additional_lookup_dict), source_env_dict, validate_expansion = validate_expansion, ) -def _expand_with_toolchains_attr( +def _expand_dict_strings_with_toolchains_attr( ctx, env_attr_name = "env", additional_lookup_dict = None, @@ -693,14 +692,14 @@ def _expand_with_toolchains_attr( A new dict with all key/values from source attribute (default "env" attribute), where all values have been recursively expanded. """ - return _expand_with_toolchains( + return _expand_dict_strings_with_toolchains( ctx, getattr(ctx.attr, env_attr_name), additional_lookup_dict = additional_lookup_dict, validate_expansion = validate_expansion, ) -def _expand_with_toolchains_and_location_attr( +def _expand_dict_strings_with_toolchains_and_location_attr( ctx, deps_attr_name = "deps", env_attr_name = "env", @@ -743,7 +742,7 @@ def _expand_with_toolchains_and_location_attr( A new dict with all key/values from source attribute (default "env" attribute), where all values have been recursively expanded. """ - return _expand_with_toolchains_and_location( + return _expand_dict_strings_with_toolchains_and_location( ctx, getattr(ctx.attr, deps_attr_name), getattr(ctx.attr, env_attr_name), @@ -751,14 +750,107 @@ def _expand_with_toolchains_and_location_attr( validate_expansion = validate_expansion, ) +def _expand_list_strings_with_manual_dict(resolution_dict, source_strings, validate_expansion = False): + """ + Recursively expands all values in `source_strings` using the given lookup data. + + All values of `source_strings` are returned in the resultant list with values expanded by + lookups via `resolution_dict` dict. + Note that the recursion performed is only among `resolution_dict`, as lists are not associative + datatypes (no mapping via `source_strings`). + This function does not modify any of the given parameters. + + Args: + resolution_dict: (Required) A dictionary with resolved key/value pairs to be used for + lookup when resolving values. This may come from toolchains (via + `ctx.var`) or other sources. + source_strings: (Required) The source for all desired expansions. All values will + appear within the returned list, with all values fully expanded by + lookups in `resolution_dict`. + validate_expansion: (Optional) If set to True, all expanded strings will be validated to + ensure that no unexpanded (but seemingly expandable) values remain. If + any unexpanded values are found, `fail()` will be called. The + validation logic is the same as + `expansion.validate_expansions_in_dict()`. + Default value is False. + + Returns: + A new list of strings with all values from `source_strings`, where all values have been + recursively expanded. + """ + expanded_vals = [] + for unexpanded_val in source_strings: + expanded_val = _expand_all_keys_in_str( + None, # No `wrapped_expand_location` available + resolution_dict, + {}, # No `env_replacement_dict` used. + unexpanded_val, + ) + if validate_expansion: + _validate_all_keys_expanded(expanded_val, fail_instead_of_return = True) + expanded_vals.append(expanded_val) + return expanded_vals + +def _expand_list_strings_with_manual_dict_and_location( + wrapped_expand_location, + resolution_dict, + source_strings, + validate_expansion = False): + """ + Recursively expands all values in `source_strings` using the given logic / lookup data. + + All values of `source_strings` are returned in the resultant dict with values expanded by + location expansion logic via `wrapped_expand_location` and by lookups via `resolution_dict` + dict. + Note that the recursion performed is only among `resolution_dict` and + `wrapped_expand_location`, as lists are not associative datatypes (no mapping via + `source_strings`). + This function does not modify any of the given parameters. + + Args: + wrapped_expand_location: (Required) A function that takes in a string and properly + replaces `$(location ...)` (and similar) with the corresponding + values. This likely should correspond to + `ctx.expand_location()` via `expansion.wrap_expand_location()`. + resolution_dict: (Required) A dictionary with resolved key/value pairs to be used for + lookup when resolving values. This may come from toolchains (via + `ctx.var`) or other sources. + source_strings: (Required) The source for all desired expansions. All values will + appear within the returned list, with all values fully expanded by the + logic expansion logic of `wrapped_expand_location` and by lookup in + `resolution_dict`. + validate_expansion: (Optional) If set to True, all expanded strings will be validated to + ensure that no unexpanded (but seemingly expandable) values remain. If + any unexpanded values are found, `fail()` will be called. The + validation logic is the same as + `expansion.validate_expansions_in_dict()`. + Default value is False. + + Returns: + A new list of strings with all values from `source_strings`, where all values have been + recursively expanded. + """ + expanded_vals = [] + for unexpanded_val in source_strings: + expanded_val = _expand_all_keys_in_str( + wrapped_expand_location, + resolution_dict, + {}, # No `env_replacement_dict` used. + unexpanded_val, + ) + if validate_expansion: + _validate_all_keys_expanded(expanded_val, fail_instead_of_return = True) + expanded_vals.append(expanded_val) + return expanded_vals + def _validate_expansions(expanded_values, fail_instead_of_return = True): """ Validates all given strings to no longer have unexpanded expressions. Validates all expanded strings in `expanded_values` to ensure that no unexpanded (but seemingly expandable) values remain. - Any unterminated or unexpanded expressions of the form `$VAR`, $(VAR)`, or `${VAR}` will result - in an error (with fail message). + Any unterminated or unexpanded expressions of the form `$VAR`, `$(VAR)`, or `${VAR}` will + result in an error (with fail message). Args: expanded_values: (Required) List of string values to validate. @@ -779,12 +871,39 @@ def _validate_expansions(expanded_values, fail_instead_of_return = True): found_errors += _validate_all_keys_expanded(expanded_val, fail_instead_of_return) return found_errors +def _wrap_expand_location(ctx, deps): + """ + Returns a function which is a wrapped version of `ctx.expand_location()`. + + Creates a function which takes a single string input parameter and returns that string after + expanding all `$(location ...)` (and similar) substrings. The returned function is backed by + `ctx.expand_location()`, where the given `deps` are used for expansion. + + Args: + ctx: (Required) The bazel context object. This is used to access `ctx.expand_location` + method to handle `$(location ...)` (and similar) expansion logic. + deps: (Required) The set of targets used with `ctx.expand_location` for expanding + `$(location ...)` (and similar) expressions. + + Returns: + A wrapped function which works similar to `ctx.expand_location()`, but takes the input string + as the only argument. + """ + + def _wrapped_expand_location(input_str): + return ctx.expand_location(input_str, deps) + + return _wrapped_expand_location + expansion = struct( - expand_with_manual_dict = _expand_with_manual_dict, - expand_with_manual_dict_and_location = _expand_with_manual_dict_and_location, - expand_with_toolchains = _expand_with_toolchains, - expand_with_toolchains_attr = _expand_with_toolchains_attr, - expand_with_toolchains_and_location = _expand_with_toolchains_and_location, - expand_with_toolchains_and_location_attr = _expand_with_toolchains_and_location_attr, + expand_dict_strings_with_manual_dict = _expand_dict_strings_with_manual_dict, + expand_dict_strings_with_manual_dict_and_location = _expand_dict_strings_with_manual_dict_and_location, + expand_dict_strings_with_toolchains = _expand_dict_strings_with_toolchains, + expand_dict_strings_with_toolchains_attr = _expand_dict_strings_with_toolchains_attr, + expand_dict_strings_with_toolchains_and_location = _expand_dict_strings_with_toolchains_and_location, + expand_dict_strings_with_toolchains_and_location_attr = _expand_dict_strings_with_toolchains_and_location_attr, + expand_list_strings_with_manual_dict = _expand_list_strings_with_manual_dict, + expand_list_strings_with_manual_dict_and_location = _expand_list_strings_with_manual_dict_and_location, validate_expansions = _validate_expansions, + wrap_expand_location = _wrap_expand_location, ) diff --git a/tests/expansion_tests.bzl b/tests/expansion_tests.bzl index 9f0b33e6..a21bd391 100644 --- a/tests/expansion_tests.bzl +++ b/tests/expansion_tests.bzl @@ -23,6 +23,7 @@ _WIN_FASTBUILD_SUBPATH = "x64_windows-fastbuild" # Test input dicts +# buildifier: disable=unsorted-dict-items _ENV_DICT = { "SIMPLE_VAL": "hello_world", "ESCAPED_SIMPLE_VAL": "$$SIMPLE_VAL", @@ -36,6 +37,9 @@ _ENV_DICT = { "TOOLCHAIN_ENV_VAR2_RAW": "$TOOLCHAIN_ENV_VAR2", "TOOLCHAIN_ENV_VAR2_PAREN": "$(TOOLCHAIN_ENV_VAR2)", "TOOLCHAIN_ENV_VAR2_CURLY": "${TOOLCHAIN_ENV_VAR2}", + "TOOLCHAIN_INDIRECT_ENV_VAR_RAW": "$TOOLCHAIN_INDIRECT_ENV_VAR_RAW", + "TOOLCHAIN_INDIRECT_ENV_VAR_PAREN": "$(TOOLCHAIN_INDIRECT_ENV_VAR_PAREN)", + "TOOLCHAIN_INDIRECT_ENV_VAR_CURLY": "${TOOLCHAIN_INDIRECT_ENV_VAR_CURLY}", "TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": "$TOOLCHAIN_TO_LOCATION_ENV_VAR", "TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": "$(TOOLCHAIN_TO_LOCATION_ENV_VAR)", "TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": "${TOOLCHAIN_TO_LOCATION_ENV_VAR}", @@ -85,10 +89,14 @@ _ENV_DICT = { "UNRECOGNIZED_FUNC": "$(nope :" + _TEST_DEP_TARGET_NAME + ")", } +# buildifier: disable=unsorted-dict-items _TOOLCHAIN_DICT = { "TOOLCHAIN_ENV_VAR": "flag_value", "TOOLCHAIN_ENV_VAR2": "flag_value_2", "TOOLCHAIN_TO_LOCATION_ENV_VAR": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "TOOLCHAIN_INDIRECT_ENV_VAR_RAW": "$TOOLCHAIN_ENV_VAR", + "TOOLCHAIN_INDIRECT_ENV_VAR_PAREN": "$(TOOLCHAIN_ENV_VAR)", + "TOOLCHAIN_INDIRECT_ENV_VAR_CURLY": "${TOOLCHAIN_ENV_VAR}", "BACK_TO_ENV_DICT_VAR_RAW": "$SIMPLE_VAL", "BACK_TO_ENV_DICT_VAR_PAREN": "$(SIMPLE_VAL)", "BACK_TO_ENV_DICT_VAR_CURLY": "${SIMPLE_VAL}", @@ -96,6 +104,7 @@ _TOOLCHAIN_DICT = { # Test expected output dicts +# buildifier: disable=unsorted-dict-items _EXPECTED_RESOLVED_DICT_NO_LOCATION = { "SIMPLE_VAL": "hello_world", "ESCAPED_SIMPLE_VAL": "$$SIMPLE_VAL", @@ -109,6 +118,9 @@ _EXPECTED_RESOLVED_DICT_NO_LOCATION = { "TOOLCHAIN_ENV_VAR2_RAW": "flag_value_2", "TOOLCHAIN_ENV_VAR2_PAREN": "flag_value_2", "TOOLCHAIN_ENV_VAR2_CURLY": "flag_value_2", + "TOOLCHAIN_INDIRECT_ENV_VAR_RAW": "flag_value", + "TOOLCHAIN_INDIRECT_ENV_VAR_PAREN": "flag_value", + "TOOLCHAIN_INDIRECT_ENV_VAR_CURLY": "flag_value", "TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": "$(location :" + _TEST_DEP_TARGET_NAME + ")", "TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": "$(location :" + _TEST_DEP_TARGET_NAME + ")", "TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": "$(location :" + _TEST_DEP_TARGET_NAME + ")", @@ -170,6 +182,7 @@ _EXPECTED_RESOLVED_DICT_NO_LOCATION = { "UNRECOGNIZED_FUNC": "$(nope :" + _TEST_DEP_TARGET_NAME + ")", } +# buildifier: disable=unsorted-dict-items _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION, **{ "LOCATION_VAL": _MOCK_LOCATION_PATH_OF_DUMMY, "EXECPATH_VAL": _MOCK_EXECPATH_PATH_OF_DUMMY, @@ -219,6 +232,7 @@ _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_L ), }) +# buildifier: disable=unsorted-dict-items _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION, **{ "LOCATION_VAL": _GENRULE_LOCATION_PATH_OF_DUMMY, "EXECPATH_VAL": _GENRULE_EXECPATH_PATH_OF_DUMMY, @@ -268,6 +282,43 @@ _EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION = dict(_EXPECTED_RESOLVED_DICT_NO_ ), }) +# "No recursion" dicts leave most ("TO_ENV_DICT" and "INDIRECT") entries unexpanded. +# buildifier: disable=unsorted-dict-items +_EXPECTED_RESOLVED_DICT_NO_LOCATION_NO_RECURSION = dict(_ENV_DICT, **{ + "SIMPLE_VAL": "hello_world", + "ESCAPED_SIMPLE_VAL": "$$SIMPLE_VAL", + "LOCATION_VAL": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "EXECPATH_VAL": "$(execpath :" + _TEST_DEP_TARGET_NAME + ")", + "ROOTPATH_VAL": "$(rootpath :" + _TEST_DEP_TARGET_NAME + ")", + "RLOCATIONPATH_VAL": "$(rlocationpath :" + _TEST_DEP_TARGET_NAME + ")", + "TOOLCHAIN_ENV_VAR_RAW": "flag_value", + "TOOLCHAIN_ENV_VAR_PAREN": "flag_value", + "TOOLCHAIN_ENV_VAR_CURLY": "flag_value", + "TOOLCHAIN_ENV_VAR2_RAW": "flag_value_2", + "TOOLCHAIN_ENV_VAR2_PAREN": "flag_value_2", + "TOOLCHAIN_ENV_VAR2_CURLY": "flag_value_2", + "TOOLCHAIN_INDIRECT_ENV_VAR_RAW": "flag_value", + "TOOLCHAIN_INDIRECT_ENV_VAR_PAREN": "flag_value", + "TOOLCHAIN_INDIRECT_ENV_VAR_CURLY": "flag_value", + "TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": "$(location :" + _TEST_DEP_TARGET_NAME + ")", + "TOOLCHAIN_TO_ENV_DICT_ENV_VAR_RAW": "$SIMPLE_VAL", + "TOOLCHAIN_TO_ENV_DICT_ENV_VAR_PAREN": "$(SIMPLE_VAL)", + "TOOLCHAIN_TO_ENV_DICT_ENV_VAR_CURLY": "${SIMPLE_VAL}", +}) + +# buildifier: disable=unsorted-dict-items +_EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION_NO_RECURSION = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION_NO_RECURSION, **{ + "LOCATION_VAL": _MOCK_LOCATION_PATH_OF_DUMMY, + "EXECPATH_VAL": _MOCK_EXECPATH_PATH_OF_DUMMY, + "ROOTPATH_VAL": _MOCK_ROOTPATH_PATH_OF_DUMMY, + "RLOCATIONPATH_VAL": _MOCK_RLOCATIONPATH_PATH_OF_DUMMY, + "TOOLCHAIN_TO_LOCATION_ENV_VAR_RAW": _MOCK_LOCATION_PATH_OF_DUMMY, + "TOOLCHAIN_TO_LOCATION_ENV_VAR_PAREN": _MOCK_LOCATION_PATH_OF_DUMMY, + "TOOLCHAIN_TO_LOCATION_ENV_VAR_CURLY": _MOCK_LOCATION_PATH_OF_DUMMY, +}) + # Unresolved/unterminated validation test input values and expected values _UNRESOLVED_SUBSTRINGS = [ @@ -384,7 +435,7 @@ _test_toolchain = rule( implementation = _test_toolchain_impl, ) -def _mock_expand_location(input_str): +def _mock_wrapped_expand_location(input_str): return input_str.replace( "$(location :" + _TEST_DEP_TARGET_NAME + ")", _MOCK_LOCATION_PATH_OF_DUMMY, @@ -410,14 +461,14 @@ def _fix_platform_dependent_path_for_assertions(platform_dependent_val): # Test cases -def _expand_with_manual_dict_test_impl(ctx): - """Test `expansion.expand_with_manual_dict()`""" +def _expand_dict_strings_with_manual_dict_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_manual_dict()`""" env = unittest.begin(ctx) env_dict_copy = dict(_ENV_DICT) toolchain_dict_copy = dict(_TOOLCHAIN_DICT) - resolved_dict = expansion.expand_with_manual_dict(_TOOLCHAIN_DICT, _ENV_DICT) + resolved_dict = expansion.expand_dict_strings_with_manual_dict(_TOOLCHAIN_DICT, _ENV_DICT) # Check that the inputs are not mutated. asserts.equals(env, env_dict_copy, _ENV_DICT) @@ -439,17 +490,17 @@ def _expand_with_manual_dict_test_impl(ctx): return unittest.end(env) -_expand_with_manual_dict_test = unittest.make(_expand_with_manual_dict_test_impl) +_expand_dict_strings_with_manual_dict_test = unittest.make(_expand_dict_strings_with_manual_dict_test_impl) -def _expand_with_manual_dict_and_location_test_impl(ctx): - """Test `expansion.expand_with_manual_dict_and_location()`""" +def _expand_dict_strings_with_manual_dict_and_location_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_manual_dict_and_location()`""" env = unittest.begin(ctx) env_dict_copy = dict(_ENV_DICT) toolchain_dict_copy = dict(_TOOLCHAIN_DICT) - resolved_dict = expansion.expand_with_manual_dict_and_location( - _mock_expand_location, + resolved_dict = expansion.expand_dict_strings_with_manual_dict_and_location( + _mock_wrapped_expand_location, _TOOLCHAIN_DICT, _ENV_DICT, ) @@ -474,18 +525,18 @@ def _expand_with_manual_dict_and_location_test_impl(ctx): return unittest.end(env) -_expand_with_manual_dict_and_location_test = unittest.make( - _expand_with_manual_dict_and_location_test_impl, +_expand_dict_strings_with_manual_dict_and_location_test = unittest.make( + _expand_dict_strings_with_manual_dict_and_location_test_impl, ) -def _expand_with_toolchains_test_impl(ctx): - """Test `expansion.expand_with_toolchains()` without extra dict""" +def _expand_dict_strings_with_toolchains_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_toolchains()` without extra dict""" env = unittest.begin(ctx) env_dict_copy = dict(_ENV_DICT) toolchain_dict_copy = dict(env.ctx.var) - resolved_dict = expansion.expand_with_toolchains(env.ctx, _ENV_DICT) + resolved_dict = expansion.expand_dict_strings_with_toolchains(env.ctx, _ENV_DICT) # Check that the inputs are not mutated. asserts.equals(env, env_dict_copy, _ENV_DICT) @@ -507,21 +558,22 @@ def _expand_with_toolchains_test_impl(ctx): return unittest.end(env) -_expand_with_toolchains_test = unittest.make(_expand_with_toolchains_test_impl) +_expand_dict_strings_with_toolchains_test = unittest.make(_expand_dict_strings_with_toolchains_test_impl) -def _expand_with_toolchains_with_additional_dict_test_impl(ctx): - """Test `expansion.expand_with_toolchains()` with extra dict""" +def _expand_dict_strings_with_toolchains_with_additional_dict_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_toolchains()` with extra dict""" env = unittest.begin(ctx) env_dict_copy = dict(_ENV_DICT) toolchain_dict_copy = dict(env.ctx.var) + # buildifier: disable=unsorted-dict-items additional_lookup_dict = { "TOOLCHAIN_ENV_VAR2": "expanded from additional dict instead", "NOPE": "naw, it's fine now.", } - resolved_dict = expansion.expand_with_toolchains( + resolved_dict = expansion.expand_dict_strings_with_toolchains( env.ctx, _ENV_DICT, additional_lookup_dict = additional_lookup_dict, @@ -534,6 +586,7 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) + # buildifier: disable=unsorted-dict-items updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION, **{ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", @@ -554,18 +607,18 @@ def _expand_with_toolchains_with_additional_dict_test_impl(ctx): return unittest.end(env) -_expand_with_toolchains_with_additional_dict_test = unittest.make( - _expand_with_toolchains_with_additional_dict_test_impl, +_expand_dict_strings_with_toolchains_with_additional_dict_test = unittest.make( + _expand_dict_strings_with_toolchains_with_additional_dict_test_impl, ) -def _expand_with_toolchains_attr_test_impl(ctx): - """Test `expansion.expand_with_toolchains_attr()` without extra dict""" +def _expand_dict_strings_with_toolchains_attr_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_toolchains_attr()` without extra dict""" env = unittest.begin(ctx) env_dict_copy = dict(env.ctx.attr.env) toolchain_dict_copy = dict(env.ctx.var) - resolved_dict = expansion.expand_with_toolchains_attr(env.ctx) + resolved_dict = expansion.expand_dict_strings_with_toolchains_attr(env.ctx) # Check that the inputs are not mutated. asserts.equals(env, env_dict_copy, env.ctx.attr.env) @@ -587,26 +640,27 @@ def _expand_with_toolchains_attr_test_impl(ctx): return unittest.end(env) -_expand_with_toolchains_attr_test = unittest.make( - _expand_with_toolchains_attr_test_impl, +_expand_dict_strings_with_toolchains_attr_test = unittest.make( + _expand_dict_strings_with_toolchains_attr_test_impl, attrs = { "env": attr.string_dict(), }, ) -def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): - """Test `expansion.expand_with_toolchains_attr()` with extra dict""" +def _expand_dict_strings_with_toolchains_attr_with_additional_dict_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_toolchains_attr()` with extra dict""" env = unittest.begin(ctx) env_dict_copy = dict(env.ctx.attr.env) toolchain_dict_copy = dict(env.ctx.var) + # buildifier: disable=unsorted-dict-items additional_lookup_dict = { "TOOLCHAIN_ENV_VAR2": "expanded from additional dict instead", "NOPE": "naw, it's fine now.", } - resolved_dict = expansion.expand_with_toolchains_attr( + resolved_dict = expansion.expand_dict_strings_with_toolchains_attr( env.ctx, additional_lookup_dict = additional_lookup_dict, ) @@ -618,6 +672,7 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) + # buildifier: disable=unsorted-dict-items updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_NO_LOCATION, **{ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", @@ -638,21 +693,21 @@ def _expand_with_toolchains_attr_with_additional_dict_test_impl(ctx): return unittest.end(env) -_expand_with_toolchains_attr_with_additional_dict_test = unittest.make( - _expand_with_toolchains_attr_with_additional_dict_test_impl, +_expand_dict_strings_with_toolchains_attr_with_additional_dict_test = unittest.make( + _expand_dict_strings_with_toolchains_attr_with_additional_dict_test_impl, attrs = { "env": attr.string_dict(), }, ) -def _expand_with_toolchains_and_location_test_impl(ctx): - """Test `expansion.expand_with_toolchains_and_location()` without extra dict""" +def _expand_dict_strings_with_toolchains_and_location_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_toolchains_and_location()` without extra dict""" env = unittest.begin(ctx) env_dict_copy = dict(_ENV_DICT) toolchain_dict_copy = dict(env.ctx.var) - resolved_dict = expansion.expand_with_toolchains_and_location( + resolved_dict = expansion.expand_dict_strings_with_toolchains_and_location( env.ctx, [ctx.attr.target], _ENV_DICT, @@ -678,26 +733,27 @@ def _expand_with_toolchains_and_location_test_impl(ctx): return unittest.end(env) -_expand_with_toolchains_and_location_test = unittest.make( - _expand_with_toolchains_and_location_test_impl, +_expand_dict_strings_with_toolchains_and_location_test = unittest.make( + _expand_dict_strings_with_toolchains_and_location_test_impl, attrs = { "target": attr.label(), }, ) -def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): - """Test `expansion.expand_with_toolchains_and_location()` with extra dict""" +def _expand_dict_strings_with_toolchains_and_location_with_additional_dict_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_toolchains_and_location()` with extra dict""" env = unittest.begin(ctx) env_dict_copy = dict(_ENV_DICT) toolchain_dict_copy = dict(env.ctx.var) + # buildifier: disable=unsorted-dict-items additional_lookup_dict = { "TOOLCHAIN_ENV_VAR2": "expanded from additional dict instead", "NOPE": "naw, it's fine now.", } - resolved_dict = expansion.expand_with_toolchains_and_location( + resolved_dict = expansion.expand_dict_strings_with_toolchains_and_location( env.ctx, [ctx.attr.target], _ENV_DICT, @@ -711,6 +767,7 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): # Check that the output has exact same key set as original input. asserts.equals(env, _ENV_DICT.keys(), resolved_dict.keys()) + # buildifier: disable=unsorted-dict-items updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION, **{ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", @@ -731,21 +788,21 @@ def _expand_with_toolchains_and_location_with_additional_dict_test_impl(ctx): return unittest.end(env) -_expand_with_toolchains_and_location_with_additional_dict_test = unittest.make( - _expand_with_toolchains_and_location_with_additional_dict_test_impl, +_expand_dict_strings_with_toolchains_and_location_with_additional_dict_test = unittest.make( + _expand_dict_strings_with_toolchains_and_location_with_additional_dict_test_impl, attrs = { "target": attr.label(), }, ) -def _expand_with_toolchains_and_location_attr_test_impl(ctx): - """Test `expansion.expand_with_toolchains_and_location_attr()` without extra dict""" +def _expand_dict_strings_with_toolchains_and_location_attr_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_toolchains_and_location_attr()` without extra dict""" env = unittest.begin(ctx) env_dict_copy = dict(env.ctx.attr.env) toolchain_dict_copy = dict(env.ctx.var) - resolved_dict = expansion.expand_with_toolchains_and_location_attr(env.ctx) + resolved_dict = expansion.expand_dict_strings_with_toolchains_and_location_attr(env.ctx) # Check that the inputs are not mutated. asserts.equals(env, env_dict_copy, env.ctx.attr.env) @@ -767,27 +824,28 @@ def _expand_with_toolchains_and_location_attr_test_impl(ctx): return unittest.end(env) -_expand_with_toolchains_and_location_attr_test = unittest.make( - _expand_with_toolchains_and_location_attr_test_impl, +_expand_dict_strings_with_toolchains_and_location_attr_test = unittest.make( + _expand_dict_strings_with_toolchains_and_location_attr_test_impl, attrs = { "deps": attr.label_list(), "env": attr.string_dict(), }, ) -def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx): - """Test `expansion.expand_with_toolchains_and_location_attr()` with extra dict""" +def _expand_dict_strings_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_toolchains_and_location_attr()` with extra dict""" env = unittest.begin(ctx) env_dict_copy = dict(env.ctx.attr.env) toolchain_dict_copy = dict(env.ctx.var) + # buildifier: disable=unsorted-dict-items additional_lookup_dict = { "TOOLCHAIN_ENV_VAR2": "expanded from additional dict instead", "NOPE": "naw, it's fine now.", } - resolved_dict = expansion.expand_with_toolchains_and_location_attr( + resolved_dict = expansion.expand_dict_strings_with_toolchains_and_location_attr( env.ctx, additional_lookup_dict = additional_lookup_dict, ) @@ -799,6 +857,7 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx # Check that the output has exact same key set as original input. asserts.equals(env, env.ctx.attr.env.keys(), resolved_dict.keys()) + # buildifier: disable=unsorted-dict-items updated_expected_dict = dict(_EXPECTED_RESOLVED_DICT_WITH_GENRULE_LOCATION, **{ "TOOLCHAIN_ENV_VAR2_RAW": "expanded from additional dict instead", "TOOLCHAIN_ENV_VAR2_PAREN": "expanded from additional dict instead", @@ -819,14 +878,86 @@ def _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl(ctx return unittest.end(env) -_expand_with_toolchains_and_location_attr_with_additional_dict_test = unittest.make( - _expand_with_toolchains_and_location_attr_with_additional_dict_test_impl, +_expand_dict_strings_with_toolchains_and_location_attr_with_additional_dict_test = unittest.make( + _expand_dict_strings_with_toolchains_and_location_attr_with_additional_dict_test_impl, attrs = { "deps": attr.label_list(), "env": attr.string_dict(), }, ) +def _expand_list_strings_with_manual_dict_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_manual_dict()`""" + env = unittest.begin(ctx) + + env_dict_values = _ENV_DICT.values() + env_dict_values_copy = list(env_dict_values) + toolchain_dict_copy = dict(_TOOLCHAIN_DICT) + + resolved_list = expansion.expand_list_strings_with_manual_dict(_TOOLCHAIN_DICT, env_dict_values) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_values_copy, env_dict_values) + asserts.equals(env, toolchain_dict_copy, _TOOLCHAIN_DICT) + + # Check that the output has exact same length as original input. + asserts.equals(env, len(env_dict_values), len(resolved_list)) + + # Check all output resolved values against expected resolved values. + for env_key, env_orig_val in _ENV_DICT.items(): + index_in_unexpanded = env_dict_values.index(env_orig_val) + expected_val = _EXPECTED_RESOLVED_DICT_NO_LOCATION_NO_RECURSION[env_key] + resolved_val = resolved_list[index_in_unexpanded] + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_list_strings_with_manual_dict_test = unittest.make(_expand_list_strings_with_manual_dict_test_impl) + +def _expand_list_strings_with_manual_dict_and_location_test_impl(ctx): + """Test `expansion.expand_dict_strings_with_manual_dict_and_location()`""" + env = unittest.begin(ctx) + + env_dict_values = _ENV_DICT.values() + env_dict_values_copy = list(env_dict_values) + toolchain_dict_copy = dict(_TOOLCHAIN_DICT) + + resolved_list = expansion.expand_list_strings_with_manual_dict_and_location( + _mock_wrapped_expand_location, + _TOOLCHAIN_DICT, + env_dict_values, + ) + + # Check that the inputs are not mutated. + asserts.equals(env, env_dict_values_copy, env_dict_values) + asserts.equals(env, toolchain_dict_copy, _TOOLCHAIN_DICT) + + # Check that the output has exact same length as original input. + asserts.equals(env, len(env_dict_values), len(resolved_list)) + + # Check all output resolved values against expected resolved values. + for env_key, env_orig_val in _ENV_DICT.items(): + index_in_unexpanded = env_dict_values.index(env_orig_val) + expected_val = _EXPECTED_RESOLVED_DICT_WITH_MOCKED_LOCATION_NO_RECURSION[env_key] + resolved_val = resolved_list[index_in_unexpanded] + resolved_val = _fix_platform_dependent_path_for_assertions(resolved_val) + + # Replace `expected` with proper value given bazel version (may differ for rlocation path). + if "_main" in expected_val and "_main" not in resolved_val: + expected_val = expected_val.replace("_main", "bazel_skylib") + asserts.equals(env, expected_val, resolved_val) + + return unittest.end(env) + +_expand_list_strings_with_manual_dict_and_location_test = unittest.make( + _expand_list_strings_with_manual_dict_and_location_test_impl, +) + def _validate_expansions_on_fully_resolved_values_test_impl(ctx): """Test `expansion.validate_expansions()` with fully resolved strings""" env = unittest.begin(ctx) @@ -949,52 +1080,58 @@ def expansion_test_suite(): name = "expansion_tests__test_toolchain", ) - _expand_with_manual_dict_test( - name = "expansion_tests__expand_with_manual_dict_test", + _expand_dict_strings_with_manual_dict_test( + name = "expansion_tests__expand_dict_strings_with_manual_dict_test", ) - _expand_with_manual_dict_and_location_test( - name = "expansion_tests__expand_with_manual_dict_and_location_test", + _expand_dict_strings_with_manual_dict_and_location_test( + name = "expansion_tests__expand_dict_strings_with_manual_dict_and_location_test", ) - _expand_with_toolchains_test( - name = "expansion_tests__expand_with_toolchains_test", + _expand_dict_strings_with_toolchains_test( + name = "expansion_tests__expand_dict_strings_with_toolchains_test", toolchains = [":expansion_tests__test_toolchain"], ) - _expand_with_toolchains_with_additional_dict_test( - name = "expansion_tests__expand_with_toolchains_with_additional_dict_test", + _expand_dict_strings_with_toolchains_with_additional_dict_test( + name = "expansion_tests__expand_dict_strings_with_toolchains_with_additional_dict_test", toolchains = [":expansion_tests__test_toolchain"], ) - _expand_with_toolchains_attr_test( - name = "expansion_tests__expand_with_toolchains_attr_test", + _expand_dict_strings_with_toolchains_attr_test( + name = "expansion_tests__expand_dict_strings_with_toolchains_attr_test", env = _ENV_DICT, toolchains = [":expansion_tests__test_toolchain"], ) - _expand_with_toolchains_attr_with_additional_dict_test( - name = "expansion_tests__expand_with_toolchains_attr_with_additional_dict_test", + _expand_dict_strings_with_toolchains_attr_with_additional_dict_test( + name = "expansion_tests__expand_dict_strings_with_toolchains_attr_with_additional_dict_test", env = _ENV_DICT, toolchains = [":expansion_tests__test_toolchain"], ) - _expand_with_toolchains_and_location_test( - name = "expansion_tests__expand_with_toolchains_and_location_test", + _expand_dict_strings_with_toolchains_and_location_test( + name = "expansion_tests__expand_dict_strings_with_toolchains_and_location_test", target = ":" + _TEST_DEP_TARGET_NAME, toolchains = [":expansion_tests__test_toolchain"], ) - _expand_with_toolchains_and_location_with_additional_dict_test( - name = "expansion_tests__expand_with_toolchains_and_location_with_additional_dict_test", + _expand_dict_strings_with_toolchains_and_location_with_additional_dict_test( + name = "expansion_tests__expand_dict_strings_with_toolchains_and_location_with_additional_dict_test", target = ":" + _TEST_DEP_TARGET_NAME, toolchains = [":expansion_tests__test_toolchain"], ) - _expand_with_toolchains_and_location_attr_test( - name = "expansion_tests__expand_with_toolchains_and_location_attr_test", + _expand_dict_strings_with_toolchains_and_location_attr_test( + name = "expansion_tests__expand_dict_strings_with_toolchains_and_location_attr_test", deps = [":" + _TEST_DEP_TARGET_NAME], env = _ENV_DICT, toolchains = [":expansion_tests__test_toolchain"], ) - _expand_with_toolchains_and_location_attr_with_additional_dict_test( - name = "expansion_tests__expand_with_toolchains_and_location_attr_with_additional_dict_test", + _expand_dict_strings_with_toolchains_and_location_attr_with_additional_dict_test( + name = "expansion_tests__expand_dict_strings_with_toolchains_and_location_attr_with_additional_dict_test", deps = [":" + _TEST_DEP_TARGET_NAME], env = _ENV_DICT, toolchains = [":expansion_tests__test_toolchain"], ) + _expand_list_strings_with_manual_dict_test( + name = "expansion_tests__expand_list_strings_with_manual_dict_test", + ) + _expand_list_strings_with_manual_dict_and_location_test( + name = "expansion_tests__expand_list_strings_with_manual_dict_and_location_test", + ) _validate_expansions_on_fully_resolved_values_test( name = "expansion_tests__validate_expansions_on_fully_resolved_values_test", ) @@ -1031,16 +1168,18 @@ def expansion_test_suite(): native.test_suite( name = "expansion_tests", tests = [ - ":expansion_tests__expand_with_manual_dict_test", - ":expansion_tests__expand_with_manual_dict_and_location_test", - ":expansion_tests__expand_with_toolchains_test", - ":expansion_tests__expand_with_toolchains_with_additional_dict_test", - ":expansion_tests__expand_with_toolchains_attr_test", - ":expansion_tests__expand_with_toolchains_attr_with_additional_dict_test", - ":expansion_tests__expand_with_toolchains_and_location_test", - ":expansion_tests__expand_with_toolchains_and_location_with_additional_dict_test", - ":expansion_tests__expand_with_toolchains_and_location_attr_test", - ":expansion_tests__expand_with_toolchains_and_location_attr_with_additional_dict_test", + ":expansion_tests__expand_dict_strings_with_manual_dict_test", + ":expansion_tests__expand_dict_strings_with_manual_dict_and_location_test", + ":expansion_tests__expand_dict_strings_with_toolchains_test", + ":expansion_tests__expand_dict_strings_with_toolchains_with_additional_dict_test", + ":expansion_tests__expand_dict_strings_with_toolchains_attr_test", + ":expansion_tests__expand_dict_strings_with_toolchains_attr_with_additional_dict_test", + ":expansion_tests__expand_dict_strings_with_toolchains_and_location_test", + ":expansion_tests__expand_dict_strings_with_toolchains_and_location_with_additional_dict_test", + ":expansion_tests__expand_dict_strings_with_toolchains_and_location_attr_test", + ":expansion_tests__expand_dict_strings_with_toolchains_and_location_attr_with_additional_dict_test", + ":expansion_tests__expand_list_strings_with_manual_dict_test", + ":expansion_tests__expand_list_strings_with_manual_dict_and_location_test", ":expansion_tests__validate_expansions_on_fully_resolved_values_test", ":expansion_tests__validate_expansions_on_unresolved_values_test", ":expansion_tests__validate_expansions_on_two_unresolved_values_test",