fix: align compilers, analyzers and the decompiler with what the API Management gateway accepts

docs/ExpressionHelpers.md

@@ -62,7 +62,7 @@ if (IsCompanyIp(context.ExpressionContext) && !IsHealthCheck(context.ExpressionC
 
 A policy configuration argument may be a call to a factory method, also from a referenced project. The factory must
 return a single `new TConfig { ... }` expression and take no parameters other than policy section contexts, such as
-`IInboundContext`.
+`IInboundContext`, or `IHaveExpressionContext`.
 
 ```csharp
 context.RateLimitByKey(Limits.PerProduct(context));
@@ -94,6 +94,44 @@ public static class TextHelpers
 Helpers are inlined from source, so the library must be a project reference. Calling a helper, property or
 non-constant field of a compiled library is reported, although its constants can still be used.
 
+## What the analyzer checks
+
+The analyzer reports code that API Management rejects when the policy is saved: types and members outside the set it
+allows in policy expressions (`APIM001`, `APIM002`), and type names it can't resolve on their own (`APIM003`).
+
+It checks:
+
+- every helper declared in a policy document or fragment class, including helpers in a class nested inside it.
+  A helper it inherits is checked when the base class is a document itself, not otherwise. A helper is a method
+  that takes the expression context, or one the document's sections, policy configuration factories or other
+  helpers call; it may take a section context instead, whose `ExpressionContext` is the gateway's `context`. The
+  `[Expression]` attribute isn't needed for these.
+- methods marked `[Expression]` and every method of a class marked `[Expression]`.
+- expression lambdas passed to a policy.
+
+A helper in any other class is only checked when its class is marked `[Expression]`. Code marked `[Expression]` may
+only call document helpers and other marked helpers, while an unmarked document helper may call any helper declared in
+source.
+
+The starting points are the sections, the configuration factories, the methods marked `[Expression]` and the helpers
+that take the expression context or a section context, of every document and helper library in the project. A
+section is what the compiler compiles: `Inbound`, `Outbound`, `Backend` or `OnError` of a document, or the first
+`Fragment` method of a fragment, taking one section context. A method of a document that none of them reaches, such as a `ToString()` override or a `void` method that isn't a
+section, never becomes a policy expression and isn't checked, nor are the helpers only it calls. Code of another
+document or library reaches a helper when it names the document, `Shared.Helper()`, directly or through a using alias.
+Calls are matched by name, so an overload of a helper is checked with it. Properties declared in source can't be used
+as helpers; the compiler reports them.
+
+### Ambiguous type names
+
+API Management resolves type names against all the namespaces it imports, whatever the `using` directives of the
+source file are. `Formatting` exists in both `Newtonsoft.Json` and `System.Xml`, so it has to be written with its
+namespace:
+
+```csharp
+JsonConvert.SerializeObject(value, Newtonsoft.Json.Formatting.Indented)
+```
+
 ## Named values
 
 `context.NamedValue("name")` compiles to a raw `{{name}}` token, so the named value is used as code.
@@ -116,7 +154,9 @@ non-constant field of a compiled library is reported, although its constants can
   pair of parentheses to keep them.
 - At either end of an interpolation hole a named value is always parenthesized, `{({{name}})}`, so it can't run into
   the hole's braces.
-- An expression that is only a named value compiles to the plain attribute value, `{{name}}`.
+- An expression that is only a named value, returned from a `string` helper or a `[NamedValue]` helper, compiles to
+  the plain attribute value, `{{name}}`. Returned from a helper of another type it stays an expression, `@({{name}})`,
+  which the gateway evaluates as code.
 
 ## C# language version
 

docs/OutputFormat.md

@@ -21,6 +21,27 @@ The generated document is only valid for the matching API Management content for
 `Microsoft.ApiManagement/service/apis/policies` or `.../policyFragments` Bicep resource has a `format` property that
 must be set to the same value (`rawxml` or `xml`) you compiled with.
 
+## Decompiling
+
+The decompiler takes the same option for the format of its input, with the same default:
+
+```shell
+# default: rawxml
+dotnet azure-apim-policy-decompiler --s .\target --o .\src
+
+# policies written or exported as xml
+dotnet azure-apim-policy-decompiler --s .\target --o .\src --policy-format xml
+```
+
+| `--policy-format` | How a policy is read |
+| --- | --- |
+| `rawxml` (default) | A value that begins with an expression is C# as it is written: an entity in it, such as the string `"&"`, is kept. Everything else is XML, an `@(...)` further into a value or anywhere in the markup of a liquid `set-body` included |
+| `xml` | The document has to be well-formed XML, and an entity in an expression is decoded like any other |
+
+Give the format the policies were compiled or exported in. A `rawxml` policy read as `xml` usually isn't
+well-formed XML and fails to parse; an `xml` policy read as `rawxml` decompiles to expressions that keep their
+entities and don't parse as C#. The decompiler reports either and names this option.
+
 ## Which format should I use?
 
 - **`rawxml` (default)** keeps expressions verbatim, so the generated files stay readable — including multi-line

src/Analyzers/ExpressionDefinition.Analyzer.cs

@@ -26,6 +26,13 @@ public override void Initialize(AnalysisContext context)
             SyntaxKind.ParenthesizedLambdaExpression);
     }
 
+    // The types an expression helper can be declared to return. The gateway types the expression itself and
+    // does not convert its value, so this follows the types it accepts as the value of a variable, with two
+    // additions the set-variable compiler reports there: System.Uri and enums, which a helper may return for use
+    // inside a larger expression. object is here because the decompiler declares the helper of a set-variable
+    // value as object. Nullable and array forms and enums are handled in IsAllowedReturnType: a nullable type is
+    // allowed when its type is, although the gateway's list omits bool?, sbyte? and TimeSpan?, which haven't been
+    // probed.
     private readonly static IReadOnlyCollection<string> AllowedExpressionReturnTypes = new HashSet<string>()
     {
         "System.Boolean",
@@ -39,14 +46,42 @@ public override void Initialize(AnalysisContext context)
         "System.Int16",
         "System.Int32",
         "System.Int64",
+        "System.Object",
+        "System.SByte",
+        "System.Single",
         "System.String",
+        "System.TimeSpan",
         "System.UInt16",
         "System.UInt32",
         "System.UInt64",
         "System.Uri",
+        "Newtonsoft.Json.Linq.JArray",
+        "Newtonsoft.Json.Linq.JConstructor",
+        "Newtonsoft.Json.Linq.JContainer",
         "Newtonsoft.Json.Linq.JObject",
+        "Newtonsoft.Json.Linq.JProperty",
+        "Newtonsoft.Json.Linq.JRaw",
+        "Newtonsoft.Json.Linq.JToken",
+        "Newtonsoft.Json.Linq.JValue",
     };
 
+    private static bool IsAllowedReturnType(ITypeSymbol type)
+    {
+        switch (type)
+        {
+            case { TypeKind: TypeKind.Enum }:
+                return true;
+            // the gateway accepts byte[] and sbyte[] as the value of an expression, no other array
+            case IArrayTypeSymbol { Rank: 1 } array:
+                return array.ElementType.SpecialType is
+                    SpecialType.System_Byte or SpecialType.System_SByte;
+            case INamedTypeSymbol { OriginalDefinition.SpecialType: SpecialType.System_Nullable_T } nullable:
+                return IsAllowedReturnType(nullable.TypeArguments[0]);
+            default:
+                return AllowedExpressionReturnTypes.Contains(type.ToFullyQualifiedString());
+        }
+    }
+
     private const string ContextParamType =
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IExpressionContext";
 
@@ -74,7 +109,7 @@ private static void AnalyzeMethod(SyntaxNodeAnalysisContext context)
         else
         {
             var fullTypeName = type.ToFullyQualifiedString();
-            if (!AllowedExpressionReturnTypes.Contains(fullTypeName))
+            if (!IsAllowedReturnType(type))
             {
                 var diagnostic = Diagnostic.Create(Rules.Expression.ReturnTypeNotAllowed,
                     method.ReturnType.GetLocation(), fullTypeName);

src/Analyzers/ExpressionTypeUsed.Analyzer.cs

@@ -27,6 +27,37 @@ public override void Initialize(AnalysisContext context)
             SyntaxKind.ObjectCreationExpression,
             SyntaxKind.ObjectInitializerExpression,
             SyntaxKind.AnonymousObjectCreationExpression);
+        context.RegisterSyntaxNodeAction(AnalyzeTypeName, SyntaxKind.IdentifierName);
+    }
+
+    // API Management resolves type names against every namespace it imports, so a name declared in two of them is
+    // rejected unless it is written with its namespace, whatever the using directives of the source file are.
+    private readonly static IReadOnlyDictionary<string, IReadOnlyCollection<string>> AmbiguousTypeNames =
+        new Dictionary<string, IReadOnlyCollection<string>>()
+        {
+            { "Formatting", new HashSet<string>() { "Newtonsoft.Json.Formatting", "System.Xml.Formatting" } },
+        };
+
+    private static void AnalyzeTypeName(SyntaxNodeAnalysisContext context)
+    {
+        var name = (IdentifierNameSyntax)context.Node;
+        if (!AmbiguousTypeNames.TryGetValue(name.Identifier.ValueText, out var candidates) ||
+            name.Parent is MemberAccessExpressionSyntax memberAccess && memberAccess.Name == name ||
+            name.Parent is QualifiedNameSyntax qualifiedName && qualifiedName.Right == name ||
+            name.Parent is AliasQualifiedNameSyntax ||
+            !IsPartOfPolicyExpression(context))
+        {
+            return;
+        }
+
+        // The compiler writes a using alias out as the name it stands for, so that is not a bare name.
+        if (context.SemanticModel.GetAliasInfo(name) is null &&
+            context.SemanticModel.GetSymbolInfo(name).Symbol is INamedTypeSymbol type &&
+            candidates.Contains(type.ToFullyQualifiedString()))
+        {
+            context.ReportDiagnostic(Diagnostic.Create(Rules.TypeUsed.AmbiguousTypeName, name.GetLocation(),
+                name.Identifier.ValueText, type.ToFullyQualifiedString()));
+        }
     }
 
     private readonly static IReadOnlyCollection<string> AllowedTypes = new HashSet<string>()
@@ -174,12 +205,21 @@ public override void Initialize(AnalysisContext context)
 
         #region Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions
 
+        "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.Authorization",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.BasicAuthCredentials",
+        "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.ByteArrayExtensions",
+        "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.CarbonIntensityCategory",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.DictionaryExtensions",
+        "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IFoundry",
+        "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.ISustainabilityInfo",
+        "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.X509Certificate2Extensions",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IApi",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IExpressionContext",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IContextApi",
+        "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IBackend",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IDeployment",
+        "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IGateway",
+        "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IWorkspace",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IGroup",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.Jwt",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.ILastError",
@@ -188,6 +228,7 @@ public override void Initialize(AnalysisContext context)
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IOperation",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IPrivateEndpointConnection",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IProduct",
+        "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.ProductState",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IRequest",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.IResponse",
         "Microsoft.Azure.ApiManagement.PolicyToolkit.Authoring.Expressions.ISubscription",
@@ -204,6 +245,9 @@ public override void Initialize(AnalysisContext context)
         new Dictionary<string, IReadOnlyCollection<string>>()
         {
             { "Newtonsoft.Json.JsonConvert", new HashSet<string>() { "SerializeObject", "DeserializeObject" } },
+            // The gateway accepts the type but none of its members: ToString(), Equals() or GetType() called on
+            // a value typed object, or on a type that doesn't declare them itself, is rejected.
+            { "System.Object", new HashSet<string>() },
             {
                 "System.DateTime", new HashSet<string>()
                 {
@@ -287,10 +331,7 @@ private static void AnalyzeNode(SyntaxNodeAnalysisContext context)
     {
         var node = context.Node;
 
-        var underUnderExpressionMethod = node.IsPartOfPolicyExpressionMethod(context.SemanticModel);
-        var underUnderExpressionLambda = node.IsPartOfPolicyExpressionDelegate(context.SemanticModel);
-
-        if (!underUnderExpressionMethod && !underUnderExpressionLambda)
+        if (!IsPartOfPolicyExpression(context))
         {
             return;
         }
@@ -307,6 +348,33 @@ private static void AnalyzeNode(SyntaxNodeAnalysisContext context)
             return;
         }
 
+        // A type named to qualify a member, Shared.Nested.Line() or TimeZoneInfo.AdjustmentRule.X, isn't a use of
+        // its own containing type; it is checked as the type of the member used.
+        if (nodeSymbol is INamedTypeSymbol)
+        {
+            return;
+        }
+
+        // A helper may take a section context and read its ExpressionContext, which the compiler maps to the
+        // gateway's context.
+        // The receiver has to be one of the authoring library's section contexts itself.
+        if (nodeSymbol is IPropertySymbol { Name: "ExpressionContext" } &&
+            node is MemberAccessExpressionSyntax { Expression: var receiver } &&
+            context.SemanticModel.GetTypeInfo(receiver).Type?.IsSectionContext() == true)
+        {
+            return;
+        }
+
+        // Invoking a delegate typed member, like context.Trace("message"), is a use of that member and not of the
+        // delegate type.
+        if (nodeSymbol is IMethodSymbol { MethodKind: MethodKind.DelegateInvoke } &&
+            node is InvocationExpressionSyntax delegateInvocation &&
+            context.SemanticModel.GetSymbolInfo(delegateInvocation.Expression).Symbol is
+                { Kind: SymbolKind.Property or SymbolKind.Field } delegateMember)
+        {
+            nodeSymbol = delegateMember;
+        }
+
         // Members of an expression helper library ([Expression] on the class) are analysed in their own project, so
         // calls to its methods and uses of its constants are allowed, from source or a referenced assembly.
         if (nodeSymbol.IsExpressionLibraryMember() &&
@@ -316,10 +384,15 @@ private static void AnalyzeNode(SyntaxNodeAnalysisContext context)
             return;
         }
 
-        // Calls to [Expression] helpers (also from a referenced assembly) are expanded by the compiler, and source
-        // constants are folded into literals, except values of source enums, which don't exist in API Management. A
-        // helper passed as a method group isn't expanded.
-        if (nodeSymbol is IMethodSymbol && node is InvocationExpressionSyntax && nodeSymbol.HasExpressionAttribute() ||
+        // Calls to [Expression] helpers (also from a referenced assembly) and to the helpers of a policy document are
+        // expanded by the compiler, and source constants are folded into literals, except values of source enums,
+        // which don't exist in API Management. A helper passed as a method group isn't expanded. A document helper
+        // that isn't marked [Expression] may call any helper declared in source, as it could before it was analysed.
+        if (nodeSymbol is IMethodSymbol && node is InvocationExpressionSyntax &&
+            (nodeSymbol.HasExpressionAttribute() || nodeSymbol.IsDocumentMember() ||
+             !nodeSymbol.DeclaringSyntaxReferences.IsDefaultOrEmpty &&
+             !node.IsPartOfMarkedPolicyExpressionMethod(context.SemanticModel) &&
+             !node.IsPartOfPolicyExpressionDelegate(context.SemanticModel)) ||
             !nodeSymbol.DeclaringSyntaxReferences.IsDefaultOrEmpty &&
             nodeSymbol is IFieldSymbol { IsConst: true, ContainingType.TypeKind: not TypeKind.Enum } field &&
             !(field.Type.TypeKind == TypeKind.Enum && !field.Type.DeclaringSyntaxReferences.IsDefaultOrEmpty))
@@ -333,9 +406,15 @@ private static void AnalyzeNode(SyntaxNodeAnalysisContext context)
             return;
         }
 
+        // API Management accepts anonymous types: new { a = 1 }.a
+        if (symbol.IsAnonymousType)
+        {
+            return;
+        }
+
         // Indexers are named "this[]" in Roslyn; the allow lists use their metadata name ("Item").
         var memberName = nodeSymbol is IPropertySymbol { IsIndexer: true } ? nodeSymbol.MetadataName : nodeSymbol.Name;
-        var typeName = (symbol.IsGenericType ? symbol.OriginalDefinition : symbol)?.ToFullyQualifiedString() ?? "";
+        var typeName = GetTypeName(symbol);
         if (AllowedTypes.Contains(typeName))
         {
             if (AllowedInTypes.TryGetValue(typeName, out var allowed) && !allowed.Contains(memberName))
@@ -355,4 +434,24 @@ private static void AnalyzeNode(SyntaxNodeAnalysisContext context)
             context.ReportDiagnostic(Diagnostic.Create(Rules.TypeUsed.DisallowedType, node.GetLocation(), typeName));
         }
     }
+
+    private static bool IsPartOfPolicyExpression(SyntaxNodeAnalysisContext context)
+    {
+        return context.Node.IsPartOfPolicyExpressionMethod(context.SemanticModel) ||
+               context.Node.IsPartOfPolicyExpressionDelegate(context.SemanticModel);
+    }
+
+    private static string GetTypeName(INamedTypeSymbol symbol)
+    {
+        if (!symbol.IsGenericType)
+        {
+            return symbol.ToFullyQualifiedString();
+        }
+
+        // System.Nullable and System.Tuple are listed without type parameters and are allowed with any number of them.
+        var nameWithoutTypeParameters = symbol.OriginalDefinition.ToFullyQualifiedStringWithoutTypeParameters();
+        return AllowedTypes.Contains(nameWithoutTypeParameters)
+            ? nameWithoutTypeParameters
+            : symbol.OriginalDefinition.ToFullyQualifiedString();
+    }
 }