V1.8.5 - #1320
Merged
Merged
V1.8.5#1320
Conversation
Signed-off-by: xuleiming <leiming.xu@163.com>
Signed-off-by: xuleiming <leiming.xu@163.com>
Signed-off-by: xuleiming <leiming.xu@163.com>
Signed-off-by: xuleiming <leiming.xu@163.com>
body rewind when fallback
Add AI gateway mod_ai_route system design docs under docs/zh_cn/sys_design: - mod_ai_route.md: module system design - mod_ai_route_bfe_changes.md: BFE main program changes for AI gateway
…rocess and add TC08-TC10 - Replace in-process tests under tests/integration/mod_ai_route with real BFE process tests under tests/integration/implementation/scenario-SC01-*. - Add common harness (ProcessEnv, BFEConfigBuilder, MockBackend) for building config and running real bfe binary. - Add TC08: fallback preserves complete body after primary disconnects mid-stream. - Add TC09: fallback is aborted when body exceeds accessibleBodySize. - Add TC10: fallback is disabled when totalBodyBufferSize limit is reached. - Add test design documents for SC01.
- Extend AIConf with Keys array, KeyPolicy, Provider and ModelTable - Implement weighted random key selection, 429 rotation, 401/403 death, and 5xx same-key retry with exponential backoff in reverse proxy - Remove legacy single-key AI handling from ServeHTTP; AI requests now flow exclusively through ServeHTTPForAI / aiClusterInvoke - Fix 4xx early-return bug and ensure 5xx retries keep the same key - Add SC02 integration tests for multi-key selection, rotation and retry - Extend test helpers: MockBackend auth header capture, per-cluster AIConf - Update zh_cn and en_us cluster_conf.data.md documentation
- Extend AIConf.ModelTable with fixed-point RMB price conversion and price index. - Add UsedCost to TokenUsage and Unit/Currency to QuotaPlan. - Implement RMB quota deduction via single-key fixed-point Lua script. - Cache SvrDataConf in TokenAuthContext for cost calculation at response stage. - Add design doc docs/zh_cn/sys_design/rmb_quota.md. - Add SC03 integration tests for RMB quota deduction scenarios. - Bump go-lib to v0.0.2 for shared quota fixed-point helpers.
Modifications of V1.8.5
- Add MatchPrefix/StripPrefix to AIConf and validation in AIConfCheck - Implement prefix stripping in reverseproxy.doSingleAIForward - Add unit tests for AIConf validation and prefix stripping - Add SC04 integration tests and design docs for provider/model prefix strip - Update cluster_conf.data docs for MatchPrefix/StripPrefix fields
feat(bfe): provider/model prefix routing support
…aming responses - Move UsedCost calculation from HandleReadResponse to HandleRequestFinish so streaming responses (ContentLength=-1) are billed correctly. - Enable mod_body_process in SC03 integration tests. - Add streaming RMB quota deduction test case. - Update rmb_quota.md design doc and AGENTS.md. Fixes #1316
fix(bfe): 4xx status codes trigger cluster-level fallback
- Upgrade bfe-access-pb from v0.1.0 to v0.2.0 - Rename access log fields: ai_apikey->ai_apikey_id, ai_mapped_model->ai_target_model, ai_prompt_tokens->ai_input_tokens - Add AiBasicInfo fields: Provider, RetryCount, CostCurrency, ClusterKeyNames - Add AiAuthInfo.HitQuotaPlans - Populate provider/currency/retry/cluster-key in reverseproxy.go - Record HitQuotaPlans in mod_ai_token_auth - Update request_log.go and request_log_test.go - Add SC05 integration tests covering all 20 AI fields (701-900) - Add b2log parser helper in tests/integration/common - Add sys_design/ai_access_log_fields.md and modifications design doc
xuleiming
reviewed
Aug 20, 2026
| Key *string // API key for AI service | ||
| Type int // type of LLM service, reserved for future use. should be 0 now. | ||
| ModelMapping *map[string]string // model mapping, key is model name in req, value is model name in backend | ||
| Provider string // provider name in model_prices |
Collaborator
There was a problem hiding this comment.
这个Provider字段对于数据面来说只是说明性的吧，没实际用到吧？
| // strip provider/model prefix according to cluster AIConf | ||
| if cluster.AIConf != nil && aiMeta != nil && cluster.AIConf.StripPrefix && cluster.AIConf.MatchPrefix != "" { | ||
| stripProviderPrefix(basicReq, outreq, aiMeta, cluster.AIConf.MatchPrefix) | ||
| } |
Collaborator
There was a problem hiding this comment.
stripProviderPrefix 中会调用 ReqBodyJsonSet 修改json，这段代码之前和之后也都有 ReqBodyJsonSet 修改json，存在重复，不够优化。
| if cluster.AIConf != nil && aiMeta != nil && cluster.AIConf.StripPrefix && cluster.AIConf.MatchPrefix != "" { | ||
| stripProviderPrefix(basicReq, outreq, aiMeta, cluster.AIConf.MatchPrefix) | ||
| } | ||
|
|
Collaborator
There was a problem hiding this comment.
在多个cluster 之间重试，每个cluster的strip model prefix、model mapping规则可能不同。ReqBodyJsonSet 后body就已经被改了，结果会被带到下一次重试，此处可能存在漏洞。例如，下一个cluster如果不需要修改modelname，则发给它的modelname仍是上一个cluster改后的结果。
- Refactor stripProviderPrefix to a pure string transform function. - In doSingleAIForward, compute final model (override -> strip -> mapping) and call ReqBodyJsonSet at most once. - Reset ContentLength for both outreq and basicReq.HttpRequest in one place. - Update related unit tests and sys_design docs.
…tempts - In doSingleAIForward, compute final model from ClientModel each attempt instead of inheriting TargetModel from previous attempts. - Copy request body before rewriting only when a rewrite is actually required, avoiding shared buffer mutation on fallback clusters. - Add SC04 TC07 to verify fallback cluster without rewrite rules receives the original client model. - Document the fix and its test design.
V1.8.5 fix problem, and adapt bfe-access-pb v0.2.0
xuleiming
reviewed
Aug 21, 2026
| // the rewrite does not leak into the next fallback/retry attempt. | ||
| // bytes_body.Rewind() only resets the read position and does not restore | ||
| // the original content modified by ReqBodyJsonSet. | ||
| if req.Body != nil { |
Collaborator
There was a problem hiding this comment.
为了解决这个问题，这里的做法是完全复制了一遍body赋予outreq。这和原有的思路不同，原有思路是让req和outreq共用同一个body。原有思路中body可以部分缓存，不影响剩余部分流式读出。这样修改后假定body必须是全部缓存的。
…uster attempts" This reverts commit 1ae02e4.
Revert "fix(bfe_server): prevent model body rewrite leaking across cluster
The integration test TestTC10_TotalBodyBufferSizeExceedsLimit could hang for 10 minutes if waitForTotalBytesBodyBuffer timed out, because the holder mock backend was blocked on HoldBeforeRead and httptest.Server.Close() would wait indefinitely. Changes: - Use context.Context for HoldBeforeRead so cleanup cancels it. - Increase metric wait timeout from 5s to 30s. - Include last metric value in failure message.
fix(test): prevent TestTC10 from hanging on cleanup
…ary cache - Use context.Context for HoldBeforeRead so cleanup cancels it and does not hang when waitForTotalBytesBodyBuffer times out. - Increase metric wait timeout from 5s to 30s for slow CI environments. - Include git commit hash in integration test binary path and check source mtime so that stale cached binaries are rebuilt when source changes. - Add holder request result logging for diagnostics.
fix(test): prevent TestTC10 from hanging on cleanup and fix stale bin…
The holder request intentionally blocks until the backend is released. Under race detector / slow CI the default 30s client timeout fires before BFE finishes wrapping the body, causing total_bytes_body_buffer to stay at 0 and the test to fail. Increase the holder client timeout to 2 minutes and the metric wait timeout to 60 seconds.
fix(test): increase holder timeout for TestTC10 in slow CI
…backend The original TestTC10 relied on a holder request with a blocking backend to keep a 2 MB bytes_body buffer allocated. This was fragile in CI: under race detector / load the client or backend timeouts fired before the body was wrapped, causing total_bytes_body_buffer to stay at 0. Redesign: - Add BFE_TEST_INITIAL_TOTAL_BYTES_BODY_BUFFER env var to let integration tests pre-seed the global body buffer counter in the BFE process. - TestTC10 now sets the counter to the limit, starts BFE, verifies the monitor endpoint reports the expected value, sends the test request, and asserts that fallback is disabled. - Remove the holder request, blocking backend, and polling logic. This makes the test deterministic and independent of request timing.
refactor(test): redesign TestTC10 to avoid timing-sensitive blocking …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
body rewind when fallback
feat: multi-API-Key support for AI backend clusters
feat: support RMB quota in mod_ai_token_auth
feat: add req_body_json_prefix_in condition primitive
feat(bfe): provider/model prefix routing support