Update TESTING.md - #1283
Conversation
…ce and one for AWS Bedrock. Removed the validation on the configuration page for the Enterprise testing, the team will simply test against whatever configuration is on the instances
dominiccooney
left a comment
dominiccooney
left a comment
There was a problem hiding this comment.
TL;DR:
- Pointer to where to get creds.
- Remove duplication by instructing testers to do a set of steps for each kind of instance.
- More specificity in what those steps are.
| #### Completions | ||
|
|
||
| 1. Login to the Cody Enterprise Cody Gateway Instance | ||
| 2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go). |
There was a problem hiding this comment.
| 2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go). | |
| 2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go). A completion should appear. |
- Document the keyboard shortcut to press or the name of the command so the testers can look up the keyboard shortcut.
- This test should specify having a particular repo cloned and open, and mention specific files to open. It will make testing easier and make reproducing any bugs much easier.
@pkukielka does the tester's setup include other autocomplete providers? If so we need to ensure the completions are coming from Cody specifically somehow.
There was a problem hiding this comment.
opt \ is the manual trigger hotkey for autocomplete.
The QA testers have experience with this, I don't think we need to assume they have another completion provider. The Cody logo also "spins/loads" when it triggers a completion, so theres indication from the UI that it's coming from Cody
| 1. Login to Cody Enterprise account | ||
| 2. Create new chat | ||
| 3. User should see the default model in the dropdown but is unable to change it | ||
| 1. Login to the Cody Enterprise Gateway Instance |
There was a problem hiding this comment.
Where's the credentials for this, if I wanted to perform these tests myself? Ditto below for "Cody Enterprise Cody Gateway Instance". Is that a different instance?
There was a problem hiding this comment.
How are they getting the credentials today? I thought they were already shared outside of this doc. @kalanchan consolidated them into 1Password
There was a problem hiding this comment.
"where can I get the credentials" kind of information should live in the handbook, but we can also add a link to 1password here. For the sake of completeness, let's just add it here 😄
| ### Cody Enterprise - AWS Bedrock Instance | ||
|
|
||
| #### Chat | ||
|
|
||
| 1. Login to the Cody Enterprise AWS Bedrock Instance | ||
| 2. Create a new chat | ||
| 3. User should not be able to change the LLM model on the chat window | ||
| 4. Ask Cody a question in the new chat window. | ||
|
|
||
| #### Commands | ||
|
|
||
| 1. Login to the Cody Enterprise AWS Bedrock Instance | ||
| 2. Go to commands panel | ||
| 3. Trigger command | ||
| 4. Command should be executed | ||
|
|
||
| #### Completions | ||
|
|
||
| 1. Login to the Cody Enterprise AWS Bedrock Instance | ||
| 2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go). |
There was a problem hiding this comment.
This would be better if you removed the duplication and said something like: There are these kinds of instances (list them) and then had one set of instructions, like "For each kind of instance: ..."
There was a problem hiding this comment.
One thing to note is that we are evaluating the LLM dropdown for Enterprise customers using Cody Gateway, which might not work as elegantly if combine the instance work. I agree combining them reads better, but I'm not sure if the team is going through a check list and having it split up by instances is easier in that case. If we do consolidate, shouldn't this also apply to the entire document? Do the following with these instances: Free, Pro, Enterprise 1, and 2? Will require a rewrite of the whole thing and I don't know why we didn't do that in the first place.
Co-authored-by: Dominic Cooney <dominic.cooney@sourcegraph.com>
4 participants
Test plan
Added test plans for verifying JBs client works for Enterprise customers, specifically Cody Gateway and AWS Bedrock deployments