Add initial acceptance test README

This commit is contained in:
William Martin 2024-10-11 13:49:28 +02:00
parent eb9eeb10e1
commit 5745eb1c1c
2 changed files with 48 additions and 0 deletions

46
acceptance/README.md Normal file
View file

@ -0,0 +1,46 @@
## Acceptance Tests
The acceptance tests are blackbox* tests that are expected to interact with resources on a real GitHub instance.
*Note: they aren't strictly blackbox because `exec gh` commands delegate to a binary set up by `testscript` that calls into `ghcmd.Main`. However, since our real `func main` is an extremely thin adapter over `ghcmd.Main`, this is reasonable. This tradeoff avoids us building the binary ourselves for the tests, and allows us to get code coverage metrics.
### Running the Acceptance Tests
The acceptance tests currently require three environment variables to be set:
* `GH_HOST`
* `GH_ACCEPTANCE_ORG`
* `GH_TOKEN`
While `GH_HOST` may not strictly be necessary because `gh` can choose a host, it is required to avoid ambiguity and unexpected results depending on the state of `gh`.
The `GH_ACCEPTANCE_ORG` is an organization that the tests can manage resources in.
The `GH_TOKEN` must already have the necessary scopes for each test, and must have permissions to act in the `GH_ACCEPTANCE_ORG`. See [Effective Test Authoring](#effective-test-authoring) for how tests must handle tokens without sufficient scopes.
The acceptance tests have a build constraint of `//go:build acceptance`, this means that `go test ./...` will continue to work without any modifications. The `acceptance` tag must therefore be provided when running `go test`.
A full example invocation can be found below:
```
GH_HOST=<host> GH_ACCEPTANCE_ORG=<org> GH_TOKEN=<token> test -tags=acceptance ./acceptance
```
### Effective Test Authoring
This section will likely extend over time.
#### Test Isolation
#### Scope Validation
#### Limit Custom Commands
...
### Further Reading
https://bitfieldconsulting.com/posts/test-scripts
https://atlasgo.io/blog/2024/09/09/how-go-tests-go-test
https://encore.dev/blog/testscript-hidden-testing-gem

View file

@ -1,3 +1,5 @@
//go:build acceptance
package acceptance_test
import (