diff --git a/acceptance/README.md b/acceptance/README.md new file mode 100644 index 000000000..132336d00 --- /dev/null +++ b/acceptance/README.md @@ -0,0 +1,46 @@ +## Acceptance Tests + +The acceptance tests are blackbox* tests that are expected to interact with resources on a real GitHub instance. + +*Note: they aren't strictly blackbox because `exec gh` commands delegate to a binary set up by `testscript` that calls into `ghcmd.Main`. However, since our real `func main` is an extremely thin adapter over `ghcmd.Main`, this is reasonable. This tradeoff avoids us building the binary ourselves for the tests, and allows us to get code coverage metrics. + +### Running the Acceptance Tests + +The acceptance tests currently require three environment variables to be set: + * `GH_HOST` + * `GH_ACCEPTANCE_ORG` + * `GH_TOKEN` + +While `GH_HOST` may not strictly be necessary because `gh` can choose a host, it is required to avoid ambiguity and unexpected results depending on the state of `gh`. + +The `GH_ACCEPTANCE_ORG` is an organization that the tests can manage resources in. + +The `GH_TOKEN` must already have the necessary scopes for each test, and must have permissions to act in the `GH_ACCEPTANCE_ORG`. See [Effective Test Authoring](#effective-test-authoring) for how tests must handle tokens without sufficient scopes. + +The acceptance tests have a build constraint of `//go:build acceptance`, this means that `go test ./...` will continue to work without any modifications. The `acceptance` tag must therefore be provided when running `go test`. + +A full example invocation can be found below: + +``` +GH_HOST= GH_ACCEPTANCE_ORG= GH_TOKEN= test -tags=acceptance ./acceptance +``` + +### Effective Test Authoring + +This section will likely extend over time. + +#### Test Isolation + +#### Scope Validation + +#### Limit Custom Commands + +... + +### Further Reading + +https://bitfieldconsulting.com/posts/test-scripts + +https://atlasgo.io/blog/2024/09/09/how-go-tests-go-test + +https://encore.dev/blog/testscript-hidden-testing-gem diff --git a/acceptance/acceptance_test.go b/acceptance/acceptance_test.go index 364d073b8..ade3517af 100644 --- a/acceptance/acceptance_test.go +++ b/acceptance/acceptance_test.go @@ -1,3 +1,5 @@ +//go:build acceptance + package acceptance_test import (