- Notifications
You must be signed in to change notification settings - Fork937
Add simple e2e test#330
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Merged
Changes fromall commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| # End To End (e2e) Tests | ||
| The purpose of the E2E tests is to have a simple (currently) test that gives maintainers some confidence in the black box behavior of our artifacts. It does this by: | ||
| * Building the `github-mcp-server` docker image | ||
| * Running the image | ||
| * Interacting with the server via stdio | ||
| * Issuing requests that interact with the live GitHub API | ||
| ## Running the Tests | ||
| A service must be running that supports image building and container creation via the `docker` CLI. | ||
| Since these tests require a token to interact with real resources on the GitHub API, it is gated behind the `e2e` build flag. | ||
| ``` | ||
| GITHUB_MCP_SERVER_E2E_TOKEN=<YOUR TOKEN> go test -v --tags e2e ./e2e | ||
| ``` | ||
| The `GITHUB_MCP_SERVER_E2E_TOKEN` environment variable is mapped to `GITHUB_PERSONAL_ACCESS_TOKEN` internally, but separated to avoid accidental reuse of credentials. | ||
| ## Example | ||
| The following diff adjusts the `get_me` tool to return `foobar` as the user login. | ||
| ```diff | ||
| diff --git a/pkg/github/context_tools.go b/pkg/github/context_tools.go | ||
| index 1c91d70..ac4ef2b 100644 | ||
| --- a/pkg/github/context_tools.go | ||
| +++ b/pkg/github/context_tools.go | ||
| @@ -39,6 +39,8 @@ func GetMe(getClient GetClientFn, t translations.TranslationHelperFunc) (tool mc | ||
| return mcp.NewToolResultError(fmt.Sprintf("failed to get user: %s", string(body))), nil | ||
| } | ||
| + user.Login = sPtr("foobar") | ||
| + | ||
| r, err := json.Marshal(user) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to marshal user: %w", err) | ||
| @@ -47,3 +49,7 @@ func GetMe(getClient GetClientFn, t translations.TranslationHelperFunc) (tool mc | ||
| return mcp.NewToolResultText(string(r)), nil | ||
| } | ||
| } | ||
| + | ||
| +func sPtr(s string) *string { | ||
| + return &s | ||
| +} | ||
| ``` | ||
| Running the tests: | ||
| ``` | ||
| ➜ GITHUB_MCP_SERVER_E2E_TOKEN=$(gh auth token) go test -v --tags e2e ./e2e | ||
| === RUN TestE2E | ||
| e2e_test.go:92: Building Docker image for e2e tests... | ||
| e2e_test.go:36: Starting Stdio MCP client... | ||
| === RUN TestE2E/Initialize | ||
| === RUN TestE2E/CallTool_get_me | ||
| e2e_test.go:85: | ||
| Error Trace: /Users/williammartin/workspace/github-mcp-server/e2e/e2e_test.go:85 | ||
| Error: Not equal: | ||
| expected: "foobar" | ||
| actual : "williammartin" | ||
| Diff: | ||
| --- Expected | ||
| +++ Actual | ||
| @@ -1 +1 @@ | ||
| -foobar | ||
| +williammartin | ||
| Test: TestE2E/CallTool_get_me | ||
| Messages: expected login to match | ||
| --- FAIL: TestE2E (1.05s) | ||
| --- PASS: TestE2E/Initialize (0.09s) | ||
| --- FAIL: TestE2E/CallTool_get_me (0.46s) | ||
| FAIL | ||
| FAIL github.com/github/github-mcp-server/e2e 1.433s | ||
| FAIL | ||
| ``` | ||
| ## Limitations | ||
| The current test suite is intentionally very limited in scope. This is because the maintenance costs on e2e tests tend to increase significantly over time. To read about some challenges with GitHub integration tests, see [go-github integration tests README](https://github.com/google/go-github/blob/5b75aa86dba5cf4af2923afa0938774f37fa0a67/test/README.md). We will expand this suite circumspectly! | ||
| Currently, visibility into failures is not particularly good. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| //go:build e2e | ||
| package e2e_test | ||
| import ( | ||
| "context" | ||
| "encoding/json" | ||
| "os" | ||
| "os/exec" | ||
| "testing" | ||
| "time" | ||
| "github.com/google/go-github/v69/github" | ||
| mcpClient "github.com/mark3labs/mcp-go/client" | ||
| "github.com/mark3labs/mcp-go/mcp" | ||
| "github.com/stretchr/testify/require" | ||
| ) | ||
| func TestE2E(t *testing.T) { | ||
| e2eServerToken := os.Getenv("GITHUB_MCP_SERVER_E2E_TOKEN") | ||
| if e2eServerToken == "" { | ||
| t.Fatalf("GITHUB_MCP_SERVER_E2E_TOKEN environment variable is not set") | ||
| } | ||
| // Build the Docker image for the MCP server. | ||
| buildDockerImage(t) | ||
| t.Setenv("GITHUB_PERSONAL_ACCESS_TOKEN", e2eServerToken) // The MCP Client merges the existing environment. | ||
| args := []string{ | ||
| "docker", | ||
| "run", | ||
| "-i", | ||
| "--rm", | ||
| "-e", | ||
| "GITHUB_PERSONAL_ACCESS_TOKEN", | ||
| "github/e2e-github-mcp-server", | ||
| } | ||
| t.Log("Starting Stdio MCP client...") | ||
| client, err := mcpClient.NewStdioMCPClient(args[0], []string{}, args[1:]...) | ||
| require.NoError(t, err, "expected to create client successfully") | ||
| t.Run("Initialize", func(t *testing.T) { | ||
| ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) | ||
| defer cancel() | ||
| request := mcp.InitializeRequest{} | ||
| request.Params.ProtocolVersion = "2025-03-26" | ||
| request.Params.ClientInfo = mcp.Implementation{ | ||
| Name: "e2e-test-client", | ||
| Version: "0.0.1", | ||
| } | ||
| result, err := client.Initialize(ctx, request) | ||
| require.NoError(t, err, "expected to initialize successfully") | ||
| require.Equal(t, "github-mcp-server", result.ServerInfo.Name) | ||
| }) | ||
| t.Run("CallTool get_me", func(t *testing.T) { | ||
| ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) | ||
| defer cancel() | ||
| // When we call the "get_me" tool | ||
| request := mcp.CallToolRequest{} | ||
| request.Params.Name = "get_me" | ||
| response, err := client.CallTool(ctx, request) | ||
| require.NoError(t, err, "expected to call 'get_me' tool successfully") | ||
| require.False(t, response.IsError, "expected result not to be an error") | ||
| require.Len(t, response.Content, 1, "expected content to have one item") | ||
| textContent, ok := response.Content[0].(mcp.TextContent) | ||
| require.True(t, ok, "expected content to be of type TextContent") | ||
| var trimmedContent struct { | ||
| Login string `json:"login"` | ||
| } | ||
| err = json.Unmarshal([]byte(textContent.Text), &trimmedContent) | ||
| require.NoError(t, err, "expected to unmarshal text content successfully") | ||
| // Then the login in the response should match the login obtained via the same | ||
| // token using the GitHub API. | ||
| client := github.NewClient(nil).WithAuthToken(e2eServerToken) | ||
| user, _, err := client.Users.Get(context.Background(), "") | ||
| require.NoError(t, err, "expected to get user successfully") | ||
| require.Equal(t, trimmedContent.Login, *user.Login, "expected login to match") | ||
| }) | ||
| require.NoError(t, client.Close(), "expected to close client successfully") | ||
| } | ||
| func buildDockerImage(t *testing.T) { | ||
| t.Log("Building Docker image for e2e tests...") | ||
| cmd := exec.Command("docker", "build", "-t", "github/e2e-github-mcp-server", ".") | ||
| cmd.Dir = ".." // Run this in the context of the root, where the Dockerfile is located. | ||
| output, err := cmd.CombinedOutput() | ||
| require.NoError(t, err, "expected to build Docker image successfully, output: %s", string(output)) | ||
| } |
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.