Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

perf: add inventory cache for stateless server patterns#1636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
SamMorrowDrums wants to merge2 commits intomain
base:main
Choose a base branch
Loading
fromperf/inventory-cache

Conversation

@SamMorrowDrums
Copy link
Collaborator

Summary

AddCachedInventory to build tool/resource/prompt definitions once at startup rather than per-request. This is particularly useful for the remote server pattern where a new server instance is created per request.

Problem

In stateless deployments (like the remote MCP server), a new server and inventory is created for each incoming request. This meansAllTools(t),AllResources(t), andAllPrompts(t) are called repeatedly, rebuilding all ~130 tool definitions including JSON schema generation on every request.

Related:modelcontextprotocol/go-sdk#685

Solution

Add aCachedInventory that:

  • Builds all tools/resources/prompts once at startup
  • Usessync.Once for thread-safe initialization
  • Returns pre-populated builders for per-request use

API

// Initialize once at startup (typically in main() or server init)github.InitInventoryCache(translator)// Then on each request, get a builder with pre-cached toolsinv:=github.CachedInventoryBuilder().WithReadOnly(cfg.ReadOnly).WithToolsets(cfg.EnabledToolsets).WithFeatureChecker(checker).// Still evaluated per-request!Build()

Key Features

  • InitInventoryCache(t) - Called once at startup with your translator
  • CachedInventoryBuilder() - Returns a builder pre-populated with cached definitions
  • Per-request configuration still works - read-only, toolsets, feature flags, filters
  • Thread-safe viasync.Once
  • Backward compatible -NewInventory(t) still works without caching

Why This Works

The elegance is preserved because:

  1. Tools are still self-describing - TheGetMe(t),GetTeams(t) pattern stays the same
  2. No code changes needed for tool definitions - They work as before
  3. Feature flags work per-request - They're evaluated inAvailableTools(ctx), not at definition time
  4. Translations resolved once - Since remote server usesNullTranslationHelper anyway, this is fine

For Remote Server Integration

// In server startup/init (once)funcinit() {github.InitInventoryCache(translations.NullTranslationHelper)}// Per-request handlerfunchandleRequest(ctx context.Context,reqRequest) {inv:=github.CachedInventoryBuilder().WithReadOnly(cfg.ReadOnly).WithToolsets(cfg.Toolsets).WithFeatureChecker(createFeatureChecker(ctx,req.User)).Build()// Feature flags are evaluated here, per-requesttools:=inv.AvailableTools(ctx)// ...}

Testing

  • Added comprehensive unit tests ininventory_cache_test.go
  • All existing tests pass
  • script/lint passes

@SamMorrowDrumsSamMorrowDrums requested a review froma team as acode ownerDecember 17, 2025 21:56
CopilotAI review requested due to automatic review settingsDecember 17, 2025 21:56
Add CachedInventory to build tool/resource/prompt definitions once atstartup rather than per-request. This is particularly useful for theremote server pattern where a new server instance is created per request.Key features:- InitInventoryCache(t) initializes the cache once at startup- InitInventoryCacheWithExtras(t, tools, resources, prompts) allows  injecting additional items (e.g., remote-only Copilot tools)- CachedInventoryBuilder() returns a builder with pre-cached definitions- Per-request configuration (read-only, toolsets, feature flags) still works- Thread-safe via sync.Once- Backward compatible: NewInventory(t) still works without cachingThis addresses the performance concern raised in go-sdk PR#685 at ahigher level by caching the entire []ServerTool slice rather thanindividual schemas.Related:modelcontextprotocol/go-sdk#685
Copy link
Contributor

CopilotAI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Pull request overview

This PR adds an inventory caching mechanism to optimize performance in stateless server deployments where a new server instance is created per request. The cache eliminates redundant rebuilding of ~130 tool definitions and their JSON schemas on every request.

Key Changes

  • IntroducesCachedInventory withsync.Once for thread-safe one-time initialization of tool/resource/prompt definitions
  • ProvidesInitInventoryCache() for explicit startup initialization andCachedInventoryBuilder() for cached builder access
  • Maintains full backward compatibility with existingNewInventory() API

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

FileDescription
pkg/github/inventory_cache.goImplements the caching mechanism with global singleton, sync.Once initialization, and builder factory methods
pkg/github/inventory_cache_test.goComprehensive test coverage including initialization, idempotency, auto-initialization, per-request filtering, and comparison with uncached behavior

Comment on lines +114 to +116
})
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The ResetInventoryCache function creates a race condition. After resetting globalInventoryCache to a new instance, the old sync.Once in the previous instance is lost. If another goroutine was in the middle of calling init() on the old instance, it would complete initialization on the old instance while new callers would get the new uninitialized instance. Additionally, resetting doesn't prevent the old sync.Once from keeping the "already called" state.

While the comment correctly warns this is test-only and not thread-safe, a safer approach would be to reset the fields inside the existing CachedInventory struct rather than replacing the entire global variable. However, sync.Once cannot be reset by design.

Consider adding a mutex-based solution for tests or document more clearly that tests must ensure serial execution when calling ResetInventoryCache.

Copilot uses AI. Check for mistakes.
"github.com/github/github-mcp-server/pkg/translations"
"github.com/modelcontextprotocol/go-sdk/mcp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Tests in this file intentionally do not use t.Parallel() because they mutate global state via ResetInventoryCache(). This is correct, but it should be documented with a comment at the top of the test file to explain why these tests must run serially, especially since most other test files in this package use t.Parallel().

Suggested change
"github.com/stretchr/testify/require"
// NOTE: Tests in this file intentionally do NOT use t.Parallel().
// They mutate process-wide inventory cache state via ResetInventoryCache /
// InitInventoryCache / CachedInventoryBuilder, so running them in parallel
// could cause test flakiness and data races. Keep these tests serial even
// though most other tests in this package use t.Parallel().

Copilot uses AI. Check for mistakes.
Comment on lines +105 to +108
if len(extraTools) > 0 {
c.tools = append(c.tools, extraTools...)
}
if len(extraResources) > 0 {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The IsCacheInitialized check is unreliable. It checks if len(globalInventoryCache.tools) > 0, but this has two problems:

  1. If AllTools(t) returns an empty slice (which shouldn't happen in practice but is theoretically possible), IsCacheInitialized would return false even after successful initialization
  2. After ResetInventoryCache() is called, the check looks at the new instance which will have empty tools, making it work correctly. However, the comment at line 106 says "We can't directly check sync.Once state" which suggests the intent is to check if the Once has fired, not just if tools are populated.

A more reliable approach would be to add an explicit boolean flag that's set to true inside the once.Do callback, which would accurately reflect whether initialization has occurred regardless of whether AllTools returns an empty slice.

Copilot uses AI. Check for mistakes.
Comment on lines +76 to +77
// github.InitInventoryCacheWithExtras(
// translations.NullTranslationHelper,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The documentation is contradictory. Line 76 states "The cache must be initialized via InitInventoryCache before calling this function" but line 77 immediately says "If the cache is not initialized, this will initialize it with NullTranslationHelper."

This creates confusion about whether initialization is required or optional. Consider rephrasing to clarify the behavior, such as: "The cache should typically be initialized via InitInventoryCache before calling this function. If not already initialized, it will automatically initialize with NullTranslationHelper as a fallback."

Copilot uses AI. Check for mistakes.
)

// CachedInventory provides a cached inventory builder that builds tool definitions
// only once, regardless of how many times NewInventoryBuilder is called.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Comment uses inconsistent terminology. The comment says "NewInventoryBuilder" but the actual function is called "CachedInventoryBuilder". This creates confusion about which function is being referred to.

Suggested change
// only once, regardless of how many timesNewInventoryBuilder is called.
// only once, regardless of how many timesCachedInventoryBuilder is called.

Copilot uses AI. Check for mistakes.
- Fix inconsistent terminology (NewInventoryBuilder -> CachedInventoryBuilder)- Add explicit initialized flag for reliable state checking- Clarify contradictory documentation about initialization requirement- Improve ResetInventoryCache warning about thread safety- Add comment explaining why tests don't use t.Parallel()
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

Copilot code reviewCopilotCopilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SamMorrowDrums

[8]ページ先頭

©2009-2025 Movatter.jp