Discover MCPs & agents
Loading MCPs and agents…
Loading MCPs and agents…
Fetch only relevant tools for the current conversation and save cost while increasing the precision of your LLM Response
From the repo.
A Precision-driven Tool Recommendation (PTR) system for filtering MCP (Model Context Protocol) tools based on conversation context. Fetch only relevant tool for the ongoing conversation and save cost while increasing the precision of your LLM Response.
Developed by OppieAI
Watch the full explanation of how ToolsFilter works and its impact on LLM performance
Modern LLMs with access to large tool suites face a critical performance degradation issue: the more tools available, the lower the accuracy becomes. This phenomenon is well-documented in research and practical implementations:
Recent studies using MCPGauge evaluated six commercial LLMs with 30 MCP tool suites and revealed alarming findings:
This accuracy degradation with increased tool count is demonstrated in this analysis video, showing how model performance deteriorates as more tools are introduced.
Instead of overwhelming your LLM with 100+ tools, get precisely the 3-5 most relevant ones:
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ FastAPI App │────▶│ Message Parser │────▶│ Search Pipeline │
└────────┬────────┘ └──────────────────┘ └─────────┬───────┘
│ │
│ ┌─────────────────────────────┼─────────┐
│ │ │ │ │
┌────────▼────────┐ ┌──────▼─────┐ ┌─────────▼────────┐ │ ┌───────▼──────┐
│ Redis Cache │ │ Embedding │ │ Qdrant Vector │ │ │ LTR Reranker │
│ │ │ Service │ │ Database │ │ │ (XGBoost) │
│ • Query Cache │ │ (LiteLLM) │ │ │ │ │ │
│ • Results Cache │ │ • Voyage │ │ • Semantic Search│ │ │ • 46 Features│
│ • Tool Index │ │ • OpenAI │ │ • BM25 Hybrid │ │ │ • NDCG@10 Opt│
└─────────────────┘ │ • Fallback │ │ • Cross-Encoder │ │ │ │
└────────────┘ └──────────────────┘ │ └──────────────┘
│
┌───────────────────────┘
│
┌──────▼──────┐
│ Multi-Stage │
│ Filtering │
│ │
│ 1. Semantic │
│ 2. BM25 │
│ 3. Rerank │
│ 4. LTR │
└─────────────┘
Search Strategy Comparison: (With 300+ noise (Genuine APIs) tools to resemble real-world)
| Strategy | F1 Score | MRR | P@1 | NDCG@10 | Best For |
|---|---|---|---|---|---|
| hybrid_basic | 0.359 ⭐ | 1.000 | 1.000 | 0.975 ⭐ | General-purpose, balanced performance |
| semantic_only | 0.328 | 1.000 ⭐ | 1.000 ⭐ | 0.870 | Simple queries, exact matches |
| hybrid_cross_encoder | 0.359 | 1.000 | 1.000 | 0.964 | Complex queries requiring reranking |
| hybrid_ltr_full | 0.359 | 1.000 | 1.000 | 0.942 | Learning-based optimization |
⭐ = Best performer for that metric
Key Achievements:
Learning-to-Rank Training Results:
✅ Completed:
🎯 In Progress:
git clone https://github.com/yourusername/ToolsFilter.git
cd ToolsFilter
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env
.env and add your API keys:# Embedding Service Keys (at least one required)
VOYAGE_API_KEY=your_voyage_api_key
OPENAI_API_KEY=your_openai_api_key # Optional fallback
COHERE_API_KEY=your_cohere_api_key # Optional
# Important: Include provider prefix in model names
PRIMARY_EMBEDDING_MODEL=voyage/voyage-2
FALLBACK_EMBEDDING_MODEL=openai/text-embedding-3-small
# Start all services including the API
make up
# Or manually:
docker-compose up -d
# View logs
make logs
# Stop services
make down
# Start in development mode
make up-dev
# Or manually:
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up
docker-compose up -d qdrant redis
python -m src.api.main
The API will be available at http://localhost:8000
Once running, visit:
http://localhost:8000/docshttp://localhost:8000/redocimport requests
# Filter tools based on conversation
response = requests.post(
"http://localhost:8000/api/v1/tools/filter",
json={
"messages": [
{"role": "user", "content": "I need to search for Python files in the project"}
],
"available_tools": [
{
"type": "function",
"name": "grep",
"description": "Search for patterns in files",
"parameters": {
"type": "object",
"properties": {
"pattern": {"type": "string", "description": "Search pattern"}
},
"required": ["pattern"]
},
"strict": true
},
{
"type": "function",
"name": "find",
"description": "Find files by name",
"parameters": {
"type": "object",
"properties": {
"name": {"type": "string", "description": "File name pattern"}
},
"required": ["name"]
},
"strict": true
}
]
}
)
print(response.json())
# {
# "recommended_tools": [
# {"tool_name": "find", "confidence": 0.95},
# {"tool_name": "grep", "confidence": 0.85}
# ],
# "metadata": {"processing_time_ms": 42}
# }
POST /api/v1/tools/filter - Filter tools based on conversation contextGET /api/v1/tools/search - Search tools by text queryPOST /api/v1/tools/register - Register new tools (for batch indexing)GET /api/v1/tools/info - Get information about indexed toolsGET /api/v1/collections - List all vector store collections with metadataGET /health - Health check endpoint{
"recommended_tools": [
{
"tool_name": "find",
"confidence": 0.85,
"reasoning": "High relevance to file search operations"
}
],
"metadata": {
"processing_time_ms": 45.2,
"embedding_model": "voyage/voyage-2",
"total_tools_analyzed": 20,
"conversation_messages": 3,
"request_id": "uuid-here",
"conversation_patterns": ["file_search", "code_analysis"]
}
}
# Run unit tests
pytest tests/ -v
# Run comprehensive evaluation with all strategies
docker exec ptr_api python -m src.evaluation.run_evaluation
# Run strategy comparison
docker exec ptr_api python -m src.evaluation.evaluation_framework.comparison
# Train LTR model
docker exec ptr_api python -m src.scripts.train_ltr
# Run ToolBench evaluation
docker exec ptr_api python -m src.evaluation.toolbench_evaluator
# Run simple API test
python test_api.py
Refer to the latest comparison report: evaluation_results/comparison_20250823_153715.markdown
Key findings:
# Linting
ruff check src/
# Type checking
mypy src/
# Formatting
black src/
# Start the load test UI
locust -f tests/load_test.py
Key configuration options in .env:
PRIMARY_EMBEDDING_MODEL: Main embedding model (default: voyage-2)FALLBACK_EMBEDDING_MODEL: Fallback model (default: text-embedding-3-small)MAX_TOOLS_TO_RETURN: Maximum tools to return (default: 10)SIMILARITY_THRESHOLD: Minimum similarity score (default: 0.7)The system automatically creates model-specific collections to handle different embedding dimensions:
tools_<model_name> (e.g., tools_voyage_voyage_3)/api/v1/collections endpoint to view all collectionsImportant: When changing embedding models, you'll need to re-index your tools as embeddings from different models are not compatible.
The system supports automatic fallback to a secondary embedding model when the primary model fails:
FALLBACK_EMBEDDING_MODEL in your .env fileembedding_model field in responses indicates which model was usedSee the /documentation directory for:
Inspired by PTR Paper
This project uses a dual licensing model:
See LICENSE for full terms.
For commercial licensing, contact: sales@oppie.ai
Replace {MCP_ENDPOINT_URL} with this MCP’s endpoint URL (from its repo or docs above). No API key — you connect directly.
Tool
OS
Config file: ~/.cursor/mcp.json
{
"mcpServers": {
"mcp-server": {
"url": "{MCP_ENDPOINT_URL}"
}
}
}Paste into mcpServers in the config file. Restart Cursor after saving.
If this MCP is also published on mcpchannel.ai, you can subscribe from Browse and use the gateway config there instead.