> For the complete documentation index, see [llms.txt](https://mcp.conserver.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mcp.conserver.io/guide/search.md). # Search & Query ## Overview The vCon MCP server now exposes a recommended unified search surface plus the older specialized search tools. For new clients, prefer `vcon_search` first. It gives you one predictable response envelope, explicit `include` groups, cursor pagination, and response-size budgeting. The older search tools are still available and still useful for compatibility or specialized flows. ## Recommended Starting Point ### `vcon_search` - Unified Search **Best for:** New clients that want one search entry point with predictable parsing **Modes:** * `metadata` * `keyword` * `semantic` * `hybrid` **Key advantages:** * Stable `{ok, items, page}` envelope * Explicit `include` groups such as `core`, `summary`, `tags`, and `dealer` * Cursor pagination instead of mixed ad hoc paging behavior * Explicit `max_response_bytes` so oversized responses fail loudly with `RESPONSE_TOO_LARGE` **Example:** ```json { "mode": "keyword", "query": "billing dispute", "filters": { "tags": { "portal": "negative_experience" } }, "include": ["core", "summary", "dealer", "tags"], "limit": 25, "max_response_bytes": 120000 } ``` **Typical companion tools:** * `vcon_capabilities` to discover supported modes, includes, and byte budgets * `vcon_taxonomy` to discover the portal taxonomy and preferred dealer source * `vcon_fetch` to expand a selected result with additional include groups ## Legacy Search Tools The tools below remain supported and useful. They are especially helpful for older clients, direct low-level access, or cases where you intentionally want their narrower behavior. ## Available Search Tools ### 1. `search_vcons` - Basic Filter Search **Best for:** Finding vCons by metadata (subject, parties, dates) **Searches:** * Subject line * Party names, emails, phone numbers * Creation dates **Does NOT search:** * Dialog content * Analysis content * Attachments **Example:** ```json { "subject": "customer support", "party_name": "John Doe", "start_date": "2024-01-01T00:00:00Z", "limit": 10 } ``` **Returns:** Complete vCon objects matching the filters *** ### 2. `search_vcons_content` - Keyword Search **Best for:** Finding specific words or phrases in conversation content **Searches:** * ✅ Subject * ✅ Dialog bodies (conversations, transcripts) * ✅ Analysis bodies (summaries, sentiment, etc.) * ✅ Party information (names, emails, phones) * ❌ Attachments (not indexed for full-text search) **Features:** * Full-text search with ranking * Typo tolerance via trigram indexing * Highlighted snippets in results * Tag filtering support * Date range filtering **Example:** ```json { "query": "billing issue refund", "tags": {"department": "sales"}, "start_date": "2024-01-01T00:00:00Z", "limit": 50 } ``` **Returns:** Ranked results with snippets showing where matches were found **Result format:** ```json { "success": true, "count": 5, "results": [ { "vcon_id": "uuid", "content_type": "analysis", // or "subject", "dialog", "party" "content_index": 0, "relevance_score": 0.85, "snippet": "...regarding the billing issue and potential refund..." } ] } ``` *** ### 3. `search_vcons_semantic` - AI-Powered Semantic Search **Best for:** Finding conversations by meaning, not just keywords **Searches:** * ✅ Subject (embedded) * ✅ Dialog bodies (embedded) * ✅ Analysis bodies with `encoding='none'` or `NULL` (embedded) * ❌ Analysis with `encoding='base64url'` or `encoding='json'` (not embedded) * ❌ Attachments (not embedded) **Features:** * Finds conceptually similar content * Works across paraphrases and synonyms * AI embeddings using 384-dimensional vectors * Tag filtering support * Similarity threshold control **Requirements:** * Embeddings must be generated first (see embedding documentation) * Currently requires pre-computed embedding vector (384 dimensions) **Example:** ```json { "query": "customer angry about late delivery", "threshold": 0.7, "limit": 20 } ``` **Returns:** Similar conversations ranked by semantic similarity *** ### 4. `search_vcons_hybrid` - Combined Keyword + Semantic Search **Best for:** Comprehensive search combining exact matches and conceptual similarity **Searches:** * Everything from keyword search (subject, dialog, analysis, parties) * Everything from semantic search (embedded content) **Features:** * Combines full-text and semantic search * Adjustable weighting between keyword and semantic results * Best of both worlds: exact matches + conceptual matches * Tag filtering support **Example:** ```json { "query": "billing dispute", "semantic_weight": 0.6, "tags": {"priority": "high"}, "limit": 30 } ``` **Parameters:** * `semantic_weight`: 0-1 (default 0.6) * 0.0 = 100% keyword search * 1.0 = 100% semantic search * 0.6 = 60% semantic, 40% keyword (recommended) **Returns:** Combined results with both keyword and semantic scores *** ## What About Attachments? ### Current Status **Attachments are NOT indexed for search** in the current implementation. **Why?** 1. **Binary content**: Many attachments contain binary data (PDFs, images, audio) that isn't suitable for text-based search 2. **Encoding**: Attachments with `encoding='base64url'` contain encoded data, not searchable text 3. **Structured data**: Attachments with `encoding='json'` contain structured data that produces poor quality embeddings ### Special Case: Tags Attachments of type `tags` with `encoding='json'` ARE used for filtering, but not for content search. Example tags attachment: ```json { "type": "tags", "encoding": "json", "body": ["department:sales", "priority:high", "region:west"] } ``` These tags can be used with the `tags` parameter in any search tool: ```json { "query": "customer complaint", "tags": {"department": "sales", "priority": "high"} } ``` ### Future Enhancements Potential future support for attachment content search: 1. **Text extraction**: Extract text from PDFs, Word docs, etc. 2. **Audio transcription**: Transcribe audio attachments to searchable text 3. **OCR**: Extract text from images 4. **Selective indexing**: Index only attachments with text content If you need to search attachment content, consider: 1. Extracting text and adding it as an analysis element 2. Adding a summary of attachment content as an analysis 3. Using attachment metadata in tags *** ## Analysis Encoding and Search ### Analysis Elements ARE Searchable Analysis elements are included in search, with filtering based on encoding: | Encoding | Keyword Search | Semantic Search | Notes | | ---------------- | -------------- | --------------- | ------------------------------------ | | `none` or `NULL` | ✅ Yes | ✅ Yes | Plain text content, ideal for search | | `json` | ✅ Yes | ❌ No | Included in keyword search only | | `base64url` | ✅ Yes | ❌ No | Included in keyword search only | ### Why Filter Semantic Search by Encoding? Analysis with `encoding='none'` contains human-readable text like: * Conversation summaries * Transcriptions * Sentiment analysis results * Translation output * Natural language insights These are ideal for semantic search because they contain meaningful natural language. Analysis with `encoding='json'` or `encoding='base64url'` typically contains: * Structured data (poor quality embeddings) * Binary content (not suitable for embeddings) * Encoded data (not searchable as text) *** ## Search Comparison | Feature | vcon\_search | search\_vcons | search\_vcons\_content | search\_vcons\_semantic | search\_vcons\_hybrid | | ------------------- | ----------------------------------------------------- | ------------- | ---------------------- | ----------------------- | --------------------- | | Subject | ✅ Depends on mode | ✅ Filter | ✅ Search | ✅ Search | ✅ Search | | Dialog | ✅ Depends on mode/include | ❌ | ✅ Search | ✅ Search | ✅ Search | | Analysis | ✅ Depends on mode/include | ❌ | ✅ Search | ✅ (encoding=none) | ✅ All | | Attachments | ✅ Via explicit include groups on fetch/search results | ❌ | ❌ | ❌ | ❌ | | Party Info | ✅ Depends on mode/include | ✅ Filter | ✅ Search | ❌ | ✅ Search | | Tags | ✅ Filter and return | ❌ | ✅ Filter | ✅ Filter | ✅ Filter | | Ranking | ✅ Depends on mode | ❌ | ✅ Relevance | ✅ Similarity | ✅ Combined | | Snippets | ❌ | ❌ | ✅ Yes | ❌ | ❌ | | Requires Embeddings | Only for semantic/hybrid mode | ❌ | ❌ | ✅ | ⚠️ Optional | | Cursor Pagination | ✅ Yes | ❌ | ❌ | ❌ | ❌ | | Response Budgeting | ✅ `max_response_bytes` | ❌ | ❌ | ❌ | ❌ | *** ## Best Practices ### When to Use Each Tool 1. **`vcon_search`**: Default choice for new clients * "Use one parser for metadata, keyword, semantic, and hybrid search" * "Return lightweight summaries plus dealer info" * "Fail loudly instead of silently returning oversized payloads" 2. **`search_vcons`**: Quick metadata lookups in older clients * "Find vCons with party email " * "Show me vCons from last week" * "List vCons with subject containing 'urgent'" 3. **`search_vcons_content`**: Keyword-based content search * "Find conversations mentioning 'refund'" * "Search for 'technical support' in dialog" * "Find analysis containing 'positive sentiment'" 4. **`search_vcons_semantic`**: Concept-based search * "Find conversations where customer was unhappy" * "Show me calls about payment issues" * "Find similar conversations to this one" 5. **`search_vcons_hybrid`**: Comprehensive search * "Find all billing-related conversations" (gets both exact matches and related topics) * "Search for customer complaints" (finds variations and synonyms) * Best when you want both precision and recall ### Performance Tips 1. **Use filters**: Date ranges and tags can dramatically reduce search scope 2. **Set appropriate limits**: Start with smaller limits (10-20) for faster results 3. **Choose the right tool**: Don't use semantic search if keyword search is sufficient 4. **Pre-generate embeddings**: Semantic search requires embeddings to be generated beforehand *** ## Generating Embeddings For semantic and hybrid search to work effectively, you need to generate embeddings for your vCons. See the following guides: * [INGEST\_AND\_EMBEDDINGS.md](https://github.com/vcon-dev/vcon-mcp/blob/main/docs/development/INGEST_AND_EMBEDDINGS.md) - Complete guide to embedding generation * [EMBEDDING\_STRATEGY\_UPGRADE.md](https://github.com/vcon-dev/vcon-mcp/blob/main/docs/development/EMBEDDING_STRATEGY_UPGRADE.md) - Details on which content is embedded **Quick start:** ```bash # Generate embeddings continuously npm run sync:embeddings # Or as part of full sync npm run sync # Check embedding coverage npm run embeddings:check ``` *** ## Troubleshooting ### "No results found" for content search * Check that the content exists in dialog or analysis * Try a simpler query (fewer words) * Use wildcards or partial words * Check date range filters ### "Embedding generation not yet implemented" * Semantic search currently requires pre-computed embeddings * Use `search_vcons_content` for keyword search instead * Generate embeddings using the scripts in `/scripts/` ### "Embedding must be 384 dimensions" * The system uses 384-dimensional embeddings * If you're providing embeddings, ensure they match this dimension * Use `text-embedding-3-small` with `dimensions=384` (OpenAI) * Or use `sentence-transformers/all-MiniLM-L6-v2` (Hugging Face) ### Poor search results * For keyword search: Try simpler, more specific terms * For semantic search: Ensure embeddings are up to date * For hybrid search: Adjust `semantic_weight` parameter * Consider using tags to filter results *** ## Examples ### Find customer complaints in dialog ```json { "query": "customer complaint angry upset frustrated", "limit": 20 } ``` ### Find high-priority sales conversations ```json { "query": "pricing quote proposal", "tags": { "department": "sales", "priority": "high" }, "start_date": "2024-01-01T00:00:00Z" } ``` ### Hybrid search with keyword emphasis ```json { "query": "billing invoice payment", "semantic_weight": 0.3, "limit": 30 } ``` ### Find conversations similar to a specific vCon 1. Get the vCon's embedding from the database 2. Use it in semantic search: ```json { "embedding": [0.123, 0.456, ...], // 384 dimensions "threshold": 0.75, "limit": 10 } ``` *** ## Related Documentation * [Getting Started](/guide/getting-started.md) - Getting started with vCon MCP * [Ingest and Embeddings](https://github.com/vcon-dev/vcon-mcp/blob/main/docs/development/INGEST_AND_EMBEDDINGS.md) - Embedding generation * [Search Optimization Guide](https://github.com/vcon-dev/vcon-mcp/blob/main/docs/SEARCH_OPTIMIZATION_GUIDE.md) - Database search performance