OcularAudio MCP
Universal local-first media intelligence for AI agents.
OcularAudio lets MCP-compatible AI systems watch, listen to, search, inspect, compare, and reason over media. It turns URLs, streams, and local media into reusable timestamped evidence: metadata, transcripts, frames, OCR, retrieval results, multimodal moments, and cross-video intelligence.
v1.3.0 · MIT
Product
What it does
- Universal media intake: web URLs, direct media, local files, live/stream sources, and a broad set of social, video, audio and archive platforms.
- Speech understanding: timestamped captions with local Whisper fallback when captions are unavailable.
- Visual understanding: intelligent screenshots, exact frames, frame bursts, crops, visual indexing and perceptual similarity.
- On-screen text: optional Tesseract OCR produces searchable visual text evidence.
- Evidence search: timestamp-aware lexical, semantic and hybrid retrieval over stable, provenance-preserving canonical evidence records.
- Multimodal analysis: aligns transcript, visual and OCR evidence around the same moments.
- Batch intelligence: process multiple sources, search across them, and generate cross-video comparisons with failure isolation.
- Agentic execution: turn analysis requests into bounded plans with retries, timeouts, audit records and health reporting.
- Capability modes: Lite, Intelligence, Deep and Auto control processing depth and response richness.
- Local caching: reuse processed evidence and manage cache state through CLI or MCP.
- Operational awareness: capabilities, health, readiness, security state, metrics and the full protocol contract are machine-readable.
- MCP interoperability: tools, resources and prompts over stdio or Streamable HTTP.
- Production deployment: bearer-token protection, host/origin controls, Docker deployment, persistent cache and readiness checks.
Best use cases
Research and lecture analysis · product demonstrations · competitor analysis · interviews · technical talks · UI walkthroughs · multi-video comparison · evidence-grounded AI investigation.
Capability matrix
Canonical evidence architecture
All source artifacts are normalized into one local manifest per source. Evidence records have stable content-derived IDs, modality labels, timestamp bounds, content hashes, provenance pointers, and metadata. Transcript search, timelines, moment inspection, and hybrid retrieval share the same canonical IDs; OCR and visual-frame records are indexed alongside transcript evidence. Manifests are written atomically, validated on read, deduplicated by ID, and refreshed from the existing transcript and visual caches without requiring a migration or a database. Clearing a source removes its transcript, canonical manifest, visual index, and frame artifacts together. Manifest metadata intentionally stores a bounded summary rather than the source URL or credentials.
Four-level intelligence engine
Every request is resolved through one deterministic policy engine. Auto resolves to Lite, Intelligence, or Deep based on query intent, requested features, source count, media duration, and an optional time budget. The decision is returned as machine-readable metadata with explicit limits, disabled features, capability gaps, and adjustments.
Modes enforce evidence caps, frame limits, source limits, retry counts, concurrency, sampling intervals, and runtime ceilings. Missing OpenCV, FFmpeg, or Tesseract is reported as a capability gap; optional unavailable features are disabled rather than represented as successful. Explicit source-limit violations fail with a clear error instead of silently dropping sources. Source coverage remains shared across modes.
The CLI's existing --mode flag now reaches the Python runtime. Use --budget-seconds to impose a caller-specified runtime ceiling (0.001-900 seconds). MCP planning, agentic execution, hybrid search, visual indexing, multimodal analysis, and batch analysis expose the same decision contract. Use --mode deep for comprehensive evidence or --mode lite for a fast overview.
Quick start
NPM / MCP
Release status: this repository contains source version 1.3.0, but that version is not yet published to npm. The latest published package is 1.2.1, so the npx examples below install 1.2.1 until a separate 1.3.0 release is authorized and published. To test the current checkout, clone the repository, install the Python dependencies from requirements.txt, then run node bin/cli.js from the checkout.
npx -y ocular-audio-mcp
Local MCP hosts can launch the same command. Example:
{
"mcpServers": {
"ocular-audio": {
"command": "npx",
"args": ["-y", "ocular-audio-mcp"]
}
}
}
Claude Code:
claude mcp add ocular-audio -- npx -y ocular-audio-mcp
Codex CLI:
codex mcp add ocular-audio -- npx -y ocular-audio-mcp
Direct CLI
npx ocular-audio "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
npx ocular-audio --check
npx ocular-audio --json "URL"
npx ocular-audio --visual-index --ocr "URL"
npx ocular-audio --multimodal-search "pricing" "URL"
npx ocular-audio --plan "Find the pricing discussion and inspect its visual evidence"
npx ocular-audio --agentic "Find every pricing discussion and inspect the relevant slides"
Technical overview
MCP host / agent
|
MCP tools + prompts + resources
|
Universal source resolver
|
Evidence: metadata + transcript + timestamps + chapters
/ | \
visual OCR retrieval
\ | /
multimodal moments
|
batch intelligence
|
agentic execution
|
operations / audit / readiness
Core modules
- media_resolver.py + universal_sources.py: source classification, platform families, streams, local media and analysis policy.
- evidence_index.py: timestamped evidence segmentation, chapter association and deterministic retrieval.
- evidence_store.py: canonical transcript/chapter/metadata/visual/OCR records, stable evidence IDs, provenance, integrity-checked manifests, atomic persistence and lifecycle cleanup.
- visual_index.py: frame sampling, bursts, crops, perceptual hashes, descriptors and OCR evidence.
- semantic_index.py: local TF-IDF/lexical/semantic and hybrid retrieval.
- multimodal_index.py: transcript/visual/OCR timestamp alignment and ranking.
- batch_index.py: bounded multi-source execution, persistence, search and comparison.
- agentic_index.py: deterministic planning, retries, timeouts, audit records and execution health.
- capability_modes.py: Lite, Intelligence, Deep and Auto policy resolution.
- protocol_contract.py: machine-readable MCP interoperability contract.
- deployment_security.py: controlled HTTP authentication and transport security.
- runtime_ops.py: bounded metrics, latency observations, readiness and audit retention.
MCP surface
33 tools
Media/evidence: get_ocular_audio_transcript, get_ocular_audio_video_context, search_ocular_audio_video, get_ocular_audio_video_timeline, inspect_ocular_audio_moment, query_ocular_audio_evidence, get_ocular_audio_metadata, get_ocular_audio_chapters, inspect_ocular_audio_source, get_ocular_audio_capabilities.
Visual/OCR: get_ocular_audio_video_screenshots, index_ocular_audio_video_visuals, search_ocular_audio_visuals, get_ocular_audio_video_frame, get_ocular_audio_video_frame_burst, crop_ocular_audio_video_frame.
Retrieval/multimodal: search_ocular_audio_hybrid, analyze_ocular_audio_multimodal, inspect_ocular_audio_multimodal_moment, search_ocular_audio_cache.
Batch: analyze_ocular_audio_batch, search_ocular_audio_videos, compare_ocular_audio_videos, get_ocular_audio_batch, list_ocular_audio_batches.
Agentic/operations: plan_ocular_audio_analysis, run_ocular_audio_analysis, get_ocular_audio_health, get_ocular_audio_audit, get_ocular_audio_contract, get_ocular_audio_identity.
Cache: list_ocular_audio_cache, clear_ocular_audio_cache.
7 resources
- ocularaudio://capabilities
- ocularaudio://modes
- ocularaudio://health
- ocularaudio://contract
- ocularaudio://security
- ocularaudio://readiness
- ocularaudio://metrics
4 prompts
- inspect_video
- search_video_evidence
- review_visual_evidence
- compare_videos
Deployment
Local stdio
stdio is the default local MCP transport and requires no HTTP authentication.
Streamable HTTP
npx ocular-audio --transport streamable-http --host 127.0.0.1 --port 8000
The MCP endpoint is /mcp. For network exposure, configure authentication, real issuer/resource URLs, allowed hosts and allowed origins. Do not expose an unauthenticated instance to an untrusted network.
Docker
cp .env.example .env
docker compose -f docker/docker-compose.yml up --build -d
The supplied deployment uses Python 3.11, FFmpeg, a persistent cache volume, a read-only container filesystem, dropped capabilities, no-new-privileges, restart policy and a healthcheck.
For internet-facing deployments, use TLS and an organization identity provider. The built-in static bearer verifier is intended for controlled deployments and can be replaced by JWT validation or RFC 7662 introspection.
Runtime requirements
- Python 3.11+ recommended for the release environment.
- Node.js 18+ for the NPM CLI.
- FFmpeg for media processing.
- OpenCV for visual processing.
- faster-whisper for local transcription fallback.
- Tesseract is optional for OCR.
- Network access is required for remote sources.
The NPM package does not hide native Python/system prerequisites; OcularAudio is a local media-processing server rather than a hosted API.
Cookies and restricted media
Operator-supplied cookies.txt can be used for sources that require an authenticated browser session. Never commit cookies, access tokens or credentials.
Release v1.3.0
npm install
npm run check
npm pack
This creates ocular-audio-mcp-1.3.0.tgz. Inspect it before publishing:
npm publish ocular-audio-mcp-1.3.0.tgz --access public
See RELEASE_v1.3.0.md. A manual GitHub Actions workflow can also build and upload the exact tarball as an artifact.
Validation
python -m unittest discover -s tests -v
node --check bin/cli.js
npm run check
CI additionally validates the MCP SDK surface, tool/resource/prompt registration, Streamable HTTP authorization behavior, compilation, CLI syntax and package metadata.
Design principles
- Local-first: no mandatory cloud inference API.
- Evidence-first: preserve timestamps and source evidence.
- Composable: each analysis layer builds on the previous evidence layer.
- Bounded: steps, retries, concurrency, metrics cardinality and audit retention have explicit limits.
- MCP-native: tools, resources and prompts are protocol primitives.
- Deterministic infrastructure: OcularAudio provides execution/evidence primitives while the host remains responsible for model-level reasoning.
License
MIT. See LICENSE.