A Hidden Bottleneck in Vector-Aware Search
When Manticore Search added native vector search capabilities in 2023, the team relied on the ONNX Runtime to execute neural embedding models directly inside the search engine. The move allowed developers to perform approximate nearest neighbor (ANN) searches alongside traditional full-text queries without a separate vector database. However, behind the scenes, the integration was far from optimal. According to the engineering team’s latest retrospective, the initial ONNX path was a “first pass” implementation—it worked, but it added significant overhead in both latency and memory consumption. Now, after a complete rebuild of that integration, Manticore reports a remarkable 14× improvement in embedding inference speed.
The performance jump matters more than a typical optimization story because it directly touches one of the most expensive operations in modern AI search stacks: converting raw text into dense vector representations. For teams building retrieval-augmented generation (RAG) pipelines, customer-facing semantic search, or real-time recommendation systems, the cost and latency of embedding inference often dictate architectural choices. With this release, Manticore brings that operation back into the core engine at a fraction of the previous cost, potentially eliminating the need for external model-serving infrastructure in many use cases.
What the Original ONNX Integration Got Wrong
Manticore’s original ONNX support, introduced as part of the search engine’s broader vector capabilities, relied on a straightforward loading and execution model. The team would load an ONNX model into memory, instantiate a session, and then feed input tensors one record at a time during indexing or query time. While functionally correct, this pattern introduced several inefficiencies that compound at scale.
First, the one-at-a-time invocation model meant that the ONNX Runtime could never take advantage of vectorized operations or batch parallelism offered by modern CPUs. Each call paid the full cost of session state management, input validation, and operator dispatch, leaving hardware units underutilized. Second, memory management was generic and conservative—tensors were allocated and freed on every call, generating fragmentation and adding pressure on the allocator. Third, the inference path ran synchronously on the main search thread, creating a serial bottleneck that forced indexing and query pipelines to wait for the embedding to finish before proceeding. For high-throughput applications ingesting thousands of documents per second, even a few milliseconds of per-document latency multiplied into unacceptable slowdowns.
Beyond pure performance, the original design also limited flexibility. Different embedding models required different tokenization schemes, maximum sequence lengths, and output dimensions, but the integration treated every model as a black box. This lack of cooperation between the search engine’s internal data structures and the ONNX runtime meant that Manticore could not optimize memory layout to match the model’s expectations, leading to costly transpositions and copies on every call.
How the Rebuild Achieved a 14× Speedup
The revamped ONNX path, detailed by the Manticore Search team, introduces a multi-pronged set of optimizations that collectively deliver the headline 14× improvement. At the core is a new batch execution pipeline that aggregates individual inference requests into larger tensor batches whenever possible. Instead of calling the ONNX Runtime for every text input, the engine now collects tokens from multiple documents or query inputs, pads them to a uniform length, and runs a single inference pass that processes an entire batch at once. This allows the underlying compute kernels—often highly optimized libraries like Intel’s oneDNN or AMD’s ZenDNN—to fully saturate AVX-512 and AMX instruction sets, dramatically improving throughput per core.
Memory management received an equally thorough overhaul. The engineering team implemented a custom arena allocator that pools tensor buffers and reuses them across inference calls. By pre-allocating a slab of contiguous memory and carefully tracking free segments, the new path eliminates per-call malloc/free overhead entirely. Additionally, the engine now examines the ONNX model’s graph at load time and aligns tensor strides with Manticore’s native storage format, avoiding the transposition penalties that plagued the original version. According to the benchmarks published alongside the release, this alone accounted for a 2–3× improvement for larger embedding dimensions such as 768 or 1024.
Thread scheduling saw a crucial design change as well. Inference is now dispatched to a dedicated thread pool managed by the Manticore scheduler, decoupling it from the main search threads that handle query parsing, filtering, and result merging. This means that a single slow embedding computation no longer stalls the entire pipeline, and multiple batches can execute concurrently on multi-core systems. The team also introduced a gradient of priority: real-time queries get preferential access to inference resources, while background indexing tasks are automatically deferred when query load spikes. This ensures that interactive search latency stays low even during heavy reindexing.
Measurable Impact and Real-World Numbers
The 14× claim isn’t just a theoretical ceiling. Manticore’s internal benchmarks, reproduced on commodity hardware with a popular sentence-transformer model, show that the time to embed 10,000 short documents dropped from approximately 28 seconds to just over 2 seconds on an AMD EPYC 7313 server. Equally important, memory usage per embedding worker decreased by nearly 60%, allowing more concurrent models to be loaded on the same machine without exhausting RAM. This combination of speed and efficiency gains is especially meaningful for teams running multiple embedding models—for example, a dense text model and a separate sparse model for hybrid search.
In continuous indexing scenarios, where documents arrive in a stream and must be made searchable within milliseconds, the new ONNX path reduces the embedding contribution to end-to-end latency from over 12 ms to less than 1 ms per document. That sub-millisecond figure opens the door to use cases previously impossible on general-purpose search engines, such as real-time abuse detection in chat applications or on-the-fly personalization of product catalogs during a browsing session.
It’s important to note that these improvements apply to any ONNX-compatible model, not just a curated subset. The rebuild preserves full compatibility with Hugging Face Transformers exported via the Optimum library, as well as models trained natively in ONNX format. The team specifically tested embeddings from the BGE, E5, and all-MiniLM families, all of which demonstrated speedups within the 12–15× range, confirming the generic nature of the optimizations.
Implications for AI Search and RAG Architectures
Manticore’s achievement arrives at a moment when the lines between traditional search engines and vector databases are increasingly blurred. Projects like PostgreSQL with pgvector, Elasticsearch with learn-to-rank plugins, and dedicated vector stores like Qdrant all compete for the same AI-driven workload. By bringing embedding inference directly into the search engine and making it this fast, Manticore reduces the architectural complexity of RAG and semantic search applications. Teams no longer need to maintain a separate model-serving infrastructure just to generate vectors for indexing; a single Manticore instance can now handle both the inference and the ANN search.
Cost implications are substantial. Cloud-hosted embedding APIs from providers like OpenAI and Cohere charge per token, and self-hosted model servers demand dedicated GPU or high-core CPU instances. Manticore’s rebuilt ONNX path runs entirely on CPU, which the team says can cut embedding costs by an order of magnitude when compared to a comparable GPU-based microservice—especially for moderate-scale deployments where GPU idling would be significant. For teams that have already adopted local AI models for data privacy or latency reasons, the ability to co-locate inference with the search engine eliminates a hop over the network and simplifies deployment topologies.
Looking ahead, the rebuilt ONNX integration paves the way for dynamic model selection and in-place model upgrades without downtime—features the Manticore team has hinted are in development. As embedding models continue to specialize (multimodal, multilingual, code-aware), the ability to efficiently swap or chain models inside the search engine will become a competitive differentiator. While dedicated vector databases still hold an edge in raw ANN performance at extreme scale, Manticore’s 14× embedding speedup repositions it as a compelling unified platform for teams that value simplicity, performance, and open-source transparency.
Commentaires