Google has announced a significant update to its voice search technology, introducing a new AI-driven system that processes voice queries directly without the need for conversion to text. This update, called Speech-to-Retrieval (S2R), marks a major step forward in improving voice search accuracy and speed.

How the New System Works

Previously, Google’s voice search used Cascade ASR—a system that first converted spoken queries into text before ranking them. This method was prone to errors due to the loss of contextual information during the speech-to-text conversion. The new Speech-to-Retrieval (S2R) system, however, bypasses this step entirely.

Instead, S2R uses a neural network-based machine-learning model trained on vast datasets of paired audio queries and documents. This allows the system to directly process spoken queries and match them with relevant documents, providing a faster and more accurate search experience.

Dual-Encoder Model: Two Neural Networks Working Together

The Speech-to-Retrieval system employs two neural networks working in tandem:

• Audio Encoder: Converts spoken queries into a vector-space representation, capturing the semantic meaning of the voice query.
• Document Encoder: Turns written documents into their own vector representations, making it easier for the system to match spoken queries with relevant content.

These two encoders create a shared semantic space, meaning that both audio queries and related documents are placed close together based on their similarity in meaning.

Rich Vector Representation for Enhanced Context Understanding

One of the key innovations of the S2R system is the use of rich vector representations. Unlike older models that relied on keyword matching, S2R “understands” the context and intent behind a voice search. For example, even if someone says “show me Munch’s screaming face painting,” the system will still recognize the query’s relevance to Edvard Munch’s The Scream.

Improved Search Ranking Process

Once the audio query is converted into a vector, it is compared to Google’s index of documents to find the most relevant matches. The system then goes through a ranking layer that combines similarity scores with hundreds of other ranking signals to determine which results are most relevant and should appear first in the search results.

Benchmarking and Performance

Google has tested the S2R system against its previous Cascade ASR model and a perfect version of it, Cascade Groundtruth. While the new system performed significantly better than the old Cascade ASR, Google acknowledges that there is still room for improvement, indicating that the technology is evolving.

Voice Search Now Live

Despite some areas for improvement, Google has confirmed that the new S2R-based voice search system is now live, rolling out in multiple languages. This new system promises to deliver faster and more reliable voice search by directly processing spoken queries without the need for text conversion.

Google calls this update a “new era” in voice search, and it’s expected to enhance the search experience across the board, improving both speed and accuracy.

Summary

Google’s new Speech-to-Retrieval (S2R) system for voice search revolutionizes how spoken queries are processed by bypassing the text conversion step, offering faster, more accurate results. The system’s dual-encoder model uses AI to better understand the meaning and context of spoken queries, ensuring users get relevant answers more efficiently.