Google’s AI Summaries in Hindi Require an Enhancement in Quality

Facing the linguistic diversity of India, companies creating digital content are faced with the significant task of accurately presenting and translating their materials. Google, too, is encountering similar obstacles with its recent introduction of AI summaries in the nation.

In August, the tech giant rolled out support for Hindi in its AI summaries feature, enabling users within India the flexibility to toggle between Hindi and English directly from the search interface. However, this integration has proven to be less reliable than anticipated.

A notable example encountered was taken from an entry on Google’s own blog. A GIF illustrated a response to the query “Cheeni ki jagah chai mai kya daal sakte hai?”, which is asking for a substitute for sugar in tea. Altering the wording slightly to “Chai mai cheeni ke jagah kya daal sakte hai?” led to no response from Google, demonstrating the system’s inconsistency.

An additional issue identified was the literal translation of English phrases into Hindi. When inquiring in Hindi about recommended foods for summers, one translated answer suggested “Chiknai wali cheezien”, equating to “Sticky things” — a baffling and misleading translation. Switching to English, the recommendation shifted to “Oily”, indicating a significant difference in dietary advice and posing a rather strange suggestion.

Moreover, misinformation regarding YouTube’s ownership was detected, where AI summaries inaccurately mentioned “Until 16 February 2023, Neal Mohan was Google’s CEO” in Hindi, contrasting with the accurate English statement “As of 16 February 2023, Neal Mohan is Google CEO”.

Qualitative inaccuracies were also found, such as overly complex answers to simple questions. For example, querying the date for Diwali yielded an elaborate response including information on the previous year’s celebration before eventually providing the sought-after date.

Inquiries regarding the consumption of spicy food yielded variably inconsistent results from the AI overview, further demonstrating the system’s irregularities.

Concerning inaccuracies extended to sensitive topics like menstruation and pregnancy. An inquiry regarding the advisable time to consider parenthood post-marriage was met with a confusingly worded suggestion to wait for at least two years, without clarification.

Queries about dietary recommendations during menstruation led to vague suggestions such as “Drinking milk with many things” and using Lemon for mood swings.

The AI also failed to provide crucial context in some answers. For example, recommending Bangla Sahib Gurudwara in Delhi for tea and snacks without making it clear these items are available outside the religious venue showcased a lack of nuanced understanding.

While English queries notably provided better outcomes, possibly due to a larger volume of source material, the issues we’ve outlined can largely be attributed to errors within Google’s systems, such as context misinterpretation and translation inaccuracies. Despite Google’s efforts toward enhancing language and search functionalities in India, much improvement is needed.

Google shared with TechCrunch that AI summaries are generated only for queries with a high confidence level in the output’s quality, emphasizing that this feature is built upon their core search quality systems and is designed to represent information supported by leading web results.

Despite striving for accuracy akin to other search features, misinterpretations by Google’s systems or inaccuracies from web sources can occur, which they aim to rectify using these instances as opportunities for refinement.

While Google may attribute errors to the quality of source materials, the ultimate responsibility of determining the relevance and trustworthiness of these sources before summarization falls upon Google’s algorithms—leaving users pointing fingers at Google for ambiguous or incorrect information.

The introduction of AI overviews has been less than stellar for Google, with previously flagged mistakes like absurd suggestions based on unreliable sources. Such errors underline the necessity for Google to enhance its language and factual accuracies to better serve India’s vast internet user base, now exceeding 830 million.