In the dynamic field of data science, Nishchay Mahor, a seasoned Data Scientist at Course5i, emerges as a trailblazer, bringing a wealth of expertise in natural language processing (NLP) and large language models (LLMs). With a distinguished Bachelor’s degree in Computer Science and profound consultancy experience, Mahor’s work extends across major sectors, including technology, personal computing, advisory, semiconductor, and fast-moving consumer goods. Mahor’s presentation, titled “Revolutionising Market Surveys through Unprecedented Generative AI for Efficient Data Synthesis,” goes beyond theoretical discussions during the Machine Learning Developers Summit (MLDS) 2024. Currently implementing his groundbreaking research for clients, including a notable consumer packaged goods (CPG) company and a leading tech giant, Mahor’s work is making tangible impacts.
Mahor sheds light on the genesis of his research, tracing its roots from a conceptual discussion in a room with four individuals to a practical solution actively benefiting industry clients. This journey highlights the transition from ideation to real-world implementation.
Mahor begins by addressing a fundamental challenge in market research – the perception of a product. The traditional methodologies face issues of prolonged cycles, suboptimal responses, and biases, hindering timely decision-making and exposing businesses to missed opportunities.
To tackle these challenges, Mahor introduces a visionary fivefold approach: Placing advanced AI technologies at the core of surveys to revolutionize insights gathering. Introducing synthetic responses in surveys through generative AI, redefining the entire survey experience. Streamlining survey processes for optimized data collection and the synthesis of meaningful insights. Acknowledging the financial landscape and significantly reducing costs tied to resource-intensive data preprocessing. Enabling real-time decision-making with the incorporation of synthetic survey data to prevent missing business opportunities.
Gathering sample data and, when necessary, using Faker to replicate data points for an acceptable volume. Utilizing pre-trained Transformers and Ensemble large language models for generating synthetic data. Refining the results through fine-tuning based on different personas input by the user, ensuring the generated data closely mimics actual responses.
A pivotal component of Mahor’s solution is the contextualization engine, which automates the process of setting objectives, defining target respondents, and fine-tuning responses. By providing clarity on the survey’s objective, target audience, and response instructions, the engine ensures that the generative AI produces contextually relevant and meaningful data.
Mahor illustrates the contextualization engine with a sample use case involving a tech company seeking to understand its product’s perception within the tech community. The generated synthetic data is then compared to real-world statistics, showcasing the remarkable accuracy and closeness to actual user reviews.
The overarching goal of Mahor’s solution is to minimize survey responses, leading to swift decision-making. The impact is profound – minimizing missed opportunities, mitigating misguided decisions, and fostering customer satisfaction and loyalty.
Nishchay Mahor’s presentation opens a gateway to the future of market surveys, where generative AI transforms the conventional approach into a dynamic, efficient, and precise method. As businesses strive to stay ahead in the competitive landscape, embracing such innovative solutions becomes imperative. Mahor’s work not only envisions this transformation but is actively driving it, making a substantial impact on how businesses perceive and act upon market insights. The journey from concept to implementation is a testament to the practicality and efficacy of Mahor’s groundbreaking research, marking a paradigm shift in the realm of data science and market research.
