ML Systems — Chat
1. Jio Copilot
Medium blog by Fynd where they walkthrough the ML model experiments of creating Jio Copilot (which probably powers all chatbots of JioMart, TiraBeauty etc) — https://lnkd.in/dnv6RFR7
Following is just a summary -
1. Objective is to build something that can tell you when your JioMart order is out for delivery, provide a smart summary of the last episode of your favourite show you watched on JioCinema or help you troubleshoot your JioFiber broadband connection.
2. They started with closed-source LLMs, but faced challenges such as high latency, limited API usage quota, and constraints with data transfer policies.
3. They chose Llama 2, beacuse of User privacy, Time & cost savings, State-of-the-Art research and Faster and better community support
4. Llama 2 base models occasionally ‘hallucinated’ with product names, and price and size details, was inconsistent in format and a few other issues on the two tasks they were targeting — Intent and Entity Classification and Chat completion
5. For data required for finetuning the model, they started with generating synthetic data across all product categories from Jio Storefronts like JioMart, TiraBeauty, Netmeds, MilkBasket, JioCinema and JioFiber covering a diverse set of scenarios
6. They fine-tuned Llama2 7B & 13B models using PEFT and QLORA-based parameter and memory efficient techniques (We had done something similar over here — https://lnkd.in/dgky-a9X)
7. To evaluate the readiness they checked for Bias Detection and Content Filtering for profanity, hate speech, violence, and other harmful content. and Model API response time to be less than 1 second for the Intent-Entity model and less than 3 seconds for the chat model response.
8. Prompts they used to train the models -
