FIERRO LTD

Fine-Tuning LLMs in 2024: Making AI Talk Like Anyone Posted on: May 17, 2024 | 00:00

Large Language Models (LLMs) have shown incredible progress, mastering complex tasks from translation to code generation. We see models like GPT-4, Claude 3, and Llama 3 engaging in nuanced conversations and generating remarkably human-like text. But what if you need an LLM to go beyond general capabilities and adopt a specific persona, style, or domain knowledge? What if you want an LLM that speaks exactly like a specific person, famous or otherwise?

As the video from Machinelearnear asks: "Che, y si queremos hacer un LLM que hable de la misma forma que un famoso ... como hacemos?" (Hey, and if we want to make an LLM that speaks the same way as someone famous... how do we do it?)

This is where fine-tuning comes in. While base LLMs are trained on massive, diverse datasets, fine-tuning allows us to adapt these powerful models using smaller, targeted datasets to specialize their behavior. This post walks through a practical, step-by-step tutorial, inspired by the Machinelearnear video and its accompanying milei-gpt repository, on how to fine-tune an LLM using extensive interview data.

The Fine-Tuning Workflow: From Interviews to a Custom Chatbot

The process involves several key stages, transforming raw interview footage into a trained model that mimics a specific speaking style:

  1. Data Collection: The first step is gathering the raw material. This involves finding all available interviews of the target person (in the example, Javier Milei) on platforms like YouTube and organizing the links, perhaps in a .csv file. The target was over 400 hours of footage.

  2. Transcription & Diarization: Raw video/audio needs to be converted to text.

    • Transcription: Tools like OpenAI's Whisper can convert the speech to text.
    • Speaker Diarization: Crucially, we need to identify who is speaking when. This process, called diarization, separates the different voices in the audio. Tools like pyannote-audio or NVIDIA NeMo are used for this. This step often requires speaker verification (using a reference audio sample of the target person) to ensure segments are correctly attributed. Challenges arise when people speak over each other, which can confuse the diarization process.

  3. Dataset Preparation: The transcribed and diarized text needs to be structured for LLM training.

    • Formatting: Convert the segments into a conversational format. Many models use specific templates, like ChatML, which involves specifying roles (e.g., user, assistant) and content for each turn. The target speaker's segments become the assistant role, and the interviewer's segments become the user role.
    • Cleaning & Merging: Short conversational turns might be merged to create more substantial context for the model to learn from. Data cleaning is essential here.
    • Tokenization: Convert the text into tokens the LLM understands.

  4. Base Model Selection: Choose a pre-trained LLM to adapt. The example uses Llama-3-8B-Instruct, but other models like Phi-3-mini-128k-Instruct could also be suitable depending on resource constraints and desired performance.

  5. Fine-Tuning: Train the base model on the prepared conversational dataset. This adjusts the model's weights to better predict text in the style of the target speaker (the assistant). Libraries like Hugging Face's TRL (Transformer Reinforcement Learning) and tools like Unsloth can significantly speed up this process.

  6. Evaluation & Inference: Test the fine-tuned model. Does it sound like the target person? How does it respond to different prompts? This involves qualitative assessment and potentially quantitative metrics if an evaluation set was created.

  7. (Optional) Adding RAG: To improve factual accuracy and allow the model to reference specific interview content, Retrieval-Augmented Generation (RAG) can be added. This involves:

    • Indexing the original interview transcripts (or relevant segments) into a vector database (e.g., using Milvus).
    • When a user asks a question, retrieve relevant interview snippets.
    • Provide these snippets as context to the fine-tuned LLM when generating the response. This grounds the stylized response in actual data.

Example: Milei-GPT

The Machinelearnear tutorial culminates in milei-gpt, a Llama-3-8B model fine-tuned on hundreds of hours of Javier Milei's interviews.

  • Dataset: machinelearnear/multiturn_chat_milei_gpt on Hugging Face
  • Model: machinelearnear/llama-3-8b-milei-gpt on Hugging Face
  • Code Repository: github.com/machinelearnear/milei-gpt

You can test the model's inference directly using the provided notebooks in the repository. Here's a sample interaction shown in the video:

[
  {"role": "user", "content": "Che pero que pensas de que el estado financie los trenes"},
  {"role": "assistant", "content": "No sé si me entendés o no. ¿Me podés explicarlo? No te entiendo. Yo creo que es una locura porque vos tenés un problema con la inflación y lo querés resolver aumentando impuestos. Es decir... vos fijate esto. Si yo tengo un problema en mi casa con las cucarachas, digamos, por ejemplo, para arreglarlo voy a tener que comprar más insecticida. O sea, eso va contra todo sentido común. Entonces, bueno, ahora vamos al caso argentino. El Estado tiene un déficit monetario enorme. Y ese déficit monetario se financia emitiendo dinero. Ese dinero entra a la economía y genera presión sobre los precios. Por ende, cuando hay aumento del nivel generalizado de precios, sube la tasa de interés real. Cuando suben los intereses reales, caemos en actividad económica. Consecuentemente, bajan los salarios reales. Bajo los salarios reales, baja la demanda de trabajo. La oferta excede la demanda, cae el empleo formal. En paralelo, como todos somos seres humanos, necesitamos comer. Nosotros nos alimentamos, entonces tenemos que ir a trabajar informalmente. Ahora bien, esa gente trabajadora que está ganándole a la pobreza, le van a estar robando sus ingresos con impuesto inflacionario. Pero además, ya no solo que le roban su salario, sino que también pierde el seguro social. Así es que terminás perdiendo dos veces. Primero perdés tu poder adquisitivo y después perdés tus derechos sociales. Esto quiere decir que cada vez que hacemos este tipo de cosas estamos generando pobres. Está claro que no puede haber crecimiento económico sin generar riquezas. Para que haya crecimiento económico hay que producir más valor neto. De vuelta, eso requiere invertir. Quiere decir que necesita capital. Dicho capital viene de ahorrar. A ver, vos vas a hacer inversiones, ¿cómo haces esas inversiones? Con plata. ¿Cómo sacaste la plata? Ganandote. ¿Y cómo ganabas? Trabajando. ¿Cuánto ganaba? Más. ¿Por qué ganabas más? Porque había menos competencia. ¿Quién era menor competitividad? Los sindicalizados. ¿Qué hicieron estos imbéciles? Se metió el gobierno en el mercado laboral. Fue tan mal hecho que hoy tenemos cincuenta por ciento de pobres. Tenías dieciocho por ciento de pobres. Estabas cerca del veinte. ¿Sabés cuál fue el gran error? Que entró el Estado en el sector privado. Bueno, ¿qué pasó? Entraron los políticos ladrones. ¿Saben quiénes son los grandes ladrones? Son los políticos. Sí, sí, están acá sentadas todas estas personas que votaron a esta casta política corrupta. ¿Vos sabés qué es lo peor de todo? Lo peor de todo es que ellos tienen privilegios. Mientras nosotros pagamos nuestros impuestos honestamente, ellos pagan impuestos de manera fraudulenta. ¿Puedo preguntarte algo? ¿Usted cree que alguien"}
]

(Note: The length and specific phrasing capture some of the characteristic style)

Why Fine-Tune? The Benefits

Fine-tuning offers several advantages over using base models directly or relying solely on prompting:

  1. Performance & Style: As demonstrated by resources like Predibase's LoRA Land, fine-tuned open-source models (even smaller ones like Mistral-7B) can outperform much larger models like GPT-4 on specific tasks they were trained for. They excel at capturing specific styles and nuances.
  2. Efficiency & Cost: Running inference on smaller, fine-tuned models (like an 8B parameter model) is significantly cheaper and faster than using large, proprietary APIs. These smaller models can potentially run locally or even on edge devices like modern smartphones.
  3. Data Privacy & Ownership: Fine-tuning on your own data means the resulting model is yours. If you train on proprietary or sensitive data, you retain control, unlike sending data to third-party APIs.

Getting Started: Tools & Resources

To replicate this process, you'll need:

  • Compute: A GPU with sufficient VRAM (>= 8GB recommended for 8B models, more for larger ones). Options include local hardware, Google Colab, Sagemaker Studio Lab, or platforms like Lighting AI.
  • Python: Version 3.10 or higher.
  • Key Libraries:
    • yt-dlp: To download YouTube videos.
    • whisperX (or similar): For audio transcription.
    • pyannote-audio or NVIDIA NeMo: For speaker diarization.
    • pandas: For data manipulation.
    • Hugging Face Libraries (transformers, datasets, trl): For model loading, dataset handling, and fine-tuning.
    • Unsloth: For faster fine-tuning of Llama models.
  • Reference Audio: A clean audio sample of your target speaker for speaker verification during diarization.

Further Reading & Exploration:

  • How to Fine-Tune LLMs in 2024 with Hugging Face (philschmid.de)
  • Hugging Face Alignment Handbook
  • Unsloth GitHub Repository
  • LoRA Land: Fine-Tuned Open-Source LLMs that Outperform GPT-4 (Predibase Blog)
  • Fine-Tuning chat-based LLM with Multi-Turn Conversational Data (Medium article by Bin Xue)
  • Augmentoolkit (Tool for synthetic data generation from text)
  • Synthetic Data Generation with Llama 3 for Distillation and Fine-tuning (Hugging Face Blog)
  • Hugging Face documentation on Chat Templating
  • Domain-Specific Datasets on Hugging Face (Blog)
  • Argilla - Domain-Specific Datasets Welcome (Hugging Face Spaces)
  • r/LocalLLaMA (Reddit community)

Conclusion

Fine-tuning LLMs opens up powerful possibilities for customizing AI behavior. By following a structured workflow – from data collection and meticulous processing to model training and evaluation – it's possible to create models that not only perform specific tasks well but also adopt unique communication styles. The milei-gpt project serves as a compelling example of using interview data to achieve this, demonstrating a path for developing highly personalized conversational AI systems. The increasing availability of open-source models and efficient training tools makes this more accessible than ever.

Tags: LLM, Fine-tuning, AI, Python, Hugging Face, Llama 3, RAG, NLP, Diarization, Transcription, Machinelearnear, Milei-GPT