How to Fine-Tune Phi-3 LoRA Locally 2026

Introduction

Phi-3, Microsoft's family of lightweight models, is revolutionizing on-device AI with performance rivaling giants like Llama-3 8B but in just 3.8B parameters. This advanced tutorial guides you through fine-tuning Phi-3-mini-4k-instruct locally with LoRA (Low-Rank Adaptation), a VRAM-efficient technique that adapts the model to your data without full retraining. Imagine customizing Phi-3 for business datasets like legal Q&A or specialized code: gain 20-30% accuracy in <1 hour on an RTX 4090.

Why it matters in 2026? Cloud costs are skyrocketing, privacy is paramount, and edge devices (phones, IoT) demand models under 5GB. We cover 4-bit quantization, SFTTrainer with PEFT/TRL, optimizations like gradient checkpointing, and API deployment. Result: a mergeable LoRA ready for production, tested on datasets like Alpaca. Prep your CUDA 12+ GPU and follow step-by-step for a reproducible workflow.

Prerequisites

Python 3.10+ with CUDA 12.1+ (RTX 30/40 series, ≥12GB VRAM recommended)
pip ≥23
Hugging Face access (free token for private datasets/models)
50GB free disk space
Familiarity with PyTorch and Transformers (intermediate+ level)

Installing Dependencies

install_deps.sh

#!/bin/bash
pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers==4.44.2 datasets==2.21.0
pip install peft==0.12.0 trl==0.9.6 accelerate==0.33.0
pip install bitsandbytes==0.43.3
pip install huggingface_hub
huggingface-cli login

This script installs PyTorch with CUDA 12.1 for NVIDIA GPUs, followed by essential libraries: Transformers for Phi-3, PEFT/TRL for LoRA/SFT, and bitsandbytes for 4-bit quantization (cuts VRAM by 70%). HF login is required to pull the gated model. Run as root if needed; check nvidia-smi after install.

Loading and Testing the Base Model

Before fine-tuning, load Phi-3 in 4-bit to validate your setup. Phi-3-mini-4k-instruct shines at instruction-following thanks to pretraining on 3.3T synthetic tokens. Use device_map="auto" for multi-GPU sharding if available, and trust_remote_code=True due to Phi's custom decoder.

Loading Quantized Phi-3 Script

load_phi.py

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "microsoft/Phi-3-mini-4k-instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
    attn_implementation="flash_attention_2"
)

prompt = "<|user|>\nExplique LoRA en une phrase.<|end|>\n<|assistant|>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

This script loads Phi-3 in NF4 4-bit (2.2GB VRAM), applies FlashAttention2 for 2x speed, and tests inference. Double quantization boosts accuracy. Pitfall: without trust_remote_code, it crashes on the custom tokenizer. Example output: concise LoRA definition.

Preparing the Custom Dataset

For efficient fine-tuning, format your data in Phi's chat template (<|user|> ...<|end|> <|assistant|>). We use an Alpaca subset (100 samples) + 5 custom ones for a quick demo. Tokenize into input_ids and labels with masking on the prompt.

Preparing and Formatting Dataset

prepare_dataset.py

from datasets import Dataset
from transformers import AutoTokenizer

model_name = "microsoft/Phi-3-mini-4k-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

def format_prompt(example):
    prompt = f"<|user|>\n{example['instruction']}<|end|>\n<|assistant|>\n{example['output']}<|end|>\n"
    return tokenizer(prompt, truncation=True, max_length=2048)

# Dataset Alpaca subset + custom
data = [
    {"instruction": "Qu'est-ce que LoRA ?", "output": "LoRA adapte LLMs via low-rank matrices sans toucher poids base."},
    {"instruction": "Avantages de Phi-3 ?", "output": "Léger (3.8B), rapide, précis sur code/math, open weights."}
] * 50  # x50 pour 100 samples
alpaca_subset = Dataset.from_list(data)

tokenized_dataset = alpaca_subset.map(format_prompt, remove_columns=alpaca_subset.column_names)
tokenized_dataset = tokenized_dataset.train_test_split(test_size=0.1)
print(f"Train: {len(tokenized_dataset['train'])}, Eval: {len(tokenized_dataset['test'])}")
tokenized_dataset.save_to_disk("phi_dataset")

This script formats 100 Alpaca-like samples into Phi's template and tokenizes to 2048 max length. It fixes the pad_token bug with padding/EOS. Splits 90/10 for evaluation. Saves locally for reuse; pitfall: without truncation, long texts cause OOM.

Fine-Tuning with LoRA and SFTTrainer

finetune_phi.py

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
from datasets import load_from_disk

model_name = "microsoft/Phi-3-mini-4k-instruct"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
    model_name, quantization_config=bnb_config,
    device_map="auto", trust_remote_code=True,
    attn_implementation="flash_attention_2"
)

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    inference_mode=False, r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["qkv_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
)
model = get_peft_model(model, lora_config)

train_dataset = load_from_disk("phi_dataset")['train']
eval_dataset = load_from_disk("phi_dataset")['test']

training_args = TrainingArguments(
    output_dir="phi-lora-finetuned",
    num_train_epochs=3, per_device_train_batch_size=2,
    gradient_accumulation_steps=4, optim="paged_adamw_8bit",
    learning_rate=2e-4, fp16=True, logging_steps=10,
    save_steps=50, evaluation_strategy="steps", eval_steps=50,
    warmup_steps=20, report_to=None, gradient_checkpointing=True,
    max_grad_norm=0.3, remove_unused_columns=False
)

trainer = SFTTrainer(
    model=model, tokenizer=tokenizer,
    train_dataset=train_dataset, eval_dataset=eval_dataset,
    dataset_text_field=None, max_seq_length=2048,
    args=training_args, packing=False, neftune_noise_alpha=None
)
trainer.train()
trainer.save_model("phi-lora-final")
model.config.use_cache = True

Full script: applies LoRA to optimal Phi modules (qkv/gate/up/down), uses SFTTrainer on tokenized dataset. Optimizations: adamw_8bit, gradient checkpointing (saves 3x VRAM), effective batch 2x4=8. 3 epochs ~30min on 4090. Pitfall: without packing=False, excessive padding slows training.

Inference and Post-Training Merge

After training, test the LoRA adapter alone then merged (for ONNX/edge deployment). model.config.use_cache=True enables fast KV-cache inference.

LoRA Inference + Merge

inference_merge.py

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False)

base_model_name = "microsoft/Phi-3-mini-4k-instruct"
peft_model_path = "phi-lora-final"

tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(base_model_name, torch_dtype="auto", device_map="auto", trust_remote_code=True)
model = PeftModel.from_pretrained(model, peft_model_path)

prompt = "<|user|>\nQu'est-ce que LoRA après fine-tune ?<|end|>\n<|assistant|>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.1, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Merge
model = model.merge_and_unload()
merged_path = "phi-merged"
model.save_pretrained(merged_path)
tokenizer.save_pretrained(merged_path)
print(f"Modèle mergé sauvé : {merged_path}")

Loads PEFT model, infers on custom prompt (improved post-tuning), then merges base + adapters into full model. SDP kernel boosts FlashAttn. Pitfall: forget merge_and_unload() and LoRA stays active in production. ~4GB file ready for ONNX export.

FastAPI Deployment

api_phi.py

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

app = FastAPI(title="Phi-3 LoRA API")
model_path = "phi-merged"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)

class ChatRequest(BaseModel):
    prompt: str

@app.post("/chat")
def chat(req: ChatRequest):
    full_prompt = f"<|user|>\n{req.prompt}<|end|>\n<|assistant|>\n"
    inputs = tokenizer(full_prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return {"response": response.split("<|assistant|>")[1].strip()}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

FastAPI exposes /chat endpoint for merged LoRA inference. Uses bfloat16 for production speed/accuracy. Pydantic validates inputs. Run with uvicorn api_phi:app --reload; pitfall: without no_grad, VRAM leaks in loops.

Best Practices

Always quantize: NF4 + double_quant for <4GB VRAM on 3.8B models.
Precise target modules: qkv_proj + gate/up/down for Phi (avoids overfitting).
Regular evaluation: steps=50, track perplexity for early stopping.
Merge before production: simplifies deployment, ONNXRuntime compatible.
High-quality dataset: 1000+ diverse samples > raw volume; manually validate template.

Common Errors to Avoid

OOM despite quantization: drop batch_size to 1, enable gradient_checkpointing=True.
Pad_token None: explicitly set to eos_token, or loss becomes NaN.
Malformed template: test apply_chat_template before dataset map.
No warmup: LR spikes cause divergence; min 10% of steps.

Next Steps

Fine-tune Phi-3-vision for multimodal (docs HF Phi-3).
Export to ONNX: optimum-cli export onnx --model phi-merged phi_onnx/ for edge.
Scale with Axolotl or Unsloth (5x faster).
Check out our AI training courses at Learni for LLM mastery.

How to Fine-Tune Phi-3 with LoRA Locally in 2026