模型与 Agent 编程

# Implement LoRA on a Linear Layer **Difficulty**: 🟡 Medium --- ### Problem Statement Low-Rank Adaptation (LoRA) is one of the most important parameter-efficient fine-tuning techniques in modern deep learning. Instead of fine-tuning all parameters of a pretrained model, LoRA freezes the original weights and injects small, trainable low-rank matrices into each layer. This dramatically reduces the number of trainable parameters while maintaining model quality. The key idea: instead of learning a full weight update ΔW (d_in × d_out), LoRA decomposes it into two low-rank matrices: ΔW = B × A, where A is (d_in, r) and B is (r, d_out), with rank r << min(d_in, d_out). The forward pass becomes: output = Wx + (α/r) × x @ A @ B Your task is to implement LoRA from scratch, apply it to an existing model, and merge the adapted weights back. --- ### Requirements 1. **LoRALinear** — A module that wraps an existing `nn.Linear`, adds low-rank matrices A and B, and freezes the original weights. 2. **apply_lora** — A function that injects LoRA into specified layers of an existing model. 3. **merge_lora** — A function that merges LoRA weights back into the original linear layer (W_new = W + (α/r) × BᵀAᵀ transposed appropriately). 4. **Parameter Efficiency** — Show that trainable parameters are much fewer than total parameters. 5. **Training** — Train a simple model with LoRA on synthetic data. --- ### Constraints - A should be initialized with Kaiming uniform; B should be initialized to zeros (so ΔW starts at zero). - Original weights must be frozen (requires_grad=False). - Merged model must produce identical outputs to the LoRA model. - Do **not** use any existing LoRA libraries (peft, loralib, etc.). ---