Low-Rank Adaptation (LoRA) has become the default way to fine-tune large language models, offering large computational savings by updating only a small fraction of parameters. The authors of the original LoRA paper found that fine-tuning works even at a rank as low as 1 or 2.
Our recent experiments with vision-language models on a parametric CAD dataset reveal a critical limitation.
Setup
We ran fine-tuning experiments on a CAD dataset using two vision-language models — models that take text and image concurrently as input.
- LLaVA 1.6 (34B parameters) — full fine-tuning compared against LoRA-based fine-tuning.
- GPT-4 (1T+ parameters) — fine-tuned via an undisclosed method, likely LoRA given the parameter count and OpenAI's guidance to use hundreds to low thousands of examples.
The task required models to develop spatial reasoning and geometric understanding — domains significantly outside their original training distributions.
Evaluation metric
We used Intersection-over-Union (IoU) as the primary metric for geometric accuracy — the overlap between predicted and ground-truth regions.
Values range from 0 (no overlap) to 1 (perfect agreement).
Results
The gap between fine-tuning approaches was substantial.
Not low-rank
LoRA assumes that fine-tuning updates to pre-trained weights have low intrinsic rank — that the required weight changes are well-approximated by low-rank matrices. That holds for tasks inside the model's existing capability distribution: domain adaptation, instruction following.
Learning spatial reasoning and 3D geometric understanding almost certainly requires high-rank updates. The task is simply too far from anything in the model's original training set.
The trillion-parameter paradox
Despite roughly 30× more parameters than LLaVA 1.6, GPT-4 reached far lower geometric accuracy (0.49 vs 0.75). We can't determine OpenAI's methodology definitively, but if GPT-4 was fine-tuned with LoRA — as the API-based fine-tuning limits suggest — the lack of improvement is exactly what we'd expect.
Implications
These findings matter for StoryGold as we train a foundation model for generating parametric, editable 3D models.
Much more data needed
Full fine-tuning needs ~100,000+ high-quality examples — orders of magnitude beyond LoRA's thousands.
Nontrivial compute
Training runs on many of the most expensive GPUs (A100/H100+), which has to be budgeted for.
Data is a moat
No parametric CAD dataset approximating professional design exists. Whoever builds it trains the frontier model.
Open source is superior
Open models outperformed proprietary ones here — so we can train openly and release back to the community.
Conclusion
LoRA remains valuable for efficient adaptation in many scenarios, but it should not be treated as universally equivalent to full fine-tuning. Where a model has to develop genuinely new capabilities — spatial reasoning, geometric understanding, anything far from the pre-training distribution — full fine-tuning will likely be necessary.
As an aside, this underscores the value of open-source model development: it lets researchers make informed decisions about fine-tuning strategy, understand model limitations, and keep full control over training.
