Language modelsState as of: 2026-10

Document generation with small language models

What it takes to get usable project documents out of free language models with 82M to 355M parameters, on hardware that cannot run large models.

Research question: How can small, freely available language models produce usable project documentation on minimal hardware?

Project requirement

A web application that drafts project descriptions and specifications, without hardware for running large models.

Five models compared: DistilGPT2, GPT-Neo 125M, Facebook OPT 125M and 350M, GPT-2 Medium, all through the Hugging Face text-generation pipeline.
Prompting techniques and generation parameters tested alone and in combination.
The application runs the same prompt on every model and shows the outputs side by side.
Quality judged by how well the output meets user expectations, because specification quality is hard to measure with numbers.

There is no universal configuration: a setting that helps one model can degrade another.
One shared, conservative configuration still worked adequately on all five models for the chosen task: temperature 0.3, top_p 0.9, repetition_penalty 1.2, no_repeat_ngram_size 2 and a per-task output length limit.
Long documents work better section by section: each section has its own stored prompt, and the prompts of related sections are added as context.
Completion-style prompts help models that are not instruction-tuned: the prompt ends with the start of the answer, for example "The project title is:".
Complex relationships between parts of a document remain weak and would need fine-tuning.

One shared sampling configuration for all models instead of per-model tuning, and section-by-section generation with related sections as context.

Quality was judged by people, without an automatic metric.
The results hold for the five models tested in 2025; stronger small models have appeared since.
The model is loaded on every request, which is slow; performance work is still open.