Synthetic Data Generation via LLMs: Self-Instruct, STaR, and Constitutional AI
A startup building a step-by-step math tutor for JEE and CBSE Class 12 board prep has a problem that looks, at first, like a data problem but is really a supervision problem. NCERT solution manuals and reference books give final numeric answers for thousands of problems — definite integrals, related-rates, probability, matrices — but very few give the kind of dense, step-annotated reasoning trace a model needs to learn to *explain itself*. Hiring subject-matter teachers to write out 200,000 fully worked solutions at scale is slow and expensive. But an existing large language model can already solve a good fraction of these problems on its own, and — critically — the tutor company already has the one thing that makes this tractable: the ground-truth final answer, from the answer key. That asymmetry — cheap verification, expensive generation — is the entire premise of modern LLM-based synthetic data generation, and it is what this chapter is about.
GANs and diffusion models solve a related but different problem: they approximate a data distribution (usually images, sometimes audio or tabular rows) by learning to fool a discriminator or to reverse a noising process, and the diffusion-model and GAN chapters already in this curriculum walk through those mechanics — adversarial minimax training, the forward/reverse diffusion SDEs, U-Net denoisers, and so on — in detail. This chapter does not repeat that. Instead it goes to the third, increasingly dominant family of synthetic data: text generated by one language model to train another, or to train a later version of itself. This is now how a large share of frontier instruction-tuning, reasoning, and safety data gets made, and it has its own algorithms, its own failure mode, and its own literature — Self-Instruct, STaR, Textbooks Are All You Need, Constitutional AI, and, on the failure side, model collapse. (For completeness: GAN-style synthetic data generation also shows up outside images — CTGAN and similar conditional-GAN architectures generate synthetic tabular rows for privacy-preserving analytics, e.g. synthetic patient records for hospital research — but the generator/discriminator mechanics there are identical to what the sibling chapter already covers, so we will not re-derive them.)
Why "just hire more annotators" stopped scaling
Three constraints made LLM-generated training data unavoidable rather than optional. First, cost: a human-written, expert-checked chain-of-reasoning example for a hard problem can take a trained annotator ten to twenty minutes; at frontier-lab scale, millions of such examples are needed per fine-tuning round. Second, ceiling effects: once a model is already competent, the marginal human annotator often cannot write examples harder or more diverse than what the model itself can already attempt, which means human effort increasingly goes into *judging* outputs rather than *authoring* them. Third, coverage: a fixed human-written dataset has a fixed distribution of instruction types, phrasings, and edge cases; a generative model can be steered — via prompting, temperature, or few-shot seeds — to sweep a much larger space of instruction styles cheaply. The shift, across three landmark papers between 2022 and 2023, was to stop treating a large language model purely as the *product* being trained and start treating it as *labor* — the thing that manufactures the training data for the next model.
Self-Instruct: bootstrapping an instruction set from 175 examples
Wang, Kordi, et al. (Self-Instruct: Aligning Language Models with Self-Generated Instructions, ACL 2023) start from a seed pool of just 175 human-written tasks — an instruction plus one example input/output pair each. The pipeline then iterates: sample a handful of instructions from the pool (a mix of the original seed tasks and instructions the model has already generated in earlier rounds) as in-context few-shot examples, and prompt a base LLM to produce a *new* instruction in the same style. The model then generates plausible input/output instances for that new instruction — using an "input-first" strategy for generation tasks and an "output-first" strategy for classification-style tasks, since asking a model to invent inputs for a fixed set of output labels tends to produce a more balanced label distribution than asking it to invent labels for a fixed input. Every candidate is passed through a cheap filter before being added to the pool: instructions with high n-gram (ROUGE-L) overlap against anything already in the pool are dropped to preserve diversity, and heuristic checks strip out instructions a text-only model plainly cannot execute (asking it to "look at the attached image," for instance) or where the instruction and its own output are near-identical. Iterated to convergence, this bootstrapped roughly 52,000 instructions and about 82,000 instance pairs from that 175-example seed — and fine-tuning the same base GPT-3 model on its own bootstrapped output produced a 33-percentage-point absolute improvement on the Super-NaturalInstructions benchmark over the un-tuned base model, getting it into the same neighborhood as InstructGPT despite using zero additional human-written instructions beyond the tiny seed set.
The diversity filter is the load-bearing piece, and it is worth seeing as code, because it is the part students most often skip past as "just deduplication":
def rouge_l_overlap(text_a, text_b):
# (assumed helper, not shown) returns the ROUGE-L F1 score
# between two strings, in the range [0, 1].
...
def is_diverse_enough(candidate_instruction, existing_pool, threshold=0.7):
"""Self-Instruct's diversity gate: reject a freshly generated
instruction if it overlaps too heavily with anything already
accepted into the pool, so the model cannot collapse onto
generating the same handful of instruction templates."""
for existing in existing_pool:
if rouge_l_overlap(candidate_instruction, existing) > threshold:
return False
return True
Without this gate, an LLM asked "write a new instruction like these" repeatedly gravitates toward a small number of high-probability phrasings — "Write a short story about...", "Summarize the following text" — because those are exactly the completions the model's own training distribution makes most likely. The filter is not cleaning noise out of the generation process; it is actively correcting for the generator's own mode-seeking bias, which is precisely the mechanism that, left unchecked across generations, becomes model collapse (see below).
Textbooks Are All You Need: curriculum quality over corpus size
Gunasekar et al. (Textbooks Are All You Need, Microsoft Research, June 2023) took a different angle on the same idea: instead of using an LLM to generate more instructions, use one to generate *better-organized explanatory content* — synthetic textbook passages and synthetic coding exercises, written expressly to be pedagogically dense rather than to mirror the noisy diversity of raw internet text. Their resulting model, phi-1, has roughly 1.3 billion parameters — roughly an order of magnitude smaller than the code models it was benchmarked against — and was pretrained on the order of a few billion tokens: a filtered slice of web code selected by a quality classifier (itself bootstrapped by having GPT-4 judge a small sample of documents for "textbook-like" educational value), combined with GPT-3.5–synthesized textbook explanations and small synthetic Python exercises used for a final fine-tuning stage. Despite its size, phi-1 reached roughly 50.6% pass@1 on the HumanEval code-generation benchmark, competitive with or ahead of code models more than ten times its parameter count trained on far larger, unfiltered corpora. The headline finding was not "synthetic data is magic" — it was that *data quality and pedagogical structure can substitute for raw data quantity*, and that an LLM is a remarkably effective tool for manufacturing exactly the kind of clean, worked-example-dense text that textbooks contain and the open web mostly does not.
STaR: rejection sampling turns "generate" into "generate and verify"
Self-Instruct and phi-1 both still rely, ultimately, on a separate teacher model (or classifier) judging quality. Zelikman, Wu, Mu, and Goodman's STaR — Self-Taught Reasoner (NeurIPS 2022) — replaces that with something stronger whenever it is available: a *ground-truth check*, exactly the situation our JEE tutor startup is in. The algorithm: for each problem with a known correct final answer, sample one or more full chain-of-thought solutions from the current model. Check whether the model's final answer matches the known answer. Keep the reasoning chains that got the right answer and use them as fine-tuning data — this is rejection sampling, and it is doing something Self-Instruct's ROUGE filter cannot: it is checking *correctness*, not just novelty. For problems where every sampled chain got the wrong final answer, STaR adds one more trick — "rationalization": show the model the correct answer as a hint and ask it to generate a chain of reasoning that would justify that answer, then keep that backward-rationalized chain too, since even a hint-derived explanation still gives the model useful practice at connecting reasoning steps to conclusions. Fine-tune on the combined pool of self-generated, verified chains, and repeat the entire loop with the improved model as the next generation's problem-solver. STaR is the paper that made explicit something that generalizes across almost every technique in this chapter: the reason model-generated data can improve a model, despite coming from that same family of models, is that the *filter adds information the generator alone does not have*. In Self-Instruct that filter is diversity; in STaR it is ground-truth correctness; in Constitutional AI, discussed next, it is an explicit written principle applied by a critique step.
Worked example: how much does rejection sampling actually cost?
Model the tutor startup's problem concretely. Take a five-dependent-step CBSE Class 12 calculus problem (say, a related-rates question requiring five chained algebraic substitutions) and suppose the teacher LLM has, independently at each of those five critical steps, a 90% chance of getting the step right — a simplifying assumption (steps are not really independent, and 0.9 is illustrative, not measured), but a standard first approximation for reasoning about compounding error.
The probability that an entire five-step chain is fully correct is the product of the five per-step probabilities:
p_step = 0.9 # per-step correctness probability (illustrative)
m = 5 # critical reasoning steps in this problem
N = 8 # chains sampled per problem (rejection-sampling budget)
p_chain = p_step ** m
p_at_least_one = 1 - (1 - p_chain) ** N
expected_correct = N * p_chain
print(round(p_chain, 4)) # 0.5905
print(round(p_at_least_one, 4)) # 0.9992
print(round(expected_correct, 2)) # 4.72
Tracing it by hand: p_chain = 0.9^5 = 0.59049, so a single sampled chain has just under a 59% chance of reaching the right final answer purely from step-level compounding, even with a 90%-reliable teacher at every step. Sample N = 8 independent chains for the same problem, and the chance that at least one of them is fully correct is 1 - (1 - 0.59049)^8 = 1 - 0.40951^8 ≈ 1 - 0.00079 = 0.9992 — over 99.9%. That is the STaR promise: even a teacher that is individually unreliable on long chains becomes a near-certain source of at least one usable, correct chain per problem once you sample enough times and verify against the answer key. The expected *number* of usable chains per problem is 8 × 0.59049 ≈ 4.72 — meaning roughly 59% of everything generated survives the filter, and the other 41% is discarded compute. Put differently, generating one verified training example costs, on average, 1 / 0.59049 ≈ 1.69 sampled chains — that ratio is the real unit economics of rejection-sampling-based synthetic data generation, and it is the number a production team would actually put in a GPU-hours budget, not "we need N=8."
Constitutional AI: synthetic preference data instead of synthetic answers
Bai et al. (Constitutional AI: Harmlessness from AI Feedback, Anthropic, 2022) apply the same "generator plus a verifying principle" pattern to a different target: not reasoning correctness, but preference labels for RLHF-style training. Stage one (supervised): the model produces a response, then critiques its own response against a short written list of principles (the "constitution" — things like "choose the response that is least likely to be harmful"), then revises the response according to its own critique; this generate-critique-revise loop runs a few times and the revised outputs become supervised fine-tuning data. Stage two (reinforcement learning from AI feedback, RLAIF): the model generates pairs of candidate responses to the same prompt, and a separate evaluation pass — again guided by the constitution rather than by a human rater — labels which response better satisfies the principles. That AI-generated preference dataset trains a preference model, which then supplies the reward signal for RL fine-tuning, in place of the human-labeled comparisons that ordinary RLHF requires for the harmlessness objective specifically (helpfulness comparisons in the paper's setup still came from humans). The synthetic artifact here is not a reasoning chain or an instruction — it is a *preference judgment*, and the "ground truth" it is checked against is not a fact but a stated, inspectable rule, which is what lets this scale past what a team of human safety reviewers could label by hand.
Diagram: verified-generation pipeline versus unfiltered recursive collapse
The failure mode: model collapse when nobody verifies
The bottom half of the diagram above is the cautionary half. Shumailov et al. (AI models collapse when trained on recursively generated data, Nature, 2024) show what happens to the STaR-style loop if the "verifier" step is removed — that is, if a model is repeatedly retrained on its *own unfiltered* output, or on a growing pool of web text that increasingly consists of earlier models' unverified generations, generation after generation. Every time a model samples from a learned distribution and that sample is then used to re-estimate the distribution, two error sources compound: finite-sample statistical error (rare events in the tail of the true distribution are underrepresented in any finite sample, and can vanish entirely by chance) and the learner's own approximation error (a neural network is itself a smoothing, mode-seeking approximator of whatever distribution it is trained on). Run this loop across several generations with no correction step, and the effect is directional, not random: variance shrinks, low-probability but real phenomena (rare word senses, minority dialects, unusual but valid problem-solving strategies) disappear first, and the distribution the model represents collapses toward a narrow, low-diversity peak around whatever was already most common — exactly the shrinking-bars pattern in the diagram's Gen 0 → Gen 1 → Gen 2 sequence. Nothing in the model's own training loss ever signals that this is happening; each generation's model can look, by ordinary loss metrics, like it is fitting its training data just fine, because it is — it is fitting a training set that is itself already narrower than the one before it.
Common misconception, corrected
The intuitive objection students raise at this point is: "If model B is trained on model A's own output, B cannot possibly know more than A — you cannot get information out of a system that you did not put in, so synthetic data from an LLM is just recycled information and training on it is a waste of compute." This is wrong, and the STaR arithmetic above shows exactly where the intuition breaks. Model A does not simply hand its raw output to model B; the pipeline inserts an external, independent source of information at the filtering step — the ground-truth answer key, in STaR's case, or the constitution's stated principles, in Constitutional AI's case, or the ROUGE-based diversity constraint, in Self-Instruct's case. That filter is not generated by the model at all; it comes from outside the model-generation loop, and it is what lets the *curated subset* of A's output carry more reliable signal than A's raw output distribution does. Concretely: model A might only get 59% of five-step problems fully right per attempt (the p_chain from the worked example), but the training set built by keeping only the verified 59% is, by construction, 100% correct on the criterion that was checked — a strictly higher-quality dataset than A's unfiltered output, even though every sentence in it originated from A. Phi-1 tells the same story from a different angle: a synthetic corpus curated for pedagogical density beat a synthetic corpus that was just "the internet, but bigger," which means quality-filtering, not sheer volume of model output, is what converts self-generated text into a genuine capability gain. Model collapse is the demonstration of what goes wrong specifically when that external filter is removed or bypassed — it is not evidence against synthetic data broadly, it is evidence about *unverified* synthetic data specifically, and the whole point of STaR, Self-Instruct's diversity gate, and Constitutional AI's principle-based critique is to be the filter that model collapse shows you cannot skip.
Active recall
Attempt each question before reading its answer.
Q1. Self-Instruct filters candidate instructions by novelty (ROUGE-L overlap against the existing pool); STaR filters candidate reasoning chains by correctness (final-answer match against a ground-truth key). Why can't STaR's filter be used for the kind of open-ended instructions Self-Instruct generates (e.g., "write a haiku about monsoon season")?
A1. STaR's filter requires an externally checkable ground truth — a single correct final answer to compare against. Open-ended generative instructions (write a poem, summarize an article, brainstorm names) have no unique correct output to check against, so there is nothing to verify a sampled response against. Self-Instruct's novelty filter sidesteps this by checking a property that *is* well-defined for any text regardless of task type — surface-level overlap with existing pool entries — at the cost of never confirming that the generated instruction-output pair is actually *correct*, only that it is *different*. This is exactly why Self-Instruct needs additional heuristic quality checks (length limits, keyword filters) that STaR does not: it has no correctness signal to lean on, so it must approximate quality through several weaker proxies at once.
Q2. Using the worked example's numbers (p_step = 0.9, m = 5, so p_chain = 0.59049), recompute p_at_least_one and expected_correct if the team doubles the sampling budget from N = 8 to N = 16. What changes, and what stays the same, and why does that matter for a production budget?
A2. p_chain is unaffected by N — it depends only on the teacher's per-step reliability and the number of reasoning steps, both fixed here — so it stays at 0.59049. p_at_least_one = 1 − (1 − 0.59049)^16 = 1 − 0.40951^16 ≈ 1 − 0.0000006 ≈ 0.9999994, essentially 100%; it was already 99.92% at N = 8, so doubling the budget buys almost nothing on "did we get at least one usable chain." expected_correct = 16 × 0.59049 ≈ 9.45, almost exactly double the N = 8 value of 4.72 — and inference cost (16 forward generations per problem instead of 8) also doubles. So the ripple is: N linearly scales both the *size* of the resulting training set and the compute *cost*, at a constant cost-per-usable-example of about 1/0.59049 ≈ 1.69 generations per verified chain, regardless of N. The lesson for a production budget: once p_at_least_one is already near-saturated (as it is by N = 8 here), increasing N further is not "insurance against getting zero good examples" — it is a straightforward, linear-cost decision about how large a training set you want, at a fixed and unavoidable price per example set by p_chain, not by N.
Q3. Same setup, but now the teacher model is weaker: p_step drops from 0.9 to 0.8 (keep m = 5, N = 8). Recompute p_chain, p_at_least_one, and expected_correct, and explain the cost implication.
A3. p_chain = 0.8^5 = 0.32768 (down sharply from 0.59049 — a small per-step drop compounds a lot over five steps). p_at_least_one = 1 − (1 − 0.32768)^8 = 1 − 0.67232^8 ≈ 1 − 0.04175 ≈ 0.9583, still high (95.8%) but noticeably lower than the 99.92% with the stronger teacher. expected_correct = 8 × 0.32768 ≈ 2.62, roughly half of the 4.72 chains yielded by the stronger teacher at the same N and same compute spend. Cost per usable example rises to 1/0.32768 ≈ 3.05 generations, about 1.8× more expensive than the p_step = 0.9 case (1.69). The broader point: "chance of getting at least one correct chain per problem" (p_at_least_one) degrades gracefully and can hide a much steeper collapse in *yield* (expected_correct) and *unit cost* — a team monitoring only "did we get at least one" would see 95.8% and assume things were basically fine, while their per-example compute cost had actually risen by 80%.
Q4. Phi-1 reached roughly 50.6% pass@1 on HumanEval at 1.3B parameters, heavily trained on synthetic textbook and exercise data, outperforming models many times its size trained on much larger unfiltered corpora. What's the actual mechanism that makes this possible, given that the synthetic content ultimately traces back to a larger teacher model (GPT-3.5/GPT-4) that phi-1 itself does not exceed on general capability?
A4. Two separate mechanisms combine. First, curation compresses out low-information text: ordinary web-scraped code corpora are dominated by boilerplate, poorly explained snippets, and near-duplicate content, so a much smaller volume of *densely explanatory, worked-example-rich* text can carry more learnable signal per token than a much larger volume of noisy text — the quality classifier (itself bootstrapped from a small GPT-4-labeled sample) is doing the same "filter adds information" job that STaR's answer-check and Self-Instruct's diversity check do elsewhere in this chapter. Second, the comparison being made is not "phi-1 versus GPT-4" but "phi-1 versus other models of similar or larger size trained the ordinary way" — phi-1 is not exceeding its teacher's general capability, it is using teacher-generated, curated content to reach a *narrower* target (Python code generation specifically) far more parameter-efficiently than models trained on unfiltered corpora of similar or greater size. Distillation-plus-curation lets a small model punch above the weight class set by parameter count, without ever punching above the teacher that generated its training data.
Q5. In Constitutional AI's two-stage pipeline, what specific category of human input is being replaced, and what replaces it?
A5. Human-provided harmlessness preference labels — a person reading two candidate responses and marking which one is safer/less harmful — are replaced by AI-generated preference labels: the model itself compares response pairs against a written constitution (a short explicit list of principles) and produces the preference judgment that would otherwise require a human rater. This is used to train the preference model that supplies the reward signal in the RL stage (RLAIF). Note what is *not* replaced in the paper's original setup: helpfulness preference labels still came from human raters — the substitution is scoped specifically to the harmlessness objective, where the "correct" judgment can be operationalized against explicit written rules rather than requiring nuanced human judgment about what a user actually wanted.
Q6. STaR's loop explicitly iterates — a model's own verified output becomes training data for a next-generation model, which then generates the data for the generation after that. Given the model-collapse mechanism described above, why doesn't STaR's iterated loop eventually collapse the same way, and under what condition *would* it start to?
A6. STaR's loop is protected specifically because every example entering the training set at every generation has passed through the same external, un-degrading filter — the ground-truth answer key does not get blurrier or narrower as generations pass, since it is fixed, human-curated data outside the generation loop, not something re-estimated from the model's own samples. Model collapse arises specifically when a distribution is *re-estimated from samples of the previous generation's model* with no outside correction; STaR never re-estimates the *set of problems or their correct answers* from model output — it only ever re-estimates the *reasoning chains*, which are always checked against a fixed external target. It would start to degrade toward collapse if the verifier itself became a proxy that could be gamed or was itself model-generated and imperfect — for example, if the "ground truth" answers used for later generations were themselves harvested from an earlier generation's own high-confidence outputs rather than from an independent source, or if the verifier only checked the final numeric answer while reasoning styles narrowed unnoticed onto a small number of templated argument structures generation after generation (correct answers, decreasingly diverse justifications) — a subtler, style-level version of the same collapse the diagram shows for full-distribution unfiltered retraining.
Think About It
Think about this: How would you explain synthetic data generation: gans, diffusion models, and llm-based data creation to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind synthetic data generation: gans, diffusion models, and llm-based data creation, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.