Prepare a one-page proposal for an individual investigation. Survey-only projects are not allowed. Include an implementation, experiment, or tested artifact, with evidence that answers your question.
Question and motivation. Explain what you want to learn, why it matters, and what prior work leaves unclear.
First experiment. Name accessible data or a generator, a simple baseline, one comparison, and an evaluation that can answer the question. Keep development and evaluation examples separate.
Feasibility. State CPU cores and RAM, GPU count and model, memory per GPU, storage, runtime per run, and total runs. Separate minimum hardware from optional parallel runs; total GPU-hours must include baselines and retries. Give a smaller CPU, one-GPU, or cached-output fallback and plan an early timing/memory pilot.
October 21One-page proposal
November 25Progress update, up to two pages
December 30Main report, up to six pages; code and experiment record
Submit privately through eLearning by 23:59, Asia/Shanghai (UTC+08:00). References and appendices are outside the final report’s six-page limit. All students use the same requirements and rubric. A sound negative result can receive full credit.
Use the ACL template for the final report. Record commands, code/data/model versions, seeds, configurations, compute use, and artifact locations; disclose material AI assistance and how you checked it. If reusing assignment work, identify the additional project question and experiments. No presentation, poster, or oral defense is required.
2. Resource requirements
Prefer CPU studies and short inference runs. Our GPU pool is shared. Choose the smallest hardware setup that can answer your question; one- and two-GPU plans should keep total use modest.
CPU · recommended starting point
Tokenizer and data audits, retrieval, verifier tests, or a measured cache-storage experiment. State CPU cores, RAM, and elapsed time; the minimum study needs no GPU.
1 GPU · inference or short training
Reuse a small checkpoint, train a 5M–30M teaching model, or fit adapters on a 0.6B model. Cards give a memory target per device and a total time estimate to check in a pilot.
2 GPUs · a bounded alternative
Run independent seeds or conditions in parallel, or test a supported draft/target split. Explain each device’s role and keep a one-GPU fallback. Two GPUs are optional in the suggested studies.
3+ GPUs · high demand
A project that needs this many devices is a poor fit for our small shared pool. Reduce the model, data, or number of runs before selecting it. Larger plans can use resources you obtain yourself; state their source and availability in the proposal.
Be careful with GPU usage; our course pool is limited.
Card budgets are unpiloted planning estimates, not confirmed allocations. The proposed shared-pool ceiling is 8 GPU-hours; it does not limit resources you obtain elsewhere. Count every device, baseline, comparison, and retry: two GPUs for two hours use four GPU-hours. Memory targets are per device; two GPUs do not automatically combine their memory.
Start with a topic you care about: a hobby, language, research question, community, or practical problem. The suggested projects are starting points, and you are welcome to propose a different idea.
Student-proposed project
Question:Which specific question about language processing would you be excited to investigate, and what evidence would change your mind?
Proposed ceiling:24 GB memory target per GPU; up to 8 GPU-hours total across all devices and runs.
Propose CPU, one-GPU, or two-GPU use and distinguish necessary devices from optional parallel runs. Larger setups are possible with resources you can obtain yourself. State the source, access, and total cost of all baselines, ablations, and reruns; scope any request to the small course pool separately. The displayed budget is an initial shared-pool planning target.
Follow this course’s individual scope, checkpoints, and rubric.
CMSC 723 project proposal guidance — Mohit Iyyer and the UMD CMSC 723 course staff; Fall 2026 course; proposal template supplied by the instructor. Course planning reference; our course requirements apply · UMD proposal template
Use the planning questions only; Fudan’s dates and page limits apply.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Choose one question with a clear NLP or LLM component that one student can investigate this semester. Explain its motivation and connection to related work, identify accessible data or a small generator, and define one main comparison. For example, study terminology preservation in translations, search over historical documents, or a language tool for a hobby you know well.
Baseline and controls
Begin with the simplest credible solution: a rule, majority prediction, lexical retrieval, or an existing small model. Identify one change to compare against it, and explain which outcome would support or weaken your hypothesis.
Evaluation
Define a held-out set, relevant quantitative or qualitative criteria, a baseline comparison, and an error analysis. Implement and test the proposed method, system, or audit; a literature survey or app demonstration alone is insufficient. Record the time and resources used.
Minimum complete outcome
An individual investigation with a reproducible implementation or tested artifact, a meaningful comparison, and a clear explanation of findings and limitations. A well-supported negative result is valid and uses the same course rubric as a listed topic.
Preparation
Use the one-page proposal to cover motivation, related work, approach and baseline, data access, evaluation, compute, tools, and a short schedule. Identify what you will implement and what you will reuse, including AI coding assistance. Plan an early baseline and reserve time for analysis and writing. Connect your question to relevant prior work; the links below provide planning guidance.
Planning budget
Describe a CPU, one-GPU, two-GPU, or larger setup, separating the minimum from optional hardware. Identify CPU cores/RAM, GPU models and memory per device, storage, total runs, total GPU-hours, and a smaller fallback. State which resources you can obtain yourself and which you would request from the shared course pool.
Risks and smaller fallback
Narrow to one language, one dataset, fewer examples, or a smaller model if data or compute access is uncertain. Discuss feasibility with the instructor through the normal proposal feedback process. Submit the proposal by October 21; the final report has up to six pages of main content, excluding references and appendices.
Possible upstream contribution
A custom project may also produce an upstream contribution if it meets a real maintainer need. Document its review or acceptance status; the same course rubric and maximum score apply, with no merge requirement or extra-credit track.
4. Open-source contributions
Build and evaluate a useful contribution to an existing project: a task, verifier repair, model adapter, or tool. Include tests or a controlled comparison and describe your individual work.
Contribution routes and current limits
Grading: open-source projects use the same requirements, rubric, and maximum score as every other individual project. External acceptance can support the evidence of usefulness and quality; it does not earn extra points under the current policy. A well-supported local investigation can receive full credit even if an upstream contribution is not merged.
A new benchmark task, a verified evaluator repair, a useful model adapter, or a reusable dataset/tool can be a strong project. Agree on a useful contribution with upstream maintainers early. Keep a local experiment and report that answer the project question even if review takes longer than the semester.
When reporting an upstream contribution, include its PR link and review status, a merge-commit link if accepted, your individual contribution, tests and before/after evidence, and how review changed the work. A self-merged personal fork, raw PR count, or repository star count is not evidence of substantive external acceptance. This guidance does not promise a maintainer decision or paper authorship.
One real scientific workflow task, or an evidence-backed verifier/environment repair.
The October 9 guide lists October 5 as the 0.2 PR cutoff. New tasks need an approved proposal and confirmation of a future intake. Task-fix requests have a documented route.
Scoring/calibration robustness, a code fix, or a well-supported question correction.
Question feedback uses the official form; HLE-Rolling contributions go through maintainers. Question acceptance and GitHub code merges are different outcomes; document the actual outcome. Preserve canaries and do not use benchmark answers for training.
One reviewed task, dataset integration, evaluator fix, or useful documentation backed by a working example.
Check the current contribution guide, licenses, existing issues, and maintainer interest before building a duplicate contribution.
These are contribution destinations, not guaranteed openings. Check current rules again before submitting. The instructor's requested public-contribution route concerns an agreed upstream artifact; proposals, graded reports, experiment records, grades, and course assignment solutions still belong on eLearning. Keep upstream reference solutions and evaluation material in the channels required by that project's maintainers, not in this public course repository.
5. Previous projects
For more ideas, see the previous projects and papers. You may implement a paper’s method and evaluate it with a small experiment, following this semester’s requirements. Earlier team sizes, deadlines, report lengths, and grading rules do not apply.
6. Candidate projects
Choose a topic from the sidebar or filter these 48 suggestions by approach, topic, or compute. Your own proposal has a separate section above. Each card shows suggested readings and its resource plan before the detailed experiment. “Unreviewed” means the scope still needs proposal feedback and a small resource pilot.
48 of 48 projects
No projects match these filters. Try a broader term or clear the filters.
Data and tokenization5 projects
Data and tokenizationCPU · no GPU requiredUnreviewed
Understand the byte-level alternative and noise-robustness question.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Freeze 1,000 short documents from two languages and code; compare three released tokenizers including Qwen3.5. Add a 100-pair, manually checked English/Chinese parallel subset.
Baseline and controls
Byte and character counts, plus the course BPE tokenizer. Keep normalization and documents fixed.
Evaluation
Tokens per UTF-8 byte, tokens per character, paired token counts, round-trip failures, and robustness to spaces or Unicode variants. Token counts alone do not measure model quality.
Minimum complete outcome
A reproducible tokenizer audit with paired plots and 20 explained outliers.
Preparation
L01 tokenization; Python text processing and sampling.
Planning budget
A laptop with roughly 8–16 GB RAM. Use bounded data and cache repeated work. The minimum study needs no GPU; time a small CPU pilot before fixing the scope.
Risks and smaller fallback
Use 200 documents and two tokenizers; no model inference or training is needed.
Choose the small checkpoint, tokenizer, and inference settings.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Generate 200 exact arithmetic problems with three meaning-preserving renderings. Use a 0.6B-1.7B model, at most 512 output tokens per problem.
Baseline and controls
Canonical numeral formatting and a Python exact-arithmetic oracle; pair every variant with its original.
Evaluation
Exact answer accuracy, consistency within pairs, token count, and errors by digit length. Do not attribute every formatting effect to tokenization alone.
Minimum complete outcome
A paired error analysis and one tested formatting intervention.
Preparation
L01 tokenization; L02 evaluation; learn model inference and robust answer parsing.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Restrict to integer addition and multiplication, 100 examples, one model.
Data and tokenization1 GPU · short trainingUnreviewed
Understand duplicate detection and train/test overlap controls.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Sample a small, versioned FineWeb2 slice in one language. FineWeb2 is already filtered and deduplicated: compare the as-released slice, additional deduplication, and additional quality filtering. Audit residual duplicates and removed examples before training a 10M-30M decoder on at most 5M tokens per run; use two seeds if the pilot fits.
Baseline and controls
Identical architecture, tokenizer, optimizer, token budget, and a document-disjoint evaluation set formed before filtering. Deduplicate across splits first.
Evaluation
Held-out bits per byte, loss by document type, unique training bytes, residual duplicate rate, and wall time; distinguish additional curation from extra resampling. If further deduplication removes almost nothing, report that limitation rather than treating this as an experiment on raw web data.
Minimum complete outcome
One controlled three-condition result with uncertainty and examples of removed documents.
Preparation
L01 preprocessing and L02 loss; learn a small Transformer training loop after L05-L06.
Planning budget
Start with one shared GPU, usually 16–24 GB. Limit training to a 5M–30M teaching model or adapters on a 0.6B checkpoint. Where a card offers two GPUs, independent seeds or configurations can run in parallel. Count all devices and runs toward the same total GPU-hours; pilot memory and time first.
Risks and smaller fallback
Use a 5M model and 1M tokens; a corpus-only audit is a pilot, not a claim about model improvement.
Data and tokenizationCPU · no GPU requiredUnreviewed
Compare byte-level processing with learned token boundaries.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Fit a smoothed byte n-gram predictor on a separate training slice, then apply fixed-length and entropy-threshold patching to 1M held-out bytes. Use disjoint development text to match average patch length.
Baseline and controls
Fixed-size byte blocks at the same average patch length. Hold predictor and text fixed; count the predictor and patcher runtime.
Evaluation
Patch-length distribution, boundary locations around code/Unicode, allocation near high-surprisal bytes, round-trip correctness, and CPU time. These measurements do not establish downstream model quality.
Minimum complete outcome
A working patcher, a paired boundary audit, and an explanation of where it allocates finer segments. Label the n-gram predictor as an adaptation of BLT.
Preparation
L01 bytes and L02 n-grams; learn conditional entropy and development-set threshold selection.
Planning budget
A laptop with roughly 8–16 GB RAM. Use bounded data and cache repeated work. The minimum study needs no GPU; time a small CPU pilot before fixing the scope.
Risks and smaller fallback
Use 200 KB of text and a byte bigram predictor. Keep the boundary-allocation question; do not train a BLT model.
Data and tokenization1 GPU · short trainingUnreviewed
Understand how byte hashing changes tokenizer dependence.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Add a modest lookup table to a 10M-30M decoder; train on at most 5M tokens. Compare ordinary token hashing with the 2026 byte-based proposal only if the first comparison works.
Baseline and controls
No-memory model and a parameter-matched wider model; identical corpus, tokenizer, training tokens, and seeds.
An ablation that distinguishes memory benefit from capacity or data leakage.
Preparation
L01-L02 tokenization and n-grams; PyTorch; learn residual model components.
Planning budget
Start with one shared GPU, usually 16–24 GB. Limit training to a 5M–30M teaching model or adapters on a 0.6B checkpoint. Where a card offers two GPUs, independent seeds or configurations can run in parallel. Count all devices and runs toward the same total GPU-hours; pilot memory and time first.
Risks and smaller fallback
Use synthetic local-dependency sequences and a 5M model; upstream supplies a demo, not a turnkey small-model training recipe.
Adaptation and training7 projects
Adaptation and training1 GPU · short trainingUnreviewed
Use the 0.6B student rather than reproducing a large training recipe.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use one documented SmolTalk2 subset and LoRA on Qwen3-0.6B. Compare 500-1,000 random, length-balanced, and embedding-diverse examples.
Baseline and controls
Untuned checkpoint and random selection; match training tokens and training steps, and exclude evaluation prompts before selection.
Evaluation
Exact checks on 100 verifiable held-out prompts, domain coverage, output length, and regressions on a separate general set.
Minimum complete outcome
A data-selection experiment with selected IDs, fixed splits, and an explanation of tradeoffs.
Preparation
L01-L04 data and neural training; learn chat templates, response masking, and LoRA.
Planning budget
Start with one shared GPU, usually 16–24 GB. Limit training to a 5M–30M teaching model or adapters on a 0.6B checkpoint. Where a card offers two GPUs, independent seeds or configurations can run in parallel. Count all devices and runs toward the same total GPU-hours; pilot memory and time first.
Risks and smaller fallback
Use 300 examples and two selection policies; reduce context to 1,024 tokens.
Adaptation and trainingCPU · no GPU requiredUnreviewed
Read the small-data classification design; distinguish a frozen probe from SetFit training.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Choose one public dataset from the authors with at most five classes. Use 8, 32, and 128 labeled examples per class for TF-IDF and a frozen small encoder plus linear head. Compare a quantized 0.6B prompted decoder on 100 fixed test examples with a documented few-shot selection rule.
Baseline and controls
TF-IDF plus logistic regression. Keep splits and label access fixed; state the decoder’s few-shot/context limit rather than assuming all training examples fit into its prompt.
Evaluation
Macro-F1, calibration, latency, training/inference cost, and performance under one controlled domain shift.
Minimum complete outcome
Learning curves that support a concrete deployment decision; a leaderboard score alone is insufficient.
Preparation
L03 embeddings and classification; learn pretrained encoders and dataset splits.
Planning budget
A laptop with roughly 8–16 GB RAM. Use bounded data and cache repeated work. The minimum study needs no GPU; time a small CPU pilot before fixing the scope.
Risks and smaller fallback
Use three classes and 50 test examples. Cache decoder outputs on CPU and keep the encoder frozen.
Adaptation and training1 GPU · short trainingUnreviewed
Construct 500 synthetic preference pairs from arithmetic or structured extraction in two conditions: correct-short versus incorrect-long, and correct-long versus incorrect-short. Prefer the correct answer in both conditions. Apply one small LoRA DPO run per condition; isolate any deliberately flipped labels in a separate noise ablation.
Baseline and controls
Untuned model and an SFT model trained on the same preferred answers; control training tokens and preference-pair content.
Evaluation
Held-out exact correctness, response length, sensitivity to 10% flipped preferences, and invalid output rate.
Minimum complete outcome
A shortcut audit with balanced evaluation pairs and examples where the tuning objective fails.
Preparation
Neural LM loss; learn preference pairs and the DPO loss before beginning training.
Planning budget
Start with one shared GPU, usually 16–24 GB. Limit training to a 5M–30M teaching model or adapters on a 0.6B checkpoint. Where a card offers two GPUs, independent seeds or configurations can run in parallel. Count all devices and runs toward the same total GPU-hours; pilot memory and time first.
Risks and smaller fallback
Reduce to 200 pairs and a 0.6B model; do not run a large online preference collection.
Adaptation and trainingCPU · no GPU requiredUnreviewed
Understand the GRPO setting that motivates verifier auditing; no RL run is required.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Generate 100 fresh Countdown-style problems with known valid solutions. Create controlled invalid candidates by changing allowed numbers, operations, or expression structure. Score the same candidates with permissive and strict verifiers; optionally add a small cached model-output set with documented provenance.
Baseline and controls
Strict-oracle scoring, random candidate selection, and permissive scoring. Keep problem-generation seeds disjoint between development and evaluation.
Evaluation
False accepts/rejects by error type, true validity of the selected expression, reward/validity disagreement, and CPU runtime. Controlled invalid examples measure verifier behavior, not the frequency of real model errors.
Minimum complete outcome
A tested verifier, a controlled selection comparison, and concrete reward-hacking cases. This is an offline verifier study, not an RL training result.
Preparation
Python tests, exact arithmetic, and restricted expression parsing; read the reward/GRPO motivation without implementing training.
Planning budget
A laptop with roughly 8–16 GB RAM. Use bounded data and cache repeated work. The minimum study needs no GPU; time a small CPU pilot before fixing the scope.
Risks and smaller fallback
Use 50 problems and three invalid-candidate families. No model training or fresh rollout collection is required.
Adaptation and training1 GPU · short trainingUnreviewed
Distill a small, verified subset of reasoning traces.
Research problem
Question:Does concise reasoning retain accuracy better per training token than full reasoning traces?
Proposed ceiling:24 GB memory target per GPU; up to 8 GPU-hours total across all devices and runs.
Manual trace checking and sequence length are the main risks. Begin with 30–50 short verified examples, one 0.6B student, and two supervision conditions.
Compare rationale supervision with answers-only supervision.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Choose 30–50 short, verifiable public s1 examples or an original exact-arithmetic generator. Document the source and inspect all traces. Compare full reasoning, checked shortened reasoning, and answers only using adapters on one 0.6B student; cap the initial context at 1,024 tokens.
Baseline and controls
Untuned student; run both an equal-example comparison and a clearly labeled equal-token comparison.
Evaluation
Held-out answer accuracy, generated length, training tokens, and error types. Avoid using s1 training questions as test examples.
Minimum complete outcome
A distillation ablation that separates supervision content from training-budget effects.
Start with one shared GPU, usually 16–24 GB. Limit training to a 5M–30M teaching model or adapters on a 0.6B checkpoint. Where a card offers two GPUs, independent seeds or configurations can run in parallel. Count all devices and runs toward the same total GPU-hours; pilot memory and time first.
Risks and smaller fallback
Use 20 short generated problems and two supervision conditions. If the available traces cannot fit without losing essential steps, change the task before training.
Adaptation and training1–2 GPUs · short trainingUnreviewed
Test the reliability of a scaling prediction on tiny models before using it to allocate a larger budget.
Case studyResearch problem
Question:How well does a curve fitted to smaller runs predict a held-out larger configuration, and how sensitive is that prediction to the fitting choices?
Proposed ceiling:24 GB memory target per GPU; up to 8 GPU-hours total across all devices and runs.
Minimum: one GPU running the four tiny configurations sequentially. Optional: use two GPUs for independent configurations or seeds. Two devices for two hours consume four GPU-hours; all runs share the eight-GPU-hour planning total. Pilot the smallest model and reduce token counts if needed.
Read first
CS336 Assignment 3: Scaling — Stanford CS336 course staff; Spring 2026. Official assignment and local-training code
Use the local scaling exercise as a starting point.
Distinguish model size, training tokens, and compute allocation.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Train one decoder family at roughly 5M, 8M, 12M, and 20M parameters on one fixed licensed corpus. Save checkpoints at 0.5M, 1M, and 2M training tokens; use two seeds if the pilot fits. Reserve the largest model for evaluating predictions rather than choosing the fitted curve.
Baseline and controls
Compare a predeclared low-parameter power-law fit with a simple log-linear extrapolation and a constant-loss predictor. Keep tokenizer, validation set, data order, and optimizer policy fixed; report any size-specific tuning.
Evaluation
Held-out loss-prediction error, ranking of candidate allocations, sensitivity to omitted fitting points and seeds, actual training time, and approximate FLOPs. Report uncertainty and distinguish interpolation from extrapolation.
Minimum complete outcome
An auditable run table, an out-of-sample prediction, and evidence explaining whether the prediction is useful within the measured range.
Preparation
Course language-model training and evaluation; learn nonlinear fitting, log-scale plots, compute accounting, and uncertainty estimation. CS336 supplies a larger reference exercise.
Planning budget
Start with one shared GPU, usually 16–24 GB. Limit training to a 5M–30M teaching model or adapters on a 0.6B checkpoint. Where a card offers two GPUs, independent seeds or configurations can run in parallel. Count all devices and runs toward the same total GPU-hours; pilot memory and time first.
Risks and smaller fallback
Use four smaller models and fewer tokens, preserving the held-out configuration. Run locally: Stanford student API access is not assumed. A narrow tiny-model fit cannot establish a frontier-model scaling law.
Adaptation and training1 GPU · short trainingUnreviewed
Use a profiler to identify the bottleneck, then test one optimization including its setup cost.
Case studyResearch problem
Question:When do faster attention or graph compilation repay their overhead in the short training runs a student actually performs?
Proposed ceiling:24 GB memory target per GPU; up to 8 GPU-hours total across all devices and runs.
One GPU, one optimization, and a fixed set of short step counts. Include compilation and profiling in the total; no distributed training or custom kernel is needed.
Read first
CS336 Assignment 2: Systems — Stanford CS336 course staff; Spring 2026. Official assignment and starter code
Plan profiling, correctness checks, and fair timing.
Measure compilation overhead using the installed backend.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use a 5M-20M decoder, one fixed input batch stream, and two sequence lengths. Profile a baseline, then choose either supported optimized attention or graph compilation. Compare runs of 20, 100, and 500 steps on one GPU; check the installed backend supports the chosen method.
Baseline and controls
Unoptimized execution with identical weights, batches, precision, optimizer, and hardware. Check outputs and gradients within documented numerical tolerances, and compare short loss trajectories before interpreting speed.
Evaluation
End-to-end elapsed time including compilation, steady-state tokens/second, peak allocated memory, repeated-run variation, and time spent in attention, optimizer, and data loading. Synchronize accelerator timing and separate warm-up from measurement.
Minimum complete outcome
A profiler-backed bottleneck diagnosis, a correctness-checked change, and a break-even estimate showing when it helps or fails.
Preparation
PyTorch training and tensor shapes; learn profiling, asynchronous GPU timing, warm-up effects, and numerical equivalence checks.
Planning budget
Start with one shared GPU, usually 16–24 GB. Limit training to a 5M–30M teaching model or adapters on a 0.6B checkpoint. Where a card offers two GPUs, independent seeds or configurations can run in parallel. Count all devices and runs toward the same total GPU-hours; pilot memory and time first.
Risks and smaller fallback
Use the smaller model and one sequence length. If GPU kernels are unavailable, analyze CPU execution and one supported optimization; do not report it as a GPU speedup. Distributed training and writing a Triton kernel are outside the minimum scope.
Reasoning and evaluation5 projects
Reasoning and evaluation1 GPU · inferenceUnreviewed
Use 150 generated math/logic problems at two difficulty levels. Compare one long sample, several short samples with majority vote, and answer-verified selection.
Baseline and controls
Greedy direct answer. Match total sampled tokens, including failed attempts; report verifier access separately.
Evaluation
Exact accuracy versus total tokens and wall time, abstentions, and uncertainty over problems.
Minimum complete outcome
A budget/accuracy curve with cases where extra reasoning hurts.
Preparation
L02 probability and evaluation; learn generation settings and answer normalization.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 80 easy problems and two strategies; cap each total problem budget at 2,048 tokens.
Reasoning and evaluation1 GPU · inferenceUnreviewed
Use 150 exact-answer problems and a Qwen3 0.6B-1.7B reasoning checkpoint. Compare a fixed budget with answer-stability and a small thought-sufficiency check.
Baseline and controls
Direct answer and full-budget reasoning; select thresholds only on a development split.
Evaluation
Accuracy, early-stop error, all generated tokens including sufficiency checks, and latency.
Minimum complete outcome
A calibrated stopping policy and a failure analysis of premature confidence.
Preparation
Sampling and evaluation; learn how the selected model separates reasoning from its final answer.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use answer-stability checks at two fixed cutoffs. Label this as an adaptation, not an exact DTSR implementation.
Reasoning and evaluation1 GPU · short trainingUnreviewed
Revisit a 2025 candidate using its 2026 methodological critique.
Paper replication
Question:Does targeted reasoning-step unlearning change answers more than irrelevant or shuffled-text unlearning?
Proposed ceiling:24 GB memory target per GPU; up to 8 GPU-hours total across all devices and runs.
Repeated unlearning runs are costly. Pilot five examples with 10 updates per intervention; restrict updates to one layer or adapters and use 12–20 examples initially.
Choose irrelevant-text controls and measure collateral damage.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Begin with 12–20 short reasoning examples on one 0.6B checkpoint and one layer or adapter. Compare targeted, irrelevant, and shuffled interventions with matched update counts. Pilot five examples with 10 updates per intervention first.
Baseline and controls
Unmodified checkpoint and unrelated-question retention checks. Freeze the tuning protocol before the held-out comparison.
Evaluation
Answer/log-probability changes, intervention success, and general capability damage. A changed answer alone does not establish faithful reasoning.
Minimum complete outcome
A controlled assessment of the metric and a clear account of collateral model changes.
Preparation
PyTorch gradients and LM loss; learn unlearning objectives and intervention controls.
Planning budget
Start with one shared GPU, usually 16–24 GB. Limit training to a 5M–30M teaching model or adapters on a 0.6B checkpoint. Where a card offers two GPUs, independent seeds or configurations can run in parallel. Count all devices and runs toward the same total GPU-hours; pilot memory and time first.
Risks and smaller fallback
Use a descriptive pilot with five examples and one layer, reporting uncertainty and unsuccessful interventions. If only text perturbation is feasible, rename the question because it measures a different notion.
Reasoning and evaluation1 GPU · inferenceUnreviewed
Read about position and verbosity bias in model judges.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Select 100 objective answer pairs from JudgeBench or generate checked math pairs. Swap A/B order and add meaning-preserving verbosity to one answer.
Baseline and controls
Exact verifier where available, random choice, and the unmodified judge prompt.
Evaluation
Agreement with verified labels, swap consistency, verbosity-induced reversals, and judge-token cost.
Minimum complete outcome
An evaluator audit and one validated mitigation; keep ambiguous pairs separate.
Preparation
L02 evaluation; learn paired experimental design and judge prompt formatting.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use one local 0.6B judge and 50 math pairs; stronger performance is not needed to study failure.
Reasoning and evaluation1 GPU · inferenceUnreviewed
Strengthen tests and check semantic equivalence of task variants.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Pin a LiveCodeBench release and select 30 easy problems, or author 30 small programming tasks. Create one meaning-preserving variant each and verify both with the same tests.
Baseline and controls
Original statements, fixed generation budget, and known-correct reference programs.
Evaluation
Paired pass@1, timeout/error categories, and confidence intervals. A drop is evidence of brittleness, not proof of training contamination.
Minimum complete outcome
A versioned robustness set, correctness evidence for transformations, and a paired result table.
Preparation
Python and tests; learn sandboxed execution, parsing, and benchmark versioning.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 15 original small tasks with a deterministic generator; do not copy competition problems without checking reuse terms.
Compare lexical and dense retrieval using a consistent evaluation.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Index 200-500 public course-note passages and author 100 questions with evidence labels. Compare BM25, Qwen3-Embedding-0.6B, and reciprocal-rank fusion.
Baseline and controls
BM25; keep documents, chunk boundaries, and evaluation questions fixed.
Evaluation
Recall@5, MRR, query latency, index bytes, and results by query type; no answer generator is needed.
Minimum complete outcome
A retrieval benchmark with audited relevance labels and a practical recommendation.
Preparation
L01 tokenization and L03 embeddings; learn BM25 and vector similarity.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 100 passages and 50 queries; embeddings can be precomputed once on a shared GPU.
Retrieval and evidenceCPU · no GPU requiredUnreviewed
Use public course notes with 80 evidence-labeled questions, including cross-paragraph cases. Compare fixed 128/256-token chunks and heading-aware chunks using the same BM25 retriever.
Baseline and controls
Fixed-token chunking; compare both fixed top-k and fixed retrieved-token budget so larger chunks do not win merely by supplying more text.
Evaluation
Evidence recall under fixed top-k and fixed retrieved-token budgets, missing conditions at boundaries, redundancy, and CPU latency. Inspect 20 failures manually.
Minimum complete outcome
A chunking ablation with annotated boundary failures.
Preparation
Text processing and embeddings; learn retrieval evaluation and document-level splits.
Planning budget
A laptop with roughly 8–16 GB RAM. Use bounded data and cache repeated work. The minimum study needs no GPU; time a small CPU pilot before fixing the scope.
Risks and smaller fallback
Use 40 questions and two chunking settings. Evidence retrieval is a complete study; no answer model is required.
Choose one documented retrieval subset and metric.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use one small English or Chinese MMTEB retrieval subset and Qwen3-Embedding-0.6B. Compare three supported output dimensions and float versus integer storage.
Baseline and controls
Full-dimensional, full-precision exact search with the same encoder and document embeddings.
Evaluation
nDCG@10, recall, total index bytes, query latency, and quality by query length; re-normalize after truncation.
Minimum complete outcome
A quality/storage frontier and a reproducible encoding manifest.
Preparation
L03 vector representations; learn retrieval metrics and quantization.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 1,000 documents and 100 queries; omit approximate nearest-neighbor indexing.
Use a small answer model with a fixed prompt format.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Author 120 questions over public notes: sufficient, incomplete, and absent evidence. Include an oracle-context condition and ordinary retrieval.
Baseline and controls
Always answer and always abstain; compare retrieval-score thresholding with a model-based sufficiency check.
Evaluation
Accuracy among answered questions versus coverage, unsupported-answer rate, citation support, and checking cost.
Minimum complete outcome
A risk/coverage curve and separate diagnoses of retrieval and generation failure.
Preparation
Embeddings and evaluation; learn retrieval, evidence annotation, and development-set threshold selection.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 60 questions and fixed supplied contexts, retaining the same abstention question.
Use 100 human-checked Chinese, English, and mixed query triplets over one fixed corpus, following the 2026 code-switching study.
Baseline and controls
BM25, a multilingual dense encoder, and translating the query before retrieval.
Evaluation
Paired recall@5 and nDCG, errors by terminology type, translation errors, and added latency.
Minimum complete outcome
A small auditable query set and evidence about when translation helps or loses intent.
Preparation
L01 multilingual tokenization and L03 embeddings; bilingual annotation and retrieval metrics.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 50 course-note queries; no need to rebuild the full CS-MTEB suite.
Possible upstream contribution
A carefully licensed dataset or evaluator improvement may fit MTEB; check current contribution requirements before proposing it.
A tool-call error taxonomy and a controlled decoding comparison.
Preparation
Python functions and JSON; learn chat/tool templates and constrained generation.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 50 locally generated cases and prompt/repair only if the inference backend lacks grammar support.
Understand feedback-driven agent retries without weight updates.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use 30 small Python tasks with flawed candidate solutions. Compare direct repair, generic retry, and execution feedback, with at most three attempts.
Baseline and controls
Equal total output budget; separate feedback tests from final evaluation tests; run all candidate code in a sandbox.
Evaluation
Final pass rate, regression rate, tool calls, elapsed time, and failure-trace categories.
Minimum complete outcome
A reproducible repair loop with hidden-to-the-agent evaluation tests and explained failures.
Preparation
Python debugging and tests; learn bounded agent loops and process isolation.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 15 deterministic seeded bugs and one repair round.
Compare explicit memory management with ordinary context replay.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Generate 100 short episodes with dated updates and delayed queries. Compare append-only retrieval, summary memory, and an update-on-write fact store.
Baseline and controls
No memory and full-history replay when the complete history fits the cap; use the same model, visible facts, context cap, and query budget for the main comparison. If full-history replay needs a larger context, label it as a separate higher-budget reference and account for its tokens.
A memory benchmark and a policy comparison with inspectable write/read traces.
Preparation
Python records and retrieval; learn agent state, timestamp semantics, and memory accounting.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 50 synthetic key-value episodes with exact scoring; label this as an adaptation of the memory question.
Study attack-success and legitimate-task-success evaluation together.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Select 20-30 public AgentDyn cases and paired benign controls, or adapt them to mocked local tools. Compare no defense, clear data boundaries, and tool-output filtering.
Baseline and controls
Fixed model, task set, and action budget. Use synthetic secrets and simulated destinations only.
A paired security/utility evaluation; a refusal-rate number alone is insufficient.
Preparation
Tool calls and evaluation; learn trusted instructions versus external tool data.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use a 20-case local retrieval assistant without network actions; do not claim full AgentDyn replication.
Generate 100 retention/update episodes at three input lengths. Give each method the same model, file access, context cap, and turn/output limits; log every context edit.
Baseline and controls
Sliding window, summary compaction, and editable context; use ordinary inference without retraining or custom cache serving.
Evaluation
Exact retention/task success, evidence deletions, total input/output tokens, elapsed time, and actual API cost if used. Keep modeled FLOPs separate from observed latency.
Minimum complete outcome
A three-method comparison and an explanation of which edits preserve or lose decisive evidence.
Preparation
Python, token counting, and evaluation; learn context budgets and a bounded file-editing tool.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
ContextBench is still listed as coming soon at the checked October 9 revision. Use an original synthetic generator and label this an adapted experiment; begin with 30 episodes.
Read ambiguity detection and targeted clarification.
CS224N project gallery: clarification for code generation — Stanford CS224N students; clarification project by Alexandra Suriya Kim, Julia Xi, and Ria Garg; Winter 2026. Course project listing; inspiration, not a verified research result
See a student-scale topic example; the gallery is not evidence of results.
Use a small local checkpoint for the bounded comparison.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Author 40 small Python tasks with hidden specification fields such as tie-breaking, units, or case handling. Create paired complete and incomplete requests. Use a deterministic simulated user that reveals only the field explicitly requested, and allow at most one clarification.
Baseline and controls
Answer immediately, always ask a fixed question, selectively ask, and an oracle given the full specification. Use the same model and total generation-token cap. Keep development templates separate from evaluation tasks and run code in a sandbox.
Evaluation
Hidden-test pass rate, useful and unnecessary question rates, unresolved ambiguities, interaction count, and total tokens. Inspect whether questions reveal task requirements rather than reference implementations.
Minimum complete outcome
A paired task generator, a reproducible clarification policy comparison, and an analysis of when the additional interaction earns its cost.
Preparation
Python tests, basic prompting, and evaluation; learn controlled user simulation and ambiguity annotation. The CS224N gallery supplies topic inspiration; this is an original reduced protocol.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 20 tasks, two ambiguity types, and a 0.6B model. Human user studies, unrestricted dialogue, and a full replication of the Stanford project are outside the minimum scope.
Check supported quantization formats and timing tools.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use Qwen3.5-0.8B or Qwen3-0.6B and 150 short prompts. Compare a high-precision reference with supported 8-bit and 4-bit variants from the same weights.
Baseline and controls
Same prompts, chat template, backend where possible, and decoding. If kernels differ, describe a system comparison rather than attributing all changes to precision.
Evaluation
Exact task scores, peak memory, prefill/decode latency, cold versus warm runs, and per-task regressions.
Minimum complete outcome
A measured quality/memory tradeoff and a list of brittle behaviors.
Preparation
Model inference; learn quantization formats and reliable timing.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Two precisions and one task; a CPU run is valid if hardware and runtime are documented.
Compare a KV-specific asymmetric quantization method.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
On a conventional-attention small model, compare full precision, scalar KV quantization, and a rotation-based quantizer on synthetic retrieval at 1K, 2K, and 4K context.
Baseline and controls
Uncompressed cache; identical weights, prompts, and output lengths. Count scale/rotation metadata in stored bytes.
Evaluation
Actual cache bytes, vector/dot-product distortion, retrieval accuracy, and end-to-end latency where an integrated implementation exists.
Minimum complete outcome
A compression/quality comparison with a clear distinction between mathematical simulation and realized memory savings.
Preparation
L03 vectors; learn attention, KV caching, and quantization.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Quantize cached vectors offline and analyze attention error; do not claim a serving speedup from fake quantization.
Study the draft-and-verify tradeoff behind modern speculative decoders.
Paper replicationCase study
Question:At what draft length and acceptance rate does verification pay for its overhead?
Proposed ceiling:12 GB memory target per GPU; up to 4 GPU-hours total across all devices and runs.
Minimum: one GPU holding a 0.6B draft and at most 1.7B target, or using n-gram proposals. Optional: place draft and target on two GPUs if the implementation supports it; measure transfer overhead and compare with the one-GPU baseline. All runs share the four-GPU-hour planning total; drafter training is outside the scope.
Read a modern acceleration design; drafter training is outside this study.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use a compatible small draft/target pair with identical tokenization on 100 short prompts. Implement greedy speculative verification or use a supported existing implementation.
Baseline and controls
Target-only greedy generation; first verify identical outputs and account for draft and verification time.
Evaluation
Output equality, accepted tokens per iteration, wall time, memory per device, total GPU-hours, and variation by prompt domain. If testing two devices, include transfer overhead and the one-device baseline.
Minimum complete outcome
A break-even plot and a correctness-tested implementation; EAGLE-3 is a recent reference, not a required drafter-training reproduction.
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use n-gram proposals with a small target, or a correctness-first CPU study; do not promise acceleration.
Efficient inference1 GPU · short trainingUnreviewed
Train a 5M-15M decoder on synthetic associative recall. Compare full, local-window, and simple selected-block attention across two sequence lengths.
Baseline and controls
Same training examples, parameters, and optimization; report different attention/FLOP budgets explicitly.
Evaluation
Held-out retrieval accuracy, extrapolation to longer inputs, memory, and runtime.
Minimum complete outcome
An accuracy/access-budget ablation and a faithful description of how the simplified pattern differs from NSA.
Preparation
Learn Transformer attention/masks and training; suitable for students who enjoy implementation.
Planning budget
Start with one shared GPU, usually 16–24 GB. Limit training to a 5M–30M teaching model or adapters on a 0.6B checkpoint. Where a card offers two GPUs, independent seeds or configurations can run in parallel. Count all devices and runs toward the same total GPU-hours; pilot memory and time first.
Risks and smaller fallback
Use dense masked attention at short lengths to test the idea. Dense masking does not demonstrate sparse-kernel speedups.
Efficient inferenceCPU · no GPU requiredUnreviewed
Build and test a cache-memory estimator using controlled CPU tensor experiments.
Case studyResearch problem
Question:When does a simple cache-size formula predict live tensor storage, and what explains its errors?
Compute:Laptop CPU, roughly 8–16 GB RAM.
Use small CPU tensors and cap the entire sweep’s live allocations at 256 MB. Keep measured storage and process-memory observations separate. No model download, training, or GPU run is required.
Understand cache precision and why theoretical bits per value may differ from allocated bytes.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Implement a cache-memory calculator and a CPU harness that allocates and updates tiny key/value caches. Sweep batch size, context length, KV-head count, and dtype across at least 12 bounded configurations. Test a dense cache and a sliding-window cache under identical dimensions; document each storage layout.
Baseline and controls
Use an explicit dense key/value cache as the reference. Compare predicted byte counts with deduplicated live tensor storage; distinguish allocated capacity from logical sequence length. Hold batch size, head dimensions, and dtype fixed when comparing cache layouts.
Evaluation
Report absolute and relative prediction error, boundary cases where the window fills, and memory-versus-context plots. Add regression tests for an incorrect dtype size, an omitted key/value factor, and shared storage counted twice. Separate tensor bytes from process memory and measured results from estimates.
Minimum complete outcome
A working estimator, executable allocation harness, regression tests, and a measured explanation of when each estimate succeeds or fails. Reading papers alone does not complete this project.
Preparation
L03 memory/parameter accounting; learn attention, KV caching, tensor storage, and model configuration fields. Use the references to define assumptions before running the measurements.
Planning budget
A laptop with roughly 8–16 GB RAM. Use bounded data and cache repeated work. The minimum study needs no GPU; time a small CPU pilot before fixing the scope.
Risks and smaller fallback
Use one layer, two cache layouts, and shorter contexts on CPU while retaining the measured comparison and regression tests. Do not infer end-to-end GPU speed or model quality from storage measurements.
Interpretability and language5 projects
Interpretability and language1 GPU · inferenceUnreviewed
Read reconstruction, sparsity, and intervention evaluation criteria.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use the exact Qwen3.5-2B-Base checkpoint matched to a released SAE, one layer, and 100 short bilingual continuation prompts. Compare three intervention strengths.
Baseline and controls
No intervention and a norm-matched random feature/direction; include unrelated control prompts.
Evaluation
Language adherence, answer/content preservation, perplexity change, activation sparsity, and off-target effects.
Minimum complete outcome
A controlled feature-intervention result with both successes and failures.
Preparation
L03 embeddings and PyTorch; learn residual streams, hooks, and SAE reconstruction.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Analyze activation specificity without intervention if steering is incompatible, and rename the question to feature association: this fallback cannot establish causal control. Never apply a base-model SAE to an unrelated instruct checkpoint.
Interpretability and language1 GPU · inferenceUnreviewed
Retain INCLINE with a two-language, one-layer study.
Paper replication
Question:Does a cross-lingual intervention improve Chinese answers without merely forcing English output?
Proposed ceiling:12 GB memory target per GPU; up to 4 GPU-hours total across all devices and runs.
Use one supported small checkpoint and one layer. If the official code needs a larger model or unsupported hooks, choose a simpler prompting comparison and rename the claim.
Plan aligned questions and language-independent answer checks.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use 100 parallel, objectively scored English/Chinese questions and one small open model. Implement one intervention from the official code and tune it on a separate development set.
Baseline and controls
No intervention and translate-to-English prompting; use paired questions and matched generation budgets.
Evaluation
Accuracy, output-language compliance, translation errors, and sensitivity to intervention strength.
Minimum complete outcome
A bilingual comparison with a clear accounting of where transfer helps or fails.
Preparation
Bilingual evaluation and PyTorch hooks; learn Transformer hidden states and the selected intervention.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 50 pairs and one layer; switch to a supported older checkpoint if the latest architecture is incompatible.
Interpretability and language1 GPU · inferenceUnreviewed
Compare how question context changes the intended bias criterion.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Select two documented scenarios and 100-200 examples from the authors' benchmark. Compare a neutral prompt with a difference-unaware and a context-sensitive prompt.
Baseline and controls
Unmodified prompt and original benchmark labels; distinguish descriptive, normative, and associative judgments.
Evaluation
Scenario-specific accuracy/agreement, paired changes, inconsistent responses, and limitations of the reference labels.
Minimum complete outcome
An evidence-based analysis of the evaluation criterion, avoiding a single unqualified fairness score.
Preparation
Evaluation and careful reading; learn paired analysis and limitations of normative annotations.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
One scenario and 80 examples, with manual inspection of ambiguous labels.
Interpretability and language1 GPU · short trainingUnreviewed
Read the motivation for data-limited language learning; use a much smaller course experiment.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Train a 5M-15M model on matched short formal sequences, then the same small natural-language corpus. Include direct natural-language training with matched total compute.
Baseline and controls
Random-symbol warm-up and no formal warm-up; report natural-language tokens and total training tokens separately.
Evaluation
Held-out language loss, a small syntax test, convergence curves, and two seeds if feasible.
Minimum complete outcome
A controlled transfer experiment that separates structure from extra optimization.
Preparation
L04 recurrent limitations and language modeling; learn Transformer training and synthetic generators.
Planning budget
Start with one shared GPU, usually 16–24 GB. Limit training to a 5M–30M teaching model or adapters on a 0.6B checkpoint. Where a card offers two GPUs, independent seeds or configurations can run in parallel. Count all devices and runs toward the same total GPU-hours; pilot memory and time first.
Risks and smaller fallback
A 5M model with at most 1M tokens per stage and a single formal grammar.
Interpretability and languageCPU · no GPU requiredUnreviewed
Plan overlap detection and false-positive analysis.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Index 10-50 MB of licensed text and inject a separate, labeled set of exact, paraphrased, and boilerplate overlaps. Compare exact n-gram search with a simple near-duplicate baseline.
Baseline and controls
Substring/hash-based search; evaluate on planted cases and manually inspect natural matches.
Evaluation
Precision/recall for each overlap type, index size, query time, and false-positive causes.
Minimum complete outcome
An overlap auditor and an explanation of why corpus overlap alone cannot prove a model saw a benchmark.
Preparation
L01 preprocessing and L02 n-grams; learn indexing and retrieval precision/recall.
Planning budget
A laptop with roughly 8–16 GB RAM. Use bounded data and cache repeated work. The minimum study needs no GPU; time a small CPU pilot before fixing the scope.
Risks and smaller fallback
Use 5 MB and exact search if FM-index installation fails; label the implementation change.
Diffusion and multimodal5 projects
Diffusion and multimodal1 GPU · inferenceUnreviewed
Understand masked prediction and unmasking schedules.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use the Qwen3-0.6B-derived Tiny-A2D masked-diffusion checkpoint, 100 short exact-answer or completion prompts, and three step budgets.
Baseline and controls
Random unmasking with the same checkpoint, output length, and number of forward passes; include the corresponding AR model as a separately qualified reference.
Evaluation
Exact correctness, repetition, wall time, model calls, and output length. Do not compare diffusion likelihood directly with AR perplexity without matching definitions.
Minimum complete outcome
A sampler comparison and failure cases showing the step/quality tradeoff.
Preparation
Learn masked prediction, iterative unmasking, and the released sampler interface.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 50 prompts and outputs capped at 64 tokens; no conversion training or 8B model is needed.
Diffusion and multimodal1 GPU · inferenceUnreviewed
Use the small released checkpoint and sampler interface.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use the Tiny-A2D block-diffusion checkpoint and 100 short structured completions. Compare only block sizes supported by its published inference recipe.
Baseline and controls
Default block size; hold total forward-pass budget and maximum output length fixed, and separately report supported cache behavior.
Evaluation
Constraint satisfaction, exact answer accuracy, repetition, throughput, and actual latency.
Minimum complete outcome
A block-size ablation with clear checkpoint/training-distribution caveats.
Preparation
Learn autoregressive versus masked generation, attention masks, and inference timing.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Two supported block settings and 50 prompts; if changing blocks is unsupported, compare step allocation within the documented default.
Diffusion and multimodal1 GPU · inferenceUnreviewed
Consider an alternative small multimodal checkpoint; run only one model family.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Create 60 original document/table images with exact questions and source values. Compare whole-page resize, question-independent tiling, and OCR-text input on SmolVLM-500M or Qwen3.5-0.8B.
Baseline and controls
Whole-page input and a text-only oracle supplied with the actual page text; preserve the same question and output budget.
Evaluation
Exact field/number accuracy, errors by font size/layout, total image and text input size, tile count, and latency. Use question-independent tiling to avoid supplying answer-location information.
Minimum complete outcome
A reproducible document-QA study with annotated visual failure cases.
Preparation
Python image/data handling; learn image preprocessing and multimodal prompts.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 30 single-page tables and one model. Avoid projects requiring large VLM fine-tuning.
Diffusion and multimodal1 GPU · inferenceUnreviewed
Use a text retriever as a separately labeled system baseline.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use 100 public course pages or original figures and 60 evidence-labeled queries. Compare extracted-text BM25, text embeddings, and Qwen3-VL-Embedding-2B page embeddings.
Baseline and controls
Use fixed pages and labels. Within Qwen3-VL-Embedding-2B compare page images with clean extracted text to study modality. Report BM25 and Qwen3 text embeddings separately as system comparisons; they also change the encoder.
Evaluation
Recall@5, nDCG, errors by text/table/diagram query, encoding time, and index size.
Minimum complete outcome
A modality comparison and an explanation of which visual information changes retrieval.
Preparation
L03 embeddings; learn document rendering, multimodal encoders, and retrieval labels.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 40 pages and one-time shared-GPU embeddings; generation is outside the minimum scope.
Diffusion and multimodal1 GPU · inferenceUnreviewed
Compare ASR evaluation and text-conditioning design; no second model run is required.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Record 60-100 short consented or self-recorded clips with manually checked transcripts. Compare Qwen3-ASR-0.6B with no context, a relevant glossary, and a distractor glossary using supported context input.
Baseline and controls
No-context ASR with the same audio. Split by sentence template and speaker when possible; preserve uncued controls.
Evaluation
Chinese character error rate, English word error rate, terminology recall, false term insertions, and runtime.
Minimum complete outcome
An annotated terminology benchmark and a measured context/faithfulness tradeoff.
Preparation
Text normalization and evaluation; learn audio loading, transcript alignment, and ASR context formatting.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 30 self-recorded clips and two glossary conditions; keep private voices and transcripts out of the course repository.
Build and evaluate one reusable skill with a possible SkillsBench contribution.
Research problemNew task
Question:Does a concise tested skill improve held-out workflow success beyond generic instructions at a matched budget?
Proposed ceiling:12 GB memory target per GPU; up to 4 GPU-hours total across all devices and runs.
One local model, one workflow, and 30 instances. Cap tool calls and include all four prompt conditions; upstream frontier-model trials need a separate plan.
Study reusable skills and transfer; adapt the idea to a local file workflow.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Choose one local workflow such as structured file conversion. Create 10 development and 20 held-out instances with deterministic checks. Compare no skill, a concise relevant skill, a generic workflow prompt, and an irrelevant skill.
Baseline and controls
Use the same model, tools, tasks, and total token cap. Construct a generic prompt at the same prompt length as the relevant skill, then optionally a longer version within the same total cap. Match the irrelevant skill’s length too. Keep task answers out of all prompts.
Evaluation
Artifact correctness, tool calls, total tokens, generalization failures, and skill-induced regressions.
Minimum complete outcome
A useful skill, held-out tests, a comparison report, and a reviewable contribution if it matches the upstream scope.
Preparation
Python tooling and tests; learn agent traces, skill loading, and contribution etiquette.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Use 10 held-out instances and one 0.6B model with at most three tool calls. Keep the generic-prompt control; frontier-agent trials are outside the local minimum.
Possible upstream contribution
SkillsBench: inspect the current task registry and contribution guide, agree on an unclaimed need, and propose one task or verifier improvement. Merge is an external decision.
Develop a Terminal-Bench-Science-style task from a real research workflow.
New taskCase study
Question:Can an agent complete a scientifically meaningful workflow whose result can be verified independently?
Proposed ceiling:12 GB memory target per GPU; up to 4 GPU-hours total across all devices and runs.
Choose a CPU scientific workflow and two small agent baselines run sequentially. A task requiring GPU simulation, model training, or full benchmark reruns is unsuitable.
Choose a CPU-only workflow in a domain you already understand, such as checking a numerical convergence result from a small published dataset. Package inputs, a bounded environment, an independently checked solution, and a verifier; analyze two small agent baselines sequentially.
Baseline and controls
Known-correct solution and a no-op/invalid solution; include plausible wrong outputs so the verifier cannot pass superficial artifacts.
One well-specified task and a defensible verifier; an upstream-quality result requires maintainer/domain review.
Preparation
Scientific-domain knowledge, Python and Docker; learn Harbor task packaging and agent evaluation.
Planning budget
Start with one shared GPU, usually 12–24 GB, or a suitable local quantized runtime. Reuse a released 0.6B–2B model, cap context/output lengths, and cache generations. A card may also offer a two-GPU plan. Its total GPU-hours include every device, comparison, and rerun.
Risks and smaller fallback
Complete the local CPU task and a small-model pilot as the course project. If the scientific workload itself needs GPUs, select a different task. The posted TB-Science 0.2 cutoff was October 5; confirm a future intake before promising submission.
Possible upstream contribution
New TB-Science tasks require a proposal approved by maintainers before full implementation. Toy textbook exercises do not satisfy the scientific-workflow scope. A future intake is unconfirmed as of October 9.
Open-source contributionsCPU · no GPU requiredUnreviewed
Make an existing scientific task more trustworthy through one evidence-backed repair.
Case studyResearch problem
Question:Can a verifier accept a scientifically wrong answer or reject an equivalent correct one?
Compute:Laptop CPU, roughly 8–16 GB RAM.
Choose a CPU-only verifier defect. Local correctness cases and regression tests are the minimum; expensive upstream agent trials are a separate dependency.
Design regression cases that expose weak correctness checks.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Choose one existing TB-Science task with CPU-only local oracle and verifier checks. Reproduce one numerical-tolerance, unit, schema, or environment bug; build a minimal failing case and a focused fix.
Baseline and controls
Current verifier, independently valid solutions, and deliberately wrong outputs; rerun the oracle and no-op checks.
Evaluation
False accepts/rejects, robustness across equivalent outputs, runtime, and preserved task difficulty.
Minimum complete outcome
A reproducible bug report, regression tests, focused patch, and evidence explaining why the task meaning is preserved.
Preparation
Python testing, numerical reasoning, Docker; learn the chosen task domain and upstream repair process.
Planning budget
A laptop with roughly 8–16 GB RAM. Use bounded data and cache repeated work. The minimum study needs no GPU; time a small CPU pilot before fixing the scope.
Risks and smaller fallback
Use a CPU reproduction of the verifier defect if the complete environment is too costly. State the checks not rerun; maintainer response and any required frontier-agent trials are outside the local minimum.
Possible upstream contribution
The current guide explicitly supports task-fix requests. Discuss the defect with maintainers and follow the repair workflow; do not publish hidden solutions or exploit details beyond their approved channel.
Open-source contributionsCPU · no GPU requiredUnreviewed
Compare agent evaluation interfaces and failure categories.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Choose one Linux/CLI-compatible public ALE task with CPU-only dependencies. Use a mock agent that emits known valid and invalid outputs, then repair one timeout, trajectory, or grading path after identifying a maintainer need. A small real-agent run is optional.
Baseline and controls
Existing integration plus valid/invalid-artifact and timeout cases. Keep references unavailable during the mock or real agent run, then stage them for grading.
A focused infrastructure patch, regression tests, and a reproducible lifecycle comparison. Mock-agent results establish infrastructure behavior, not agent intelligence.
Preparation
Python processes, containers, and tests; learn ALE deployment and task lifecycle.
Planning budget
A laptop with roughly 8–16 GB RAM. Use bounded data and cache repeated work. The minimum study needs no GPU; time a small CPU pilot before fixing the scope.
Risks and smaller fallback
Use the task lifecycle with small local fixtures if the full task requires a VM, GUI, or expensive dependencies. Document what was adapted.
Possible upstream contribution
ALE documents how to add an agent. Confirm maintainer interest in the specific patch; many full benchmark tasks need VMs or GUI software, so the complete benchmark is outside this project.
Open-source contributionsCPU · no GPU requiredUnreviewed
Contribute evaluation reliability rather than treating a benchmark score as a mergeable artifact.
Case studyResearch problem
Question:Can answer formatting or confidence parsing change the measured score without changing the substantive answer?
Compute:Laptop CPU, roughly 8–16 GB RAM.
Use synthetic answer/confidence strings and public reference examples for local parsing tests. Requiring paid frontier judges or a full HLE run would change the scope.
Plan false-accept and false-reject checks for an evaluator.
Experiment plan, evaluation and smaller fallback
Minimum scope and data
Use 20–30 public, domain-understood HLE examples and synthetic answer/confidence strings. Audit one local parsing or calibration path using equivalent formatting and known incorrect variants. A model-judge audit is optional only when a bounded cached-output set is available.
Baseline and controls
Current scorer, manually verified equivalence classes, and deliberately incorrect answers. Use only the public split and pin dataset revision.
Evaluation
Parsing false accepts/rejects, score changes under equivalent formatting, and correctness of confidence/calibration calculations. Report small-sample uncertainty; no model capability claim follows from synthetic strings.
Minimum complete outcome
An evaluation bug reproduction or well-supported correction, regression cases, and a small empirical analysis.
Preparation
Evaluation, Python tests, and subject knowledge; learn HLE answer formats and calibration.
Planning budget
A laptop with roughly 8–16 GB RAM. Use bounded data and cache repeated work. The minimum study needs no GPU; time a small CPU pilot before fixing the scope.
Risks and smaller fallback
Audit parsing with synthetic outputs when small-model accuracy is too low. No full HLE run, frontier API spend, or new expert question is required.
Possible upstream contribution
HLE question feedback uses its official form; rolling contributions are coordinated with maintainers. Its GitHub repository has issues disabled. Confirm a code-PR route before assuming that question acceptance will be represented by a merge; preserve benchmark canaries and keep evaluation data out of training.
7. Others
Use other resources you can access
You may use your own computers or GPUs, laboratory or research-group machines, cloud credits, hosted APIs, and other resources you can obtain. You are not limited to the course pool or its proposed GPU-hour budget. State the resource source, availability, estimated cost, and total use in your proposal and experiment record.
Request only the portion you need from the shared course pool. Keep a smaller fallback if outside access is uncertain. The same rubric applies to everyone; a larger compute budget does not itself earn a higher grade.
Complete practical or experimental work
Students may not submit a survey-only project. Related work provides background. Your own contribution must include an implementation, experiment, tested dataset or benchmark, or evaluated repair, together with a baseline or other meaningful comparison and analysis. CPU projects qualify when they produce this evidence.