Autoresearch Github

AutoResearch refers to three separate open-source frameworks on GitHub, each solving distinct research automation problems. The most widely referenced is Karpathy's AutoResearch, which constrains AI agents to edit a single Python file (train.py) and run experiments within a 5-minute wall-clock budget per iteration. The second is EvoMap's AutoResearch, a full research pipeline automating idea generation, experiment design, and paper-ready evidence collection. The third is JLX0's AutoResearch, a Python framework focused on academic paper search, organization, and summarization. Conflating these three creates confusion, each targets a different user: ML researchers optimizing training code, research teams automating empirical workflows, and academics automating literature review. Understanding which framework matches your use case is essential before diving into implementation. - Karpathy's version: single-file training optimization with strict time constraints

• EvoMap's version: end-to-end research pipeline from hypothesis to publication
• JLX0's version: paper discovery and analysis automation For instance, autoResearch by Andrej Karpathy is a framework that lets AI agents autonomously optimize machine learning training code by editing a single Python file (train.py) and running experiments within a fixed 5-minute time budget.

| Aspect | Summary | |---|---| | What Is AutoResearch on GitHub and Why Three Projects Share One Name | AutoResearch refers to three separate open source frameworks on GitHub, each solving distinct research… | | How Does Karpathy's 5-Minute AutoResearch Loop Actually Work? | Karpathy's AutoResearch operates a tightly constrained autonomous research loop where an AI agent modifies… | | What Hardware, Software, and Performance Requirements Does AutoResearch Need? | Karpathy's AutoResearch requires a single NVIDIA GPU (tested on H100), Python 3.10+, and the uv package… | | How Do AI Agents Autonomously Modify and Iterate on Training Code? | The AI agent receives program.md (human written research direction) and the current train.py, then… | | Which AutoResearch Framework Should You Use and How to Get Started | Choose Karpathy's AutoResearch if you are optimizing machine learning training code and want rapid,… |

Karpathy's AutoResearch operates a tightly constrained autonomous research loop where an AI agent modifies train.py, trains a model for exactly 5 minutes, evaluates the validation metric (bits-per-byte or val_bpb), and commits only improvements to Git history. The framework consists of three core files: prepare.py (fixed data prep, never modified), train.py (the single file the agent edits), and program.md (baseline instructions guiding the agent), according to the GitHub README. This constraint design, one editable file, one metric, one time budget, makes autonomous research tractable by eliminating combinatorial explosion. The agent proposes architectural or hyperparameter changes, the system trains for 5 minutes, and Git stores only successful improvements, creating a monotonic progress log. The framework runs approximately 12 experiments per hour, enabling rapid iteration. Each experiment's result is recorded in results.tsv, giving researchers a complete audit trail of what worked and what didn't. - Modify train.py → Train for 5 minutes → Check val_bpb → Commit if improved

• Only successful changes persist in Git; failed experiments are discarded
• Complete experiment log stored in results.tsv for analysis

Karpathy's AutoResearch requires a single NVIDIA GPU (tested on H100), Python 3.10+, and the uv package manager, according to the quick-start guide. The framework is intentionally minimal, only three files matter, and humans edit program.md to set research direction while the agent modifies only train.py. The validation metric, val_bpb (validation bits per byte), is lower-is-better and vocabulary-size-independent, enabling fair architectural comparison across different model designs. Performance gains are concrete: AutoResearch achieved an 11% improvement through 700 automated experiments in 2 days. This speed comes from the 5-minute constraint, it forces the agent to propose high-impact changes rather than tweaking hyperparameters endlessly. The ratchet approach (keeping only improvements) means the final model is always better than the baseline, with zero wasted compute on failed directions stored permanently. - Single H100 GPU sufficient; Python 3.10+ required

• val_bpb metric enables vocabulary-independent comparison
• 700 experiments in 2 days = ~12 experiments/hour throughput

The AI agent receives program.md (human-written research direction) and the current train.py, then proposes a code edit, changing the architecture, adding regularization, or adjusting hyperparameters. The system applies the edit, trains for 5 minutes, and compares the new val_bpb to the previous best. If the metric improves, the change is committed to Git; if not, it is discarded. This loop repeats continuously, with the agent learning from the Git history which types of changes yield improvements. The framework uses a 'ratchet' approach where Git history stores only successful improvements, creating monotonic progress with a complete experiment log in results.tsv. Humans retain control by editing program.md to steer the agent toward specific research directions, for example, "try reducing model width" or "experiment with different activation functions." The agent interprets these hints and proposes concrete code changes. Because the 5-minute budget is strict, the agent must propose meaningful changes; trivial tweaks fail to show improvement within the window and are dropped. - Agent reads program.md (human guidance) + current train.py

• Proposes code edit → trains 5 minutes → compares val_bpb
• Improvement → commit to Git; no improvement → discard
• Humans steer direction via program.md edits The core loop involves an AI agent modifying train.py, training for 5 minutes, checking the validation metric (bits-per-byte or val_bpb), and keeping only improvements—according to the GitHub repository.

Choose Karpathy's AutoResearch if you are optimizing machine learning training code and want rapid, autonomous iteration on a single Python file. Choose EvoMap's AutoResearch if you need to automate a full research pipeline from hypothesis generation through paper-ready evidence. Choose JLX0's AutoResearch if your primary goal is automating academic paper discovery, organization, and summarization. For Karpathy's version, start by cloning the GitHub repository, installing Python 3.10+ and uv, and preparing your train.py file with a clear validation metric. Write program.md with your research direction, be specific about what you want the agent to explore. Run the agent and monitor results.tsv for improvements. The tight constraints (one file, 5 minutes, one metric) mean you see results quickly; most users report meaningful improvements within hours. The framework is intentionally small and transparent, making it easy to understand what the agent is doing and why. - Karpathy's: best for training code optimization with rapid feedback

• EvoMap's: best for automating full research workflows end-to-end
• JLX0's: best for academic literature automation
• Start with GitHub clone, Python 3.10+, and a clear program.md AutoResearch achieved an 11% performance improvement through 700 automated experiments in 2 days, according to researchagent.dev.