RL·TRADER GitHub
Deep reinforcement learning research

Is the agent learning, or just lucky?

A from-scratch study of the two ways deep-RL trading agents fool you: memorising the training path, and one lucky seed. Tested on crypto and stocks, where markets are noisy enough to make both easy.

Every experiment runs in your browser. Research, not a trading product.

See the result ↓ Run an experiment →
Held-out backtest daily bars
Agent Buy & Hold
Total return+0.0%
Sharpe0.00
Max DD0.0%

What this project is

What this is

A research study: does one PPO recipe generalize across market regimes, and does an apparent edge survive proper testing? The experiments ship with it, so you can re-run them.

What it is not

Not a trading bot or investment advice. It places no orders and connects to no broker or exchange. The headline finding is negative: no seed-robust edge over buy-and-hold.

Real-time? No.

Daily closing bars: never intraday, never streaming. The figures here are a fixed snapshot that changes only when the pipeline is re-run.

The one live path is the lab's out-of-distribution test, which runs the frozen policy on fresh bars for any ticker you type.

The result

The most impressive number was the least trustworthy.

Trained on one asset, the agent posts returns that look like a bug, then loses money on data it has never seen. Trained across a basket, the headline shrinks to something modest that holds up. The only difference between the two is how many assets they saw.

16
Real tickers backtested
10
Seeds in the significance study
28
Engineered features
418
Tests passing
What I got wrong

Every result here held up — until the next test.

Each step below is a result I believed until the next test broke it. The numbers are read live from this repository's committed experiments — the same files the charts above use — except where a step quotes a superseded run, which is cited by commit so you can check it against history rather than against my description of it.

  1. 01

    The number was enormous.

    A crypto agent trained on one asset finished its training period up …, depending on the seed. My first instinct was that something had gone right.

  2. 02

    It was a receipt for memorisation.

    Scored on held-out tickers it had never traded, the same recipe averaged … across 10 seeds. It had not learned to trade; it had learned one price path.

    tools/ablation.py

  3. 03

    The fix was the data, not the algorithm.

    I changed nothing about PPO. Training across the whole basket — domain randomization — left nothing single to memorise, and held-out return moved to …. That is the contrast at the top of this page.

    rl_trader/envs/ · tools/ablation_multiseed.py

  4. 04

    Then one run looked spectacular again.

    The published crypto agent returned +275.45% against +19.57% buy-and-hold — a 14× outperformance. This was the version of the result I nearly wrote up as the finding.

    Superseded · results.js at d4c0ef9, test window ending 2026‑06‑17

  5. 05

    Reseeding dissolved it.

    Retraining the identical recipe under 10 independent seeds gave returns spanning … — one seed more than tripling capital, another losing money. Against buy-and-hold across the held-out pairs the paired test gives …: not distinguishable. The spectacular run was one draw from a wide distribution.

    evaluation/statistics.py · tools/real_significance.py

  6. 06

    And the window mattered as much as the seed.

    Rebuilding on a data snapshot pinned two months later — same code, same seed 42, same 200k steps, only more days of price history — the same run returned … against … buy-and-hold. Moving the evaluation window alone erased the headline. It was not only seed-lucky; it was window-lucky.

    Current run · data/SNAPSHOT.json pins the window

  7. 07

    The honest answer is a negative result.

    On stocks the agent is not merely tied with buy-and-hold — across 10 seeds it averaged …, and the paired test against buy-and-hold over the 10 held-out tickers gives …. It is significantly worse. That is the result, and it is the one I am publishing.

What survives all of this is narrow, and worth stating plainly: domain randomization measurably improves held-out performance — that gap is large, reproduced across seeds, and its intervals do not overlap. Nothing here demonstrates a profitable trading strategy, and none of it is investment advice.

What I learned

Four things that transfer past trading.

The timeline above is what happened in this project. These are the parts I would carry into any reinforcement-learning result, in any domain — markets were only the testbed that made each one cheap to demonstrate.

01

A huge training number is a symptom, not a result.

The single-asset agent's in-sample return moved between … and … depending only on the random seed. A quantity that unstable is not measuring skill; it is measuring how thoroughly one price path was memorised.

02

The fix was the training distribution, not the algorithm.

PPO was not touched — no new loss, no tuning. Only the number of assets the agent saw changed, and held-out return moved from … to … on crypto. Where the data comes from is a modelling decision.

03

Sample size can invalidate a test before any data is collected.

Repeated runs of the same recipe spanned …. Worse, a two-sided sign-flip test over five pairs draws from 2⁵ sign assignments, so its smallest attainable p-value is 0.0625 — at five seeds it could not reach significance at all, whatever the effect size. The study was rerun at … seeds for that reason, which drops the floor to 0.002. Building the calculator that says this is easy; acting on what it says about your own design is the part that counts.

04

Evaluation choices move results as much as modelling ones.

Identical code, identical seed, a data window extended by two months: the headline fell to … against a benchmark that had itself risen. Nothing about the model changed. The window is part of the claim, which is why this one is pinned.

None of this makes the agent a trading strategy — on stocks it is significantly worse than holding the index, and that is the published finding. What it does support is narrower and, I think, more useful: the evaluation is where the science is.

The lab

Don't just read the result. Test it.

Nine experiments, each one runnable against the deployed agent or the project's real committed results. Nothing on these pages is a mock-up.

01

Signal or Noise?

Before you test the agent, test yourself. Tell trending charts from random walks, then see the exact binomial verdict on your score.

Live 02

Your Turn

Trade the same bars the agent trades, one at a time with no lookahead, then get scored against it and against buy-&-hold.

Live 03

Agent Playground

Pick a market, a data source, costs and a reward function. Run a real episode and scrub it bar by bar against buy-&-hold.

Live 04

Agent X-Ray

At any bar, see the actual 563-dimensional observation the agent consumed, and follow it through to the action and the reward.

Live 05

Can You Break It?

One agent memorised a single price path; the other saw a new one every episode. Then push the deployed policy onto distributions it never met.

Real data · live stats 06

Walk-Forward

Disjoint chronological folds, each with its own scaler — plus a direct measurement of what fitting that scaler on the test block costs you.

Live evaluation 07

Real or Luck?

A single-seed headline beside what ten seeds actually say. Re-run the bootstrap and the permutation test yourself.

Real data · live stats 08

Anything There?

Shuffle the returns and buy-&-hold is unchanged, but every pattern is gone. The sharpest question in the project: is the agent weak, or is the market empty?

Real · precomputed 09

Research Notebook

Every experiment you run, with the exact config that produced it — and a button that replays it to the same numbers.

Live

What isn't live: training. Each seed of a multi-seed study is a full PPO run, so those results are served from the repository's real committed experiments, labelled with the command that regenerates them — never simulated. The statistics computed over them do run live.

The framework

Built from the algorithm up.

PPO, both environments, the data pipeline, and the evaluation are implemented here — no stable-baselines, no wrappers. Each layer is a separate, individually tested module.

PPO, from scratch

Clipped surrogate objective, GAE advantages, entropy bonus, orthogonal init and gradient clipping — implemented in PyTorch, not wrapped from a library.

Dual market environments

StockTradingEnv and CryptoTradingEnv share one BaseTradingEnv, so accounting, costs, and rewards never silently diverge.

Leakage control

Chronological train / validation / test splits with scalers fit on training data only. The agent is scored on data it has never seen.

Risk-aware rewards

Return net of transaction costs, plus drawdown and turnover penalties — discouraging reckless, over-leveraged noise-trading.

Honest backtests

Total return, annualised Sharpe, and max drawdown on a held-out split — the metrics a quant actually cares about.

Swap the policy or the market

The same PPO agent trades one asset or allocates across a whole basket (cross-sectional long/short). A recurrent (LSTM) policy is fully implemented, and new markets/algorithms drop in behind the same API.

About

Why I built this.

I wanted to understand modern reinforcement learning by building it, not importing it — so I implemented PPO from scratch in PyTorch, wrote two custom environments, and engineered the whole data pipeline. But the real question I set out to study is a general one in RL: how do you know an agent has learned something real, rather than memorised its data or gotten lucky?

The hardest — and most valuable — part wasn't the algorithm; it was the evaluation. An early run made the crypto agent look like it crushed the market. Instead of shipping that, I built a multi-seed significance test — and it showed the "win" was mostly luck of the random seed, statistically tied with buy-&-hold. That's exactly the warning of Henderson et al.'s Deep Reinforcement Learning that Matters (2018), and I reproduced it in a domain of my own.

What it demonstrates: I can take a non-trivial ML research question end to end — algorithm, environment, training, and an evaluation held to a real scientific standard — and report the honest answer even when it's a negative result.

Reproduce it

Don't take my word for any of it.

Every figure on this site traces to a command in this list and an artifact in the repository. If you think a number here is wrong, this is how you show it.

  1. pip install -e . PPO, both environments and the feature pipeline — no stable-baselines, no wrappers. ppo_agent.py →
  2. python tools/ablation_multiseed.py --seeds 42 43 44 45 46 47 48 49 50 51 Regenerates the single-asset vs cross-asset contrast at the top of this page — and rewrites the page's own copy of it. source →
  3. python tools/real_significance.py --seeds 10 --timesteps 150000 Retrains under 10 seeds and runs the paired permutation test behind steps 05 and 07 above. statistics.py →
  4. python tools/surrogate_test.py --mode real Destroys the temporal structure and checks the edge disappears with it, with a positive control that proves the test has power. source →
  5. python tools/build_site_data.py --real Rebuilds every chart on this site from a fresh seeded training run. source →
  6. bash tools/rebuild_all.sh All of the above in order, then re-syncs every table in this repository's documents from the artifacts it just produced — so a partial rebuild cannot leave the write-up describing runs that no longer exist. source →
  7. pytest The suite, including the tests that pin these honesty rules in place. tests/ →

What is actually pinned

  • DataA dated snapshot manifest rather than a live feed, so re-fetching cannot silently move the evaluation window. SNAPSHOT.json →
  • SeedsEvery published run states its seed, and every multi-seed study lists all of them.
  • SplitsChronological, with scalers fit on the training slice only. preprocessor.py →
  • StatisticsBootstrap CIs and paired permutation tests, with the resolution floor reported next to every p‑value.
  • ArtifactsCommitted JSON, regenerated by a named command that is printed beside the result it produced.
  • The write-upEvery result table in the README, RESULTS.md and the paper is generated from those artifacts, and the test suite fails if any of them drifts. sync_docs.py →

How RL·Trader works

How it works

Trading as a reinforcement-learning problem.

There's no labelled "right answer" for what to do on a given day — only the consequences of what you did. That's the exact setting reinforcement learning is built for: an agent takes actions in an environment, collects a reward, and learns a policy that maximises reward over the long run. Here the environment is a market, the action is a position, and the reward is risk-adjusted profit.

The loop

Observe, act, get rewarded — millions of times.

Every simulated trading day, the agent runs the same three-step loop. Repeat it over millions of steps and a strategy emerges from the consequences alone — no one ever tells it the "correct" trade.

State

What it sees

A rolling window of 28 features — multi-horizon momentum, RSI, MACD, Bollinger %B, ATR and a volatility-regime flag, plus cross-asset signals (its strength relative to the SPY/BTC index) — concatenated with its own account state: position, cash, and normalised equity.

Action

What it does

One continuous number in [−1, +1] — the target position. +1 is fully long, −1 fully short, 0 is flat. The environment moves the book toward that target and charges transaction costs and slippage on every change.

Reward

What it optimises

Not raw profit — a risk-adjusted signal (differential Sharpe) net of costs, with drawdown and turnover penalties. That's what keeps it from chasing returns by over-leveraging or churning the account into the ground.

The algorithm

What PPO actually does.

Proximal Policy Optimization is the learning rule — implemented here from the math up, no stable-baselines.

Try, then nudge

It gathers a batch of experience under the current policy, estimates which actions did better than expected — the advantage, via GAE — and shifts the policy toward those.

But not too far

A clipped surrogate objective caps how much the policy can move in one update — the "proximal" part. It's what makes PPO stable instead of collapsing, the classic failure of naive policy gradients.

Stay curious

An entropy bonus keeps it exploring early instead of locking onto one strategy too soon; orthogonal initialisation and gradient clipping keep the optimisation well-behaved.

Two heads, one network

An actor picks the position; a critic estimates how good the current state is. The critic's value is the baseline the advantage is measured against.

Architecture

Unified agent. Separate environments.

The PPO agent speaks only in observations and actions — completely market-agnostic — so the exact same network trains on both stocks and crypto.

Unified agent PPO ActorCritic policy head · value head
select_action ↓↑ update(rollout)
On-policy Rollout Buffer + GAE
Stock StockTradingEnv low cost · ~252 bars/yr
Crypto CryptoTradingEnv high cost · 365 bars/yr
Pipeline OHLCV → indicators → scale → split synthetic generator or your own CSVs
End to end

From raw prices to a trader you can trust.

01

Ingest & engineer

Load real OHLCV (or generate synthetic data) and build the 28 features — returns, RSI, MACD, MA ratios, volatility, and cross-asset context.

02

Split without leakage

Chronological train / validation / test splits, with scalers fit on the training slice only — so the agent is scored purely on data it has never seen.

03

Train with PPO

Run the loop above over millions of steps, with domain randomization across tickers so the policy can't memorise one asset's path.

04

Evaluate honestly

Backtest deterministically on the held-out split, then repeat across 10 seeds and run a permutation test to see whether the edge is real.

Overfitting, caught

Why train across many tickers, not one.

Train the agent on a single asset and it posts absurd returns on the data it trained on — then loses money on data it has never seen. It memorised one price path. Training across many tickers (domain randomization) forces it to learn something transferable: in-sample and held-out returns line up.

Training setup In-sample (range over 10 seeds) Held-out return (mean, 95% CI)

Those astronomical in-sample figures aren't a win — they're the overfitting tell: a single-asset agent memorises the exact training path, so it "earns" impossible returns there and then collapses out of sample. The honest column is held-out, and only domain randomization keeps it positive — across 5 seeds all four intervals exclude zero. In-sample is shown as a range rather than a number because it varies by an order of magnitude across seeds; quoting one run of it would imply a precision that isn't there. (bootstrap 95% CI, 30 held-out paths held identical across arms; reproduced by tools/ablation_multiseed.py.)

The honest part

And then: does it actually work?

A single training run can look brilliant by luck. So every result is held to a multi-seed permutation test — the whole thing is retrained under 10 independent random seeds, and the edge only counts if it survives. Watch it play out per market on the Crypto and Stocks tabs, or read the full methodology in RESULTS.md.

RL·Trader markets explorer

Real markets · walk-forward · out of sample

Explore what the agent trades.

Click any ticker to see that name's out-of-sample backtest, scrub through the positions the agent held day by day, and run the policy live on fresh prices.

Equity curve REAL

PPO Agent Buy & Hold

Scorecard

Agent · B&H

Watch the agent act price & the position it held

longshortflat

Agent drawdown

Action distribution

target position over the run

How it learned episode reward over training

smoothedper update

Every held-out result click a row to load it · click a header to sort

Ticker Agent Buy & hold Δ vs B&H Sharpe Max DD Result

Reproduce: python tools/build_site_data.py --real regenerates every chart here from a fresh, seeded training run.

RL·Trader Lab

Can an RL agent actually learn a generalizable strategy — or is it exploiting its training data and getting lucky? Run the experiments yourself.

connecting…

Signal or noise?

Before testing the agent, test yourself. Half of the charts below carry the exact return autocorrelation the agent was trained to find. The other half are random walks with no structure at all. Every chart is standardised to the same volatility and given a drift from one shared distribution, so nothing but the pattern itself can give the answer away.

Take the test Live

Now you try

Trade the same bars, through the same environment, paying the same transaction costs. You are shown one new bar per decision and nothing beyond it — the price series stays on the server, so there is nothing here to read ahead in. At the end your run is scored against the deployed agent and against buy-&-hold over exactly the bars you traded.

Set up a run Live

Agent playground

Configure experiment

The policy is fixed and not learning. Costs change the trajectory (they feed back through account state); the reward selector changes what is measured, not what is done.

No episode yet.
Configure a run and press Run episode.
Every number will come from the backend stepping the real environment.
No episode loaded.
Run an episode in Agent Playground, then return here to inspect what the agent actually saw at any bar.

Can you break the agent?

Strong training performance is not evidence of a strategy. These two experiments separate memorising a price path from learning something transferable.

1 · Trained two ways Real · precomputed

Two identical PPO agents. A trained on one fixed price path; B trained on a freshly resampled path every episode. Both evaluated on paths neither had seen.

Held-out difference, tested live

A paired permutation test over the same seed set, recomputed on request.

2 · Break the deployed agent Live

Now take the actual deployed policy and move the world out from under it. Each regime is a controlled distribution: the agent is unchanged, only the data generating process differs.

Does it hold up over time?

One backtest is one slice of history, and that slice might just have been kind. Rolling walk-forward re-asks the question on several disjoint, chronological blocks — and refits the feature scaler on each block's training rows only, so no fold ever sees its own test data. Then it runs the same folds the wrong way, with the scaler fit on everything, to measure what that shortcut actually costs.

Run the folds Live evaluation · fixed policy

Is the result real — or luck?

A single training run is one draw from a distribution. Below is the number this project headlines from one seed, next to what the same recipe produces across ten.

Single seed vs. the seed distribution Real · precomputed

Every training run

Each row is one independent PPO training run, evaluated on held-out data.

Bootstrap distribution of the mean

Resampling the five outcomes with replacement, recomputed live.

observed mean zero

Recompute the inference

These are the estimators the paper uses, run on request against the same real data.

The published test Real · precomputed

The paper's significance test pairs across held-out tickers, not across seeds. Different axis, different question — and enough pairs to actually resolve a p-value.

Or is it the recipe? Real · precomputed

Seeds are one way to get lucky; hyper-parameters are another. Each configuration below moves one knob either side of its published default, everything else held fixed, and is retrained from scratch. It is deliberately not a grid search — the question is whether the conclusion is fragile, not which recipe wins.

So how many runs would it take? Live

Every panel here reports the floor 2/2ⁿ — when a design cannot reach significance. This answers the question that follows. Put in an effect size and its spread, and it simulates the real sign-flip test to find the sample size that would detect it.

Is there anything there to find?

Every other panel shows the agent failing to beat buy-&-hold. None of them can tell you why. This one can: shuffle a series' daily returns and re-integrate them. The sum is unchanged, so the price ends in exactly the same place and buy-&-hold is identical — but every pattern a timing agent could exploit is gone. Train the same recipe on both and compare.

Research notebook

Every experiment run in this session, with the exact configuration that produced it. Any of them can be replayed — the config is a fixed point, so a replay rebuilds an identical environment.

Run a new experiment

Session history

Clone it. Train it. Break my results.

Runs out of the box with zero external data — then point it at real markets and check the numbers yourself.

Explore the repo → 
pip install -e .
python -m rl_trader.scripts.compare_markets --plot