Test the selection process
This article explains a validation method. The dates and settings below are hypothetical. No strategy performance was tested here.
Fit on an earlier block, choose among predeclared candidates on a later past block, then evaluate the chosen procedure on the next window. Repeat on a fixed schedule. Keep a separate final interval outside development decisions. Once a result changes your rules, it is development evidence.
Hyndman describes evaluation with a moving forecast origin and past-only training. Cawley and Talbot explain why model selection itself can overfit and must be part of the procedure being evaluated. The chronological selection design here adapts those principles; their papers do not establish crypto returns.
Our teaching market is Binance spot BTCUSDT: unlevered long-only or cash, completed 1h candles, UTC boundaries, BTC quantities and USDT accounting. No price series was read and no backtest exists for this example. These historical dates were not a real prospectively quarantined study. Calendar and unit checks establish neither data coverage nor execution quality.
A rolling schedule in fixed days
Rolling history drops old observations as the origin advances. Anchored history keeps its original start and expands. Hyndman shows both growing history and a fixed-length variation. Neither choice, nor any window length below, is proven best for BTC.
Use rolling windows here. At evaluation origin e, fit on [e−180 days, e−60 days): 120 days. Select on [e−60 days, e): 60 days. Fix the candidate set, cost-aware selection score and tie rule before scoring. Optionally refit the selected candidate on [e−180 days, e), only if that refit was declared in advance. Evaluate on [e, e+30 days), then advance e by 30 days.
All ranges are half-open [start, end), assigned by candle open timestamp. Every date below is 00:00 UTC; the right endpoint is excluded. Days are fixed 24-hour periods, not calendar months. The six adjacent development evaluation blocks are:
- Fold 1 — fit 2025-01-01 → 2025-05-01; selection 2025-05-01 → 2025-06-30; evaluation 2025-06-30 → 2025-07-30.
- Fold 2 — fit 2025-01-31 → 2025-05-31; selection 2025-05-31 → 2025-07-30; evaluation 2025-07-30 → 2025-08-29.
- Fold 3 — evaluation 2025-08-29 → 2025-09-28; shift the fit and selection windows by the same 30 days.
- Fold 4 — evaluation 2025-09-28 → 2025-10-28.
- Fold 5 — evaluation 2025-10-28 → 2025-11-27.
- Fold 6 — evaluation 2025-11-27 → 2025-12-27.
Lock learned parameters; let trailing inputs update
For the anchored alternative, keep 2025-01-01 as the fit/refit start and the latest 60 days for selection. Fit still ends at e−60 days; refit ends at e. Choose the history rule during development, before opening the final interval.
Calendar membership is only one filter. Use inputs actually final and available at the fit or decision cutoff. Lock the candidate, learned parameters and preprocessing before the first evaluated decision. If fitting or selection finishes late, wait in cash. Evaluation observations cannot finish a fit intended to use only pre-origin data.
Keep that choice and learned state fixed within the 30-day evaluation block. Trailing indicators may update from newly available past candles. Updating an input is different from retuning a fitted threshold. At a later origin, prior evaluation candles are now past and may enter the predeclared selection/refit. Human changes prompted by their scores are additional development trials.
Warm-up supplies trailing price context, not scored P&L or a free starting position. Choose context and initialization for the actual rule; stateful or exponential models need their own convergence and carry-state contract. Fit scaling, imputation and feature selection only inside the permitted fit/refit block. Full-history normalization, centered features and backward filling can leak future information despite a chronological trade split.
A closed candle may still be unavailable
Binance spot REST documentation identifies klines by open time, supports 1h intervals and defaults interval timeZone to 0 (UTC). Its startTime and endTime request timestamps remain UTC even when the interval timezone changes. Fix the interval timezone explicitly in the data contract.
The official UTC spot stream example distinguishes exchange event time E, kline start k.t, close k.T and closed flag k.x. Timestamps default to milliseconds. None is your local receive time, processing completion or decision lock time. A documented stream update speed does not prove delivery by a particular open.
A proposed audit record should retain received_at, available_at after validation, data version and decision/lock timestamps. Every input must be available by its decision; accept only final candles. Historical archives may lack receive times. Disclose that gap and label simulated latency as an assumption. This article does not establish that SeekAlgo records or replays these fields.
Leave time between the signal and its fill
Suppose the 2025-06-30 00:00 UTC bar closes at 00:59:59.999. Assume receipt at 01:00:02 and a processed, locked decision at 01:00:03. The first eligible hypothetical fill is 02:00:00, not 01:00:00. Those seconds are made-up latency assumptions, not measurements or a delivery promise.
This coarse model leaves an intervening hour: a signal from the completed bar opening at t can first trade at t+2h, only if received, processed and locked strictly before that open. At each eligible open, use the latest eligible available signal under a deterministic rule. Never fill at the signal candle’s earlier open or close.
If the lock is late, use the first later hourly open for which the decision is eligible. No retroactive fills: if there is no eligible entry before the block’s entry cutoff, remain cash and record why. An exact open adjusted for adverse slippage remains a model, not a guarantee that an order could execute there.
Close the block without erasing its costs
Choose flat boundaries for this example. Each selection/evaluation block starts with no position or pending order; ignore prior-block signals for new entries. At the last hourly open strictly inside the block, liquidate any position, charge adverse sell slippage and the sell fee, cancel pending entries and prohibit new entries at or after that time. Fold 1 exits at 2025-07-29 23:00 UTC. Its last hour is cash. This is a scheduled exit, with no need to see the final candle early.
Carry net USDT equity through all development evaluation blocks, along with the running drawdown peak. Do not supply fresh capital at every fold or average fold returns and call it a compounded path. Inner candidate comparisons may start from a shared normalization; that does not replenish outer evaluation capital.
Flat boundaries create artificial turnover and cash time. A carry-position alternative must preserve BTC quantity, cash, entry costs, pending orders, model identity and marked P&L, and state whether a new model can alter existing positions. Neither free liquidation nor choosing the more flattering policy after results is acceptable.
Keep quantity, cash and costs in the same ledger
Start development evaluation with 10,000 USDT. Each flat-to-long entry has an all-in budget B of 20% of current marked equity, including its entry fee. No borrowing, repeated-long pyramiding or rebalancing. Price moves can change the position’s weight: 20% is an entry fraction, not an ongoing exposure cap.
Use arbitrary per-side model inputs: fee f = 10 basis points = 0.0010 and adverse slippage s = 5 basis points = 0.0005. Predeclare a stress with s = 15 basis points = 0.0015 and the same fee. These are not Binance fee quotes. Do not reduce costs after seeing a failure.
Assume quote-paid fees and fractional BTC, without fee-token discounts. This toy excludes real order filters, quantity rounding, minimum notionals, depth, partial fills and measured latency. It is not an exchange-executable order specification.
- For eligible reference open O in USDT per BTC, buy fill = O(1+s). Buy quantity q = B/[O(1+s)(1+f)] BTC. Entry fee = q × buy fill × f; deduct buy notional plus that fee from USDT cash.
- At exit, sell fill = O(1−s). Add q × sell fill × (1−f) to cash. The fill already includes slippage; do not subtract that same slippage again.
- Require nonnegative cash and BTC. Mark equity as cash plus BTC quantity times the declared eligible mark price, then reconcile every fill and fee to ending equity.
Use a comparison that obeys the same clock
Compare with no-yield USDT cash and a block-aligned BTC benchmark. The BTC account enters all-in, including its entry fee, at e+2h in the illustrative on-time case. A delayed availability schedule needs a declared matched late-entry rule; do not give the benchmark an earlier impossible fill. It exits at the same scheduled final open, pays the same per-side costs and carries its own net equity into the next block.
This is not uninterrupted buy-and-hold. The all-in benchmark also has more market exposure than a 20% strategy entry budget. A return difference alone would not establish risk-adjusted alpha. USDT quote value does not guarantee fiat purchasing power.
Reject incomplete evidence
For a supervised target, track the full information interval through the time its future endpoint becomes available. Exclude targets unresolved at the fit cutoff. A universal fixed row-count purge gap cannot establish this. Sharing older trailing feature context is not automatically the same as leaking a future target.
López de Prado’s author material relates purging and embargo to label overlap and serial conditionality. That narrow context supplies no universal gap duration or verified implementation here. A forward-only split has no future-side training rows at its origin, but delayed inputs, unresolved targets and holdings still need checks. Clean splits do not make trades or returns independent.
Missing, conflicting or duplicate candles, wrong product/timezone/units, unverified bounds or unreconciled accounting make a result incomplete. Do not fabricate candles or silently substitute another product. A last pre-origin candle that arrived late cannot justify a fit or a trade as if its final close had been known earlier.
Freeze the protocol before the final interval
The reserved final teaching interval is [2025-12-27, 2026-01-26), with both boundaries at 00:00 UTC. Its rolling fit is [2025-06-30, 2025-10-28), selection is [2025-10-28, 2025-12-27), and optional predeclared refit is [2025-06-30, 2025-12-27). The final interval was not run here. No price data, candidate scores or holdout outcomes exist for this example.
Before a real final evaluation, freeze candidates, rolling/anchored windows, preprocessing, selection score, ties, refit, cadence, availability, fills, boundaries, sizing, costs, benchmarks and acceptance criteria. Select/refit only on eligible pre-final information. For this single interval, lock the learned parameters and do not reselect within it. No intermediate holdout plot or metric may guide human tuning.
Keep failed calls, rejected candidates, window changes, hidden and manual trials in the development ledger. Bailey and co-authors warn that holdout evaluation does not account for the number of attempted configurations. Their February 2015 author draft discusses a different procedure, CSCV/PBO; no PBO estimate or multiple-testing correction was computed here.
If a final outcome prompts a change, that interval becomes development data. Retesting it is not fresh untouched confirmation; a revised claim needs fresh evaluation. Disclose prior knowledge of public historical periods. One untouched interval cannot guarantee protection from multiple testing or regime change.
Next step: write an audit manifest
Before requesting any study, write a small manifest another researcher could inspect. It should name the question, the frozen procedure and the evidence needed to reject the claim. No measured return, drawdown, trade count, best window or released optimizer capability follows from this article.
- Record the protocol revision, candidate/window choices, score/ties/refit, information cutoffs, boundary policy, sizing/costs, benchmark and acceptance criteria. Preserve every development trial and final-access decision.
- For a future actual study, retain source/engine versions, real data snapshot hashes, exact market identity, UTC bounds, finality/availability evidence and the complete fill, fee and mark ledger. Do not invent missing run IDs or hashes.
- Keep one chronological marked equity path, net returns, drawdown, a defined turnover measure, USDT fee totals, closed trades, exposure/time in market, benchmarks and cost stress. Retain weak folds and failures, including a gross gain that disappears after costs or a failed cash/benchmark claim. No such results exist here.