trading systems
regime detection approach to systematic trading
markets transition between different volatility regimes.
in which i make the case for Hidden Markov Models for regime detection in systematic trading.
regime detection is a crucial component in any risk management system for institutional investors. It plays an important role in deciding when to buy tail protection as well as shield portfolios from systematic and systemic risk.
traditional approaches to regime detection often rely on simple moving averages, technical indicators, or discretionary judgment. however, these methods can lag behind actual regime changes or generate false signals during volatile markets.
this article introduces a systematic approach using Hidden Markov Models (HMMs) for regime detection and portfolio allocation. the HMM framework provides a probabilistic approach to identifying underlying market states that are not directly observable but can be inferred from market data.
my expertise is in financial risk management and systematic trading, so the focus of this article will primarily relate to volatility regimes and asset allocation.
this model is not perfect, and i will detail limitations in a later section, but to start off, let’s define the key question:
how can we systematically detect market regime changes and adjust portfolio allocations accordingly?
through a systematic, model-driven approach, we can create a beta-neutral asset allocation strategy that adapts to changing market conditions.
understanding hidden markov models in finance
hidden markov models assume the system being modeled is a markov process with hidden, unobservable states. in finance, these hidden states represent the underlying market regimes, while observable data like price action, volatility, and volume provide signals about the current regime.
the HMM framework consists of:
- hidden states: the unobservable market regimes (low, neutral, high volatility)
- observations: market data that depends on the hidden state (VIX, volume, returns)
- transition probabilities: likelihood of moving from one state to another
- emission probabilities: likelihood of observing specific data given the current state
for our trading strategy, we use a 3-State HMM that identifies three distinct volatility regimes:
- low volatility
- neutral volatility
- high volatility
the number of states was selected using Akaike information criterion (AIC) and Bayesian information criterion (BIC).
model construction and features
the model uses intraday VIX and SPX data with the following features:
- close of VIX
- range of VIX
- log return of VIX
- volume of SPX
these features were chosen for their ability to characterize different market conditions. the model is trained using walk-forward cross-validation, ensuring that only data available at the time of training is used, making every prediction an out-of-sample prediction.
t = 0 1 2 3 4 5
Out of Sample Predictions
Data available at t
through each training iteration, the model updates a transition probability matrix and an emission probability matrix, which are then used to determine the current state.
the clearest way to visualize these states is through the VIX closing price during each regime:
Low Volatility State
VIX Close
80 |
60 |
40 |
20 |___*___*_*_*_**_*_*__*___*__*_*___*___*__*___*_*__
|
Neutral Volatility State
VIX Close
80 |
60 |
40 | _/\_ _/\__ /\ _/\ /\ /\
20 |____/ \__/ \__/ \_/ \___/ \___/ \___
|
High Volatility State
VIX Close
80 | /\ /\
60 | / \ / \
40 | / \ /\ /\ /\ /\ \
20 |___/\_/ \__/ \____/ \___/ \_____/\/ \____\_
+--------------------------------------------------------
2006 2008 2010 2012 2014 2016 2018 2020
Date
what this model is most concerned with is the lead-up to regime changes. as evident graphically, the lead-up to each large volatility spike shows early signs of transpiring, most notably as a disappearance of low volatility state observations. the strategy is built around these regime changes and allocates assets accordingly.
signal generation and portfolio allocation
the model outputs a time series of states at the training frequency. to generate actionable signals, we:
- convert the model outputs to daily frequency
- assign optimized weights to each signal
- compute the trailing percentile rank of the average daily state over the last 252 days
- determine an optimized cutoff value
when the percentile rank is below the cutoff value, it is considered “safe” to trade risk assets, and the portfolio longs 1-day SPX (NYSE:SPY). when the signal is above the cutoff (indicating “unsafe” conditions), the strategy rotates into a weighted combination of Gold (NYSE:GLD) and long-dated US Treasuries (NYSE:IEF).
to prevent catching negative returns during regime shifts, the strategy tracks the trailing maximum of the rolling percentile rank. when the current percentile falls below the maximum by a predetermined amount, the portfolio holds cash until it’s safe to buy risk assets again.
here’s a simplified representation of the allocation process:
if percentile_rank < cutoff_value:
# Safe regime - invest in equities
SPY_allocation = 1.0
GLD_allocation = 0.0
IEF_allocation = 0.0
else:
# Unsafe regime - rotate to defensive assets
SPY_allocation = 0.0
GLD_allocation = 0.4 # Weight determined by optimization
IEF_allocation = 0.6 # Weight determined by optimization
performance
this regime detection strategy demonstrates robust performance metrics:
- annual return: 10.51%
- cumulative returns: 213.90%
- annual volatility: 6.55%
- sharpe ratio: 1.56
- max drawdown: -5.68%
- beta to SPX: approximately 0
the strategy excels at avoiding major market drawdowns while maintaining consistent returns. during the out-of-sample period (2017-2020), the strategy significantly outperformed with:
- annual return: 17.00%
- sharpe ratio: 2.54
- max drawdown: only -2.67%
the strategy avoided major drawdowns in late 2018 and early 2020 (COVID-19 selloff), while benefiting from the rally in long-dated rates during market stress. it then rotated back to equities in late May 2020 to capture the subsequent equity rally.
the monthly return distribution shows a slight positive skew with an average monthly return of approximately 1%. this right-tailed distribution indicates a higher probability of achieving positive returns.
the drawdown profile of the strategy never exceeds 6% and typically recovers in a reasonable timeframe. most drawdowns are minor and quick to recover from.
a key characteristic of this strategy is its consistently low beta to the SPX index, making it an excellent diversifying source of returns in a broader portfolio.
limitations
while the model is effective, there are several limitations and areas for improvement:
-
correlation breakdown: during extreme market stress (like March 2020), traditional correlations between asset classes can break down, with safe-haven assets selling off alongside equities. the strategy should monitor real-time correlation matrices to ensure assumed relationships hold.
-
data limitations: the model would benefit from additional data sources like VIX futures term structure, which could provide earlier signals of regime shifts.
-
extension to other assets: while focused on equity markets, the approach could be extended to other asset classes with their own volatility indices.
-
parameter sensitivity: the performance depends on optimized parameters for signal weights and thresholds, which must be regularly reviewed.
-
implementation costs: transaction costs and slippage were not fully accounted for in the backtest, which could impact real-world performance.
ideas for further research
several promising directions could enhance this strategy:
real-time correlation matrix for asset universe
a real-time correlation matrix would ensure that the negative correlation between risk and non-risk assets still exists. during periods when correlations break down, the strategy could adjust its allocations accordingly.
volatility strategy
since this model is based on underlying market volatility, it might perform even better by directly trading volatility instruments rather than using proxy assets. going long and short volatility based on the detected regimes could potentially generate higher returns.
asset universe expansion
the strategy could be extended to other asset classes with their own volatility indices, such as commodities and bonds. this would create a multi-asset class regime detection framework.
VIX futures curve
incorporating the VIX futures term structure could provide earlier signals of regime shifts by detecting changes in the slope and convexity of the curve.
@misc{salman2020regimedetection,
title = {A Regime Detection Approach to Systematic Trading},
author = {Raahim Salman},
year = {2020},
url = {https://rsalman.xyz/regime-detection}
}