Regime Detection via Hidden Markov Models in Futures Markets

1. Why Regimes Matter for Systematic Traders

Every systematic trader has experienced it: a strategy that prints steady returns for months suddenly begins haemorrhaging. The signals haven't changed, the code hasn't changed, but the market has shifted into a different mode of behaviour. Momentum strategies that thrived in the calm trending conditions of 2017 were crushed by the sharp reversals of late 2018. Mean-reversion strategies that captured every dip during the V-shaped recoveries of 2019–2020 bled steadily during the sustained trending decline of 2022.

The existence of distinct market regimes — qualitatively different modes of behaviour that persist for weeks to months before abruptly shifting — has been documented since Hamilton's (1989) seminal work on business-cycle regime switching. For systematic traders, the practical question is not whether regimes exist (they do) but whether they can be identified in real time with sufficient accuracy to improve trading decisions. A regime classifier that is right 80% of the time is valuable; one that is right 55% of the time may be worse than useless because the 45% misclassification events trigger precisely the wrong trades.

2. Hidden Markov Model Specification

We model daily futures returns as emissions from a Hidden Markov Model with K = 2 states. In each state k ∈ {1, 2}, the return r_t follows a Gaussian distribution with state-specific mean μ_k and variance σ²_k. The unobserved state s_t follows a first-order Markov chain with transition probability matrix P, where P_ij = Pr(s_t = j | s_{t-1} = i).

The model parameters Θ = {μ_1, μ_2, σ²_1, σ²_2, P} are estimated via the Baum-Welch algorithm (a special case of EM), and the most likely state sequence is inferred using the forward-backward algorithm. For real-time trading, We use the filtered state probabilities Pr(s_t = k | r_1, ..., r_t) rather than the smoothed probabilities, because the smoothed probabilities use future data.

from hmmlearn.hmm import GaussianHMM
import numpy as np

class RegimeDetector:
    """
    Two-state Gaussian HMM for regime classification.
    State 0: low-volatility (trending)
    State 1: high-volatility (mean-reverting)
    """
    def __init__(self, n_states=2, lookback=504):
        self.n_states = n_states
        self.lookback = lookback
        self.model = None

    def fit(self, returns):
        """Fit HMM on trailing window of returns."""
        X = returns[-self.lookback:].reshape(-1, 1)
        self.model = GaussianHMM(
            n_components=self.n_states,
            covariance_type='full',
            n_iter=200,
            random_state=42,
            tol=1e-6
        )
        self.model.fit(X)

        # Ensure State 0 = low-vol, State 1 = high-vol
        vols = np.sqrt(self.model.covars_.flatten())
        if vols[0] > vols[1]:
            # Swap state labels
            self.model.means_ = self.model.means_[::-1]
            self.model.covars_ = self.model.covars_[::-1]
            self.model.transmat_ = self.model.transmat_[::-1, ::-1]
            self.model.startprob_ = self.model.startprob_[::-1]

    def current_regime(self, returns):
        """Return filtered probability of high-vol state."""
        X = returns[-self.lookback:].reshape(-1, 1)
        probs = self.model.predict_proba(X)
        return probs[-1, 1]  # P(high-vol) for most recent obs

    def regime_stats(self):
        """Return regime characteristics."""
        means = self.model.means_.flatten() * 252  # annualise
        vols = np.sqrt(self.model.covars_.flatten()) * np.sqrt(252)
        P = self.model.transmat_
        durations = [1/(1-P[i,i]) for i in range(self.n_states)]
        return {
            'ann_means': means,
            'ann_vols': vols,
            'avg_duration_days': durations,
            'transition_matrix': P
        }

3. Asset Universe and Training

We fit the model independently to six liquid futures contracts: E-mini S&P 500 (ES), Nasdaq 100 (NQ), WTI Crude (CL), Gold (GC), 10-Year Treasury Note (ZN), and EUR/USD (6E). The training window is 504 trading days (approximately 2 years), re-fitted monthly. The evaluation period is January 2017 to December 2025, providing 9 years of data with approximately 108 monthly re-estimations per contract.

A critical implementation detail: We impose a minimum duration constraint by ignoring regime switches that last fewer than 3 days. Without this filter, the model produces spurious one-day regime changes driven by single extreme returns, leading to excessive turnover and whipsaw losses.

4. Regime Characteristics

Table 1: Regime characteristics from the two-state HMM fitted to ES daily returns, 2017–2025. Statistics are computed from days classified into each regime with >70% probability.

The two states correspond to immediately recognisable market environments. State 0 is the "normal" market: low volatility, positive drift, slightly positive return autocorrelation (consistent with gradual trend development). State 0 has a Sharpe ratio of 1.06 — equity buy-and-hold is an excellent strategy in this regime. State 1 is the "stressed" market: more than double the volatility, slightly negative drift, and negative autocorrelation (consistent with choppy, mean-reverting behaviour). Buy-and-hold has a negative Sharpe in State 1.

The average durations — 47 days for the low-vol state, 22 days for the high-vol state — have practical implications. The low-vol regime persists long enough for momentum strategies to establish and profit from trends. The high-vol regime is shorter but intense; its 22-day average duration is long enough for significant drawdowns but short enough that the model can detect the transition and respond before the worst damage is done.

5. Cross-Asset Regime Synchronisation

An important question is whether regimes are synchronised across asset classes. If all six futures markets enter the high-vol state simultaneously, the diversification benefit of a multi-asset portfolio disappears precisely when it is most needed.

Table 2: Regime concordance between asset pairs. "State Concordance" = fraction of days both assets are in the same regime. "Sync During Stress" = concordance conditional on at least one asset being in the high-vol state.

Equity indices (ES, NQ) are nearly perfectly synchronised (91%). Other pairs show moderate concordance in the 48–67% range during normal times, rising to 58–78% during stress. This increase in regime synchronisation mirrors the well-documented correlation increase during crises (see our article on correlation breakdown during stress) and has the same implication: multi-asset portfolios provide less diversification during high-vol regimes than their calm-period statistics suggest.

6. Regime-Conditional Position Sizing

We implement a straightforward regime-aware sizing rule: when the filtered probability of the high-vol state exceeds 0.7, reduce position sizes to 50% of the base level across all assets. When it drops below 0.3, restore full sizing. Between 0.3 and 0.7, linearly interpolate. This avoids the binary switching that would create excessive turnover at the regime boundary.

def regime_position_scalar(high_vol_prob, threshold_high=0.7,
                            threshold_low=0.3, min_scalar=0.5):
    """
    Compute position size scalar based on regime probability.
    Returns a value between min_scalar and 1.0.
    """
    if high_vol_prob >= threshold_high:
        return min_scalar
    elif high_vol_prob <= threshold_low:
        return 1.0
    else:
        # Linear interpolation
        frac = (high_vol_prob - threshold_low) / (threshold_high - threshold_low)
        return 1.0 - frac * (1.0 - min_scalar)

We apply this to a diversified trend-following strategy across all six futures contracts, with volatility targeting at 10% annualised as the base sizing method. The regime scalar is applied as a multiplier on top of the volatility-targeted weights.

7. Results

Table 3: Performance comparison of regime-conditional vs. naive position sizing, 2017–2025. The HMM overlay reduces absolute return slightly (due to lower average exposure) but materially improves risk-adjusted metrics.

The regime overlay adds 0.18 to the Sharpe ratio and reduces maximum drawdown by 5.6 percentage points. The mechanism is clear: the model correctly identifies the high-vol regime during the major drawdown events (Q4 2018, March 2020, H1 2022) and reduces exposure before the worst losses accumulate. The cost is modestly lower absolute returns (7.1% vs. 7.8%) because the model also reduces exposure during some profitable volatile periods — particularly the sharp recovery rallies that follow stress events.

8. Model Selection and Overfitting

The choice of K = 2 states is not arbitrary. We compare 2, 3, and 4-state models using three criteria: the Bayesian Information Criterion (BIC), out-of-sample log-likelihood, and the stability of estimated parameters across re-estimation windows.

Table 4: Model comparison across state counts. "Param Stability" = average cosine similarity of parameter vectors across consecutive re-estimation windows (1.0 = perfectly stable). The 3-state model fits better in-sample (lower BIC) but generalises worse.

The 3-state model achieves a better BIC — its third state captures a "transition" regime that exists in-sample. But this third state is unstable: its parameters shift substantially across re-estimation windows (stability = 0.64 vs. 0.82 for 2-state), and the out-of-sample Sharpe degrades to 0.74. The 4-state model overfits clearly, with parameter stability of only 0.43 and OOS Sharpe of 0.61 — worse than the naive baseline. The 2-state specification wins on every out-of-sample metric.

9. Practical Implementation Notes

Three details that matter for live implementation. First, the Baum-Welch algorithm is sensitive to initialisation. We run each fit from 10 random initialisations and select the model with the highest log-likelihood, which is computationally inexpensive but dramatically reduces the frequency of convergence to local optima.

Second, the regime classification at the boundary (probabilities near 0.5) is inherently uncertain. The linear interpolation scheme described above avoids binary switching, but a more conservative approach is to define a "no man's land" between 0.4 and 0.6 where the position scalar remains at its previous value. This reduces turnover by approximately 20% with negligible impact on Sharpe.

Third, the model should be re-fitted monthly rather than daily. Daily re-fitting produces unstable parameter estimates because the new data point has too much influence on a 504-day window. Monthly re-fitting provides a stable update cadence that tracks slow regime shifts without reacting to daily noise.

10. Conclusion

A two-state Gaussian HMM provides a parsimonious and empirically validated framework for regime detection in futures markets. The model identifies regimes that differ in volatility (2.2× higher in State 1), drift (negative in State 1), and return autocorrelation (negative in State 1 vs. positive in State 0). Regime-conditional position sizing improves the Sharpe ratio by 0.18 and reduces maximum drawdown by 5.6 percentage points, primarily by reducing exposure before major stress episodes. The 2-state specification consistently outperforms 3 and 4-state alternatives on out-of-sample metrics, reinforcing the principle that simpler models generalise better with limited financial data.