Parameter Robustness Curve – A novel measure to evaluate parameter influence on trading systems, strategies and expert advisors
September 21, 2011 13 Comments
Sven Schmidt^{* }
*Corresponding author: Scientific-Trading.Com, Germany, Munich, Email: scientific-trading@web.de
September 2011. All rights reserved.
Abstract
Parameters of indicators and expert systems used in technical analysis and automated trading can have a huge impact on their usefulness and performance. Many systems are published and evaluated with parameters that perform optimal on selected data. However, this gives no information of the influence of the parameter space on the performance of the indicator, expert advisor or trading system. Even worse, no intuitive measure addresses the question how likely it is that the chosen parameters will lead to a positive result of the trading system.
Here, we systematically evaluated a simple trading system based on the Super-Trend indicator for the major currencies EUR, GBP, USD, CHF and JPY. The analysis was run for 12 years and six different time frames on each of the ten currency pairs. For each currency pairs and time frame, 640 back test runs with different parameters were performed and the result noted.
In this study, we introduce the Parameter Robustness Curve (PRC) and demonstrate its usage on a large-scale evaluation of a Super-Trend based trading system. The PRC gives an intuitive impression about the probability that a random parameter combination of a trading system would lead to a positive result. Its application on different markets shows the generality of the used strategy by the Parameter Robustness Curve’s down-grade. The PRC may allow the comparison of the influence of parameter space on the performance on expert advisors and trading strategies. Here, we demonstrated on which markets our Super Trend strategy performs most robust and that this observation is very little influenced by the applied time frame.
Introduction
Parameters of indicators and expert systems used in technical analysis and automated trading can have a huge impact on their usefulness and performance. Many systems are published and evaluated with parameters that performed optimal on their tested data. However, this gives no information of the influence of the parameter space on the performance of the indicator, expert advisor or trading system. Even worse, no intuitive measure addresses the question how likely it is that the chosen parameters will lead to a positive result of the trading system.
Although various measures to evaluate the performance of a single run of an expert advisor system are available like Profit Factor and Risk of Ruin, no information about the influence of the parameters is available usually. Here, we evaluate a simple trading strategy based on the Super-Trend indicator (Robinson, 2008) and introduce a novel measure the Parameter Robustness Curve (PRC). We demonstrate that the PCR gives an impression about the likelihood to obtain a positive net result using the trading system with a random parameter choice. In addition, we show that the slope of the PRC can be used to estimate the generality of the strategy in different markets. Besides, we provide the best parameters for the Super-Trend indicator on 6 time frames. Finally, we show that parameter robustness is mainly dominated by the market and less by the time frame in our analyzed strategy.
Methods
Data
Date data was obtained from MetaQuotes Historical data download (MetaQuotes) for the currencies EUR, GBP, USD, CHF, JPY and its pairs (Table 1). For all pairs, data on the time frames 5, 15, and 30 minutes, hourly, all 4 hours and daily were used starting from January 1999 until December 2010. Here, open prices were used only.
Table 1 Currency pairs analyzed
EUR | GBP | USD | CHF | JPY | |
EUR | / | X | X | X | X |
GBP | / | X | X | X | |
USD | / | X | X | ||
CHF | / | X |
Performance Evaluation
The Super-Trend indicator algorithm as published by (Robinson, 2008) was used to define an up- or down trend based on historical daily rates data. The indicator’s parameter “window size” and “multiplier” were analyzed between [5 to 20] step size 1 and [0.5 to 20] step size 0.5, respectively. Shorter windows than five are hardly informative. Thus, in total 16*40 = 640 parameter pairs were tested for each currency pair.
Trades were opened upon any trend change indicated by the indicator and closed at the next trend change. For example, the trend is predicted to change to an up-trend a long order was opened and closed as soon as the trend is signaled to change to down-trend according to the Super-Trend indicator. If a closed order yields a loss of more than 10% of its open price than the parameter pair’s performance is labeled with -1. Likewise, any simulation in which more than 30% drawback (relative to the order open price) occurs during the holding time is denoted with a performance of -1. Parameter pairs that are labeled with -1 are referred as “failed” parameter pair.
To avoid any tampering that the account-currency and its exchange rate to the currency pair and lot size is influencing the performance evaluation of the indicator, only the absolute rate difference between open and closing of the orders is noted. Just as well, swaps and further costs like commission were not considered. Therefore, the performance evaluation results need to be multiplied by a usual leverage like 100 and a margin for example like 100. With this example leverage, margin and a 0.1 lot, a performance of 0.70619 for the best parameters in EUR/USD would yield around 70,619 EUR net winnings.
Implementation
The Super-Trend indicator as well as the performance test was implemented in Delphi 2009 (CodeGear). Since the original Super-Trend indicator repaints the last bar, i.e. in MetaTrader (MetaQuotes) bar 1, we used the open price of the following bar as trading open price. In addition our simulation executes the indicator every time a new bar is started. Here, this means that each day the indicator is run upon its open time. In a MetaTrader back test this is reflected by an Open-Price simulation. The rationale here is that a sudden out-break which leads to a Super-Trend indicator signal is frequently followed by a rebound and a immediate order opening is of often found to be less efficiency than waiting until the next bar (here day).
Results
Here, we evaluated a simple Super-Trend (Robinson, 2008) based trading strategy. Simulated trading was based on opening an order upon a trend-change and closing it on the next trend change i.e. every time a signal of the Super-Trend indicator was issued. In total 640 parameter pairs on 10 FOREX currency pairs were evaluated on six time frames starting from January 1999 until December 2010. Details about the data are shown in (Table 1).
We found that the choice of the applied parameter multiplier and window size influence the outcomes significantly. This observation supports our findings on the daily time frame (Schmidt, 2011). Therefore, we addressed the question how likely it is that a random parameter combination would result in a positive result on different markets with the Parameter Robustness Curve. Beyond, we revealed that the market is the major determinant for the performance of the analyzed strategy and less the chosen time frame.
Parameter Robustness Curve (PRC)
Any trading strategy – as our Super-Trend example here – should return a positive net win. This is the first criteria which we applied for a parameter choice to be counted as successful. Further, we consider all parameters as ‘failed’/’no-success’ that yielded a draw-back of more than 30% or a single loss of 10% on any open trade (percentages relative to the order open price).
As Parameter Robustness, we now define the percentage of parameter combinations relative to the tested total number of parameter combinations (here 640) that fulfilled these success criteria. A high robustness indicates the likelihood that a random choice of the parameters in the given market would result in a positive trading performance. Figure 1 visualizes the robustness of the Super-Trend trading strategy on major FOREX markets and on the time frames all 5 minutes (M5), all 15 minutes (M15), all 30 minutes (M30), hourly (H1), all four hours (H4) and daily (D1).
Figure 1 Parameter Robustness Curve (PRC) of the Super-Trend trading strategy. Robustness shows the percentage of possible parameters (on the evaluated parameter space and markets) that yielded a positive total net win without inacceptable drawbacks or single losses. Color and shape of the dots represent the applied time frame (M5= 5 minutes, H1=hourly, H4=all four hours, D1=daily). A high robustness indicates that almost all parameter combinations will result in a positive performance of the trading. For example a robustness of 84.1% for the currency pair EUR/USD and 15 minutes time freame means that a random choice of the parameters (within the parameter range) will lead with a likelihood of 84% to a positive result on that market and time frame. The slope of the curve further shows how well the strategy is applicable to different market types. An interesting observation is that the parameter robustness of the Super Trend strategy is comparable within a currency pair no matter which time frame was analyzed.
Best parameters of the Super-Trend indicator for each currency pair
The Parameter Robustness Curve as shown for the Super-Trend example gives an immediate visualization of the likelihood that a parameter choice is successful. In Table 2 the optimal parameters for the Super-Trend indicator are shown for the 15 minutes time frame. A complete listing of all data and time frames can be found in the supplementary material.
It should be noted that leverage and margin was assumed to be 1, therefore the resulting performance values of Table 2 need to be multiplied to represent real net winnings. For example a leverage and margin of 100 each would means to multiply the performance value by 10,000.
Table 2 Best parameters of the Super-Trend trading system on M15 time frame
Symbol |
Best-Mult.^{1} |
Best-Period^{1} |
Performance^{2} |
Trends^{3} |
Failed^{4} |
Number-Params-Loss^{5} |
Failed-Total^{6} |
Robust-ness |
CHF/JPY |
0 |
0 |
0 |
0 |
640 |
0 |
640 |
0.0% |
EUR/CHF |
13 |
9 |
0.49418 |
407 |
0 |
388 |
388 |
39.4% |
EUR/GBP |
7.5 |
17 |
0.50674 |
980 |
0 |
333 |
333 |
48.0% |
EUR/JPY |
6 |
5 |
150.126 |
2023 |
0 |
267 |
267 |
58.3% |
EUR/USD |
12.5 |
20 |
0.90632 |
733 |
0 |
102 |
102 |
84.1% |
GBP/CHF |
6 |
18 |
1.3284 |
1433 |
0 |
356 |
356 |
44.4% |
GBP/JPY |
9 |
14 |
213.087 |
963 |
0 |
240 |
240 |
62.5% |
GBP/USD |
3.5 |
19 |
0.93325 |
4563 |
0 |
162 |
162 |
74.7% |
USD/CHF |
3 |
20 |
0.9797 |
5541 |
0 |
328 |
328 |
48.8% |
USD/JPY |
5 |
8 |
60.956 |
2516 |
0 |
461 |
461 |
28.0% |
^{1} Multiplier and window denotes the best multiplier and window size parameter of the Super-Trend indicator on the 5 minutes timeframe for the data used in this study.
^{2} Performance shows a raw net win of the simulated trading given the best parameters. As outlined in the Methods sections this raw value needs to be multiplied according to real life leverage and margin as well as lot size. For a leverage and margin of 100 and a lot size of 0.1 the multiplier would be 10,000. The best Super-Trend indicator parameters for EUR/USD for the data and timeframe evaluated would thus yield around 60,110 EUR net winnings.
^{3} Trends show the number recognized trend phases i.e. how many up- and down episodes were predicted.
^{4} Failed shows the number of parameter pairs that yield either more than 30% draw-back or a single loss of more than 10%. A low number indicates that the indicator is very applicable to this currency pair no matter which parameters were tested and may serve as indicator for parameter robustness.
^{5} Loss denotes the number of parameter pairs that resulted in a negative performance result.
^{6} Failed total is the sum of ‘failed’ and ‘loss’.
Discussion
In this article, we presented a novel measure of parameter robustness on trading systems. Our defined Parameter Robustness denotes the percentage of ‘successful’ parameter choices in a parameter search space. We recommend to use i) positive total net result ii) no draw-backs greater than 30% and iii) no single losses greater than 10% as criteria for a successful run of the trading strategy. It should be noted that Parameter Robustness not only depends on the ‘success’ criteria but also on the number of evaluated time points and the parameter space. Therefore, a sufficient large parameter and rate data space is obligatory. Given comparable criteria and data, the Parameter Robustness allows comparing the likelihood of choosing ‘good’ parameters of a trading system. Even if a parameter space is not covered completely, a representative and sufficiently large- and fine grained sampling would allow estimating a pseudo-probability.
To address the comparison of different markets, we further introduced the Parameter Robustness Curve (PRC) which allows estimating the influence of parameter choice on different markets. Considering the success-criteria of a trade system evaluation as constant and parameter space equivalently large, PRC allows to compare trading systems with regard to their parameter influence on various markets at a single glance easily.
References
CodeGear. (n.d.). Delphi 2009. Retrieved from http://www.embarcadero.com/products/delphi
MetaQuotes. (n.d.). Forex Trading Platform MetaTrader 4. Retrieved 08 2011, from http://www.metatrader4.com/
Robinson, J. (2008, 07 20). Codebase MQL4, Super Trend source code. Retrieved from http://codebase.mql4.com/4725
Schmidt, S.-O. (2011, 08 15). Evaluation of Super-Trend indicator’s parameters for all major FOREX pairs over 12 years. Retrieved from TradingResearch Blog: http://wp.me/1M1iZ
Supplementary Material
Robustness_Curve_SuperTrend.xlsx is available on request from http://www.scientific-trading.com/contact.php