In/Out Sample metrics and WFA

In/Out Sample metrics and WFA

allan nathan
2013-03-31 22:15:14

Hi,
Have been using the Walk Forward Optimization and was wondering how the In and Out of Sample Metrics were calculated and what exactly they are?

hanks,

Allan

QuantShare
2013-04-02 11:54:08

Hi Allan,

The In-Sample period is used for optimization and the best parameters are then backtested against the Out-of-Sample period to see if whether the result is robust or not.

Take a look at this post:
http://www.automated-trading-system.com/walk-forward-testing/

clonex
2013-04-02 12:26:04

Hi I have few suggestions.

As you wrote ,
"The In-Sample period is used for optimization and the best parameters"

But there i see one problem. Many times best parameters doesent work in OTS testing. In ITS period WHO find one best combination of parameters which could be not profitable in OTS period. But maybe second/thir result is more robust. So the crucial qustion is about robustness.

So I have two ideas how to solve this problem.
1. Creating option for testing best 5%/10% results. ( And then make an average of theese results)
2. Have an option for 3d optimization charts . ( This ist better applicable for classis backtester)

I found some sites related to this issue:
http://mechanicalforex.com/2012/05/walking-forward-analysis-more-robust-development-with-more-realistic-expectations.html
http://help.tradestation.com/09_01/tswfo/topics/cluster_analysis_evaluating_re-optimization.htm

Clonex

allan nathan
2013-04-03 01:09:28

Hi,
I understand the dynamics of a walk foward optimisation and in sample testing vs out of sample.What I am not familiar is the term In Sample/Out of Sample Metric?

For instance,I have an average in sample metric of 1.37 and a 17.32 In sample annual return.The annual return is self explanitory,but I dont know what a 1.37 IS metric means or relates to.

Thanks,

Allan

QuantShare
2013-04-05 10:45:13

The metric is the measure you choose in the walk forward optimization settings.
It will be used to choose the best parameters from the In-Sample optimizations.
By default, it is set to "Annual Return".

Alpha Trader
2014-06-02 17:35:27

Hello - Do you plan to add walk forward testing in the PBIL optimizer? This would be great! Any progress on Clonex's ideas? These would be very helpful.

Thanks,
Alpha Trader

QuantShare
2014-06-03 10:40:59

-1

Hi, It is not possible to use walk forward testing with PBIL.

PBIL optimizer is a tool that uses a specific algorithm to speed up optimization process.

Vangelis M.
2014-06-03 13:43:03

+1 for Clonex's idea. Best parameters (i.e., overfitted) usually underperform in Out Of Sample data. It would be interesting to choose a 'average" set and walk-forward.

clonex
2014-06-03 14:46:48

Yes theese improvements is a must. For me is WFA unusable in this setup

Alpha Trader
2014-06-04 05:00:27

I also strongly agree with the average statements above... However, I beg to differ that PBIL is not possible in WFA.

You would set the PBIL parameters first. It would be set to say, 50 generations, etc.etc. then set the WFA parameters including the average concept above. Once the first InSample Data set is run, and the following Out of sample is done, a new PBIL would start on the second In sample/Out of Sample Set. (completely reset from the first InSample).

In this case, lets say we had 200 IS and 200 OOS. Let's call this a "trial run" which is one lap through the data set. Or in theory, one potential result set over the period assuming someone used a PBIL to find the best parameters, run them, and then reset the PBIL and run again.

Furthermore, if we could then tell the system how many "trial runs" through the IN/OUT of sampling set to take....

i.e. Run 1000 Trial Runs (200 IS/OS *1000 = 200,000 Unique PBIL Optimizations)In/Out of sample sets. This could be quite useful when there are Tens or Trillions of iterations.

Additionally as a separate request item, it would be great to run a more robust range of IS/OS dates. i.e. I would like to see the results of not just resetting every month, or quarter, but all of them in aggregate with Anchoring on and off at differing lengths of IS data sets. This way it may help identify data sets that may benefit from longer term memory with Anchoring on and those that benefit from more flexibility with anchoring off.

That reminds me of a 3rd request... is it possible to save the out of sample performance strung together in a equity curve ? Or show the out of sample performance relative to something? i.e. A benchmark or the population that was tested(percentile) etc. etc.

Thanks for all the great work! Great stuff.

Get more messages

No more messages

Back


Trading Items Risk Metrics Adaptive Trading System - Minimum Percent Winners and Percent Inv... MAE and MFE of a Trading System Entry Price and Exit Price in Trading System Report System Metrics v2.0	How-to Lessons How to add custom position-based metrics to your trading system Difference between the watchlist and the screener tools How to download EOD quotes for active and valid stocks only How to create trading rules based on Put and Call volume data How to create and trade a Neural Network model
Related Forum Threads Performance Metrics of Buy and Hold Request: Example of Custom Metrics addition into Rules Analyzer Accessing Symbol Info in Screener, Strategy script and in MM tool... Quantshare release notes and in the works GURU, GOLD AND SILVER	Blog Posts Volatility and trading systems Create custom metrics for the statistical data analysis tool How to Create Custom Metrics for Your Trading Positions Trading System Analysis: Backtesting report and custom measures Download historical EOD data for the stock, futures, ETF and Fore...