Assessing the Forecasting Accuracy of Competing Prepayment Models: A Case Study

Introduction

By David Sykes, Ph.D, David Sykes Partners, LLC – How does a mortgage investor decide which prepayment model to use? Mortgage prepayment models are available from a number of third-party vendors and/or a portfolio manager may be confronted with multiple in-house alternatives. The choice of prepayment model is clearly a critical decision given that prepayment forecasts underlie the calculations of both the value and risk (e.g. duration and convexity) of a mortgage portfolio.

Unfortunately, however, determining the "best" model is generally not a traightforward process. "Best" has numerous dimensions. To sort some of them out, consider two extreme modeling perspectives:

1) Empirical: this perspective reflects the observation that history often repeats itself; thus a model is designed so that for given economic conditions (rate scenario etc), the prepayment forecast replicates as closely as possible the prepayments actually experienced when those same conditions occurred in the past.

2) Behavioral/theoretical: here the priority and emphasis is on accurate, detailed modeling of the decision-making process; a model’s accuracy in replicating historical scenarios is a secondary consideration; behaviorists argue that fundamentals change and that the only hope of capturing the prepayment implications of these kinds of changes is detailed attention to modeling the essence of the decision-making process.

In practice, the "best" model will strike a happy balance between the two extreme positions. No one can be sure that the past will repeat itself and, of course, until it happens, one cannot know with certainty how well a given behavioral model will anticipate scenarios never before experienced. Thus, the modeler must judge as to the proper balance between the empirical and the theoretical, and the end-user (investor) is left to judge the modeler’s acumen with regard to modeling issues as well the integrity of the modeler’s claims.

The objective of this note is to demonstrate the application of statistical methods useful for assessing the relative accuracy of competing models' forecast. These methods are empirical in their orientation in that no judgment is made as to how a model’s forecast is derived. However, while the analysis necessarily entails evaluating historical forecasts, the forecasts evaluated by the analysis may be temporarily out-of-sample. For example, consider two models that were estimated and put into service, say, two years ago. The methods discussed is this note can be used to evaluate the forecasts made by the model in the most recent two years. Thus, the results of the statistical analysis can be interpreted as an indirect assessment of the performance of the embedded behavioral model. Some standard statistical procedures will be discussed, however, the primary focus is on the Diebold-Mariano (DM) statistic. In essence, the DM statistic compares the forecast errors of two competing models to derive a probable statement as to which model is relatively more accurate.

The Models and the Data

For purposes of illustration, we conduct a comparative analysis of the forecast accuracy of two mortgage prepayment models: (1) AFT's "Espiel" model and (2) a hypothetical "Brand X" model. For expediency, the analysis is limited to conventional 30-year fixed rate product, and in particular, pools issued by Fannie Mae.

A caveat with regard to assessing forecast accuracy via back testing is in order. Over a period of 10 years, prepayment models undergo revisions and updates to reflect changes in the fundamentals of the prepayment decision. For instance, changes in application processing, technology, and marketing have shortened the lag time for a refinancing by 1-2 months; blitz marketing, among other factors, has heightened borrower sensitivity to refinancing opportunities; and, the overall long-term housing turn over rate is higher than it was 10 years ago. Consequently, it is inappropriate to apply today’s models too far back in the past. To mitigate this problem, the retrospective forecasts are limited to five years. Thus, on a pool originated in May 1993, forecasts are restricted to the period November 1998 to the present.

Methods and Analysis

The methods used to evaluate forecast accuracy exploit the following observation regarding model structure. Rigorous prepayment models, such as the two evaluated in this study, are non-linear time series models that generate path dependent forecasts. As such, the prepayment forecast for any given month depends on the model's forecast for the immediately preceding month. Thus, the process of producing a lifetime forecast entails generating each monthly forecast in a "one-step ahead" fashion; that is, the model structure requires that the forecasts be generated in sequence from the first month to the final month projected. From the sequential forecasts, a sequence of one-step ahead forecasts errors can be calculated: , where, the actual CPR for month t, and the forecasted CPR for month t given the predicted CPR at month t-1.

Gain access to the complete article and statistical results.