Criticism of forecasting accuracy

Adam Creighton has criticised the accuracy of modelling used in forecasting, conflating the performance of pandemic, climate, and economic forecasting. He is an economics editor of The Australian newspaper and his article on June 17 2020 was headed “Coronavirus: Inflated pandemic estimates weaken climate forecasts”.

Here are some excerpts from his article.

“so spectacularly bad was expert modelling of the spread and lethality of the coronavirus, faith in all modelling must surely suffer”.

“Why trust the experts to forecast the climate decades into the future when they were so wrong about a disease related to the common cold?”

“Climate modelling was struggling even before the pandemic, given the planet has warmed about half as much as forecast by the first Intergovernmental Panel on Climate Change report in back 1990”.

“It’s remarkable we put so much faith in expert models, given their history of failure. The Club of Rome in 1972 notoriously forecast that growth would collapse as the world’s resources ran out, ignoring human ingenuity and the shale revolution”.

“Financial models failed to account for — indeed they probably facilitated — the global financial crisis. And as almost every utterance by a central bank governor since has reminded us, economists struggle to know what happened last month, let alone forecast the impact of a policy change tomorrow”.

He has cited modelling from the 1970’s, the 1990’s, a failure to predict the global financial crisis in 2008-09, and modelling of the Pandemic in 2020 – which is still spreading and claiming lives. He identifies only 4 “failures” in the last 50 years!

There are many forecasting errors throughout recent decades, which lead one to question why he selected these particular four to demonise.

Many other forecasting failures are documented elsewhere in my book Forecasting: the essential skills. One standout is political leaders in USA, UK, Australia, and Spain predicting that Iraq had weapons of mass destruction, despite UN arms inspectors on the ground never having found any. The experts got it right and the politicians got it wrong (perhaps wantonly). The cost of this failed prediction was thousands of US armed forces lives, tens of thousands of civilian lives, and arguably a power vacuum filled by ISIS. The financial cost was in the trillions of dollars, as documented by Joseph Stiglitz and Linda Bilmes in the 2008 book “The three trillion dollar war”.

The Australian newspaper’s proprietor, Rupert Murdoch, predicted that the invasion of Iraq would result in the price of oil falling from over $US30 a barrel to $US20 a barrel. He said that the whole world would benefit from cheaper oil. This was presumably a judgemental forecast which was at least as inaccurate as any modelling. In the event, the price never fell below $28 and was over $34 in February2004. The price soared to $74 by mid-2006 and to $134 by mid-2008.

In fact the end of cheap oil was predicted, by modelling. In 1956, M. King Hubbert, a geologist with Shell Oil, correctly predicted that the rate of oil production from the lower 48 American states would peak in 1969. His prediction was based on the observation that oil production from any large region starts to fall when about half the crude is depleted. The output from a region as a whole follows a bell shaped curve. Hubbert extrapolated this to output from whole countries. Campbell and Laherrere extrapolated this pattern to world production (“The End of Cheap Oil”, Scientific American, March 1998). Their modelling was very accurate! More recently, shale oil has increased supply which has contributed to a fall in prices – but not sustainably to $20 a barrel.

There is a common element to the four cases cited by Creighton – perceived damage to economic growth if the forecasts are acted on, in the cases of the Club of Rome Limits to Growth, Climate change, and the COVID-19 pandemic. The reason that the failure of economists to predict the GFC is included may be that economic recession in the USA was not avoided by appropriate policy which could have implemented if recession was predicted. This would have been a self-defeating prophecy!

In the case of the Club of Rome limits to growth, the dire predictions did not occur (yet) because of action, such as the green revolution, in response to the predictions. A self-defeating prophecy.

Similarly in the case of climate change predictions: some actions in response have been taken. Not enough action to slow the rate of growth of carbon dioxide in the atmosphere, but enough to stop an acceleration of it.

At least Creighton acknowledges there has been global warming – half as much as predicted is his claim – so the climate model predictions were directionally accurate. The models used by climate scientists have improved since the 1990’s and will continue to improve in the future as more is learned about feedback mechanisms and other climate effects.

The predictions of epidemiologists concerning COVID-19 also resulted in action to reduce the rate of spread, which did have severe economic consequences. This pandemic has a long way to run yet, with global cases and deaths still increasing rapidly. Hopefully their predictions were also a self-defeating prophecy.

The failure of the vast majority of economic forecasters to predict the global financial crisis was consistent with their track record – they fail to predict turning points.

In the case of the impact of the COVID-19 pandemic in Australia, I was shocked at the scenarios. Australia’s Deputy Chief Medical Officer Paul Kelly said that the number of infections would be in the range 20% to 60% of the population. Deaths would range from 50,000 to 150,000 (The Age 17 March 2020). Australian economist Warwick McKibbin estimated that almost 100,000 Australians could die from COVID-19 (the range was 21,000 to 96,000). This is based on modelling seven different scenarios, building on the experience of the SARS outbreak in 2003 and Spanish Flu in 1918 (Australian Financial Review, 3 March 2020). The media reporting did not mention the specific assumptions used by the experts.

As at 4 August 2020, the number of confirmed infections in Australia was 18,730 and the number of deaths was 232. The accuracy of the forecasts was abysmal. To some degree, this is attributable to the policy response – a shutdown of travel, workplaces, and schools combined with social distancing. It might be argued that without the dire forecasts, politicians may never have agreed to the policies which were effective but which were known to cause an economic recession. Self-defeating prophecy.

There is, however, a second wave occurring concentrated in Victoria. The number of active cases has surged from 129 on 14 June to 6,755 on 4 August.

The actual number of infections is likely to be much higher than the confirmed number – one estimate made on the basis of analysis of blood test from pregnant women and pathology test for other purposes puts the figure at 500,000 (Australian Financial Review, July 22 2020.  If this figure, representing 2% of the population, is accurate it is far lower than the 20% to 60% predicted.

Had Australia experienced the same death rate as Sweden, for example, there would have been 15,000 deaths instead of 232.  Future analysis will ascertain the relative impacts of government policies, population density, inherent population characteristics such as vitamin D levels, and other factors on death rates by country, but Australia does appear to have fared exceptionally well so far.

Suggesting that poor forecasting performance in the case of the pandemic experts automatically casts doubt on the forecasting performance of climate experts is misguided.  The forces at play are very different.

In the case of the pandemic, the characteristics of the virus, interactions with the human body, and the policy response are the key drivers of the outcome.  Early on in the pandemic these were all unknown and so scenarios were needed to crystallise the range of outcomes.  In the case of Australia, a fourth, lower, scenario was not provided initially.  Later, more detailed, modelling guided the policy response of social distancing which has proven to be effective.

In the case of climate change, it is the behaviour of carbon dioxide molecules and the responses of the atmosphere and oceans which is modelled.  The behaviour of carbon dioxide molecules, and those of other greenhouse gasses, is well known and proven empirically.  There are uncertainties in the responses of the atmosphere and oceans – for example, some types of cloud reflect incoming sunlight into space, while other types prevent heat radiation from Earth from escaping into space.  To handle these uncertainties, and the range of responses in terms of reducing emissions, scenarios are constructed to span the range of likely outcome.

I am no staunch defender of current practices in modelling.  Throughout my book I have criticised forecasts based on modelling.   My aim, however, is to contribute to improved modelling and forecasting accuracy.  Creighton, on the other hand demonises modelling without offering any solution to the problem of forecasting inaccuracy.

Creighton did not question why the pandemic itself was not predicted.  Pandemics are rare, but do they occur randomly?  Or is there something about our interaction with nature which is making coronavirus outbreaks more common?

Previous coronavirus outbreaks have included MERS (Middle East respiratory syndrome) in 2012 and SARS (Severe acute respiratory syndrome) in 2003.  While it has been known since the 1960’s that the common cold is caused by a coronavirus, there have been no recorded deadly coronavirus outbreaks until SARS.

While influenza pandemics are rare, are coronavirus pandemics becoming more frequent given three within 20 years?  If so, why?  Could it be due to the clearing of forests, for example.

We need more experts analysing this problem, not a pandemic of mistrust in experts and modelling based on misunderstanding.

Creighton’s article, and similar opinions emanating from some prominent politicians and conservative commentators, present a challenge to expert professionals involved in modelling and forecasting.  The challenge is twofold: improve accuracy and build trust in the court of public opinion.

Charlie Nelson

 
Charlie Nelson

 

Coming boom in electric vehicle sales in Australia

The overall new vehicle market in Australia has been in decline for two years, but sales of hybrid and electric vehicles have increased very substantially, from a low base, over this period. Momentum is strong and the drivers of this upsurge are expected to continue in the longer term.
A unique segmentation of the general public based on their expected future concerns indicates potential for continuing significant share growth by low fossil fuel consumption vehicles. As the overall market recovers from the current slump, this means that sales will continue to grow strongly.
Just how quickly depends on factors such as purchase price differential compared with internal combustion vehicles and government encouragement for the development of the recharging network. Three scenarios to 2030 have been developed to span the range of plausible outcomes.

Report available at foreseechange.com.au

spainrx 013 50

Charlie Nelson

Risk management

The emergence of COVID-19 was not something that we were prepared for. Not enough protective equipment for healthworkers, not enough ventilators, no economic plan, and some political leaders and commentators dismissed it as another hoax (just as the same people dismiss climate change).

Catastrophes, whether local or global, seem to happen much more frequently than humans expect. These include share market crashes, extreme weather events, geological events, terrorism, and pandemics. Perhaps we are mentally working on the normal distribution when in fact catastrophic events have fatter tails than the normal distribution. For example, we seem to hear about once in a thousand year floods or droughts almost every year.

Since 2000 some of the catastrophes that have occurred include:
• The September 11 2001 attacks in USA with the loss of nearly 3,000 lives;
• Hurricane Katrina which devastated New Orleans in August 2005, with the loss of over 1,800 lives;
• The Boxing Day 2005 great earthquake in Sumatra and the resulting tsunami, with the loss of over 200,000 lives;
• The March 2011 great earthquake in Japan and the resulting tsunami, with the loss of 16,000 lives and the destruction of a nuclear power plant which could have had even more widespread consequences;
• The European heatwave of summer 2003, which killed over 70,000.

This is just a partial list selected to show that both rich and poor countries are affected. Other earthquakes with a high death toll which have occurred since 2000 include: Haiti 2010, 220,000 killed; Sichuan 2008, 87,000 killed; Kashmir 2005, 73,000 killed; Bam 2003, 27,000 killed. There have been deadly cyclones in Myanmar (138,000 killed) and the Philippines (6,000 killed).
Then there have been three recent incidents of flight disruptions due to volcanic ash clouds. Much more severe eruptions occur at the rate of one per 100 to 200 years which are capable of disrupting air travel globally. Such eruptions occurred in Iceland in 1783 and in 1815 in Indonesia. Less frequently, mega eruptions can severely cool the planet for several years leading to huge numbers of deaths by starvation and freezing.

Solar flares are a hazard for electronic communications. The last major disruption was the Carrington event in 1859, in the days of the telegraph. A huge solar flare followed by a large coronal mass ejection struck Earth induced huge currents into wires causing extensive and costly damage. Lesser events struck in 1921 and in 1989, the latter causing a huge blackout in Canada. In 2012, there was an event of similar magnitude to the 1859 Carrington event which passed through Earth’s orbit – fortunately, the planet was in a different quadrant of its orbit. A repeat of the Carrington event today would severely disrupt all forms of communication, power transmission, GPS and other navigation, and damage pipelines. The recovery time has been estimated at four to 10 years and the cost at up to $2 trillion in the first year.

In 1908, a large comet or asteroid caused a major explosion over Siberia (the Tunguska event). The energy released was up to 1,000 times greater than the atomic bomb dropped on Hiroshima in 1945. Fortunately the area was sparsely populated, but such an impact on a major city would be catastrophic on an unprecedented scale.

The 1918 to 1920 Spanish Flu pandemic infected up to 500 million people worldwide and killed 50 to 100 million of them. Today, the speed of transmission would be much greater (due to air travel) and so there would be little time to develop a vaccination. There have been lesser epidemics since, such as Hong Kong flu. It was only a matter of time before there was another pandemic and 2020 happened to be the year it happened. We don’t know how long it will last, whether immunity or a vaccine will be developed, or when the next pandemic will occur.

This listing of potential catastrophes suggests that planners should develop scenarios to cope with such rare but devastating events. Some are local but some are global.

Planners should consider how they can prepare to survive in the event of such catastrophes. In some cases, they may also represent a profitable opportunity. At a presentation to the Australian Mushroom Growers Association, I said that they could save the world in the event of crop failures due to a major volcanic eruption blotting out sunlight. Mushrooms grow in the dark!

The annual World Economic Forum Global Risks Report is a good source of information about potential catastrophes, including human caused ones. Their January 2020 analysis perhaps under-rated the likelihood and potential severity of a disease pandemic. We need to learn from that understatement and be better prepared. We also need to factor Government response in to the analysis.

Charlie Nelson

 

A year of shocking forecasting inaccuracy

clouds0315 026 50

There have been huge forecasting errors in 2020.  While this year may be an extreme case, we are experiencing very poor accuracy about one year in ten.  Even in other years, forecasting accuracy can be poor.  How then should business and government plan?

Forecasts for the number of deaths attributable to COVID-19 were very pessimistic.

Australia’s Deputy Chief Medical Officer Paul Kelly said that the number of infections would be in the range 20% to 60% of the population.  Deaths would range from 50,000 to 150,000 (The Age 17 March 2020).

Australian economist Warwick McKibbin  estimated that almost 100,000 Australians could die from COVID-19 (the range is 21,000 to 96,000).  This is based on modelling seven different scenarios, building on the experience of the SARS outbreak in 2003 and Spanish Flu in 1918 (Australian Financial Review, 3 March 2020).

As at 16 April, the number of deaths attributable to COVID-19 is 63 and the daily number of new deaths is declining.  The number of new confirmed cases is declining, despite increased testing.  On current trends, and assuming continued adherence to restrictions such as social distancing, there will be no active cases by the end of May.

The lowest of the predicted number of deaths, 21,000, is likely to be at least 210 times too high.  The highest of the predicted number of deaths, 150,000, is likely to be 1,500 times too high

These are dreadfully inaccurate forecasts.  They may not have allowed for how Australia would respond to the pandemic – if so, they should have stated this assumption (or the reporters should have asked about the assumptions).

Economic growth forecasts were not much better.

On February 6 2020, the Reserve Bank of Australia’s forecasts were for GDP growth of 2.75% and an unemployment rate of 5.0%.  The 90% confidence interval on their GDP growth forecast included +5%, but not 0% or lower!

A survey of economic forecasters by The Age, published on 1 February 2020, produced an average predicted GDP growth of 2.18% (range 1.0 to 3.3) and an average predicted unemployment rate of  5.23% (range  4.8 to 6.4).

On March 18, the governor of the Reserve Bank said “While it was not possible to provide an updated set of forecasts for the economy given the fluidity of the situation, it was likely that Australia would experience a very material contraction in economic activity, which would spread across the March and June quarters and potentially longer. The size of the fall in economic activity would depend on the extent of the social distancing requirements, and potential lockdowns, put in place to contain the virus”.

Australia will almost certainly be in recession and the unemployment rate is likely to exceed 10%.

We only have to think back to 2008 and 2009 for another example of very poor economic forecasting.  In 2008, the Reserve Bank was lifting interest rates until March, seemingly oblivious to the imminent global financial crisis.  By September 2008 they were cutting rates as fast as they could.  In May 2009, the federal budget predicted a recession.  So too did the Reserve Bank and 99% of economic forecasters.  The recession never arrived!

Economic forecasting seems to experience very bad economic forecasting errors at least once every decade.

With frequent large forecasting errors and an increasingly turbulent world, how can planners prepare for the future?

They could look at risk assessments, such as that produced by the World Economic Forum (WEF) at the start of each year – it is an assessment of long term risks.  At the start of 2020, the WEF Global Risks Report, prepared in partnership with Marsh & McLennan and Zurich Insurance Group, rated a set of 30 risks on the basis of impact and likelihood.  Infectious disease was rated 27th out of 30 on the likelihood scale and 9th on impact.  No doubt many planners would not include infectious disease at the centre of their plans.  The WEF most likely risk was extreme weather followed by climate action failure.  The highest rated risk in terms of impact was climate action failure.

Within weeks of the release of this report, an infectious disease has had devastating impacts on lives and economies with long term repercussions.  Events which are considered unlikely do happen!  Accordingly, there is a need for a well-rehearsed plan ready for implementation to minimise the impacts of such events.

Scenarios are a valuable input to developing robust plans in the face of uncertainty.  Scenarios are plausible futures, based on a range of assumptions.  While scenarios can map out an envelope of future possibilities, they also provide a basis for developing plans which are resilient across all plausible futures.

A frequent refrain from some business leaders is a call for certainty, especially from governments.  There is no such thing as certainty, as shown by the global financial crisis and the COVID-19 pandemic!  Even government policy must change in the face of uncertainty

Tracking surveys of Australian consumer expectations about the year ahead, conducted by foreseechange, showed that in November 2019 the average estimated likelihood of a major disease outbreak in the year ahead was 42% (the Wisdom of the Masses).  This is a perceived risk that people would expect governments to manage.  The Wisdom of the Masses also indicated economic downturn and a rise in unemployment.

The general public has information that is important in the development of scenarios.  The Wisdom of the Masses should complement information from experts in the development of scenarios.

Future posts will provide more information on scenarios and the Wisdom of the Masses.

Charlie Nelson

 

 

Australia’s population growth rate may be slowing

IMG_20200110_192401

Australia’s fertility rate continues to be below the long-term replacement level, although births will outnumber deaths for many years to come.  The number of births is not increasing while deaths are increasing slowly, so natural increase is slowing.

Could Australia follow some other countries on a path to even lower fertility?  It is unlikely at present, based on our research.

Net migration may be slowing and so our population growth rate may also be about to slow.  The largest component of net migration is temporary visas – for foreign students, long-term visitors, and working holidays.  The largest source of foreign students is China and the recent coronavirus outbreak there may prevent some students from coming to Australia.

It is important that Australia’s international image is one of high educational standards and a good quality natural environment.  Recent tragic, extensive bushfires along with very poor air quality even in capital cities is not good for our image.  Nor is the intransigence of many politicians on the issue of effective action to limit dangerous climate change.

Population growth is one of three important drivers of long-term economic growth.  Let’s not turn young people away due to complacency about climate change.

Details are available at foreseechange.com.au.

Charlie Nelson

 

 

A change in the expectations and concerns of the Australian general public

hhcev0919

There has been an abrupt shift in consumer psychology which has implications for government policy, the Reserve Bank, and business decision makers.  Economic pessimism has increased and the level of belief in climate change has lifted.

These changes seem to be at odds with the federal government’s “sticking to our policy” mantra.  The heightened expectation of a rise in unemployment is inconsistent with the Reserve Bank’s hope for a decline in the unemployment rate to 4.5%.  Other shifts in consumer psychology are more positive and provide an opportunity to boost consumer spending growth.

The level of belief in imminent climate change in late 2019 is the second-highest recorded and is slightly higher than in 2007, when John Howard lost his seat in parliament and his government lost office.

The federal government and many businesses need to take more decisive action on climate change to satisfy voter (and customer) expectations.

For several years, prominent Australian economist Ross Garnaut has warned of “the great Australian complacency” which has significantly slowed Australia’s economic growth rate.  That description clearly also applies to the issue of climate change.

As the new decade dawns, more Australians are experiencing the costs of these policy complacencies.

Our tracking survey update was in field in November and early December 2019.

A summary report is available at foreseechange.com.au.

Charlie Nelson

 

The collapse of the Australian consumer: a remedy

bourkest

Symptoms of a collapse in consumer spending include four successive quarters of declining per capita retail sales volume, to the September 2019 quarter.  Declining new vehicle sales to private buyers for nearly two years to October provides confirmation of the breadth of the slump.

The recent cuts in interest rates, boosted personal income tax returns, and lower income tax rates have had no measurable impact on consumer spending so far.

Causes of this damaging slump are easily identified: low income growth and a fall in residential property prices.  There are, however, several causes of low income growth and these are identified in this report.

My report identifies the characteristics of those consumers who are most willing to spend.  These have been deserted by recent monetary and fiscal policy and should be the targets of immediate stimulus.  The forms this remedy should take are described and an implementation strategy is outlined.

My report uses proprietary consumer tracking data which measures consumer willingness to spend by demographic.

Industries exposed to consumer spending have an interest in understanding how to stimulate spending and recommending our strategy proposals to the Treasurer and the Governor of the Reserve Bank before they make any more mistakes.  Industries which would benefit include retailers and their suppliers, service industries, media companies, and media agencies.

My report is available at www.foreseechange.com.au

 

Australia’s population outlook

cakecrop

Population growth is an important driver of economic growth, lifting both demand and productive capacity.  Growth is currently 1.6% per year and may be falling.  The peak growth rate was 2.2% in 2008.

Peak births?  Births may have peaked in 2018, at 314,900.  The 1971 peak of 274,400 was not exceeded until 2007 despite significant population growth.  Fertility peaked in 1961 at 3.55 and the arrival of the oral contraception pill quickly caused a rapid decline.  Fertility has averaged 1.85 since 1985.

Peak deaths? The increase in the number of deaths has slowed recently and deaths may be at a peak.

Peak life expectancy? Not yet: life expectancy is still increasing at all ages.

Peak natural increase?  Natural increase (births minus deaths) appears to have peaked at 165,200 in 2012.

Peak net migration?  The peak was 315,700 in 2008.  The trend is currently rising, but not fast enough to break the record for several years.

Our report projects Australia’s births, deaths, net migration, and population for 2020 and 2021.  It also discusses the uncertainties surrounding births and net migration.

Charlie Nelson

 

Why Labor lost the 2019 Australian federal election

Predictions of the outcome of the 2019 federal election were inaccurate.  Both polls of voting intentions and betting markets failed to predict a Coalition victory.

parliamentcrop

 

I have explored whether the Wisdom of the Masses, the general public’s expectations about the future on several issues, can explain the election outcomes between 2007 and 2019.  Elections are intrinsically about the future.

My findings are positive and this provides a new basis for predicting Australian federal election outcomes.

Expectations about three issues have explained the outcomes of the past five elections: belief in imminent climate change; fear of terrorism; and economic pessimism.  Expectations about each of these three issues have varied considerably over this period.  The findings provide insights relevant to the focus of election strategies of political parties.

My report is available at foreseechange.com.au.

Charlie Nelson

 

 

Predicting inaccuracy in economic forecasts

 

sunkenkirk

Government and business planners need accurate economic forecasts to make optimal decisions.  Investors too need accurate economic forecasts as part of evaluating future company profits.

There are times when economic forecasts are quite accurate: this is usually when there is little variation from recent trends.  There are also times when accuracy is very poor.

It would be valuable to be able to predict when economic forecasts can be relied upon and when they are likely to be inaccurate.  This is important for risk management and appears to be possible, at least in Australia.  There are circumstances when official forecasts are likely to be most inaccurate.

I have identified the conditions under which GDP forecasting inaccuracy is greatest.  My report is available at foreseechange.com.au.

Charlie Nelson