“Predictive models” rarely are predictive

Guest “Yogi Berra’ism ” by David Middleton

Source: First Coast Advisers

Predictive models dominate our lives — not always for the better
BY MERRILL MATTHEWS, OPINION CONTRIBUTOR — 05/06/20

The vast majority of Americans are completely unaware of how much models that predict the future of the economy and the climate – and now disease – dominate our lives. There’s no escaping their reach. Those models drive many of our public policy debates and much of the major legislation passed by Congress. 

That might not be so bad if the models’ predictions were generally accurate. But they aren’t. Indeed, they are often wildly wrong.

As the late statistician Prof. George E. P. Box warned us: “All models are wrong, but some are useful.”

[…]

All models are built on a multitude of assumptions, and many of those assumptions increasingly reflect the ideological and political views of the modelers. If the assumptions are skewed, so will be a model’s predictions.

[…]

Environmentalists, the left and most of the media accuse skeptics of being “climate deniers.” But what many on the right are skeptical of isn’t actual scientific data, but some climate models’ predictions of temperatures and sea level rise 50 or 100 years in the future. 

And yet the media regularly conflate the two. If you don’t believe the predictions of a climate model then you are denying the science, when what’s actually being questioned is many of the assumptions built into the model.

Here’s an example. Nearly all climate models in the late 1990s and early 2000s greatly overestimated rising temperatures because they didn’t take into account what’s now known as the “warming hiatus” that lasted about 14 years – from about 1998 to 2012 – when global temperatures remained relatively flat. 

In other words, the actual data did not match the models’ predictions, which left climate modelers and environmentalists scrambling to explain the discrepancies.

[…]

And yet leftists and environmentalists want us to dramatically alter the economy and our way of life – e.g., through the Green New Deal – based on predictions that might, but probably won’t, be correct.

And speaking of predictions that aren’t correct, can we talk about those coronavirus pandemic models? A new National Bureau of Economic Research (NBER) working paper highlights just how influential – and wrong – some of the pandemic models have been.

Both U.S. and UK leaders were advocating a measured response to the coronavirus pandemic until the UK’s Imperial College-London released its model’s results predicting 500,000 deaths in the UK and 2.2 million deaths in the U.S. 

[…]

Within a couple of weeks, the Imperial College scaled back its predictions, to no more than 20,000 UK deaths. And most pandemic modelers have been revising their worst-case scenarios.

[…]

Even so, the media have been obsessed with the worst-case numbers. And anyone who raised doubts about those predictions was pilloried by the media and the left as denying the “science.” 

[…]

And yet models increasingly control our lives because policymakers use them to justify their actions and their votes.

As Dr. Anthony Fauci, the lead U.S. epidemiologist in this pandemic, recently warned, “I know my modeling colleagues are going to not be happy with me, but models are as good as the assumptions you put into them.” He’s right. 

[…]

Merrill Matthews is a resident scholar with the Institute for Policy Innovation in Dallas, Texas. Follow him on Twitter @MerrillMatthews.

The Hill

The IHME (Institute for Health Metrics and Evaluation at the University of Washington) is anther model policymakers have relied upon.

Influential Covid-19 model uses flawed methods and shouldn’t guide U.S. policies, critics say
By SHARON BEGLEY @sxbegleAPRIL 17, 2020

A widely followed model for projecting Covid-19 deaths in the U.S. is producing results that have been bouncing up and down like an unpredictable fever, and now epidemiologists are criticizing it as flawed and misleading for both the public and policy makers. In particular, they warn against relying on it as the basis for government decision-making, including on “re-opening America.”

“It’s not a model that most of us in the infectious disease epidemiology field think is well suited” to projecting Covid-19 deaths, epidemiologist Marc Lipsitch of the Harvard T.H. Chan School of Public Health told reporters this week, referring to projections by the Institute for Health Metrics and Evaluation at the University of Washington.

Others experts, including some colleagues of the model-makers, are even harsher. “That the IHME model keeps changing is evidence of its lack of reliability as a predictive tool,” said epidemiologist Ruth Etzioni of the Fred Hutchinson Cancer Center, who has served on a search committee for IHME. “That it is being used for policy decisions and its results interpreted wrongly is a travesty unfolding before our eyes.”

[…]

The chief reason the IHME projections worry some experts, Etzioni said, is that “the fact that they overshot will be used to suggest that the government response prevented an even greater catastrophe, when in fact the predictions were shaky in the first place.” IHME initially projected 38,000 to 162,000 U.S. deaths. The White House combined those estimates with others to warn of 100,000 to 240,000 potential deaths.

[…]

IHME uses neither a SEIR nor an agent-based approach. It doesn’t even try to model the transmission of disease, or the incubation period, or other features of Covid-19, as SEIR and agent-based models at Imperial College London and others do. It doesn’t try to account for how many infected people interact with how many others, how many additional cases each earlier case causes, or other facts of disease transmission that have been the foundation of epidemiology models for decades.

Instead, IHME starts with data from cities where Covid-19 struck before it hit the U.S., first Wuhan and now 19 cities in Italy and Spain. It then produces a graph showing the number of deaths rising and falling as the epidemic exploded and then dissipated in those cities, resulting in a bell curve. Then (to oversimplify somewhat) it finds where U.S. data fits on that curve. The death curves in cities outside the U.S. are assumed to describe the U.S., too, with no attempt to judge whether countermeasures —lockdowns and other social-distancing strategies — in the U.S. are and will be as effective as elsewhere, especially Wuhan.

[…]

While other epidemiologists disagree on whether IHME’s deaths projections are too high or too low, there is consensus that their volatility has confused policy makers and the public:

— Last week IHME projected that Covid-19 deaths in the U.S. would total about 60,000 by August 4; this week that was revised to 68,000, with 95% certainty that the actual toll would be between 30,188 and 175,965.

— On March 27, it projected that New York would see 10,243 deaths (and that the total had a 95% chance of falling between 5,167 to 26,444) by early August. Three days later, the New York projection was 15,546, and on April 3 it was 16,262, Jewell and her colleagues pointed out in another analysis, published in JAMA on Thursday.

— On April 8, IHME projected 5,625 deaths for Massachusetts by August; on April 13, it was 8,219.

[…]

A different, data-driven model from researchers at the University of Washington predicts “about 1 million cases in the U.S. by the end of the epidemic, around the first week in June, with new cases peaking in mid-April,” said UW applied mathematician Ka-Kit Tung, who led the work. “By the first week of June, we project that the number of new cases will be close to zero if current social distancing policies are maintained.” That model predicted two weeks ago that the number of new daily cases would peak around now, as seems to be the case.

Stat News

One of the major pitfalls in using predictive models to drive policy decisions, is that no matter what happens, it always would have been worse, if we hadn’t followed the model-driven opinions of “experts”…

The chief reason the IHME projections worry some experts, Etzioni said, is that “the fact that they overshot will be used to suggest that the government response prevented an even greater catastrophe, when in fact the predictions were shaky in the first place.”

Stat News

Rarely do we ever have a way to determine whether or not “the government response prevented an even greater catastrophe.” One of the clearest examples of being able to demonstrate that “the predictions were shaky in the first place.” was the 2009 economic stimulus bill.

The Grand Obama Illusion: Major Promises Never Delivered

Kyle Smith

Summarizing his wonderful, magisterial book, The Discoverers, in an interview, historian Daniel Boorstin said, “The great obstacle to progress is not ignorance, but the illusion of knowledge.”

Fast forward to one of the great illusions of our time, the infamous chart that precisely laid out exactly what the unemployment rate would be at each stage of the recovery, with and without the Obama stimulus package. Today the chart is a monument to folly. It is not merely incorrect; it is stunningly off. It might as well have been produced by a witch doctor or by random guessing.

[…]

The chart by Obama economic elves Jared Bernstein and Christina Romer — it would be granting them far too much dignity to call them “economists” — tells us that, were it not for the miracle of the stimulus, we would be stuck with unemployment of about 5.7 percent today, but with the stimulus we were told to expect a jobless rate of about 5.2 percent. Instead, unemployment is at 7.8 percent, and the $800 billion we spent on snake oil stimulus has vanished as the disease it purported to cure continues to ravage us.

There is a direct line between the arrogance of the chart and the personality of the Commander in Chief. Obama is a frightening combination: He possesses both a proudly non-empirical mind — he admitted on The Tonight Show this week that his math skills began to fail him as early as seventh grade and that homework in the subject done by his ninth-grader daughter baffles him — and an absolute faith in those who call themselves scientists. Like the most devout churchgoers, he admits to no understanding of how those he worships works, yet is prepared to defend everything they do. The difference is that people of faith don’t get to redirect hundreds of billions of dollars of other people’s money to their belief system.

[…]

Forbes, October 31, 2012

“It’s tough to make predictions, especially about the future”… But it’s easy to get away with bogus predictions, if there’s no way determine what would have happened under different conditions. In the case of the worst COVID-19 model, Covid Act Now, they just keep shifting doomsday to the day after people are allowed to go back to work.

Texas began lifting restrictions on May 1. The Covid Act Now model is particularly useless, because it has two options 1) current trend and 2) no restrictions at all. In the no restrictions scenario, Texas would have had about 25,000-30,000 COVID-19 hospitalizations by now…

Covid Act Now, Texas, April 18, 2020

As of May 7, Texas has 1,750 COVID-19 hospitalizations…

TXDSHS

The latest Covid Act Now model, still shows Texas at the brink of doomsday…

Covid Act Now, Texas, May 7, 2020

To hammer home, the danger to our liberty and prosperity, that bureaucrats armed with models present… Dallas County government officials have steadfastly relied on the Covid Act Now models in imposing restrictions on Dallas County residents.

Dallas County Judge Clay Jenkins (Fire Marshal Gump) and Covid Act Now model. https://thefederalist.com/2020/03/25/inaccurate-virus-models-are-panicking-officials-into-ill-advised-lockdowns/
Dallas County Health and Human Services Director Dr. Phillip Huang and Covid Act Now model. https://thefederalist.com/2020/03/25/inaccurate-virus-models-are-panicking-officials-into-ill-advised-lockdowns/

They recently extended the shelter in place order to mid-May, defying the state’s decision to end shelter in place at the end of April. Dallas County Judge Clay Jenkins (Fire Marshal Gump) and Dallas County Health and Human Services Director Dr. Phillip Huang cited a sudden spike in COVID-19 cases that mysteriously began on May 1, 2020 and claimed this was not due to increased testing… But Dallas County will not release data on the number of tests being performed or recoveries.

Record coronavirus cases for 2nd consecutive day in Dallas County despite no test change
BY STEFAN STEVENSON
MAY 01, 2020

Dallas County reported a single-day high for the second consecutive day on Friday with 187 new coronavirus cases and two more deaths.

[…]

“This increase in positive cases has occurred without any significant increase in testing capacity,” Dallas County Judge Clay Jenkins said in a release. “We have seen younger people dying from COVID-19 this week and today’s victims add to that list. All this illustrates why we all must make smart decisions and follow the science to flatten the curve.”

[…]

Fort Worth Startlegram

Another COVID-19 Record for Dallas County; Masks Required, Enforcement Unclear
County judge amends county order to make Gov. Greg Abbott’s recommendations for reopened services a requirement in Dallas County
By Frank Heinz • Published May 4, 2020

Dallas County Judge Clay Jenkins amended his Safer at Home order Monday to make Gov. Greg Abbott’s recommendations for businesses reopening in Dallas County to now be requirements.

[…]

“For instance, when the governor says to the fullest extent possible wear a mask as a recommendation, we would say that’s a requirement,” Jenkins said. “When the governor says in a movie theater, let’s close every other row and put two seats of separation, that’s recommended, we say in Dallas county that’s required.”

He told NBC 5 Monday that the county doesn’t plan to fine individuals, but that code inspectors could fine businesses. The amended order can be seen below.

[…]

Dallas County is now reporting a total of 4,370 positive COVID-19 cases. Dallas County has not been releasing statistics on the number of recoveries in the county saying it’s not a surveillance variable being used nationally by the Centers for Disease Control and Prevention or state health departments.

[…]

NBC5DFW

Since Dallas County won’t report daily testing data, only reporting positive tests and deaths, the claim that “This increase in positive cases has occurred without any significant increase in testing capacity,” can’t be verified. And the Dallas County claim sticks out like a sore thumb. The State of Texas publishes statewide testing data…

Texas and Dallas County COVID-19 testing data.

The rate of statewide testing clearly has been increasing, while the rate of positive tests has steadily been decreasing. Yet Dallas County claims that the positive tests spiked right when the state began reopening because the infection rate was still increasing, therefore, restrictions must be tightened. However, Jenkins claims that “the county doesn’t plan to fine individuals”… Except…

Dallas salon owner gets 7 days in jail for reopening in defiance of countywide restrictions
Published 2 days agoDallas

DALLAS – A Far North Dallas salon owner will spend 7 days in jail after she refused to apologize for opening her business in defiance of countywide restrictions.

A Dallas County judge offered Shelley Luther, the owner of Salon a la Mode, a deal: apologize for being selfish for having her salon open while everyone else’s were closed, pay a fine, shut down until Friday and she could avoid jail time.

[…]

Fox 4 Texas

Shelley Luther wasn’t jailed for violating “countywide restrictions,” she was jailed for refusing to apologize to Dallas County officials. Texas Lieutenant Governor Dan Patrick paid her fine, the Texas Supreme Court ordered her immediate release from jail and Governor Abbott has ensured that this sort of travesty doesn’t happen again.

via Watts Up With That?

https://ift.tt/2WePlGf

May 8, 2020 at 04:43AM

Leave a comment