All Models Are Wrong. Is Simulation Still Worth It? (Part 1)

Part 1 - Does building performance simulation have value?  If so, what is the value of building performance simulation?

These questions crop up every now and then, in all sorts of contexts.  Having worked in simulation across a number of different roles, I’ve seen it add real value when it’s done well and at the right time. It’s valuable when done early in the design process to inform investment decisions.  It’s valuable later in design for comparing more detailed options like zoning, HVAC systems, and value engineering proposals.  It adds value when calibrated models are applied to existing buildings to inform capital planning and future investment decisions.  It isn’t so valuable when it’s crammed into mid or late design as an exercise to “get LEED points” or “hopefully pass code because we can’t get COMcheck to pass with our 50% glazed façade.”  Those exercises mostly add cost to the project and very little value.  Unfortunately, this is where a lot of simulation effort goes today – and it likely feeds the perception that building simulation is worth less than it truly is.

I encounter skeptics of simulation with some regularity.  One of the most common objections is the old chestnut: 

“A modeler can make their model say whatever they want it to say; garbage in, garbage out.”  

Setting aside that this objection is a bit of a knock on modelers’ intelligence and motives (we’re honestly not that sneaky), it also overlooks how much we all rely on models in our day-to-day as humans, let alone as building professionals.  

Let’s start with a few other deterministic models used every day in building design, construction, and operations. While we could include the “rules of thumb” (these are also simple models), let’s leave those aside for now.

HVAC loads and sizing. The outputs depend on the inputs.  Yet does any engineer skip the load calculation when sizing HVAC equipment?

Structural loading calculations. Also dependent on the inputs.  Does any structural engineer skip them because “garbage in, garbage out”?

We could list more, but let’s look at a few models we use every single day outside of work:

Ever check the weather forecast?  Models.  Are they right 100% of the time? No. 

Ever look at a mapping app to navigate somewhere?  Was it 100% correct?  Did it know about the construction, or the road closure because an event just let out nearby?  Probably not. 

Does the fact that these models aren’t 100% accurate 100% of the time stop you from using them – or finding value in them?  I doubt it.  If you’re like me, you have alternate models to check against: multiple weather forecasts and mapping apps to compare, rather than giving up on models altogether.

Of course, following any model blindly carries risk.  No one should just follow the mapping app that tells them to drive into the lake.  But it happens.  Should we refuse to use mapping apps because someone didn’t appropriately use that model?

Which brings me to George Box’s oft-cited line:

All models are approximations. Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind.

George E. P. Box, Empirical Model-Building and Response Surfaces (with Norman Draper, 1987)

What I take away from this:

• Simplification – no model can capture every real-world detail.

  • Right-sizing – I need the right amount of data and the right “algorithms” for the job.

• Utility over perfection – a model doesn’t need 100% accuracy to have value.

  • Do not let the perfect be the enemy of the good.

• People matter – those who understand a model’s approximations and limitations keep it useful for the question at hand.

  • Competency matters – there are good reasons professions have certifications and standards.

Think about using your mapping app (a digital model of the world around you) on your phone to navigate: it doesn’t show every sign, house number, tree, building color, or current weather condition.  While you’re driving, it shows relatively little – just enough to get you there: approximate distances, street names, turn directions, a few other things. It’s simplified for the task at hand.  Of course, modern mapping apps also have street view for spotting things the navigation view doesn’t show – and you can pull it up whenever you need more detail.  

This is a key parallel to building simulation that often gets overlooked. A good simulation practitioner doesn’t rely on a single, simple early-stage model for every decision; they know which model fits which moment, and when to add detail to have confidence in the result.  They know who to collaborate with, and when.  And they know when their model is “bad” versus “good.”  And they know this from being constant learners.

Models don’t have to involve technology at all; our brains construct all sorts of simplified models for our daily tasks and activities – whether we’re conscious of it or not.  We can use technology to improve our models and take some of the mental burden off our brains.  When we combine technology with our own learning and experience, we can often get more value than either one on its own.

Say you’re attending a sporting event in an unfamiliar place.  When the game lets out, there’s a lot of traffic leaving the stadium.  If you know from experience how game-day traffic usually works – police closing certain roads, temporarily making others one-way – you have knowledge a mapping app will likely struggle to accurately represent in the moment. But since you don’t know the area, the mapping app is still useful for getting your bearings.  Even though it’s wrong about which streets are open and which way traffic is flowing, it still has value.  

Building performance models have enormous value across the entire building lifecycle when talent, technology, and time align.  Even when they are “wrong.”  Or perhaps especially when they are wrong.  Learning happens in both instances.  In other words, you develop your talent from doing things the wrong way just as you do from doing them the right way.  You want to have the talent learn the “wrong” things in a way that doesn’t expose the enterprise to risk.

In Part 2, we’ll build on that value and explore the barriers to scaling simulation to every building and across entire portfolios.