Validation and 'entanglement' in models of complex energy systems

In January of this year, one of us (Tan) was at the National Taiwan University in Taipei as an external examiner for the final defense of a Ph.D. student working in the field of process systems engineering (PSE). PSE is a relatively new branch of chemical engineering that focuses on the development of systematic methods for the design and control of industrial processes. The student’s research work was very solid, and had been published in the form of half a dozen papers in chemical and environmental engineering journals, so the oral examination was almost (but not quite) a mere formality. Over the course of his studies, he had developed a family of computational techniques for systematically designing water recycling schemes in industrial plants with batch operations, which were demonstrated in his thesis using a variety of realistic illustrative examples based on actual industrial processes. Even so, one question was predictably raised by a local (Taiwan-based) member of the examination committee: “Have you validated your model by actually applying it in industry?”

This question betrays a fundamental misunderstanding of the role of the systems engineering researcher. In systems engineering research, the main goal is to develop models to add to the practicing engineer’s “toolbox” for dealing with various potential problems. The actual application of the tools is left to the practitioner, who plays the role of “client.” Clearly, the question was biased, since the only way this Ph.D. student could have been guaranteed to actually, physically implement his designs in an industrial plant was if he owned a factory that he could use as a test bed! In any case, the issue was quickly resolved, and the student received his Ph.D. degree without any further significant issues. Over dinner later that day, his adviser said that once one is satisfied with the internal logic of the model, the issue of validation is moot. Although such a statement may seem self-evident to those engaged in systems engineering research, the issue of “validation” will always leave nagging doubts in the minds of the practitioners and decision-makers who are our ultimate clients, especially for novel techniques that have yet to gain a faithful following. And so, for systems engineers, the issue of validation is by no means a trivial one.   

The fundamental distinction between models in engineering and in the pure sciences lies in their underlying purpose. While models in the pure sciences are meant to provide an understanding of a given system (e.g., the effect of atmospheric greenhouse gas levels on air temperature), in engineering there is an implicit desire to change the state of the system toward a desired goal. This difference gives rise to a common misconception regarding the “validation” of mathematical models. Models in the sciences can be calibrated using historical data; for as long as a given model reflects the underlying processes with sufficient fidelity, it can be used to make satisfactory a satisfactory forecast of future events. Such models can be validated retrospectively (i.e., by “predicting” the present state from past data) or simply by observing the system’s future state. By comparison, engineering models have a prescriptive (as opposed to descriptive) aspect which prevents strict validation in this sense. We cannot use historical data that contain historical actions that may not be known. For instance, can we find a way to ensure the success of the current Philippine biofuels program by looking at the reasons for the failure of similar programs (e.g., Alcogas in the 1980s) in the past? Can we derive any useful lessons from Brazil, which has somehow sustained its own ProAlcool program for nearly four decades? Physical system models where decision variables can be identified can be calibrated using historical data because the effect of old decisions can be removed. Modeling of soft and social/economic/political/operational systems presents a challenge because old decisions are embedded in the total system dynamics. In other words, borrowing terminology from quantum mechanics, we can say that intrinsic system properties and external interventions are “entangled.” Thus, applying new decisions for forecasting future system performance is very challenging. Furthermore, strict “validation” is impossible since the decision-maker deals with alternative futures — i.e., with and without the interventions prescribed by the model. Clearly, such alternative futures cannot be observed using the “wait and see” approach that works for models in the pure sciences.

In our current research, we are trying to develop a general methodology that is more appropriate for forecasting the effect of new policies and new decisions on future system performance. The key stumbling block is finding a way to “untangle” the effects of intrinsic system properties from external factors. If, or when, we are able to figure out this problem, it will become possible to create “mathematical crystal balls” to look into the future, to see the future that we want, and to find out how to get there.

* * *

Raymond R. Tan is a professor of Chemical Engineering and University Fellow at De La Salle University, Manila. His main research interests are process systems engineering, life cycle assessment and pinch analysis. He is the author of more than 60 articles in ISI-listed chemical, environmental and energy engineering journals. He is a member of the editorial board of the journal Clean Technologies and Environmental Policy and co-editor of the forthcoming book “Recent Advances in Sustainable Process Design and Optimization.” He is also the recipient of multiple scientific honors, including the 2004 Outstanding Young Scientist Award from the National Academy of Science and Technology (NAST) and the 2007 Achievement Award from the National Research Council of the Philippines (NRCP).

Jose B. Cruz Jr. received his BS degree from the University of the Philippines, MS degree from the Massachusetts Institute of Technology, and Ph.D. degree from the University of Illinois, Urbana-Champaign. He was a distinguished professor of Engineering and dean of Engineering at the Ohio State University. He is a professor emeritus at the University of Illinois, at the University of California Irvine, and at the Ohio State University. He was president of the IEEE Control Systems Society, editor of the IEEE Transactions on Automatic Control, and IEEE vice president for technical activities, and later for publication activities. He is a Life Fellow of IEEE, and a Fellow of the American Association for the Advancement of Science, the American Society for Engineering Education, and the International Federation on Automatic Control. He received the American Automatic Control Council Richard E. Bellman Control Heritage Award, the IEEE James H. Mulligan Jr. Education Medal, the ASEE Curtis W. McGraw Research Award, the IEEE Richard M. Emberson Award, and the IEEE Education Society Achievement Award. He is a member of the United States National Academy of Engineering, and a corresponding member of NAST.

Alvin B. Culaba, Ph.D., is a University Fellow and professor of Mechanical Engineering, and is the director of the Center for Engineering and Sustainable Development Research at De La Salle University, Manila. He is engaged in energy and environment-related research which have been published in local and international journals. A member of the NAST, he is designated as the focal person on energy. He also serves as member of the energy and environment expert panel of the Congressional Commission on Science and Technology and Engineering (COMSTE) and a former special technical energy adviser of the Department of Energy. He was recently appointed by President Aquino as a member of the Presidential Coordinating Council for Research and Development (PCCRD). Currently, he is the president of the NRCP.