A critical look at those surveys

In May 2019, days before the midterm polls that year, a private survey group projected that incumbent mayors in most cities in Metro Manila would win their re-election bids. It was a non-commissioned survey that had 8,000 registered voters in Metro Manila as respondents, according to a report by The Philippine STAR.

While the survey group’s methodology was not reported in detail, one aspect of the results was particularly notable: its projection in Pasig City was significantly way off. It predicted a resounding victory for then Pasig City re-electionist Mayor Roberto “Bobby” Eusebio against then challenger Councilor Vico Sotto. However, in the actual election that month, Sotto won by a large margin, receiving nearly 75% more votes than Eusebio. This means for every vote Eusebio received, Sotto received about 1.73 votes.

Surveys can sometimes be wrong due to statistical margins of error. However, given the significant ratio discrepancy I described above, it can be said that both politicians and the public must be careful in assessing the credibility of a survey, as well as the survey group itself.

This week, I read news about surveys conducted in Cebu that named leaders considered to be the best performing in their respective positions, according to the survey group’s findings. While this recognition is certainly good news, my unsolicited advice to politicians is to take these flattering survey results with a grain of salt.

You might wonder if I am contradicting myself by saying they should be flattered yet wary. I’m simply calling for a more realistic approach --acknowledge the recognition, but remain cautious about the underlying data.

My point here is that everyone --politicians and the public alike-- must be cautious about interpreting the results of surveys as truth without considering potential biases or errors, especially those from surveys with methodologies that are not thoroughly explained.

Public opinion surveys, as well as poll surveys, are conducted with a specific purpose: to provide the person or group commissioning them with accurate, actionable data. This refers to information that is both precise and reliable, at least within the projected statistical margin of error, and can be used to make informed decisions.

By “accurate,” we mean that to obtain such data, one must consider how it was collected and processed to minimize errors and biases, ensuring it truly represents the views or situations of those surveyed.

“Actionable” data, meanwhile, is information that effectively guides decision-making. It is detailed enough “to influence specific actions, such as shaping policy, adjusting campaign strategies, or understanding public sentiment to better address concerns.”

If the survey results merely present overall satisfaction percentages without exploring detailed areas of concern or commendation, they risk becoming akin to generic accolades handed out at a kindergarten graduation --where everyone gets a ribbon of commendation. “Actionable results break down the ratings into more detailed aspects of governance, such as specific programs or policy areas that require attention. Actionable data also includes insights into effective strategies.”

Thus, while the survey's intentions may be good and the results might make certain quarters feel good, making the results actionable requires a more nuanced understanding of how the data is collected, analyzed, and presented.

Often, surveys are shaped by the questions asked and the motives behind them. Understanding why the survey was conducted in the first place is crucial. Was it to truly gauge public opinion for policy or campaign adjustments, or was it aimed merely at boosting a particular political image?

One must consider the phrasing and structure of questions. Are they leading or suggestive to elicit a desired response? It is also important to examine who were surveyed and how they were selected. A non-representative sample can skew the data.

Another crucial question is who funded the survey? If the survey is non-commissioned, then who is funding the upkeep and operation of the survey firm? Anyone who has conducted serious research and surveys knows that there is no such thing as a cheap survey --financially or logistically. Therefore, it is only fair to ask how the survey firm sustains itself. Do they rely on subscriptions, donations, grants, or government funding? Are they attached to, or in any way affiliated with, academic institutions known for their independence, prestige, and scholarly rigor?

The underlying economics of any project or program is always a critical matter because they can shed light on the reliability of the survey project.