Consultation, Good Practise and Perception: An Examination of FIFA’s Research into a Biennial World Cup
Whilst this story could easily be characterised as an opportunity for the football family to fall out with itself, it does provide a fascinating case study in how good research practise has the potential to be undermined by an over enthusiastic interpretation of the results. Given the spectacular failure of the European Super League, brought about in no small part by the lack of consideration given to supporter opinion, FIFA would do well to consider how research should be used to inform the decision-making process.
In November 2019 the Market Research Society (MRS) published guidelines for how surveys and polling data should be used by journalists. So, does the FIFA research pass the MRS test or does it raise more questions than answers?
The first question to ask is who has undertaken the research. Helpfully, FIFA have published their methodology which states the study was undertaken “by market leading consulting agency IRIS and panel provider YouGov”. Founded in 2014, Intelligent Research In Sponsoring GmbH (IRIS) has established itself as one of the leading companies in market research, digital and media monitoring, intelligence as well as sponsorship evaluation in the sports business sector. YouGov is an international internet-based market research and data analytics firm, headquartered in the UK, with operations in Europe, North America, the Middle East and Asia-Pacific. Without question, these two organisations pass the recognised and reputable test with flying colours.
The next issue is whether the research has been carried out amongst an appropriate and clearly defined target audience. We know from the method statement that 23,225 people from 23 countries were involved in this research and that 15,008 of them were interested in football. Methodologically, it appears this was not initially a survey of football fans, rather it was a survey of the wider population, of which football fans were subsequently identified and surveyed separately. This doesn’t necessarily cause a concern, but it does beg the question why FIFA didn’t choose to identify and sample football fans directly.
FIFA say that the respondents selected were based on “football interest and viewership, geographic diversity, and population sizes while still obtaining a representative indication of the regions covered”. We know that amongst the 23 countries included were: England, Spain, Brazil, Germany and the United States. However, we also know that: Belgium, France, Italy, Argentina, Mexico and Uruguay were not. Indeed, six of the top ten international nations in the men’s game were missing from this research. Whilst it’s reasonable not to focus solely on the most developed footballing Nations, it is surprising that so many of the games traditional powerhouses, have been overlooked in this study.
We also need to understand how the research was undertaken. FIFA state that “email invitations to take part in surveys were randomly sent to a sample of nationally representative panellists. Once a panel member clicked on the survey link in the email invitation, they were sent to the survey they were most suitable for.” This approach is hardly unreasonable, particularly in the UK where 85% of the population have access to email, but what about email accessibility in countries like South Africa, China and Nigeria? Is this a survey of football fans or is this a survey of football fans who have access to email, as well as the ability and time to complete a ten-minute online survey?
Critically, we need to understand what questions have been asked. FIFA’s Results Snapshot shows that respondents were asked “If you consider your personal interest to see the FIFA World Cup – without taking into account any other impact of rescheduling the current cycle – how often would you like to see the FIFA World Cup?”. The two options to this question appear to be the “status quo” or “more frequent”, with the later prompting further options regarding the desired frequency.
The Market Research Society suggest that questions should be evaluated to see if they are “accurate, balanced and unambiguous and fit for the purpose intended”. Given that the methodological statement introduces this study as “research into potential feasibility of holding the men’s FIFA World Cup every 2 years”, it’s challenging that this question hasn’t been asked directly. Indeed, a YouGov Direct poll of football fans in the UK, conducted in September this year, addressed this question much more directly by asking respondents whether they would “support or oppose the FIFA World Cup tournament taking place every two years instead of every four years”.
To be fair to FIFA, it’s only possible to make these observations because they have published data tables and technical information. The MRS state that “published opinion polls should include sufficient background and contextual information to enable users …to interpret the information” and there can be no doubt that FIFA have done this. We know when the data was collected, how it was collected, who it was collected from and who it was collected by.
It’s not so much how they have presented their findings, rather what they consider their findings to be that is problematic. The MRS want to see research results “written up in an accurate and balanced way, to ensure they are a true reflection” and it’s this commitment to the analysis of the results where FIFA appear to fall short. The current 4 year cycle was the single most popular option at 45%, compared with only 30% of respondents supporting the 2 year option. So, whilst it’s not unreasonable for FIFA to report that a majority (55%) want a World Cup more frequently than every four years, it’s equally applicable to conclude from this study that 70% of fans don’t want a World Cup every two years!
The final question is that of influence, do the people who commissioned the research have a particular interest in the outcome? Fundamental to this question is understanding what the potential feasibility of holding the men’s World Cup every two years, actually means to FIFA. Is this something FIFA want to do, or is it something they are interested in better understanding? FIFA’s own consultation statement says “The aim is to conduct a comprehensive analysis taking into account the diverse views and interests of global football, with no predetermined objectives, and with an open mind in search of better solutions for the common good of the game.” Yet is this consistent with either the wording of the research question, or the way the headline finding has been presented?
The BBC have reported that European football's governing body has criticised FIFA over its proposal for a biennial World Cup and has asked for a "proper consultation" rather than a "promotional campaign". FIFA have said that “Under the leadership of FIFA’s Chief of Global Football Development, Arsène Wenger, the consultation process will involve all key football stakeholders’ groups such as players, coaches, clubs, leagues, scientist/medical doctors, media, commercial and broadcast partners and fans, as well as representatives from confederations and FIFA’s member associations.” Again, in principle this approach appears to be very sound, but it’s the rigour and good practice that will ultimately determine whether it is successful.
Research doesn’t always give us the answer we were hoping for, but it can help us to understand why people feel the way they do and what might be done to help them change their mind. FIFA may have started this process with the best of intentions, but how their research is conducted is every bit as important as the conclusions they communicate and ultimately the decisions they wish to make about the future of the global game. Good research practice is a necessity in delivering accurate insight. However, the perennial problem the market research industry faces is how to save clients from themselves.
Comments
Post a Comment