consultations_do_and_dont_guide_to_consultation_questions_0
Argyll and Bute Council · 20 June 2023
Back to Audit and Scrutiny Committee — 20 June 2023 and its agenda
Text extracted from the council's PDF. Layout and reading order may differ from the original.
Page 1
1 This guide outlines a few key principles that will: Help people to contribute their views through consultations Help you to meet your goals when carrying out a consultation The role of a consultation Carrying out a consultation is about seeking people’s views so you can take them into account when making a decision. A consultation gives people the opportunity to contribute to a decision that has not been made yet. Write your consultation step by step 1 Decide what it is you want to know Ask yourself what you want the data (the feedback you get from the consultation) to tell you. The easiest way to do this is to plan out a report or presentation that you will give once you have the data. If you want to be able to say: x% of residents feel we should focus our efforts on child care provision while y% want us to focus on fixing pot holes in the roads, that gives you a starting point for writing your questions. 2 Decide whether you need to look at views from any specific groups of people; are you interested in whether views from one group differ from another, e.g.: Gender Age ranges Disability House tenure Employment type Household composition Income brackets People with children If you are asking for personal information be clear in the consultation WHY you are collecting any personal information and make sure you comply with GDPR (general data protection regulation) requirements. Use routing: online surveys allow you to skip questions that are not relevant to certain groups of people. For example if you have a section on attitudes to child care, asking people who have no interest in child care to give views will put them off going any further with the consultation. Asking up front whether they have an interest in child care allows you to skip the section for anyone not affected or interested. Guide to writing consultations questions
Page 2
2 Similarly, if you ask people how happy they are with a service and want to ask a follow up question about why people are unhappy, you can include routing so that only those who said they were unhappy are asked the follow-on question. 3 Make sure your questions (wording and type) will provide the information you need: Keep questions short and specific. Be clear what you are asking about EXAMPLE: The council needs to prioritise spending and plans to focus on child care, pot holes, recycling, parking. The consultation document includes outlines of spending plans for each. You want to know what people think of the plans. EXAMPLE 1 Q1. Do you agree with the plans the council has for the next budget? Please tick one box X Completely agree Slightly agree Slightly disagree Completely disagree Please explain your answer (write in below) This question is: A. Vague and hard to answer. People may agree with some and disagree with others B. Check – will the answers be useful to you? C. Check – do you have time/resources to analyse answers to the open question (please explain your answer)? EXAMPLE 2 Q1. The council has suggested the following budgetary priorities and these are explained in detail in the paragraphs on page x. Please tell us to what extent you agree or disagree with each: √ Completely agree Slightly agree Slightly disagree Completely disagree Not sure The council should prioritise spending on child care The council should prioritise spending on pot holes The council should prioritise spending on recycling The council should prioritise spending on parking If you disagreed that the council should prioritise spending on these areas of our work, please give your reasons (write in below) These questions allow you to measure agreement for each proposal and collect specific information on reasons for disagreement
Page 3
3 EXAMPLE 3: Q1. The council has outlined the following as possible budgetary priorities. Please tell us which are most important to you. Please rank the possible priorities in order where 1 is most important to you and 4 is least important to you. √ RANKING ( 1 to 4) The council should prioritise spending on child care The council should prioritise spending on pot holes The council should prioritise spending on recycling The council should prioritise spending on parking Please tell us why x is most important to you (please write in) This question allows you to find out what aspects are most and least important to people. This type of question is especially useful when combined with classification groups: you can look at the most important overall, most important to men or to women, most important to those with cars or who use public transport; most important to different age groups etc. The open question can provide additional evidence to justify prioritising one aspect over another. Make sure you only ask one question: EXAMPLE 1 Q2. How happy are you with the frequency and efficiency of your bin collections? X Completely happy Quite happy Not very happy Not at all happy This asks two separate questions: one about frequency and one about efficiency. People may be happy with one but not happy with the other and therefore will be unable to answer. EXAMPLE 2 Q2. Please tell us to what extent you are happy or unhappy with the following: √ Completely happy Quite happy Not very happy Not at all happy Don’t know / Not applicable The frequency of your bin collections The efficiency of your bin collections
Page 4
4 Make sure you construct your question clearly Avoid double negatives Do you think that bin collections are not too infrequent? Yes / No X Are your bins collected often enough? Yes / No √ Avoid leading questions Do you agree with the council that it is better to cut down on waste to landfill sites? Yes / No X Do you think that the council should aim to cut down on waste to landfill sites? Yes / No √ Avoid raising expectations and try not to ask obvious questions EXAMPLE 1: How often would you like your bin to be emptied Please write in: X This would be a good question to ask only if there are unlimited resources and your objective is to create high levels of satisfaction with the frequency of bin collection or even offer a tailored service in line with individual needs EXAMPLE 2: How happy would you be if your school opened at 9am instead of 8.30am? Very happy Quite happy Not very happy Not at all happy X This would be a good question only if you needed to confirm that most people would not be happy with a reduced opening times of schools. If you want to find the most popular option for change… EXAMPLE 3: We can make savings through more efficient household waste collection, while still maintaining health and hygiene standards. Please tell us which of the possible options presented on page x would be you’re most favoured option. Option 1 Option 2 Option 3 Please say why this is your favoured option (please write in) And which would be your least favoured option? Option 1 Option 2 Option 3 Please say why this is your least favoured option (please write in) √
Page 5
5 NB: In the example above the required output from the consultation questions might seem obvious; a frequency count or a proportion of all respondents choosing each option as their first choice, thus enabling you to identify the “winner” or “best route”. In considering your outputs you may want to consider how you would reach a decision on the route to adopt if equal proportions chose each of 2 options. Framing your questions to establish a first, second and third choice from the options available would almost certainly enable an easier decision; you can take account of first choices alone or, in the case that two options are first choice for the same proportion you can consider the proportions citing each as first or second. Make sure your options match your question How satisfied are you with the frequency of litter picks? Completely agree Slightly agree Slightly disagree Completely disagree X How satisfied are you with the frequency of litter picks? Completely satisfied Quite satisfied Quite dissatisfied Completely dissatisfied √ Make sure you give people an opt-out answer (but also remember to use routing to take out any people who would not be able to answer the question) How satisfied are you with the recycling facilities in your area? Completely satisfied Quite satisfied Quite dissatisfied Completely dissatisfied Don’t know / not applicable In the example above, some respondents, perhaps those who are housebound or do not have access to cars, might not be able to answer this question. You need to give an opt-out answer such as don’t know / not sure / not applicable. In addition, you might want to find out whether the person answering is answering from experience or simply hearsay. You might, therefore, ask first: Do you use the recycling facilities in your area? Y / N Those answering no could be asked to say why, rather than being asked to rate the satisfaction of a service they do not use.
Page 6
6 Forcing Responses When consultations are conducted online you have the option to “force” a response i.e. to prevent a respondent from moving to the next question or submitting their survey until they have provided an answer. In order to maximise the number of people answering questions without forcing a response, try and include all reasonable answers as options. For example, if asking “which is your preferred day of the week for litter picks?” you would include an option for respondents to say “no preference” or “I’m not sure”. In the case of potentially sensitive questions it is helpful to include an option of “I prefer not to say”. 4. Decide what type of data you want from the question Closed questions Do you want a simple: x% said yes and y% said no: Simply use tick boxes: Yes No Don’t know Do you want to measure something like awareness, satisfaction, agreement Use a suitable tick-box scale. For example: To what extent do you agree that the Council should prioritise spending on recycling? Completely agree Slightly agree Slightly disagree Completely disagree Don’t know A 4-point scale (2 agrees / 2 disagrees as shown above) stops people sitting on the fence. There may, however, be times when a mid- point is needed and so a 5-point scale should be used. For example, : To what extent do you agree or disagree that recycling should be prioritised over bin collections? Completely agree Slightly agree Neither agree nor disagree Slightly disagree Completely disagree Don’t know
Page 7
7 Do you want very detailed measurements? A 4-point scale gives a fair amount of detail but there may be times when you want to go into even more detail, perhaps when tracking changes over time. In these cases an increased rating scale may be more useful. For example, looking at calls to the council’s call centre: Please rate your satisfaction with the speed that we answered your call. Please use a scale of 1 to 10 where 1 is not at all satisfied and 10 is completely satisfied. Gauging feelings: you might use a sliding scale for measuring things like changes in happiness and online surveys typically use a slider or smiley faces or other similar pictorial methods rather than tick boxes. For example, attitudes to the area: Please use the sliding scale to show how happy you are in general to be living in Argyll and Bute: ------------------- Do you need to ask an open question? You may want to ask about services used, how people spend their time, or other questions with multiple possible answers. You could ask an open question (i.e. allow people to write in their answer). However open questions are very labour intensive, open to interpretation and non- specific. In online surveys, you can also allow people to tick as many of the options as you like, or restrict them to one or ‘pick the top three’ as relevant and appropriate. When you phoned the council, what was your query about? Please write in When you phoned the council, what was your query about? Housing repairs Library opening hours To report a pot hole Bin collection Parking permits Etc. Something else (please write in what) However, there are times when an open question is best (e.g. to ensure respondents feel they have been given the opportunity to say everything they wish to say). For example: please use the space below to provide any other information you think would be useful to the council in deciding how frequently to collect household waste.
Page 8
8 5. Basic Analysis When conducting analysis, start with the feedback you want to find from the consultation. Do you want to know the overall total that gave one or other of your answers? Is it important to work out whether men and women gave different answers or whether different ages gave different answers? Once you know the groups that are important to you, you can work out the percentages for each. The software you use will determine the way in which you do this. Online surveys generally provide analysis software alongside the survey programming. But you may wish to download data and carry out analysis of your own. The following example is based on using Excel. We will assume that you have asked about satisfaction with some aspect of council services: How satisfied are you about the xx recycling centre’s opening hours? Very satisfied, Quite satisfied, Not very satisfied, Not at all satisfied, Don’t know. Your Excel sheet should have a column for each of the demographics (classification groups) such as gender, age ranges etc.; and one column with the answer to your question. The sheet would look like the following: ID Gender Age Response 1 Male 18-25 Very satisfied 2 Male 26-35 Not at all satisfied 3 Female 36-45 Quite satisfied 4 Male 26-35 Quite satisfied 5 Female 46-55 Very satisfied 6 Male 18-25 Very satisfied 7 Male 46-55 Quite satisfied 8 Female 46-55 Quite satisfied 9 Male 26-35 Not at all satisfied 10 Male 18-25 Very satisfied Using Excel filters you can filter on the Response column to count how many people in total gave each of the answers. You can also filter by Male and then by Female respondents to see if there is a difference between the two; similarly you can filter on the age ranges. Using filters it is also possible to look at more detailed analysis such as differences between males of different ages. So in the example above, you can see that all males aged 18 to 25 gave the response Very satisfied; this differs from males aged 26 to 35 where some said quite satisfied and others said not at all satisfied.
Page 9
9 However, most analysis software will do these calculations (and can use even more involved filtering) and will produce data tables (with %s and/or numbers) for you to use. A simple example would be: Q1: Satisfaction with the xx recycling facility opening hours TOTAL Gender Age Male (7) Female (3) 18-25 (3) 26-35 (3) 36-45 (1) 46-55 (3) Very satisfied 4 3 1 3 0 0 1 Quite satisfied 4 2 2 0 1 1 2 Not very satisfied 0 0 0 0 0 0 0 Not at all satisfied 2 2 0 0 2 0 0 Don’t know 0 0 0 0 0 0 0 REMEMBER: Be clear about your objectives The starting point for writing effective questions is to identify your objectives clearly and specifically; what information is needed and for what purpose. This starting point informs the question composition and format. What exactly do you need to know? Why? How will you use the information? Think about the feedback you need There might be additional considerations that you wish to take into account in terms of sub- groups in your population. For example, you may consider it important to place greater weight on the preferences of those living furthest from recycling points or you may wish to take particular account of the preferences of households with children. This means you need to think about including questions that will allow you to analyse responses by key sub- groups; in this example that might mean adding a question about proximity to the nearest recycling point or a question about the presence, number and age(s) of children in the house. Linked to this, think carefully about who you should consult on any topic: is the subject matter relevant to the whole group of people or should routing be used to ensure people only answer questions relevant to them? For example, is the subject matter only relevant to a specific demographic such as parents with children at a primary school? Set Context and Limitations Once you are clear about your objectives and your preferred outputs you will be in a better position to frame appropriate questions. It is important that you also consider the information that needs to be provided to enable people to respond honestly and constructively. Issues you should take into account include:
Page 10
10 Limitations facing you; these need to be identified wherever possible to ensure that expectations are not raised artificially eg don’t ask if people would like something you can’t deliver. How will you use the feedback and how will you let respondents know about the decisions made? This is important to ensure that both the quality and quantity of data you receive are maintained in the longer term. The complexity of the subject matter; the complexity of the issues on which you are seeking opinions can affect the way that questions are framed and it may be helpful to break down a subject into individual issues so that a question does not attempt to collect information on more than one issue or part of an issue. The complexity of the subject matter should not be reflected in a complex question as this can lead to reduced response rates or unreliable data. Using section headings and providing background information in small, easy assimilated chunks will yield better responses to the subsequent questions. Technical or Specialist Language relating to the topic: if you cannot avoid jargon, be sure to provide a simple explanation of each term as it is introduced. Framing Your Questions All questions should be clear and unambiguous. Avoid leading questions. Make sure only one question is being asked. Types of Question At the simplest level you need to consider whether questions should be closed questions or open questions. Closed questions typically deal with facts, are easy to answer and quick to answer e.g. “have you had contact with the planning department in the last 12 months?” or “are your bins collected on a Wednesday or a Thursday?” They are easy and quick to answer because you can provide clear, unambiguous choices from which respondents can select their answer. They are also much simpler to analyse and will provide clear quantitative data measures e.g. 65% have had contact with the planning department in the last year. Closed questions also include more than simple “yes” or “no” options and you may find some of the following question types useful within the survey mix: Likert scale: where responses are provided using a linear scale, for example in relation to agreement, frequency or importance. This type of question provides more in-depth data than a simple Yes/No question and generally has four points (for example Agree Strongly, Agree, Disagree and Disagree strongly). Semantic differential: where responses are provided using a value plotted between two extremes. This type of question allows for a more expressive response and can be used to identify how Panel members feel about a certain issue. Net Promoter Scale: again this plots a value on a scale but rather than this being between two extremes such as excellent or terrible, this uses a numeric scale generally
Page 11
11 from zero to ten (where, for example, zero is Never and ten is Always). This has the benefit over the semantic scale of allowing for more subtle responses than a five point scale. Open questions such as “how did you feel about your contact with the planning department?” hand control to the respondent and can generate long and detailed responses. They are much more resource intensive to analyse but when used appropriately can deliver a great depth of insight into feelings or the reasons for underlying opinions. Remember that every response to an open question has to be read in full and all elements of the answer need to be incorporated into some form of analysis matrix before you can arrive at a clear and balanced view of opinions being expressed. Further, entirely free text can be ambiguous, hard to understand or open to interpretation. If undertaking a survey or consultation that is likely to generate very large numbers of responses you may find it helpful to offer a list of commonly expected responses to an open question from which respondents can select any that are relevant. It is important to also include an option for “something else” or “else” where respondents can then add further views they feel are not covered in the predetermined list of options. It is likely that you will wish to use a mixture of closed and open questions in most surveys or consultations. If a survey relies entirely on closed questions it is usually good practice to add a single open question at the end of the survey to allow respondents to make any additional comments that are important to them and that they may feel have not been fully addressed by their earlier responses. It is often helpful to frame the question in such a way as to keep the focus on the issues under consideration and your objectives. For example, “please use this space to detail any further suggestions as to how we might improve the pattern of bin collections?”