Chapter 3 Developing Objectives

Step by step guide to developing a statement of objectives

Important note: the word question in this guide never refers to the questions that respondents are to answer. It refers to:

the research questions the survey is to answer
policy questions that the survey information is to be used to address, or
questions about the survey that need to be addressed.

3.1 An overview

This section covers all the steps very briefly. Each step is explained in more detail in the following pages.

Before you start. Gather existing information, document the problem and how you view it, ask for advice, and make sure that you need the survey.
Where to start. Document the overall research question tou want to answer with this survey, and what the information from the survey will be used for.
Who or what you want to ask your research questions about. Specify your ideal population.
Periodicity. Say whether you need the information at one point in time, or information at a number of points in time, or continuously over a period.
The important quantitative research questions. Document a very few specific research questions that you want the survey to answer, define the terms used, and state uses of the information.
Key outputs. Specify the outputs you must produce, and the accuracy with which you want to provide them. This specification is the basis for sample design.
Other outputs. Specify any outputs you want to produce. These must be justified by the uses the information will be put to.
List of variables. Specify all the variables needed to produce the outputs you have specified. Give each a priority rating. Make sure all the terms are defined. Make sure it is clear what all the information is to be used for.
Keep checking design work and be prepared to revisit your objectives. As the design work progresses, you need to check it. The development team will give you progress reports. It may turn out that some of your objectives cannot be met. If so, you need to decide whether and how to progress.

3.2 Before you start

Are you sure that you need a survey? Does the information you need already exist? Are there other ways of finding out what you need to know?
Find people to help you.
Do you have a background paper on the subject of the survey? Does it cover all of the following points?
- what is already known on the subject
- how you are thinking of the problem - your model or way of seeing it
- why a survey is needed - other sources such as administrative data, or other surveys are not available or appropriate
- any information on the incidence of the thing your survey is going to be looking at.
If you have such a paper prepared, include it as an introduction to your objectives statement: it is very useful to the designers. If you don’t, consider writing one. It will help you establish the need for the survey, and it will help you sort out your ideas on the data you need from the survey.
Do you have a group of users who have needs for this information? Do you have an external client or clients whose information needs you have to reflect?

You need to develop a way of working with the various parties whose ideas have to be reflected in the set of objectives. It will be easier if all parties understand what the set of objectives is for and what it has to do. You could use these guidelines to help you all reach a shared understanding of what has to be done and why.

3.3 Where to start

What is the overall research question you want to answer with this survey?

This part is allowed to be a little woolly, for example:
What problems are there with housing in New Zealand? This information is needed so that the government can plan policy to address the problems.
What will the information from the survey be used for?

It’s worth spending quite a lot of time on this before going any further. It serves two main purposes: it justifies the survey itself, and it helps you work out what information is really needed.

The sort of statement you should have at the end of this phase is:
We need this information we will get from this survey in order to:
- decide where to target funding for \(X\)
- decide whether to change our policy on \(Y\)
- see whether policy has brought about the change it was meant to
The more specific and concrete you can be, the easier it will be to work out your information needs, and ensure that you have them right.
You may only be able to talk about the information being used to ‘inform debate’ but you must be specific about what the debate is about.

e.g. don’t say
informing debate on provision of housing assistance
but rather
informing debate on how housing assistance ought to be delivered to households, in particular whether …

The statement ‘there is a lack of information on …’ is not useful unless you can add ‘and it is needed in order to …’, so you might as well leave out the first bit.

3.4 State who it is you want to ask your research questions about

This is called defining your target population.

This is the ideal population about whom you want to be able to draw inferences. It may be that later it will turn out not to be practicable to survey exactly this population. For example, you may have to decide later whether it is economic to include the off-shore islands, or people in non-private dwellings. But for now the important issue is: who is it that you need to answer your research question about?

First, say whether the research question is about households, or people, or businesses or dwellings, or something else altogether. In the housing example mentioned above there could be two populations – one of people and one of dwellings.

People. Except for the five-yearly census, you probably won’t want to know about all the people who are in NZ. In the housing example, tourists, people in prison or hospital and people in army camps would probably not be included.

You might define your population as:
People living permanently in private dwellings in NZ

Note that you still have to define what you mean by all those words. Or you might want to say:
People living long-term in NZ but in institutional housing of any kind

You then have to make clear, by definition or listing, what you mean by institutional housing, plus define ‘long-term.’
Dwellings. In the housing example, you probably wouldn’t want to know about all dwellings. For example, non-private dwellings and unoccupied dwellings may not be interest.

Occupied private dwellings in NZ
Again, note that you will still have to define what you mean by all those words.
Other. In some surveys the population will be neither people nor dwellings. It might be a population of businesses, of schools, or of something else. In any case, you need to say what is to be included, and what you mean by your terms.

3.5 Periodicity

Do you need information at one point in time, or information at a number of points in time, or continuously over a period?

The answer to this should arise from your research question. The examples suggested above would require information at one point in time. But a question such as:

What is the relationship between the quality of NZ housing and the extent of government involvement in housing provision?

would require information at a number of points, possibly annual information.

Another research question might require a longitudinal survey. For example,

How much does the quality of housing vary over the lifetime of an individual?

3.6 What are the important quantitative research questions that you want the survey to answer?

This question or questions should be based on statements you produced above. More than one question may be needed, but you should not have a long list - you haven’t got to the detailed information needs yet.

Each question should have a statement about the intended use of the information. For example:

How many people in NZ live in housing of a poor standard?
Needed because if the number of people living in poor dwellings is greater than X, the government will have to increase the budget allocation to housing. (You are going to have to say how big X is and how accurately you need to estimate it. Survey designers will need that in order to design the sample.)
How does the number of people living in housing of poor standard change as the housing sector is progressively privatised?
Needed to decide whether the effect of privatisation is positive or negative and thus to decide whether to continue with the policy.
How many dwellings are of a poor standard?
Needed to calculate the amount of money needed to reduce the number to an acceptable level. (You are going to have to decide what is an acceptable level.)

Clarification and specification: clarify each of your research questions by:

putting each important word into statements like:
What I mean by people living dwellings is …
It includes people who have been in a household for more than 3 nights a week for the past 12 weeks, except for any who have another place of residence to which they will return in the next …
expanding it with some statements like
What will count as poor housing is the existence of one or more of: crowding, leaks, dampness, lack of inside plumbing.

In the second case you still need to define the terms, e.g.

Crowding is to be measured by dividing the number of people by the number of rooms, and any dwelling with a measure greater than 1.3 is to be counted as crowded.
Rooms to be counted in this measure include … but exclude ….
People living there is to include people who … but exclude
people who are ….

3.7 Key outputs

Now you need to think about the outputs from your survey that you see as absolutely essential, and how accurate you need those outputs to be. You will probably plan a larger number of outputs, and may think of some after the survey has been run, but you must have the absolutely vital ones sorted out well in advance so that the sample can be designed to ensure that those outputs are produced at the level of accuracy you desire.

The best way to describe your key outputs is to produce skeletons of tables - the title and the labels of the axes. Attached to each should be a statement about the use of the information and the accuracy desired.

Note that if there is a standard published by Statistics New Zealand for any variable in your output, you should consider using it unless there is a good reason not to.

	Key output 1
Area of the country	Proportion of dwellings that are poor
North of the North Island
South of the North Island
South Island
Total

Definitions: North of North Island – area north of a line from Y to Z. South of North Island – area south of that line.
Poor dwellings are defined above.
This output is important so that housing assistance can be targeted to areas according to need
Cells are to have a sampling error of plus or minus 5%. For instance, if the estimate of a proportion of dwellings in the North Island that are poor is 40%, this is 40 \(\pm\) 5%.

	Key output 2
Ethnicity	Proportion of population living in crowded dwellings
NZ Māori
European/Pākehā or Other
Total

Ethnicity - use NZS standard definition
This output is important so that housing assistance can if appropriate be channelled through Māori organisations
Cells are to have a sampling error of plus or minus 5%. For instance, if the estimate of a proportion of population living in crowded dwellings is 40%, this is 40 \(\pm\) 5%.

If something more complex than tabular output is being planned as a key output, talk to whoever is working on the sampling about what they need from you in this case.

3.8 Other outputs

These are outputs that will be useful to you, but that you can if necessary get along without, if, for instance, having them would double the cost of the survey because:

you’d need a much larger sample to enable you to have them, or
you’d need a different, more expensive, interviewing method to collect the necessary information.

At some stage the survey designers will talk to you about what level of accuracy you could get for these outputs from a sample designed to provide your key outputs. At that stage, you will need to make decisions about whether to keep or drop those proposed outputs.

Again, the best way to show the outputs you want to produce is to do skeletons of tables. The other way is to write the statements you would, or will, put into a report after the survey, leaving out the numbers. For example,

: It was found that …% of parents of children aged under 5 lived in crowded dwellings, while of parents with a youngest child aged 5 or older only …% live in crowded dwellings.

Again, you should say why it is useful to know this.

: This is needed to decide whether it would be useful to target housing assistance by presence of children aged under 5, perhaps through the present family assistance provided through IRD.

Again, you should remember that if there is a standard published by Statistics New Zealand for any variable in your output, you should consider using it unless there is a good reason not to.

3.9 List of variables

At this stage you should be able to put together a list of the variables or bits of information that you need. This is very useful to designers.

It is worth putting a priority rating for each variable at this stage. Then if some variables need to be dropped because of cost or time constraints there will be no delay while decisions are made about which to drop.

Information needed	Population the information needed about	Definition	Output Categories	Priority
Crowding	all dwellings in pop. (see definition of population)	see point 2 above	Need number for each dwelling for producing medians, quartiles, etc	A
Housing quality	ditto	see X, Y	index to be developed (see X, Y)	A
Persistent leaks	ditto	leaks whenever there is rain	two-value variable - has or does not have - contributes to housing quality index	A
Lack of inside plumbing	ditto	No running water or drainage inside main structure	ditto	A
Security of Tenure	ditto	see definitions	two-value variable: secure/not secure - see X, Y	B
Ethnic group	ditto	as in SNZ standard	Māori/other	A
Area of country	ditto	NZ divided as shown in point 2	see point 2	B
Age	ditto	as in SNZ standard	10 year age-groups as in SNZ standard, or further collapsed as necessary	B
Children under 5	ditto	whether respondent is in parental role to a child less than 5	dichotomous variable - is/is not in such a parent role	C

3.10 Keep checking

Keep checking design work done for you against objectives, and be prepared to revisit your objectives.

If the questions being developed do not seem to you to match your objectives, point this out to the designers and ask them to find remedies, rather than trying to find remedies yourself.

Similarly if there are any features of the proposed methodology that make it unlikely, in your opinion, that the survey will be able to meet your objectives, point that out to the designers.

It is quite likely that some of the information you think you need will not be able to be collected, or will not be able to be collected given your budget. So you may need to look at the objectives again and decide whether the survey is still justified, given the reduced list of objectives, and consequently the reduced set of uses of the information.