How social landlords are approaching tenant satisfaction measures

In July, Inside Housing sent a survey to around 300 social landlords. More than 200 – 116 housing associations and 87 councils – shared their planned approach to the TSMs. At the time our survey was filled out – between July and August – 145 of the 203 landlords had started collecting the TSM data. 

For starters, we wanted to find out if the way landlords are approaching TSMs differs in any way from their previous methods of measuring satisfaction. We found that in a number of cases, the methods of survey have changed (see two charts, below). Just under half used only one survey method before the TSMs, the majority of which was phone surveys – at 63%.

This is an even more popular collection method for TSMs. Phone use has gone up 23 percentage points for councils, and 19 percentage points for housing associations. Overall, 55% of landlords are using different methods for collection to the ones they were using previously.

The nature of TSMs themselves, however, may partly explain this shift. This is because there are two main types of survey landlords typically carry out: transactional, which involves the landlord carrying out a survey only after it interacts with a tenant, and perception surveys, involving a completely random sample of tenants or a full census.

Previous research by data firm Housemark found that before the final TSMs were published, the majority of large housing associations had been using transactional surveys, which yield more positive results – on average 10 percentage points higher than perception surveys.

The main reason for this is how the sample is selected – for transactional surveys, the sample is made up only of residents who have recently received a service, which is generally a positive outcome.

Perception surveys are a random sample of all tenants, and will include tenants who have not had recent experience of the landlord’s service, as well as residents who have requested a service but are still waiting, in some cases a long time, for it to be completed.

Before 2010, it was a regulatory requirement to carry out a perception survey, which was then called STATUS.

The requirement was dropped following the closure of the Tenant Services Authority, the former regulator that was scrapped in 2010 in the so-called ‘bonfire of the quangos’.

Housemark then stepped in with a voluntary framework called STAR. Jonathan Cox, director of data at the firm, which will shortly be carrying out mid-year research on how landlords are faring with TSM responses, says that in 2010, over 80% of English landlords regularly carried out a STAR survey. But by 2018 this had fallen to around 60%.

He explains: “Larger housing associations in particular were more likely to cease doing perception surveys entirely and focus only on transactional surveys.

“By 2019, only 40% of housing associations with over 10,000 units were still regularly carrying out perception surveys. Many still reported an overall satisfaction score to boards, but this was based on an aggregate of transactional satisfaction survey scores – which invariably gave higher results than would be achieved with a perception survey.

“For example, in the G15 annual reports for 2019-20, all quoted an overall satisfaction figure, but only five out of the then 12 used a perception figure.”

Changing approach

Bearing this context in mind, just under half of housing associations are changing their approach for the TSMs. For councils, the difference is more dramatic, with 66% using different survey methods.

Mr Cox says: “While online/SMS surveys remain highly used and effective for transactional surveys, most landlords are avoiding relying heavily on them for perception surveys due to the negative survey bias.

“Online survey results, even when weighted for representativeness, are usually between 10 and 15 percentage points lower than telephone survey results.

“Furthermore, the negative survey bias is generally heavier for larger urban landlords that already have a more challenging context. The survey bias is around ‘self-selection’.

“An unhappy tenant is much more likely to complete an online survey when invited to do so than a tenant whose perception is positive or broadly neutral.”

The TSMs must be collected on an annual basis within the financial year, for landlords with 1,000 or more homes. Smaller landlords can choose to collect and report every two years.

Overall for 2023-24, the most popular option is tracker or rolling surveys. Places for People is conducting a continuous survey over a specific part of the year: August 2023 until January 2024, while L&Q is surveying over four months, initially June, July, September and October, although it may go into November due to technical issues.

Of the landlords using tracker surveys, almost half (46%) are surveying monthly, 37.5% quarterly and 7% weekly.

Of the landlords using single or biannual surveys, 64 specified which months they would be surveying tenants. September was the most popular at 47%, followed by October at 39%. The least popular month was April, chosen by two landlords, as was December.

Mr Cox says: “For annual surveys, landlords tend to avoid the gloomier winter months, which typically produce a slight negative survey bias, as well as the summer months when many people are away.”

Mr Cox says there are benefits to both tracker and one-off surveys, with tracker surveys providing regular figures for executives and boards, enabling landlords to act on the feedback more quickly and make in-year improvements to improve scores,” he says.

But they can present logistical challenges in terms of ensuring representativeness, require greater integration with systems and processes, and sufficient resource in-house to analyse and act on the results as they come in.

“For landlords without this level of resource, an annual survey may be a better option, at least for the short-to-medium term. With good planning, annual surveys can also lend themselves to good resident communications campaigns and can be planned to avoid seasonal survey biases,” he says.

Landlords can use internal or external staff, or both, to survey tenants. The majority (70%) say they are using external. A proportion of 16% are using internal staff and 13% are using both. L&Q, for example, is using both: an external team for surveying and an internal team for analysing the data.

Councils were slightly more likely to use an external team (75%) compared to housing associations (67%).

Of the landlords using tracker surveys, almost half (46%) are surveying monthly, 37.5% quarterly and 7% weekly.

Of the landlords using single or biannual surveys, 64 specified which months they would be surveying tenants. September was the most popular at 47%, followed by October at 39%. The least popular month was April, chosen by two landlords, as was December.

Mr Cox says: “For annual surveys, landlords tend to avoid the gloomier winter months, which typically produce a slight negative survey bias, as well as the summer months when many people are away.”

Mr Cox says there are benefits to both tracker and one-off surveys, with tracker surveys providing regular figures for executives and boards, enabling landlords to act on the feedback more quickly and make in-year improvements to improve scores,” he says.

But they can present logistical challenges in terms of ensuring representativeness, require greater integration with systems and processes, and sufficient resource in-house to analyse and act on the results as they come in.

We also asked landlords which characteristics they are taking into account in their survey samples to ensure they are representative of their tenant population.

The vast majority (182) are taking into account more than one characteristic, while five are not targeting any specifically, four are targeting one, and the remaining have yet to decide.

Overall, 136 are taking into account ethnicity, 174 age, 148 region/area, 135 property type, 70 household size, 25 tenure, 11 gender, three ‘needs category’, four length of tenancy and one sexuality.

On average, landlords are surveying 33.8% of low-cost rental accommodation tenants – 39 landlords are planning to survey 100% of tenants.

The regulator does not specify how much of the tenant population must be surveyed, but says providers must, as far as possible, generate a sample size that meets the minimum level of statistical accuracy. It also provided an illustration of what sample sizes, based on completed responses, would be needed to achieve this.

A landlord with 100,000 tenants is required to achieve 2,345 responses, a 50,000-home landlord should achieve 2,291 responses, while a 15,000-home landlord must achieve 997.

At the other end, a 100-home landlord must get 80 responses, a 1,000-home landlord 278, and a 5,000-home landlord 536.  

Landlords are raising some concerns about the TSMs. All providers must ask exactly the same set of questions to tenants – the regulator says this is to ensure the results between them are comparable.

But specialist housing providers, which have many tenants with complex needs, say the questions are more geared towards general needs.

Sophie Hayward, head of housing at Westmoreland Supported Housing Association, says: “Around 50% of our tenants either lack capacity, so wouldn’t understand what they need to do to answer these questions, or are non-verbal. The obvious one would be to speak to the care providers, but you could argue then that it’s a biased view – it might be their view rather than the tenant’s view.”

203
Responses to survey

145
Landlords that had started collecting TSMs data

55%
Landlords that are using rolling/tracker surveys

Ms Hayward says that in future, “it might be good to consider the fact that there are quite a lot of different providers providing housing for all kinds of people”.

But she “completely understands” why the TSMs have been introduced. “We support the need for improved customer engagement and knowledge, and the TSMs, however not perfect, give us the opportunity to gather information and further improve.”

Choosing a method

There is also a concern around ‘gaming’ the results by using certain methods which produce higher satisfaction scores, such as phone and face-to-face, although the requirements mean landlords have to justify why they use those methods, such as barriers to participation.

Tim Quinlan, senior insight manager at Riverside Housing Group, says it “could be argued that changing survey collection methods to higher scoring methods is one way to do this”.

“What we are seeing here is a shift from traditionally lower scoring collection methods (email and post) to telephone, which traditionally elicits higher satisfaction scores. This collection method change will very likely result in higher satisfaction levels being reported across the sector compared to the previous year.

“Our view on collection methods is that we would have preferred the regulator to have specified a set collection method. Without this there is no way tenants will be able to fairly compare the performance of their landlord against other landlords.”

In May, Kate Dodsworth, chief of regulatory engagement at the Regulator of Social Housing, warned that the regulator may become “more prescriptive” if people try to game the surveys.

Kate Roberts, principal consultant at Altair, says landlords using a single method of surveying, such as phone contact, should think carefully about whether it excludes some people, such as those where English isn’t their first language.

She adds: “Registered providers should remember that the results of the TSM survey are only part of the equation here and it’s not about whether your satisfaction scores put you at the top of a league table or not. It is about how you demonstrate you are listening and acting on the views of your residents.”

In this webinar, a sector expert will join a senior member of the Inside Housing editorial team. Together they will discuss how organisations can best collect, report and act on tenant satisfaction measures, drawing on research and real-life insight. 

The event is CPD certified. To sign up and find out more, click here.

A brand new housing sector conference shining a light on changing regulation and best practice governance – from financial and ethical governance, to ensuring a fair and good service for tenants.

Bringing together 250 UK housing governance professionals in one setting for the first time, this event is an unmissable opportunity to kick-start critical discussions around regulatory policy, tenant satisfaction, accountability, transparency and financial risk management.

Find out more and book your delegate pass