Five easy ways to get your policy guidelines delivering higher moderation accuracy

Making your policy deliver accurate results in practice can be challenging. Add more details and exceptions to your guidelines, and you end up with an unwieldy tome that’s hard to learn. Keep it brief and you can miss vital context that drives accuracy. Either way, you can end up relying on a few key moderators or team leads holding everything in their heads - and this becomes risky and hard to scale.

At Unitary, we are contractually accountable for 95%+ accuracy SLAs, and we report on these every week. This means we’ve experimented with lots of different ways to deliver highly accurate moderation day in, day out.

Accurate moderation requires lots of things, including the right feedback loops; low overhead communication; great tools; and proper training and support systems. And every platform is different.

But we’re sharing here five tactics for your policy guidelines that are easy to test, likely to boost accuracy and fast to implement.

‍

Wait! Before we start - do moderators even use policy guidelines after they’re onboarded?

Do moderators even use policy guidelines once they’re trained? Is it still worth doing if all my moderators are onboarded?

In our experience, if policy guidelines are long and difficult to navigate, experienced moderators won’t refer back to them very often, but this can lead to drift and consequently lower accuracy over time.

We find that concise and helpful policy guidance isn’t just important for training; it helps experienced moderators to work accurately and even more quickly for complex moderation tasks.

Even if only 5% of an experienced moderator’s cases require going back into the guidelines, optimising these cases for speed and accuracy will make a significant impact on your overall metrics.

‍

1) Build a moderation flowchart with Typeform or GPT

Convert your moderation task into a flowchart of questions that guides moderators to the correct classification

Example policy flowchart with Typeform

This is great for refining and optimising your policy guidelines. Having to structure your policy guidelines as a sequential list of questions, even if only for a one-off exercise, can help you to identify gaps, grey areas and overlaps. Running examples through the flowchart further enables this refinement process, and allows you to quickly adjust, test and adjust again.

Developing the flowchart also forces you to think through the right hierarchy for your categories - i.e. which classes or denial reasons should take precedence when multiple apply to a case. Having the right hierarchy helps with accuracy, speed and usefulness of labelled data (more on this below). In our experience, the optimal hierarchy is not always obvious at first - the flowchart allows you to test it with real cases.

Here are two ways to do this:

1. You can do this for free and very quickly using Typeform, combining the Outcome Quiz and Logic Branching Features. You use the Outcome Quiz to lead the moderator to different decisions, e.g. “you should deny this case and add the category Y”. You use the logic branching to show different routes depending on question answers. Here’s a handy guide
2. If you have a way of hosting website code, you can ask Chat-GPT to create the code for you for an interactive decision tree with outcomes. This allows you to create a slightly faster user experience, i.e with hotkeys and no submit button. (Reach out to us if you want some help with this!)

‍

‍2) Write your policy guidelines as sequential steps

Restructure your policy guidelines from a taxonomy index into a set of sequential steps

Many policy guidelines are written as a taxonomy: a list of categories that might apply with guidance for each. Sometimes we have found that this forces the moderator to spend a lot of time scrolling up and down between categories - especially when multiple categories can apply and some categories take precedence.

More fundamentally, classifying against a taxonomy can take a lot of mental effort. The moderator might need to assess the content against each category, and come up with their own list of questions to correctly account for exceptions.

We have found that turning policy guidelines into a set of sequential instructions or questions - ideally starting from the easiest to identify or rule out - can significantly reduce this mental effort and free up moderators to focus on the content rather than recalling or interpreting the guidelines.

Just like the flowchart approach above, it also helps you to clarify your category hierarchies and identify gaps and overlaps.

‍

3) Get your policy to fit on one screen

Say goodbye to scrolling or clicking

We’ve observed moderators navigating different pages, opening toggles or scrolling up and down in documents. This can often lead to inaccuracies through moderators missing or forgetting parts of the guidelines.

In many cases the policy guidelines can be condensed into a single screen. Even if you can’t get your whole policy onto one screen, you can still provide a one-screen version that covers the majority of cases, with more detail available in a linked page or separate tab for the cases that benefit from extra detail.

We've made a template you can use to get started: click here to view it and make a copy

‍

4) Hackathon your policy guidelines with product and engineering colleagues

Organise a short hackathon to generate out-the-box ideas for how to moderate

We’ve had our best results when we’ve had collaboration with colleagues outside of policy and operations - or even outside of trust and safety. Unencumbered by “how guidelines are normally done”, they bring first principles thinking and fresh perspectives.

An easy way to do this is a short “hackathon” session:

• 15 minutes to introduce everyone to the problem statement, objectives and any constraints, e.g. we want to maximise accuracy and minimise onboarding time. Speed is important but not more so than accuracy.
• 35 minutes in groups to design a totally new approach to moderating content. Encourage teams to be wacky, give points for most ‘out there’ idea
• 10 minutes to present back

Even if you don’t implement any of the solutions presented, you may find they give you inspiration for small tweaks to your current process.

‍

5) Get everyone, yes everyone, to do some real moderation

Crowdsource QA of your guidelines and processes with a broad group of people

Ask everyone you can think of to onboard as moderators and spend 1 hour moderating real content (curating the batch to exclude extremely distressing content). They will have tons of feedback on the policy guidelines and process. We find that this kind of stress-testing - with a high volume of real cases  - is essential for evolving guidelines to ensure they deliver accurate moderation.

A broad audience is best for this as you can make sure the guidelines accommodate a range of learning styles and experience levels.

If you want to maximise the different types of edge cases you test, get everyone to moderate different content batches. If you want to measure whether the guidelines are interpreted consistently, get everyone to ‘cross-label’ the same batch.

At Unitary we try to make this process fun - we do it on a shared google meet call on a Friday afternoon. We ask everyone to share feedback in a form afterwards.

We hope you find these tips to get your policy guidelines working for you useful. We’d love to hear your thoughts and feedback - you can email us at support@unitary.ai or reach out on LinkedIn.

‍

6) Bonus tip

Optimise your moderation accuracy with help from the Unitary team

At Unitary, we build blended teams of AI agents and expert human moderators that replace your BPO and deliver better accuracy, faster responses and lower, more scaleable costs.

Optimising policy guidelines is just one part of the what we do to deliver better, more efficient moderation.  Get in touch if you want to learn more about how this might work for you!

‍