Watch On:
Summary
User feedback is one of the most direct ways that the Government Digital Service learns about user experience. These responses can dilute our insights, cause security concerns, and prevent real problems from being identified. There was also a security risk to consider, as individuals could attempt to negatively exploit feedback mechanisms to disrupt the usual workings of GOV.UK. With ML, we could use the probability score to demand a high level of confidence in our model’s predictions, reducing the mislabelling of legitimate feedback. Once we had made the decision, we focused on the tools and techniques that would help deliver a working solution as soon as possible. We then ranked individual feature importance to assess the features used in the model’s classification judgements, to understand which derived features had the most impact on the outcome of the model’s predictions. Tools such as PyCaret and DVC meant that we were able to focus on deploying a working solution at pace. We can run it on over a month’s worth of feedback data –around 40,000 responses– in less than five minutes, a fraction of the time it takes human reviewers. Now, a careful iteration process is important for combating spammers that adjust terminology to outwit filters and cause “model drift”.
Show Notes
Unfortunately, this also creates a lot of avenues for spam responses.
The problem with GOV.UK feedback spamAt GOV.UK, we received around 540,000 feedback responses from the public and other departments in 2021.
In early 2022, we saw spam responses surge due to a technical change on the front end, peaking at 12% of total feedback.
Why we used machine learningWe quickly recognised that colleagues needed to derive their insights quickly, without needing to manually filter out spam.
We saw the challenge as a great application for a machine learning (ML) model.
