One of the main features Player XP provides is the ability to track change over time and one of the best ways to utilise this is in a post-mortem.
For this article, we're going to provide a step-by-step guide on how Player XP can be used to run a review of a recent update. Hopefully, you'll find some value in seeing how different features can be brought together to build a robust report for your team.
First of all, we're going to set a scene and imagine ourselves as part of the team that recently launched a new season pass within a live game. The data shown below is from a real game but we've chosen to remove all references to the game itself.
Building Your KPIs:
Hopefully, by this stage, you will have already worked with Player XP enough to understand what the overall KPIs are for your game within the dashboard. However, for the benefit of this article, we're going to use the following as a measure of improvement:
- Overall Sentiment- Did we see any notable spikes or drops in our overall sentiment score?
- The volume of negative comments- Did we see a major difference in negative comments from our last season's pass launch?
- Sentiment towards our "Updates" category- Did we see a positive increase in sentiment within our category dedicated to Updates?
- Sentiment towards the Topic "Season Pass"- When looking specifically at the reaction to the new Season Pass, did we see an upwards trend?
- New community members reached- Did we see a spike in the number of community members talking about the launch of the new Season Pass?
- Server Stability*- In this fictitious update scenario, one of the long-term problems for the game is the stability of the servers around the launch of new updates so we're going to track the volume, sentiment and response towards Server Stability for this update.
* Analyst Note - This is often the most overlooked factor of using Player XP to contribute to your Post-Mortem. How has the recent event changed our long-term priorities? Was there a community reaction that needs to be reviewed against long-term problems and improvement areas?
We're now going to step through the various parts of the dashboard that provide reporting on each of the above, some of which you may be aware of and some that may be new to you.
Checking Sentiment Change:
For the benefit of this article, we're going to set out that our new Season Pass launched on the 1st of November 2022 and it's now been out for 14 days.
To review sentiment change, the best place to start is to understand how your sentiment has been performing recently. Hopefully, you'll have a good idea of this based on regular reviews and the weekly emails we send, but for the benefit of the doubt, here's our game's average sentiment vs two of our closest competitors for the last 3 months...
As you can see, whilst our sentiment overall has been more negative than our 1st competitor but more positive than our 2nd competitor. Over the last 3 months, our sentiment dropped which could indicate a negative reaction from players towards our new season.
Looking closer at the period around the release, we can see that sentiment dropped on the day of the release of our new season and has remained consistently negative apart from on the 4th of November when sentiment increased to 7.
Considering the overall negative sentiment the game has received since the new season, it is fair to say the launch has not been the most successful. Further monitoring will be needed to help identify the potential reasons for a negative reaction to the launch.
Negative Comments:
As outlined in the KPIs above, one thing we wanted to avoid was a spike in the number of negative comments coming through, though currently, we can see that there has been a more negative response from players since the launch of our new season.
When we look at the split of positive vs negative comments across the release window, we can see if the community had any major negative spikes. Tracking this around the launch can help prepare your community team so they can anticipate the level of response needed.
When looking at the weekly volumes around the launch, we can see that there wasn't a major negative spike in the week that contains the launch, but a slight increase in negative interactions.
Changing this to a per-day interval, we can see that since the launch the average % of negative interactions has slightly increased compared to pre-launch.
This indicates that the community have had an issue with the new season since its launch due to the increase in negative interactions starting from the launch date of the new season. However, we can also see the majority of the discussion around our game is largely neutral.
One of the interesting things Player XP can highlight is how provocative your game is, we know there's nothing worse than a neutral comment that doesn't contain feedback but most communities do find themselves having a mostly neutral conversation. When updates happen, we look to monitor the positive/negative comment % increase to see if the community has reacted passionately one way or another.
Tracking the Updates Category:
For our next KPI, tracking of specific feedback towards updates, we will look to use Player XP's Most Discussed Categories feature that looks to report on the volume and sentiment of the categories that group various discussions together.
The first insight we found was that Updates was the 2nd most discussed category around the release of the new update, the fact that Online Systems is the third most discussed category doesn't bode well for the KPI related to server stability but we'll come onto that later.
Coming back to Updates, we do see that sentiment for the category is negative. Once again, we can see the sentiment drops the week of the new update and has remained negative.
Another thing to consider is when we look at the volume of the Updates category (by switching the toggle) we can see there have been fewer interactions in this category since the update, meaning there is less discussion about this but the discussion that does take place remains mostly negative.
Specific Feedback Towards the Season Pass:
Thanks to the Topic Finder, our search engine, where you can go in and easily find the feedback related to a specific topic, and as seen above, we've gone in and set up a custom query for "season" to identify the sentiment towards a very specific element inside the game.
Given we're looking at a major element in the game, our KPI here should be looking across the last 12 months of the game to see how the most recent pass has performed from a sentiment perspective. Unlike the other KPIs, if we looked at how the community reacted around the launch, we wouldn't have any comparison and any spike in the volume and sentiment would be expected with such a release.
Looking at the last 12 months, we can confidently see that the latest season pass has been received more negatively than the previous three passes (released in February, May & August 2022).
If we wanted, we can then go down to the Talking Points feature to look into the exact conversation taking place and dig into each talking point to see what people specifically like and dislike about the recent season pass.
Did We See The Community Increase?
Across the Player XP dashboard, we try to relate three key metrics: sentiment, number of interactions and community size. For this KPI, we're going to focus on the latter, the community size.
With any major update, the hope is to bring new players to the game and subsequently into the community. As Player XP tracks this metric, it's extremely easy to see if the volume in the community has increased.
Looking at the last 3 months, we can see the month containing the update saw a decrease of over 3000 members in Community Size.
When we zoom out one step and look at the last 12 months, we can see that the community was at its lowest in November 2022. Given the focus on the new Season Pass, there's a good chance that the new content forced players away from the game.
Tracking Long-Term Priorities:
So far, this update seems to have gone down poorly across our KPIs. Also with this game, we know they have an issue with their Online Servers.
To help quantify if the issue remains to be a problem and if the latest update saw more complaints being left, we're going to use Player XP to report on a few key areas.
The first is to look at the "Online Systems" category, this category automatically collects feedback related to the Server and if there was a problem, we should see a spike in volume and a drop in sentiment around the launch of the new update. Furthermore, we should see issues with the Server being one of the most discussed Topics inside the Category.
As we say, the first step is to check the volume of conversation being picked up by the Online Systems category. As seen above, the % of the volume being picked up by this category has increased over time, potentially suggesting that this is still an issue even after the recent changes.
This is also mirrored by a very negative sentiment which suggests this has been an issue in the game for some time. It's also important to note that inside the Online Systems category, the most discussed Topic is 'Servers'.
This is reflected when we look at the Online Systems category over the last month and see that Servers and Matchmaking are the two most discussed problem areas within the category. The sentiment for both topics is also extremely negative. We can see more by exploring the Servers topic.
We get some interesting insight into this Topic from the feature above. The green line shows how the overall sentiment would be affected if the Topic we're reviewing ("Servers" inside the Online Systems category) was removed from Player XP - for example, if the issue was fixed. We can see the sentiment of the category with or without this topic would still be very negative.
If we look over the last 12 months and then change the displayed interval to yearly, we can see that the topic has always been extremely negative and in fact has continued to drop in sentiment even further in 2022. This indicates that the issue has become more of a problem as we move through the year.
Summary:
Overall, it seems as if this update was not a success for the game. There were increasing complaints about an ongoing issue in the game. and the size of the community has gradually decreased since the launch of the update.
We hope this article has been valuable to you and helps provide you with a comprehensive guide on how you can use Player XP to improve your post-mortems.