Q: How do you calculate sentiment?
A: Sentiment accuracy is determined by a combination of its classifier's precision & recall. Precision is the task of identifying the fraction of correct evaluations made across all potential evaluations for a set of interactions. Recall is how often our system makes a prediction when there is one to be made. After being trained on over 100 million interactions, sentiment accuracy is then evaluated against pre-built tests. The system is regularly tested and adjusted to account for changes in the way players speak online.
Q: How can the Player XP sentiment system tell when data is positive, negative or neutral?
A: The AI model we use for identifying sentiment is known as a “supervised learning” algorithm. This means that it learns from being shown labelled examples of what is considered positive, negative, and neutral feedback. It uses these examples to learn patterns within the different classifications of review.
Of course, our system will not agree with a human assessment all of the time, and we can only assess the accuracy of our sentiment with our analysts or against pre-analysed data sets. When we perform sentiment analysis, we aim for greater than 90% accuracy against a manual review by an analyst.
Q: How accurate is Player XP's sentiment analysis?
A: Our sentiment performs better or worse against different sets of data, so we cannot give a single number for the core metrics: precision and recall of our system. We expect a minimum of 80% accuracy for both metrics for any given game’s data set, however, we strive to have greater than 90% accuracy in all data segments.
To ensure our sentiment system remains accurate, for every new game we add to Player XP, an analyst will manually review a significant sample of data to ensure it’s performing as expected. If we do not meet our baseline criteria we will take corrective measures such as fine-tuning our existing model or creating a dedicated model for the specific type of data to ensure that we meet our quality standards.
Q: How does the system manage different sources and data points?
A: We have built a trailblazing data pipeline that is highly scalable and extremely flexible. We have a suite of integrations for all of our data sources that bring data into our data processing pipeline. This gathers the data using these integrations and processes it with our industry-leading sentiment and semantic processing. The data is finally stored for aggregation and query purposes by our Player XP dashboard.
Q: How does sentiment analysis work for translated data?
A: The Player XP translation engine is trained on game domain data and translates non-English data into English data so we can then perform our routine sentiment and semantic analysis. The impact of machine translation on aggregate data such as that in Player XP is very minimal so we can still return high-quality insights.
Q: What is flattened sentiment?
A: Our dashboard allows the louder voices to impact your sentiment more so than quieter voices, giving you a more accurate insight into the discussion happening within your community as a whole. However, if you want to avoid giving more credence to the louder voices, Player XP provides a form of sentiment calculation called “Flattened Sentiment”. This reduces each person within the data to a single sentiment value before averaging, therefore normalising the strength of their voice. You can enable this by checking the tick box in the options menu on any dashboard page (shown below).
Q: Why are some interactions marked as positive or negative when the content is the opposite?
A: We are aware that on very rare occasions, interactions' sentiments are marked incorrectly by our sentiment algorithms. If you see an incorrectly marked comment in the Interaction Explorer, you should be able to report it by clicking the speech bubble with an exclamation mark (found underneath the comment), then select your reason for reporting from the drop-down and click "Report".
Once reported, our team will review the flagged comment and if it is deemed incorrect, a change is put in place and our AI system will adjust its results the next time the system retrains itself.
Q: What is Hertzian?
A: Hertzian Ltd is the award-winning AI development team behind Player XP. We are specialists in data analytics and are internationally recognised as experts in our field.
Q: Should I trust an automated system?
A: There are many automated systems currently that don’t do their job well enough, so it is tough to blindly trust an automated system. However, we can ensure that our system demonstrably works. It is built on current industry-leading Natural Language Processing AI technology and uses best-in-class cloud technologies to service the rest. We liaise with you to identify the Player XP services you need and show you how you can use the dashboard to get key insights into your game’s community.