THE CURAT SONIFICATION GAME: GAMIFICATION FOR REMOTE SONIFICATION EVALUATION

As sonification is supposed to communicate information to users, experimental evaluation of the subjective appropriateness and effectiveness of the sonification design is often desired and sometimes indispensable. Experiments in the laboratory are typically restricted to short-term usage by a small sample size under unnatural conditions. We introduce the multi-platform CURAT Sonification Game that allows us to evaluate our sonification design by a large population during long-term usage. Gamification is used to motivate users to interact with the sonification regularly and conscientiously over a long period of time. In this paper we present the sonification game and some initial analyses of the gathered data. Furthermore, we hope to reach more volunteers to play the CURAT Sonification Game and help us evaluate and optimize our psychoacoustic sonification design and provide feedback on the game and recommendations for future developments.


INTRODUCTION
After decades of research, the auditory display community has defined sonification as a technique that systematically represents data by means of reproducible sound that is interpretable by a human listener [1] [2, chap. 2]. This inevitably brings the user into play; a person that either only listens to the sonification and interprets it, or even interacts with the data. Therefore, it is common to evaluate sonification designs in experiments with users. In studies such as, for example, [3,4,5,6,7,8,9,10,11,12] between 6 and 24 participants are invited to the laboratory for experiments that last several minutes. They interpret and sometimes interact with the sonification. Based on objective measures and subjective ratings, the effectiveness, suitability and/or appropriateness of the tested sonification design is evaluated. These experiments are necessary as a proof-of-concept, and they reveal some benefits and drawbacks of the sonification design. Strengths of such experiments are the high level of control over conditions, tasks, parameters and processes the completeness of the experiment and the reproducibility. Such experiments also share 5 typical characteristics that imply some limitations: The small sample size implies a lack of generalizability. Small groups do not fully reflect target groups or even larger populations, which may exhibit a large variety of experience with and competence in interpreting sound. Consequently, such experiments do not constitute representative cross-sectional studies.
Being situated in a laboratory, participating in an experiment, being elucidated about the the rights of the participants, objectives of the study and potential risks (as demanded by ethics committees) is certainly an unnatural condition for many people. Carrying out an unknown, instructed task is unnatural, too. This unnaturalness may affect the behavior of the participants. They may feel tested, even though experimenters want to test the sonification design and not the participants. They may feel the pressure to achieve a certain performance or fulfill certain expectations. Consequently, the results from the laboratory experiment may not transfer to reallife scenarios, intended use-cases or everyday experience.
Many people have never analytically listened to sound, i.e., performed close, concentrated, conscious listening. Artificial sound is only ubiquitous in terms of alarms, warnings, music and primitive audio feedback in electronic devices. But a sonification of a high density of data is new to many people. Exposing novices to highly complex sonification means throwing them in at the deep end. As a consequence, participants may have reservations using sonification. The sound can make them feel stumped, overwhelmed, overstrained, stressed, distressed, confused, incompetent, inadequate, frustrated and upset. This could affect their objective performance and their subjective evaluation of the sound. Hours, weeks or even years of familiarization with sonification would be more appropriate to evaluate a certain sonification design. This is especially true if we aim at comparing it with visualizations, which most people face in their everyday lives.
Short experiment durations have the additional disadvantage that they hardly reflect the progress of learning and familiarization. Again, this progress affects the objective performance and the subjective evaluation of the sound. Consequently, such experiments do not allow to measure long-term effects. "(. . . ) it is critical to examine performance longitudinally when evaluating auditory display designs" [13], so longitudinal studies outside the laboratory are desirable.
Last but not least, experiments in the laboratory require personal contact. This naturally implies a geographic and thus demo-graphic variety of participants. Furthermore, we are facing a pandemic that has already made experiments with human participants in the laboratory almost impossible for more than a year. This situation raised the urgent need for remote evaluation of sonification designs.
In this paper we present the CURAT Sonification Game as a means to overcome the 5 limitations of laboratory experiments by using gamification. The release of the game is a starting point of our sonification evaluation study. We present and analyze initial results and give a brief outlook on future work.

RELATED WORK
One pioneering game for sonification evaluation is the Walking Game [14]. Here, users navigate an avatar to a target point, while avoiding obstacles. First, users train in a multi-modal setting, while seeing the virtual scene and hearing the guidance sonification. Then, they perform blind training tasks before the main experiment starts. Here, their execution time, number of collisions and subjective user ratings on the sonification are stored and analyzed. In the game, the control parameters are standardized, so the sonification design can be exchanged easily. This way the game allows for an evaluation and even a direct comparison of multiple sonification designs.
Other studies present playful sonifications [15,16,17] or even evaluate their sonification design in a game-like scenario [3]. However, none of the above-mentioned sonification games fully meet the game criteria or contain all gamification elements as described in Section 5.

THE SONIFICATION
With the CURAT Sonification Game we aim at evaluating our psychoacoustic sonification. The Digital Signal Processing (DSP) of the psychoacoustic sonification for two dimensions is described in [11], the expansion to three dimensions, in [18].
A three-dimensional space is illustrated in Fig. 1. In our case we have the Cartesian coordinates x, y and z. The space is not infinite but has boundaries like a box in front of you. The origin of the coordinate system lies in the center of the box. Your avatar can move freely within the box. You can now imagine the psychoacoustic sonification as pointing towards the coordinate origin. In other words, it always tells you where the origin of the coordinate system is located, relative to your avatar. Is it on the left or on the right along the x-dimension? And: how far to the left or right? Is it above or below on the y-dimension? And: how far above or below? Is it in front or behind on the z-dimension? And: how far towards the front or back?
In our psychoacoustic sonification, one continuous, complex tone has six attributes as depicted in Fig. 2   This three-dimensional sonification allows for navigation in threedimensional space. If the origin is on the left, the pitch will fall continuously, if it lies to the right, pitch will rise. The further to the left or right, the faster the pitch will fall or rise. When the origin is below, the complex tone will sound increasingly rough with increasing distance along that dimension. When the origin lies above, beats, i.e., continuous loudness fluctuations, will be heard. The further above, the faster the beats. When the origin lies towards the front, the fullness will decrease with increasing distance. When it lies in the back, the brightness will increase with increasing distance. The strength of this psychoacoustic sonification is that the single attributes of the complex tone are perceptually independent. Therefore, listeners can interpret the direction and magnitude of every single dimension even though they are always presented simultaneously [12]. No reference tone is needed. You know that your avatar has reached the origin when the sound has a steady pitch (x = 0), exhibits no loudness fluctuation and no roughness (y = 0) and when the sound is dull, but full (z = 0). The sonification serves as a guidance tool. Consequently, your task as a user is to understand where the origin is. The sonified origin represents your target point.
The psychoacoustic sonification is defined by real analytic functions. It can be updated interactively and continuously in realtime, even on budget devices with little computational power and one mono tweeter.
We have implemented the two-dimensional psychoacoustic sonification in the Tiltification App [19] and evaluated it in a passive [20] and an interactive [10] laboratory experiment with abstract localization and navigation tasks and in the context of pulse oximetry [9]. Figure 2: Description of how the sonification sounds when your avatar is at different locations relative to the origin. When, e.g., the avatar is on the upper right, the sonification will exhibit roughness and a falling pitch. From [21].

THE GAME
We developed the CURAT Sonification Game to promote sonification as such and bring it to people outside the scientific community. Therefore, it has a user-friendly graphical user interface and front end, appealing music, graphics and sound, and needs little competence to install it on the most widespread devices. Furthermore, we developed it to evaluate our psychoacoustic sonification design by long term usage of a high number of diverse users outside the laboratory so as to overcome the shortcomings of our previous experiments. The CURAT Sonification Game is available for Android, Windows, Mac, and Linux. The multi-platform game was implemented using the UNITY game engine, the programming language C# and its integrated development environments. The game contains 5 mini-games in several modes and difficulty levels that can be played as arcade game or in a campaign.

Game Menu
When you open the computer game, the game menu appears, which provides a meaningful structure that we also follow in the following explanation of the game.

Settings
In the settings you can specify your language, adjust the volume of background music, sound effects, and the sonification, and give and take permission to send your game stats via wi-fi and/or via mobile data connection.

Profile
Under profile you can create, change and delete profiles. For each profile, you can view all scores for all mini-games and dimensions. Furthermore, you can track your high-scores and achievements. The profile menu is shown in Fig. 3.

Campaign
In video games, campaigns are premade levels [22] with rising complexity. Our campaign mode acts as an introduction to the 5 mini games and an initial training. In our campaign you familiarize with the single and combined dimensions of the psychoacoustic sonification, the game control, and the 5 mini-games. You have to accomplish one or more campaign levels successfully in order to unlock new campaign levels. The campaign levels are organized in a tree-structure as can be seen in Fig. 4. Locked levels have a white background, solved levels, a green one and yet unsolved levels, a blue one. You have to solve the easier levels in order to unlock more difficult ones. The levels in the campaign are presets that do not adapt to your performance skills. The only information that is stored locally, is whether you have successfully accomplished a level or not. Figure 4: Screenshot of the campaign tree. Only after completing noisy nuts in xand in y-mode, you are free to move on with Dancing Darts, which combines the xand y-dimension.

Mini-Games
The CURAT Sonification Game is built around 5 mini-games in landscape-orientation, described in the subsequent sections. In the arcade mode you can play any of the games. Here, you choose a basic difficulty level. However, the game also adapts to your performance, so it gets more difficult if you perform well. Game stats are stored and transferred for further analysis. Basically, there are two types of games. In passive games, many points or fields are presented. You interpret the sonification to identify which of the points represents the sonified origin, i.e., the target point. If you interpreted the sonification successfully, you've got a hit. In interactive games you interact with the sonification, so every movement causes a change in the sonification. Here, you do not passively listen to the sonification in order to identify the target location. Instead, you move your avatar until you have reached the target, i.e., the sonification guides you to a target point.

Piñata Party
In the Piñata Party, you are blindfolded, but the sonification tells you where the piñata is hung. The sound updates interactively, while you get into striking position using the position bar. When you think you have found the piñata, you click the checkbox to swing your baseball bat. The closer you are, the more points you get. If you're too far off, you will miss the piñata and get 0 points. Figure 5 is a screenshot of the pinata game. After 10 trials, the mini-game is over, so you can reach a score between 0 and 1000 points. You can play the Piñata Party along the x-, y-, or z-dimension. Piñata Party adapts to your improving skills by increasing the resolution of the position bar, while reducing the size of the region that is counted as a hit. The Piñata Party is one-dimensional and interactive. The user can decide in which of the three dimensions to play.

Noisy Nuts
Noisy nuts is based on the shell game; famous street scam. A coin is prominently placed under one out of many nutshells. Then, the nutshells are shuffled. After that the sonifications tells you which nutshell hides the coin. Your task is to identify the sonified location correctly. Depending on the chosen number of dimensions and your previous performance, you will face between 2 and 9 × 9 = 81 nutshells. Only when you chose the correct shell, you get points. The more nuts are on the table the more points you can get. If you choose the wrong nutshell, you loose one life and get negative points. You have up to 5 lives. You can play Noisy Nuts along any single dimension and all possible dimension pairs, i.e., planes.
Noisy Nuts adapts to your improving skills by increasing the number of nutshells and the shuffling speed. Noisy Nuts is a passive game in one or two dimensions.

Urban UFO
In Urban UFO, you are an alien who is supposed to collect specific humans through a tractor beam. The sonification guides you towards the wanted human. You have to collect a given number of humans in a limited amount of time. In addition to the sonification, a mug shot of the wanted human and a visual distance indicator help you find the right subject. However, to your alien eyes all humans look the same except for the color of their shirt. For each correct human you get a time bonus. A screenshot of the Urban UFO mini-game is provided in Fig. 7. Your UFO has a constant altitude (y), so the mini-game takes place within the x-z plane.
Urban UFO adapts to your improving skills by lowering the precision of the visual distance indicator and by adding more distractor humans that look exactly like the wanted one. Urban UFO is an interactive game in two dimensions.

Dancing Darts
In dancing darts the sonification is your opponent. It announces one area on the dartboard and you have to hit it with your dart within a limited amount of time. In other words, the sonification represents a location in one of the dartboard areas and you have to identify the correct one, quickly. You only score if you hit the right out of 2 to 32 areas. Otherwise, you loose one life. Figure 8 is a screenshot of the Dancing Darts mini-game.
The arrangement and number of areas on the dartboard adapt to your performance skills. Dancing Darts is a passive game that can only be played in the x-y plane. Figure 8: Screenshot of the Dancing Darts mini-game. Your dart will be thrown at whatever area you click. When you miss the sonified area you loose a life.

Sounding Submarine
In Sounding Submarine you maneuver a submarine through the dark. Your sensors track checkpoints that you need to reach as quickly as possible. The sonification guides you towards the next checkpoint. You can turn on a flashlight to illuminate your surrounding, and even turn around to look into all possible directions. However, your battery has a limited capacity, so you need to rely on the sound most of the time. You loose a life if you recede from the target by more than the range of your sensor. You can play Sounding Submarine within any dimension pair or within a threedimensional space.
If you perform well, the capacity of your battery is lowered to make the mini-game more challenging. Furthermore, when you perform well, checkpoints are placed further away, so you will need longer to reach them and you can easily exceed the sensor range and loose a life. Sounding Submarine is an interactive game in two or three dimensions. You can find further explanations and engaging advertisements on our YouTube channel under https://tinyurl.com/2ynxzjx5.
In addition, we implemented gamification elements according to [25]: You experience autonomy by choosing which mini-game to play, in which mode and with how many lives. Furthermore, you have the freedom to move your character where and when you like, to turn on the submarine lights, or to decide how to circumnavigate buildings with your UFO. In the passive games, you can skip the nutshell shuffling and take your time to make a final decision. You experience competence when you realize that binary choices are so easy for you the game decides to present a much higher number of nutshells or dart fields to you. Relatedness, i.e., social connectivity is difficult to include in games. We allow for multiple user profiles on the same machine so that family members or classmates can compare each others' scores and achievements. However, to ensure anonymity, we do not offer a platform to exchange stats, thoughts and experiences online.
Autonomy, competence and relatedness are considered factors of intrinsic motivation. As factors of extrinsic motivation, we included points, leader boards, badges, rewards, and incentives [25].
You can earn points in every arcade and campaign game. Leader boards as shown in Fig. 10 are meant to motivate you to improve your personal high score and that of other players. Furthermore, all scores are listed in your personal profile so that you can admire your progress.
We think that none of the badges are easily earned right away. It takes some training to earn them, and some require prolonged and dedicated playing.
To reward you for successful training, new levels are unlocked in the campaign mode. Furthermore, winning sound effects and the colorful animation in the Piñata Party reward you, in contrast to the punishing loosing sound effect and the animation of a disappearing heart.
As a researcher in the field of auditory display or a layman with a high interest in sonification, learning something new and supporting idealistic and non-commercial research may already be an incentive to you. In addition, we like to point out that playing the CURAT Sonification Game may save lives: We aim at using the psychoacoustic sonification as a guidance tool for surgeons in image-guided interventions, and for surgical training [26]. To do so, we need long-term sonification usage data to analyze long-term training effects and identify aspects in the sound design that need improvement. We hope that potentially saving lives is an incentive to you additional to the fact that playing the game is (a) fun and (b) a helpful contribution to our scientific work.
With these gamification elements we hope to recruit a high number of long-term players with a demographic variety that is Figure 10: Screenshot of the leader board that is displayed when opening the Urban UFO arcade game.
larger than subjects we could recruit for experiments in our laboratory. This would allow us to collect both cross-sectional and longitudinal data, i.e., average performance and learning curves.

DATA COLLECTION
If players give the necessary permission, the data from all arcade games is collected locally and then transferred to the University of Bremen server as soon as an internet connection is established. The user data is associated with the chosen nickname. However, the nickname is encrypted to ensure that the chosen name does not reveal the identity of the player.
The transferred data contains information on the operating system. This gives us a cue whether the game was played on a mobile device (android) or not (windows, linux, and mac). For the Dancing Darts the sonified target area, the number of available areas and the area clicked by the player are stored in a file together with a timestamp. This allows us to analyze, e.g., how often the target has been hit. In Noisy Nuts, the sonified target nutshell, the number of available nutshells, the nutshell selected by the player and the dimension (x, y, z) or plane (x-y, x-z, y-z) are stored in a timestamped file. For the interactive games, the location of the avatar and the target at the beginning of each round are stored. In Piñata Party, a ∆x or ∆y value indicates whether the game has been played in the xor the y-dimension, and how closely the player approached the sonified target position. In Urban UFO, the time needed to collect each wanted human is stored. In Sounding Submarine is is additionally stored whether the game has been played in the x-y, x − z or y − z plane, in the threedimensional space, i.e., x-y-z. We can calculate cross-sectional averages as well as the longitudinal progress of individual players, i.e., training effects.

DATA ANALYSIS
So far the game was downloaded about 500 times. 72 individual users have transferred their game statistics within 368 sessions. We have several possible explanations for the comparably low number of players who transferred their game stats: Many players have not (yet) completed the campaign, only played interactive arcade games so far, or decided not to give permission to send their statistics. Furthermore, 20 to 25% of apps are only used once after download 2 .
As we are still waiting for a) more players to share their game statistics and b) statistics from long-term gaming that reveal their training effects, we only present a very brief overview of some initial results from the passive games in Fig. 11. The data summarizes data from Noisy Nuts and Dancing Darts. Here, the computer screen is divided into nine subregions. For every subregion, the three plots show how many targets have been identified correctly by the players, i.e., the hit-rate. The hit-rate shows how successfully players localize the exact location if a target lies somewhere within a certain region. The plot on the left refers to the x-y plane and contains data from both the Dancing Darts game and the x-y mode of Noisy Nuts. The plot in the middle refers to the x-z plane and only contains data from the x-z mode of Noisy Nuts. The plot on the right refers to the y-z mode of Noisy Nuts. The three plots show that the hit-rate is mostly better than a random guess, which indicates that players played conscientiously and the sonification is well-interpretable. We can observe that most regions in the x − y plane have a high hit rate. The average hit-rate is 0.53. We find the highest hit-rate along the axes and a slightly lower hit-rate in the corners. This is expected, because along the axis, the player only needs to identify one sonification attribute. In the corners, players need to identify both the attribute for the azimuth and the attribute for height simultaneously, which is more demanding and prone to misinterpretation.
The x − z plane exhibits a slightly higher hit-rate of 0.59 on average. Here, players identified targets better at positive z-coordinates (auditory brightness) compared to negative zcoordinates (auditory fullness).
In the y − z plane, 2 fields have always been hit and 1 fields has never been hit. The average hit-rate of 0.60 is the highest of all three.
Most data has been transferred for the x − y plane, followed by the x-z and the y-z plane. This is probably because a) the xy plane is the only plane that is available in both Noisy Nuts and Dancing Darts, and b) the x-dimension is pre-selected in Noisy Nuts. In contrast, the y-z plane is only available in Noisy Nuts and has to be selected manually by the player. The few data on the y-z data may be the reason why some fields still have a hit-rate of 0 or 1.
The falling number of player statistics from x-y over x-z to yz may also partly explain the rising hit rate: While players start with the xand x-y dimension, they gain experience and thus, training. Consequently, their initial y-z performance is already much better than their initial x-y performance. On the other hand, it is also possible that the z dimension or the combination of the y and z dimensions are simply easier to interpret. However, this is not in agreement with our experiment results from the laboratory [12].
Note that these plots show the raw hit rate. They do not indicate how many fields were available when a target was hit. The minimum number of available point is 2 nuts in Noisy Nuts and 2 areas on the dart board in Dancing Darts. However, the difficulty level of the game adapts to the performance of the player. Consequently, a player who performed well may have to identify the correct area out of up to 32 areas on the dart board in Dancing Darts or the correct nutshell out of 81 available nut shells in Noisy Nuts. This means the probability to guess the correct field lies between 1.2 and 50%. Furthermore, this overview neither considers time, nor individual performances.

CONCLUSION
In this paper we presented the CURAT Sonification Game as a means to evaluate our psychoacoustic sonification design. The game combines sonification with gamification to overcome the limitations of experiments in the lab: We hope to reach a large, diverse number of players that carry out familiar tasks in their usual environment without feeling examined. This way, they can take time to familiarize themselves with the sonification, get used to the mini games and improve their performance. The multi-platform game is available for both mobile devices and desktop systems, and does not require expertise or previous knowledge to install and run it. This way we also hope to contribute in making sonification better known to the public.
Initial analyses show that around 500 people have already downloaded the game and some dozens play it continuously. At a first glance most players seem to play the game conscientiously, as average the hit-rate lies above random guessing. There seem to be no regions where users are unable to interpret the sonification. This is evidence that the dimensions of the sonification are in fact orthogonal, linear and have a high resolution.
From previous experiments we know that both precision and accuracy improve a lot when users interact with the sonification [10] instead of interpreting it passively [8]. Consequently, we plan to analyze the statistics from the interactive games soon. After collecting some more statistics, we will deeply look into the data to find cross-sectional averages and longitudinal progress of players.

ACKNOWLEDGMENT
We thank the students of the CURAT bachelor's and CURAT master's project who did an amazing job conceptualizing, designing, implementing, testing, releasing and advertising the CURAT Sonification Game. We also thank all the gamers who play our game and spread the word for fun and for science.