Developing a Multitasking Augmented Reality Application for Theatrical, Cultural Content

1. Introduction

Theatre is an ark, in which historical information of human activity & culture from every era and period is preserved. In the eyes of the contemporary observer, researcher or creator, the everyday costume of a previous era, becomes a historical costume. The theatrical act is a “slice” of life of the heroes who live and move on stage. In this sense the theatrical act provides the future generations with information relating to the whole spectrum of a person's life. Ones habits, education, life status, financial situation, job, interests, age, even ones beliefs and ethics and a whole lot more.

Costumart project (Figure 1) is part of the National Scope Action "RESEARCH-CREATE-INNOVATE" of the Operational Programme Competitiveness, Entrepreneurship and Innovation, co-funded by the European Regional Development Fund (ERDF) and national resources, under the NSRF 2014-2020 (code number Τ1ΕΔΚ-05070) and was funded to address some of these issues. The project was structured by a set of activities aiming to develop a set of applications based on Augmented Reality (AR) technologies and Virtual Reality (VR) tools. Augmented reality (AR) is an enhanced version of the real world, a kind of interactive environment based on reality, and this new version is achieved through the use of digital visual, audio or other sensory provided by real-time technology. Virtual Reality (VR) aim to create simulated environments by using computer technology tools and to embody users in virtual worlds.

In order to achieve its goals the project took initiative and finished a set of actions and activities, while the most significant were:

• Activities in order to portray historical and urban costumes (digitization, recordings, etc.)
• Creation of an infrastructure (storage systems, creation of ontologies, documentation dictionaries in three languages, etc.)
• Creation of applications
• Publicity and communication activities (clothing display, offer of data documentation, virtual tour, storytelling)
• Educational activities (educational applications and programs for specialists, for adults, for children, for schools, etc.)
• Dictionary and index of costume terminology.

The project created open ontologies for the costume/garment, its use, its materials, its way of production as a learning machine for everyone interested in theatrical costume from an educational, theatrical or research point of view. The project addresses and presents, via a series of applications, the clothing and dress patterns of men and women coming from four major historical periods, namely: (a) Ancient Greek, (b) Roman, (c) Elizabethan and (d) European years. The whole procedure is taking part in two ways, by two distinct applications:

• Incorporation of augmented reality within the theatrical act: The activity involved pilot AR use in a short-term theatrical performance where a small number of users wearing special glasses watched a theatrical performance with real and virtual characters.
• AR Application via mobile devices / augmented reality glasses: It included the development of a special augmented reality application that was applied in a showroom of theatrical costumes. The visitors have the opportunity to "scan with a mobile device or via the AR headsets" exhibits using the application. Following 3D actor models or civilians (avatars) wearing the costumes were presenting the costumes’ historical use and cultural value.

These applications contain not only the theaters but also three-dimensional models (avatars) for the representation of clothes of each era, while a series of texts and audio for educational purposes were developed (providing information about the theatrical performances and the actors of each era). Moreover, alongside with the costume/garment and its theatrical use, the applications’ users can enrich their knowledges about the various architectures related with the above-mentioned historical periods. A series of 3D theatres were developed permitting end users to choose the appropriate clothes / building and to see well known historical monuments to be created, while there exist an operation permitting users to explore both buildings and costumes from every side by using his mobile phone.

The development of 3D models, clothes and buildings / monuments as well as the user's interaction with the application, were all achieved by using well-known technologies, for Artificial Reality (AR) such as the Unity tool and Vuforia. The application has been implemented for both mobile phones with android software and IOS so that any user has the opportunity to experience this application but also to have the opportunity to learn through it.

Current paper aims to present the project’s main result and to broaden readers’ understanding about how can augmented reality, 3D models and virtual reality can be used to enrich a pure cultural experience such as a theatrical act. In the following sections, a brief theoretical background – mainly focused on technological aspects and its usage in cultural activities – will be provided, technological issues related with the proposed application will be explained and project’s results (3D models) from an aesthetic and operational point of view will be presented.

2. Literature Review

Augmented Reality (AR) technologies alongside with Virtual Reality (VR) technologies are getting developed over the last twenty years, finding an expanding amount of usage in several kinds of cultural organizations. More and more, different cultural sectors face a need or a willingness to implement such technologies. Cultural organizations are using both VR and AR technologies in order to reach their goals in their physical places as well as in digital world (Loumos et al., 2018), including actions such as

• expanding their presence to digital world and exploiting their content to mass audience in global cultural market,
• reusing or distributing already existing digital content (moreover expanding their collections’ digitalization) and
• connect physical structures / artefacts with digital content (e.g. Augmented Reality technologies) in order to provide their physical audience with a more interactive experience.

The use of Augmented Reality technologies in various cultural organizations (e.g. museums, galleries, archaeological places) has gathered both theoretical and practical interest. Starting from earlies 2000 researchers have been interested in interactive technologies for museums (Brown et al., 2003; Grinter et al., 2002), up to 2009 when Sylaiou et al. (Sylaiou et al., 2009) compared various forms of emerging technologies, such as VR, AR and Web3D. In the field of museums and archaeological places a series of AR applications were developed with more indicative examples being (a) ARCHEOGUIDE (Augmented Reality-based Cultural Heritage On-site GUIDE) system (Vlahakis et al., 2001), (b) AR museum guide for internal navigation and data information provision on cultural artefacts (Miyashita et al., 2008), (c) Arbela Layers Uncovered (ALU) a mobile Augmented Reality guide (Mohammed-Amin et al., 2012) using and 3D models as well, (d) AR application acting as a guide in the archaeological site of Knossos in Greece (called KnossosAR) (Galatis et al., 2016).

Technological advances and price affordability for reaching and using a smartphone (Kim et al., 2014), led to a further development of a series of applications, not only as mean for digitally replace physical artefacts as it was happening in the past (Novitski, 1998), but to enrich users’ experience as a whole (Karafotias et al., 2022; Kargas, Loumos, et al., 2019) via artefacts’ information, storytelling experiences etc (M. Vayanou et al., 2019; Maria Vayanou, Ioannidis, Loumos, & Kargas, 2019; Maria Vayanou, Ioannidis, Loumos, Sidiropoulou, et al., 2019). Reaching artefacts from different time periods in a realistic way and in a dynamic environment (Ch’ng, 2013) leading to what it can be called as Industry 4.0 for cultural sector / organizations (Kargas & Varoutas, 2020; Loumos et al., 2018).

During the development stage, the project’s team used existing experience from other projects including: (a) issues related with 360o videos (Cicco et al., 2019; Liu et al., 2019; Qian et al., 2016; Yen et al., 2019; Yi et al., 2019), (b) various pure augmented reality issues (Barrow et al., 2019; Ibáñez & Delgado-Kloos, 2018; Pratt et al., 2018), (c) issues related with 3D content and virtual reality (El-Ganainy & Hefeeda, 2016; LaValle, 2019; Men et al., 2017; Petry & Huber, 2015), (d) theater and the implementation of advanced technologies (AEA Consulting, 2022; Bertola & Teunissen, 2018; Pope et al., 2017; Rajan, 2016).

3. Materials and Methods

As mentioned before, two distinct technological tools (Unity tool and Vuforia) were used, while the most interesting part was achieving the appropriate collaboration between them in order to reach the best result, from operational and aesthetic point of view. Moreover, knowledge and skills were needed to understand the environment of Unity and how this environment translated / appeared not only into a computer but also into a mobile phone. Initial tests of the application were conducted on computer’s environment, to reach a no-error status, and then the corresponding procedure was conducted on the mobile’s environment to perform the test on a smaller screen but also in different versions of both android and IOS.

There were some problems with the correct movement of the characters on different screens where without the prior knowledge they could not have been solved. Developers also had to know the correspondence of each costume with the corresponding theatre so as not to transmit incorrect information to users. Also, historical knowledge was needed in order to create the right texts for each character but also the right sound files. The main purpose of this application is not only for each user to be able to see through a game the architecture of the buildings of each era (Ancient Greek, Roman, Elizabethan, European) but also to learn what clothes the actors wore in each ancient theatre but also to learn some theatrical performances that take place nowadays so that he can go and watch them.

As mentioned above, many modern technologies have been used to create this application which will be analyzed below (Figure 2). First of all, the unity tool is the environment in which the application was implemented and the tool with which the application was converted to android and IOS, so that it can be compatible with mobile phones and also the tool in which there are all the three-dimensional models of the application.

Another tool is Vuforia, through which the application can scan a specially formatted image and with this way start the game for the user (Figure 3). The basic function of this tool is that when the image is lost from the camera of the mobile then the game stops, as all the 3D models are created on this image. Also, when the image is found the user can interact with the game and see and learn everything, she/he wants.

3D technologies have also been used to create the three-dimensional models of both the theatres and the clothes of each actor (Figure 4) as well as the dressing room where the actor enters to wear the right clothes.

Even for the movement of each character to go to dress, movement technologies have been used to make them look like they are walking like real people (Figure 5).

To achieve the result presented in the above figure, many techniques - related with specific operations - have been used. The first operation is actors’ walking, where a technique called timeline is used so the walking of the actor can be more realistic.

Furthermore, in the beginning as you can see in Figure 4 the characters are not wearing something. User has to find the correct actor to begin the process and make the actor to move. When the actor enters the dressing room there is an invisible collider where it starts another process and the actor wear the correct costume. Each time the collider is activated, a new cloth (part of the costume as a whole) is added, until everything is on. Moreover, user has to choose the right theatrical environment, among four distinct 3D theaters representing different periods of time (Figure 6).

Also, there are techniques to facilitate the user to use the application by pressing a zoom button to enlarge the theater but also by activating and deactivating the panel with the written information to see more easily the costumes and frocks.

4. Results

Proposed application incorporated four, distinct theaters and four actors, each one coming from a different time period of theater’s history. These 3D models have been reused in each application and educational game developed. Moreover, a fifth character / avatar has been developed taking the figure of a clown and acting as a” conferacie”.

Apart from 3D theaters and 3D actors – avatars (including and 3D costumes), a third category of 3D models developed, included a series of 3D objects (props), either wearables from actors or objects used as part of the play.

In order to achieve, the quality needed for the development of 3D models, extensive research was conducted. Research included, finding and collecting the appropriate reference material, coming from pictures, architecture drawings, videos, etc. of course the most valuable source, related with 3D theaters / buildings, were architecture drawings (floor plans, facades and sections), while photos and videos were greatly useful when information was needed regarding construction materials (wood, stone, plaster, etc.) that were used. An example, related with the Globe Theater is given in Figure 7.

Especially when it comes for Globe Theatre, there was a lot of reference materials as a result pf the recent restoration of the theater, which provided a treasure of information for researchers, such as the roof of the theater (Figure 8).

At a second stage, all reference material was evaluated and most accurate information was used to develop 3D models. Architecture drawings, drawings and photos were imported to Autodesk Maya (Figure 9), a 3D graphic and motion design software. By using this material and existing information as a base of design, a series of software techniques were applied until an appropriate representation of the building was constructed. Such techniques included (indicatively): Extrusion, Revolution, Lofting, Boolean functions etc.

The third stage includes the development of the 2D parts that will be used to create the 3D models (Figure 10), a procedure known as UV mapping. At this stage the two-dimensional development of the model is created which will be used to “clothe” the 3D model with textures, providing the color information (Albedo map), dielectricity-gloss (Metal-Roughness map) and details of various anomalies (Normal map). Success during this stage is to achieve the model with the smallest degree of distortions with the least possible notches in the joints between the triangles

At the next stage, emphasis was put on the building’s Texturing, so that there exists a realistic resemblance of the “materials” that were used to build the physical building. Most accurate techniques involve the development of the above – mentioned 3D pictures (Albedo map, Metal-Roughness map, Normal Map). The model developed is implemented to the specialized software of Adobe Substance Painter (Figure 11), using information gathered during the first stage of development, while there exist software’s libraries permitting the effective imitation of the properties of natural materials (in terms of how light interacts on the,) in the 3D model.

Finally, the model and the special images mentioned above are inserted in the game machine, which undertakes its final display on mobile devices. The game machine determines a series of parameters, for example the detection properties of the various materials as an obstacle to the avatars’ movement, etc. During this stage, programmers evaluate how the various 3D models operate when implemented to mobile devices (e.g. smartphones). Optimizations are made in the models and the special images in order to ensure the satisfactory performance rate (frame rate) of the application.

As far as three out of four theaters (Figure 12, Figure 13, Figure 14 and Figure 15) are concerned (Globe Theater, Ancient Greek Theater and Roman Theater), there were no problems regarding frame rate, when the proposed 3D models were implemented to mobile devices. European Theater, having hundreds of seats, seemed problematic issue. Based on the fact that seats are similar to each other, made it possible to use an algorithm called “GPU instancing” so that to keep frame rate to an accepted level, without any discount on the realism of the representation.

For the development of the 3D actors, the same above-mentioned steps were followed: collection of reference material, construction of the three-dimensional models, creation of their extensions, creation of the materials and implementation in the game machine. Two more stages were added including the production of a model with a high number of triangles (HighPoly) and adding animation. Based on the fact that mobile applications must operate at a satisfactory rate of performance, modern 3D graphic design techniques require the construction of a second identical 3D model with a large number of triangles in order to represent details and irregularities of the physical object. This second model (highPoly) will be used so that these details are captured in the two-dimensional special image (NormalMap) and to create the illusion of their existence (Figure 16).

For the development of the highpoly costumes, the Marvelous Designer software was used. The proposed software is widely used in the video game industry where the representation of clothing is particularly important. Its main characteristic is that it uses the design of the pattern of the clothes and through simulation applies it to the character imitating the folds that are created. For example, in the following Figure 17, we can see the pattern on the right and the simulated model on the left. The correspondence of the pattern pieces in the model is marked in yellow.

For the needs of the game, the corresponding characters were also created wearing simple everyday clothes. The manufacturing process remained the same, but they are stand-alone models due to the fact that most parts of the body are covered by fabric. Additionally the characters acquired special properties (rigging) that allow them to move in space (animation). These properties are given in the 3D graphics and motion design software but are controlled by the game machine. For each character, an internal skeleton was created from joints and bones (rig) which follow a logical hierarchy (parent-child hierarchy) for example shoulder – arm – hand – fingers (Figure 18 and Figure 19). The latter procedure is called weighting or skinning and aims to move the model's points in space by turning only the individual bones.

Complementary items are an important part of the app as the user has to choose the right combination of theater-costume-items to complete the game. For each character, two types of objects were created. The Ancient Greek actor is accompanied by a mask and quills, the Elizabethan and the European by a hat and shoes, while the Roman by a laurel wreath and sandals. Their difference in relation to the previous types of models lies in the fact that in their final stage they must be inserted into the skeleton hierarchy and additionally for the shoes to acquire rigging and weighting properties to follow the deformations of the foot and ankle.

5. Discussion

After the in-depth analysis of the project’s results, an initial conclusion is that through this application the user can enrich his knowledge regarding the clothing and the architecture of four different eras of antiquity (Ancient Greek, Roman, Elizabethan, European) through play and interaction with an environment. Alle these created from his mobile phone, without having to open a computer to see the clothes, masks or theatres, as all the information will be in the palm of his hand.

There were even 4 scenario for the user for each theatre of each season but in each scenario, there are small differences to understand the different options that the user can choose and the different things he can learn about the theatres of each season through this application.

As the space where the painting will exist, it will not need to have made a model for each theatre as through the possibilities of modern technology and the work of several people, the user just needs a mobile phone with a camera to scan the painting and start a very interesting and educational game.

Finally, modern augmented reality technologies and programming skills have been used, as well as a lot of research to create three-dimensional models of clothes, theatres, shoes and accessories that without this knowledge neither this items nor the application itself could be implemented.

The proposed application can be used in many different fields. Initially it can be used as a game where students from schools will be able to visit a theatre and not only see a part of it but through augmented reality to be able to have the theatre itself in their palm and enlarge it to see and to learn all the details not only from the theatre but also and from each costume its design. They can also see the costumes of each character and learn what they wore in each season, but in a much more modern and interesting way than just seeing a garment, a shoe or a mask in a shop window.

In general, the proposed application seem to create a dynamic environment, capable to deliver information more easier to end – users, achieving a higher degree of implicit learning, as it has happened in various such applications (Kargas et al., 2022), while an issue under research remains how such applications can be directly associated with social media means for further disseminate any project’s social impact (Kargas, Karitsioti, et al., 2019).