A Coordination Approach to Support Crowdsourced Software Design Process

1. Introduction

Crowdsourcing software design (CSD) is usually organized around platforms. The platforms allow clients to request to perform design tasks and allow designers in finding tasks that are required from clients. Crowdsourcing platforms rely on contests which represents all the CSD activities that occur between independent designers that create solutions towards winning a reward for the completion of tasks. CSD is used in many design areas, such as web, mobile, embedded systems, cloud computing, etc.

When performing tasks in CSD, some dependencies occur between activities as different participants collaborate and it has many forms, the most common one occurs if two activities are related to each other, and a second activity cannot start before the first activity ends. When activities are related, if there is a problem in one of the activities, it must share this information with the others [1]. Managing these dependencies among activities requires coordination to implement the activities on time [2].

Regardless of the wide use of CSD, the lack of coordination is one of the major issues in CSD, as the lack of good coordination can lead to project failure and long delivery delays [3]. The literature identified multiple limitations associated with coordinating its process. For example, many designs will likely repeat due to a lack of coordination among participants, and potential waste of time in selecting best designs from a larger pool of designs [4,5,6]. Additionally, many designers can create similar designs for the same task without considering each other [7]. A lack of coordination support in CSD activities results also in delays and missed opportunities for designers [8]. The absence of coordination among these activities may lead to disturbances in performance.

Therefore, many researchers have emphasized the need to study how to coordinate activities among a group of participants in CSD and what mechanisms need to be adopted to ensure process applied most effectively [1,6,9,10]. Additionally, CSD should support more complex and valuable work by defining tasks, interpreting the dependencies that occur when tasks are completed, finding and allocating appropriate designers to perform those tasks, monitoring and supervising their performance, and evaluating that their performance is of a high quality [7].

Coordination theory provides a lens through which to analyze tasks, participants, dependencies, and coordination mechanisms. This theory assumes that the compatibility or incompatibility of the dependencies and available coordination mechanisms may explain the issues participants face in achieving their goals [11,12,13].

The aim of this research is to support the effective management of CSD process through identifying the key activity dependencies among participants in CSD platforms and designing a set of process models to provide coordination support through managing this activity.

This paper is structured as the following: Section 2 presents a background of the software design, and the crowdsourcing concept and characteristics, and then it discusses the use of CSD. In Section 3, the approach is described in detail. Section 4 the current CSD process and activities are discussed. Section 5 identifies the activities that require coordination support. Section 6 discusses the results of the evaluation of the importance of supporting coordination in CSD activities, which have been done by conducting online surveys. Section 7 presents the design of the proposed models of CSD activities. Section 8 evaluates the proposed approach to manage the CSD process, while Section 9 concludes the paper.

2. Related Work

2.1. Crowdsourcing Software Design (CSD)

CSD is defined as the accomplishment of specified software design tasks on behalf of a client (typically, an organization) by a large and typically undefined group of external individuals with the requisite specialist knowledge through an open call [14].

In addition to the basic web and mobile design, Mao et al. [15] identified some possible design areas that can benefit from CSD. In Architectural Design, the requester asks the crowd of designers to create an architectural design for his program based on the specifications he chooses. In software design review, crowdsourcing designers are used to provide feedback on designs, enabling designers to improve their work [16].

Many research studies [2,14,17,18,19,20] defined three four main components to any crowdsourced software development system:

• Requesters: Companies or individuals, who post the issues they want to solve as a freelance job or contest.
• Crowd: The community of online workers who have signed up for the platforms and then decided as individuals or small teams whether to take on an open call challenge.
• Platforms: Computer systems that provide an online marketplace for workers and clients to meet. This marketplace mediates and connects requesters with online workers to solve tasks.

2.2. Literature Review

Many research papers discuss and contribute to the CSD process. Latoza et al. [21] proposed a limitation of the award-winner selection process, where the role is conducting two contests on user interface design and architectural design. Designers in both were able to review and rate their peers’ designs. The researchers found that the quality of the designs was higher as all the designers benefited from each other’s feedback on the initial designs and improvements implemented into the revised designs. Some designers can produce the best designs in much less time than others have produced low-rated designs. While recombination enables most designers to improve, it does not enable weak designers to produce strong designs even in adapting stronger designers’ ideas. Murray-Rust et al. [22] stated that crowdsourcing software development handles tasks in a simplified format. To support the crowdsourcing of software development processes, the system needs to activate coordination mechanisms. It is often necessary to impose certain restrictions on the coordination and quality of these collaborations in order to manage them. Coordination protocols provide a high-level organization of activities (including planning and evaluation/feedback), while a set of coordination and quality constraints direct the assignment of workers to tasks. This allows the system as a whole to strike a balance between imposed restrictions and creative freedom in software development. The approach was evaluated by implementing an initial version of the coordination model for a case study and simulating its behavior on a heterogeneous group of workers.

Latoza and Hoek [21] discusses methods of exploring software development that enable multitudes of developers to contribute effectively through creating, distributing, and coordinating tasks. Underlying these approaches are three essential considerations: analysis, coordination, and quality. All of them influence how tools enable software to be built with a crowd. In this research, each aspect was studied, including analyzing a wide range of software development work into nuanced tasks, coordinating contributions across the crowd, and ensuring the quality of software produced by the crowd. Some challenges in coordination include how workers choose tasks, allocate the most efficient workers for the task that they have to do, how the system can track the tasks to be done, and how the dependencies between tasks are revealed and managed.

Alyahya and Alamer [23] investigated the different types of dependencies between the activities of the crowdsourcing platform. Through the search, five types of dependencies occurred between activities, among them, they dealt with two types. The first type was a hard prerequisite dependency, which is if there are two activities in a task and Activity 1 depends on Activity 2. If the developer of Activity 1 tries to finish it, the system must close Activity 2 because of its association with Activity 1. This type of dependency is very complex because parameters must be set that coordinate this sequence for all the related activities. The second type was a soft prerequisite dependency. This dependency also occurs between two activities. When a developer begins work on Activity 1, the system will send a message to the developer of Activity 2 that to this effect. This dependency is simpler than the previous one.

Xiao and Paik [7] mentioned that crowdsourcing software platforms allow clients to advertise their tasks and help them find workers to do the tasks, but the problem is the coordination among workers to complete the tasks. Through this research, a conceptual framework was formed to bring coordination support to crowdsourcing software platforms. It is not enough for these platforms to support crowdsourcing for creative, complex, and organized work that needs multiple workers with varied expertise. Coordination among these highly skilled workers is needed so that each worker, as a service that can be self-described and explored, can be dynamically assembled into complex teamwork. A workflow-based scheme was defined to structure and represent the work of the crowd. Crowdsourcing was then performed as the coordinate protocol was administered. The coordination protocol was designed to initiate and manage the crowdsourcing process by finding the worker, obligating him/her to carry out the tasks, and coordinating him/her in the protocol, where work was identified that consisted of a set of interrelated activities.

Machado [24] mentioned that Crowdsourcing software development platforms depend on a competitive approach in which workers independently create solutions while they compete against each other via financial rewards for completing tasks. Contest usually reduces cooperation. This research focused on studies related to cooperation between the primary software and the client in the assigning of crowd workers, presenting tasks, and the effect of cooperation among workers of the crowd on the quality of the task solutions. The research aimed to identify the characteristics of collaboration and the barriers that crowd members face in competitive crowdsourcing software.

Amrit [25] addressed an important part of coordination in software development, which is the problem of assigning tasks among members of the software development team. This research provided an analytical framework for research on teamwork. A vision for a potential future for the crowdsourcing business is laid out and entails worker considerations, such as motivation, feedback, and pay. This can be addressed through mechanisms to preserve reputation, provide better customer interaction, and increase pay. Also, customer considerations, such as coordination, the dismantling of tasks, and quality control, are considered. These can be addressed through workflow mechanisms such as collaboration.

Although there is research supporting coordination among participants in CSD platforms [26], it has not covered all aspects. To develop and contribute to the development of CSD, the importance of this research comes in bridging the gap related to the study of coordination among participants in CSD.

3. Approach

The aim of our research is to investigate the current process of CSD practice and to search for possible advancements to the process. To achieve this aim, we use a five-stage approach as follows:

Stage 1: Investigating current CSD process: This stage investigates 14 CSD platforms and identifies the key activities involved in the CSD process.

Stage 2: Identifying coordination needs: This stage is concerned with identifying the main dependencies involved in the activities extracted from the previous stage. The potential limitations of managing these dependencies along with suggestions of coordination mechanisms that may improve dependency management are identified. We used the approach of studying coordination dependencies in order to provide process improvement. This approach has been intensively used in the literature (e.g., [11,27]).

Stage 3: Evaluation of existing and proposed coordination mechanisms: This stage is the evaluation and comparison of the existing coordination mechanisms with the proposed ones. Data will be collected from questionnaires to assess existing coordination mechanisms and then participants will be interviewed to evaluate proposed coordination mechanisms based on an understanding of the way tasks are carried out and the dependencies that occur between activities.

Stage 4: Design process models: This stage presents the design of the proposed model to manage CSD activities. A set of process models has been created for the proposed activities agreed on in the previous evaluation. These models provide a visual representation of how the activities can be developed.

Stage 5: Evaluation of the process models: This stage evaluates the proposed model and discusses the outcome through interviews with experts in CSD.

4. Stage 1: Investigating Current CSD Process

CSD platforms allow clients to create a contest to perform specific design tasks for example web design sites, (HTML), (JS) services, etc. The CSD platforms covering various regions around the world were selected based on the number of users, number of clients, and appearance in internet search results and blogs, and the platforms most used as a case study in scientific research.

In addition to the above, the selected platforms provide all software design services such as (Architectural design, Component design, Algorithm design, Database design, System Interface design, User Interface design, and Logical design), the investigation included general crowdsourcing software platforms that contain software design services. These platforms are as presented in Table 1.

The review of fourteen CSD platforms was performed to identify current CSD operations and their activities. An investigation of the activities in CSD platforms was conducted using the following methods: reading the official description of the nature of the work of the platform, which is available on the website of the CSD platform, watching the videos on the platform that explain how it works, reading the comments of the participants on the platform., contacting platform support and ask about the features offered to participants, communicating with some clients and designers on the platform and inquiring about how to work on contests, and visiting the forums of the platform that bring together the participants in the platform.

To understand the workflow of the current design process in CSD, the platforms are analyzed and accordingly the crowdsourcing workflow for software design is drawn. Figure 1 illustrates the current CSD workflow. Most CSD platforms follow a general process, which can be seen and understood from the flow of activities across the platforms. As mentioned in the Approach Section, the result of the review is presented in [99].

5. Stage 2: Identifying Coordination Needs

Coordination is important to ensure that activities are implemented in the CSD on time, so it was necessary to communicate between clients, designers and platform representatives (i.e., project managers) to build a contest. Based on coordination theory, coordination is defined as the process of managing the dependencies between activities. Thus, studying coordination means analyzing the dependencies that appear between tasks in the system and determining how to manage those dependencies [89].

An analysis of CSD activities on available CSD platforms and the results of current research in this field show that the coordination support for CSD activities needs to be improved for a better crowdsourcing contest. This section analyzes the limitations of coordination support provided by the current CSD process and suggests a set of coordination mechanisms that can help overcome these limitations.

We followed heuristics provided by Crowston and Osborn [12] for analyzing dependencies and coordination mechanisms in a situation to make dependency focused analysis. They proposed this procedure: identify dependencies, and then search for coordination mechanisms. In other words, look for dependencies, then ask which coordination support is used to manage those dependencies. Crowston and Osborn [12] suggested asking questions such as the following to discover dependencies and coordination mechanisms:

• What are the inputs/outputs to each activity (e.g., informational and other necessary pre-conditions/post-conditions)?
• What potential performance problems associated with this process? Do these problems reflect unmanaged [or poorly managed] dependencies?

Table 2 emphasizes to what extent the current CSD platforms support these activities, identifies the dependencies in each activity, discusses the potential limitations, and discusses how current CSD platforms can be improved to better coordinate and support CSD activities.

6. Stage 3: Evaluation of Existing and Proposed Coordination Mechanisms

In order to evaluate the need for the proposed coordination support, an online survey was conducted. The survey is distributed through email and social media and received 41 responses. It is taken into account that the selected people must have a strong background in the domain of CSD.

The respondents were asked to rank the limitations and suggestions affecting contest coordination in CSD platforms on a Likert scale of 1-5 where 1 means strongly disagree and 5 means strongly agree. Seventeen limitations and seventeen suggestions affecting contest coordination in CSD platforms were evaluated. The descriptive statistical analysis [94] was used to find out the respondents’ opinions regarding the limitations and suggestions in the stages of the design contest (Table 3).

The survey findings show that it is clear that a large number of participants agreed on that these CSD activities suffer from the identified coordination limitations and that the proposed coordination mechanisms are valid, although not all of them did. This is normal due to their different roles, responsibilities, and needs. However, the goal of this evaluation is to make sure that these CSD activities would provide more coordination support for CSD process. The full results of this evaluation can be found in this external webpage1.

7. Stage 4: Design Process Models

This section presents the design of new process models incorporating the proposed coordination mechanisms identified earlier in this research. The list of these models and how they cover the proposed mechanisms are shown in Table 4.

Ten process models are suggested. Due to the page limit of this article, we discuss in details the first three ones below, while the rest of models can be found in the Appendix.

7.1. P1: Create Contest Brief

This process model covers the proposed activity A1: Create contest brief. The process model in Figure 2 demonstrates that the platform shall provide a detailed form that includes all the requirements to ensure the proper functioning of the contest.

When a client registers on the platform to request a design contest, then the platform determines automatically the type of contest and sends customized form that matches that type. The platform sends a detailed form for the contest that includes the following information: the name of the client, type of design, design language category, the main purpose and objectives of the contest, the potential impact of the contest, what are the main elements of the design contest, and the actions you want the designers to take in the contest. The client fills out the contest form and optionally attaches the files he wishes, then sends them to the platform, and the platform verifies the detailed form of the contest.

7.2. P2: Assign a Project Manager and Validate Contest Brief

The process model in Figure 3 demonstrates how to assign a project manager and validate contest brief. The platform automatically verifies the project managers present on the platform in accordance with the type of design contest. After that, the platform checks the mechanism of participation of the project manager in the contests, whether full-time or part-time and the current number of contests, as the platform appoints the project manager who has the least number associated with the contests. The platform then will send notification to the project manager that he has been selected to be the project manager for the new contest.

7.3. P3: Announce the Contest

The process model in Figure 4 illustrates the importance of having multiple communication channels to send contest announcement messages to designers.

The project manager sends invitation messages to participate in the contest to designers who have been selected in accordance with the requirements and needs of the contest. The designers browse the invitation, then two decisions are made either to not accept the invitation to participate in the design contest and or accept it. The platform accumulates the acceptance responses and waits for a particular period of time (i.e., 48 hours), then the platform checks the number of acceptance responses again.

if the number of designers is incomplete to participate in the contest, invitations are sent to the designers with another communication channels, either through social media or WhatsApp messages.

8. Stage 5: Evaluation of the Process Models

In this research, semi-structured one-to-one interviews with domain experts were conducted for the evaluation of the proposed process models to manage CSD activities. The evaluation using process models, provide more thorough discussion since these models visualize the CSD process, giving a better understanding about the activities and the coordination required.

In semi-structured interviews, the interviewer prepares a list of predetermined questions and participants have the opportunity to explore issues in much depth and from as many angles as they please, during answering the open-ended questions. In addition, the interviewer freely explores different fields and asks specific questions during the interview [96].

The sample of interviewees should share critical similarities related to the research question. In order to maximize the depth and richness of collected data, the selected interview participants are practitioners from the crowdsourcing design domain. Interviewing experts on the domain made the conversations more valuable as they shared their experiences and common challenges. The interviews had an average time of 55 minutes. Their views are reported here to evaluate the proposed CSD activities.

8.1. Interviewees Background

The interviews involve 7 people who are four designers (D1, D2, D3, and D4), two clients (C1, C2) and one project manager (PM1). Table 5 shows the characteristics and experience of the participants in the domain of CSD.

8.2. Interview Findings

The participants discussed the difficulties they face with CSD projects and how they could be overcome through the proposed models. Most suggestions are agreed on by all participants. However, we have received some comments and these are detailed in Table 6. The process models not mentioned in the tables mean did not receive comments and they are agreed on as designed.

9. Conclusions

This research provides several mechanisms to support the the CSD process. Seventeen CSD improvements, related to twelve CSD activities, were identified.This complies with the emerging research in the CSD literature for more efficient solutions to overcome current limitations (e.g., [8,11,12,15]).

Although the CSD platforms have provided features to improve the CSD process (e.g., email notifications, chat, announcements), these features are insufficient to solve the coordination limitation. The proposed approach extends the scope of the existing coordination mechanisms by structuring the contest form, selecting project managers, selecting designers, avoiding inactive designers, improving notifications and reminders, , in addition to tracking the progress of the contest and giving feedback.

The authors believe that the results and suggestions that have been reached will add valuable information that contributes to the development of the field of design contests in CSD

Seventeen CSD improvements, related to twelve CSD activities, were identified along with the identification of the coordination support required for each CSD activity.