IoT Communication Based on MQTT and OneNET Cloud Platform in Big Data Environment

1. Introduction

With the gradual development of communication technology, a variety of communication technologies have emerged. HTTP is the representative of the request-response mode. HTTP/2 protocol also introduces the asynchronous request-response mode, the client can set different priorities for the request, and the server can decide which request to respond first according to the priority. In the case of HTTP, the re-establishment of a connection for each new request message results in significant overhead. The CoAP (Constrained Application Protocol) protocol is designed to be lightweight and can be used for resource constrained IoT devices. The LwM2M protocol is defined above the CoAP protocol, but it processes a step further on message transmission. It standardizes the device model based on IPSO (IP-based Smart Object), provides a set of lightweight device management and interactive interface protocols. The MQTT protocol is well suited to remote devices with limited computing power, low network bandwidth, unstable signal, and low power consumption. Meanwhile, the persistent connection used by MQTT can greatly reduce the overhead of re-connection. It gradually became the network protocol standardized for IoT communication system.

This paper proposes a combination scheme based on MQTT protocol and OneNET cloud platform considering the problems of data transmission, security, management and cost existing in the IoT communication technologies in recent years. The combination of the two models has emerged in the IoT applications recently and this scheme can effectively improve some of the problems encountered in the IoT communication, such as the slow delivery and reception speed of messaging, the real-time capabilities, the update management of message and data, etcetera. The MQTT (Message Query Transmission Transport) is a protocol under ISO / IECPRF 20922, based on the publish-subscribe messaging that uses TCP protocol [1]. For the devices with low hardware performance, the communication performance is in positive quality after joining the message broker. Currently, the IoT communication technologies have entered a new phase.

In the industrial applications of IoT, it is crucial to consider the measurement and control requirements for data type, size, and transmission rate based on big data [2,3,4]. Under such constraints, a targeted transmission protocol becomes essential.

2. Background and Principles

At present, the main domestic IoT cloud service manufacturers, Ali Cloud, Tencent Cloud, Baidu Cloud, OneNET cloud platform and so on, have been able to provide platform to realize MQTT protocol analysis and application. The MQTT protocol can be widely used in the industrial production field. The MQTT is designed as a lightweight, simple, and open transport protocol for easy implementation, based on the TCP / IP protocol [5,6,7,8,9].

Based on the MQTT protocol communication network technology, it can perform on the platform of other network protocols, which makes it quite easier to transmit data. It can be in progress orderly and ensure good bi-directional communication. The main benefit of this technology is the ability to provide authentic message support to remote users with less code and lower bandwidth. The construction principle is shown in Figure 1.

MQTT is a publish-subscribe based messaging protocol via client-server mode [10,11,12]. The technology is applicable in many cases, including in certain restricted environments, such as “machine to machine” communication and the IoT. It was widely used in satellite communication sensors, medical devices, smart-home devices, and certain small devices.

Connect with the OneNET cloud platform via this technology. OneNET, based on the structure of “cloud, network, edge and port”, has popular features including extensive computing, smart memory block loading, and APIs for typical IoT application scenarios. With OneNet, the architecture of IoT project is clear, elements such as the device, API, application, event trigger, and dataflow are organized by customer projects, and the input for every project is designed to be event triggered and derived by preset statements. OneNet is proven to be a cost-effective solution for fast-track IoT application development or concept verification [6,12].

This platform can provide developers with more possibilities to connect all intelligent hardware, and smart home products together, as well as providing comprehensive solutions for intelligent hardware and smart home products in a big data environment. Furthermore, it can also provide an efficient and low-cost hosting service, multi-protocol intelligent analysis, reliability and security in data storage and analysis, and multi-dimensional support, that is, immediacy and sustainability advantages.

2.1. MQTT Communication Principle

MQTT will build the basic network transmission: it creates an orderly, non-destructive, byte-based bi-directional transmission between the client and the server [7]. The MQTT links the quality of service (QoS) to the topic when transmitting the application data.

Implementation requirements for MQTT: clients and servers. The MQTT protocol can be divided into three categories: publishers, brokers, and subscribers. In this case, the publisher and subscription are the client, while the publisher is the server, and the publisher can be the subscriber. The message sent by MQTT is divided into two parts: topic and payload. Topic, which means the type of message, will receive message about the topic after the subscriber orders it. Payload, which could be interpreted as message content, means what will be used specifically by users. The schematic diagram of the MQTT is shown in Figure 2.

The concrete content of the message is included in the payload of MQTT protocol. The message content is well readable in JSON sequence format which easy to be analyze and generated by machines, therefore, it is encoded in JSON format generally. In the IoT environment, different information types are defined to complete various functions, including device registration, acquiring gateway establishment equipment, setting up data collection tasks and instructions, etc. Any application or system, by the MQTT protocol, commonly has a network connection to the server. Clients can publish or subscribe to messages, as well as unsubscribe or delete messages from applications, and can also disconnect from the server. The MQTT server, called the broker, can be an application or a device which between publishers and subscribers. It can accept messages and process related requests.

2.2. MQTT Characteristics

The currently available MQTT protocol is in the version MQTTv3.1.1 released in 2014, aiming to run in networks such as TCP or IP.MQTT' s publish-subscribe model allows for seamless bi-directional communication between devices, and in this model, the client that sends a message (the publisher) is decoupled from the client or clients that receive the messages (or the subscribers), which makes it much easier to decouple from the applications. Moreover, it has a compact header, which only contains 2 bytes, and simple protocol interaction, that minimizes data transmission and protocol exchange, to reduce network traffic. In the case of abnormal interruption, it has a notification mechanism to notify the publisher, combined with Last Will and Testament functions to inform the relevant clients of abnormal interruption.

MQTT has 3 defined QoS (Quality of Service) levels, making it convenient to choose flexibly according to different scenario requirements. Facilitate application according to different scenario requirements. The QoS is a key feature of the MQTT protocol that enables clients to select service levels matching with their network reliability and application logic. Since the MQTT protocol guarantees transmission, even though when the underlying transmission is unreliable, the QoS makes it easier to communicate in unreliable networks.

QoS 0 (up to once): the client publishes the message only once, regardless of whether the message has reached the broker. QoS 1 (at least once): the client publishes the message repetitively until the broker responds to confirm the receipt, which may cause the message to reach the broker multiple times. QoS 2 (exactly once): the client publishes the message once while ensuring that it reaches the broker, and the QoS 2 communication requires higher bandwidth than the QoS 0 and QoS 1.

In the MQTT protocol, one session can contain multiple subscriptions. Each subscription in each session will have a different topic filter. When every client has connected with the server, a state of interaction will occur. Session may only last for one network connection, or if the client can re-establish the connection before the session expires, the session may exist across multiple network connections. When the identifier can be matched with the server connected to, the server will publish messages to all clients that have the matching identifier.

3. Materials and Methods

In the past IoT transmission technology, only a single communication protocol technology was used to realize the data transmission. In this paper, data is transmitted based on MQTT protocol and OneNET Cloud Platform. The scheme utilizes the OneNET Cloud Platform, which leverages the characteristics of MQTT protocol technology and enhances data transmission quality. It facilitates later management and ensures data security. All the above are completely different from the traditional IoT communication technology. The overall structure of the scheme is illustrated in Figure 3.

The hardware part is composed of embedded devices which can be connected to the OneNET Cloud Platform through the MQTT protocol by using AT instructions. AT is the short form of Attention. Each AT command line can contain a single AT instruction. Only one AT instruction can be included per AT command line. The transmission of AT instructions, in addition to “AT” these two characters, can receive up to 1056 characters long (including the last blank character). For the URC instructions or responses actively reported to the PC by the terminal devices, there must also maximize one AT instruction in each AT command line, against multiple instructions or responses in the reported line. AT command terminates with the carriage return, and the response or report process is terminated with a carriage return wrap. Finally, there will be certain feedback on whether each command is executed or completed.

The purpose of applying AT instructions is to simplify the complexity of networking embedded devices. The command is short, easy to understand and can basically complete all the functions of wireless communication technologies (call, SMS, fax, etc.), additionally provides a set of standard hardware interface, and the serial port simplifies the hardware design. Common format of the AT command: AT + CMD = <xxx> [, <xxx>, <xxx>].

4. Results

4.1. MQTT Communication Debugging

Discover the product and service series corresponding to the device through the MQTT protocol, and then connect to the China Mobile OneNET Cloud Platform. After successful connection, the cloud platform would display the information of the device currently used, such as the name of the device manufacturer, device model, etc., and could receive data and send commands. The cloud platform has rich interfaces, which can facilitate communication and upgrade function in the product and service series corresponding to the device through the MQTT protocol, and then connect to the China Mobile OneNET Cloud Platform. After successful connection, the cloud platform would display the information of the device currently used, such as the name of the device manufacturer, device model, etc., and could receive data and send commands. The cloud platform has rich interfaces, which can facilitate communication and upgrade function.

Research manuscripts reporting large datasets that are deposited in a publicly available database should specify where the data have been deposited and provide the relevant accession numbers. If the accession numbers have not yet been obtained at the time of submission, please state that they will be provided during review. They must be provided prior to publication. Intervention studies involving animals or humans, and other studies that require ethical approval, must list the authority that provided approval and the corresponding ethical approval code.

When entering the OneNet Cloud Platform for the first time, it is necessary to create an account, log in, find the corresponding product services, then create the products, add the devices, and then the devices can be connected to the OneNet Cloud Platform. For ease of testing, the software MQTT.fx can be used to simulate the connection to the OneNet Cloud Platform for debugging. What’s more, it is also necessary to input the ID of the product, the device name and key. And then, convert the device key to the token by the token password generation tool is also in need. For each device has different keys, it means login passwords are in differ, thus the security is improved.

The password after conversion is the login password on the software MQTT.fx. The IP address of MQTT is 183.230.40.96 and port number is 1883. After successful login, click Connect, and when the prompt message turns green, it means the login is successful. At this time, the devices can be seen online on the OneNet Cloud Platform, and the devices will be supervised online.

Next, subscribe to the topic and publish the message on the MQTT.fx software side. The format of the published message must be in JSON format. After that, the published messages would be seen on the OneNet cloud platform. The software side shows that the message has been published and subscribed to with a quality-of-service level of QoS0. Use different levels to get messages reach the destination on demand, adapted to the needs of unstable network transmission. OneNet The uploaded real-time data messages can be seen on the cloud platform, achieving the function of receiving data messages in real time.

The cloud platform can also monitor each different message type separately, and record the message upload status of each time, which is easy for data management in the later stage. This part, observing the update and change of data information in real time based on big data visualization analysis[13,14,15], is shown in Figure 4 and Figure 5.

The performance of the MQTT protocol compared with that of HTTP IoT and the TCP message overhead are shown in Table 1.

It could be found that the connection establishment number for MQTT is larger because the MQTT CONNECT packets are exchanged after the TCP connection is completed. The equivalent "overhead" of HTTP must be included in every request, which is the reason why the overhead per message is much bigger. The MQTT disconnect overhead shown here can be 0, that is because it is not required until the TCP connection is closed. Table 2. has tabulated the comparison of response times.

Since more information were sent in a single TCP connection, it would be feasible to reduce the response time, which is because there was no extra cost that would be used to create connections for each message. HTTP does not have such capabilities and some methods, such as caching or connection pooling, can optimize the response speed and overhead, but not to the MQTT level.

These measurements were adapted to MQTT QoS 1, waiting for a response one at a time before the next message. Before receiving the above responses, performance is improved by transmitting subsequent messages (at the cost of some coding complexity), or by adopting QoS 0 not requiring a response (at the cost of some reliability).

It would be possible to gain the additional relative throughput and response time with QoS 0 and QoS 1, the following are the numbers obtained through the local HiveMQ broker using the round-trip scheme, as shown in Table 3.

Compared separately the MQTT’s total traffic consumption is under 1 hour with HTTP’s, and assuming a 5-min active heartbeat interval. This is shown in Figure 6.

4.2. Equipment Debugging

The Embedded module device uses the AT+MQTTCFG command directly to create an MQTT connection, and the product ID, device name and key should be known when using this command. The command AT+MOTTOPEN is to start the MQTT server, and the command AT+MQTTSUB is used to subscribe to a topic, and only after subscribing to the topic successfully, it will publish. The module publishes topics, topics, and messages using the AT + MQTTPUB command, resets the module using the AT+CMRB command, using the AT + CSQ command to query the signal, and using AT + CGPADDR = 1 to check whether the IP is connected (return + CGPADDR: and the IP is connected). The whole process is shown in Figure 8.

Figure 7. Total traffic consumption in one day.

Figure 7. Total traffic consumption in one day.

Figure 8. Connect to the MQTT server.

Figure 8. Connect to the MQTT server.

Connect the module devices to the OneNet Cloud Platform, open the intelligent module serial port assistant of the IoT to connect to the devices, and return some basic information, indicating that the serial port connection is successful. Send the corresponding AT instructions, and the device is connected to the OneNet Cloud Platform, as shown in Figure 9.

After the module devices are connected, it is turned to subscribe to the topic, publish the topic and message. Using the AT + MQTTPUB command, the valve information can be sent to the cloud platform, and the cloud platform can make a corresponding reply after receiving the corresponding information. The whole process is shown in Figure 10.

Deliver the ValvePos command from the cloud platform, set the valve action and delay time to 30s. During the delay time, the module terminal needs to complete the data receiving, and perform corresponding operations on the valve, as well as provide information feedback to the cloud platform. The process is shown in Figure 11.

Figure 11. The module equipment reply.

Finally, the MQTT protocol is used to achieve the information distribution effect of one-to-many and two-way transmission. Combined with the use of the OneNet Cloud Platform, it is much easier to manage data, improves data transmission efficiency, and reduces costs. The Publish-Subscribe model simplifies application development and facilitates the transmission of messages. What’s more, the QoS support ensures reliable transmission and improves message service quality.

5. Discussion

A significant revelation of our research is the pressing need for enhanced network bandwidth and data processing capabilities. In the big data era, the sheer volume of data produced by a myriad of IoT devices necessitates a system capable of handling this influx efficiently. This requirement extends beyond mere data handling; it calls for scalable and maintainable IoT infrastructure that can adapt to evolving technological demands. Addressing the compatibility issue is also paramount. To achieve a certain degree of specialization, it is often necessary to utilize equipment from various manufacturers, each choosing their own technical standards, operating systems, and protocols. This diversity in technology and specialization can create conflicts, posing a formidable challenge in ensuring seamless interoperability among different IoT components. The need to harmonize these diverse elements is critical for the cohesive functioning of IoT systems.

Furthermore, our study highlighted a noteworthy concern regarding MQTT - the potential escalation in bandwidth consumption and latency issues, especially at higher quality levels. This observation is critical for IoT applications where real-time data transmission and processing are paramount. Looking ahead, several avenues for future research have become evident. First, addressing the bandwidth and message capacity constraints is crucial. This can be achieved by developing more robust infrastructure capable of handling increased data loads without compromising on speed or efficiency.

Enhancing the security framework for MQTT is another critical area. Implementing stringent encryption protocols and complex authentication mechanisms can significantly bolster the security aspect, which is paramount in the IoT domain. Additionally, broadening the support for diverse application scenarios and pushing towards more standardized practices in IoT communications will not only streamline operations but also foster a more cohesive IoT ecosystem.

6. Conclusions

This paper introduces a groundbreaking integration of the MQTT protocol with the OneNet Cloud Platform within a big data environment, significantly enhancing IoT communication technologies. Our study delved into the intricacies of IoT communications facilitated by this integration, demonstrating how it ensures high-speed and stable data exchange, and greatly simplifies the management of the voluminous data generated by IoT devices. Importantly, this approach also establishes robust security measures, addressing a critical concern in today’s IoT applications.

These advancements offer considerable benefits for IoT applications, particularly in environments requiring reliable and efficient communication. However, our findings also highlight certain areas that need further attention, underscoring the robust potential of this integration while pointing out its limitations. For instance, while we have made strides in data management and security, there remain challenges in ensuring the sustainability and relevance of IoT systems in the rapidly evolving technological landscape.

In conclusion, the integration of MQTT with the OneNet Cloud Platform in a big data environment presents a promising frontier for IoT communications. Future work should focus on addressing the identified limitations, exploring further optimizations, and investigating potential applications in diverse IoT domains. By doing so, we can enhance not only the efficiency and security of IoT systems but also ensure their long-term viability and adaptability to future technological advancements.