PUSHPA – Predictive UAV & Smart Headset for Pyro Alerting

1. Introduction

PUSHPA (Predictive UAV and Smart Headset for Proactive Assistance) is a multi-component system designed to provide real-time situational awareness and automated decision-making for critical tasks, such as firefighting, disaster response, and industrial inspections. By integrating autonomous drones, wearable devices (Meta Glasses), a LangChain-powered AI agent, and chat-based control interfaces, PUSHPA enables users to give high-level commands in natural language and receive actionable insights with minimal latency.

Existing research and commercial solutions have shown the benefits of AI-driven autonomy in drones [1,2], as well as the potential of edge computing on wearables for real-time scene analysis [3,4]. However, bridging these components in a single system—with robust communications, security, and ease of use—remains challenging. PUSHPA addresses this gap by combining state-of-the-art embedded vision techniques, secure messaging APIs, and an agentic AI layer to interpret commands and orchestrate tasks autonomously.

In this paper, we provide:

• A deep technical overview of the Meta Glasses edge compute pipeline, including hardware specs and on-device AI models.
• Details on autonomous drone control, covering real-time decision-making loops, and obstacle avoidance.
• Real-life examples demonstrating the performance gains of convolution-based optimizations and flexible mission handling via an LLM-based planner (LangChain).
• WhatsApp bot integration via a PHP webhook, enabling conversational commands for the drone and glasses.
• A look at the UI integration built with Next.js and TypeScript for real-time telemetry, mapping, and AR overlays.
• The codebase structure, highlighting key functions, security best practices, and example modules in PHP, Node.js, and TypeScript.

2. Meta Glasses Edge Compute

2.1. Hardware Specifications and Processing Capabilities

Our Meta Glasses use a Qualcomm Snapdragon AR1 Gen1 processor, offering a multi-core CPU, Adreno GPU, and a dedicated AI accelerator. This chipset can perform several billion operations per second, accommodating real-time CNN inference on-device. The glasses have dual 12 MP cameras capturing up to 30 FPS, enabling continuous object detection, OCR, and face detection without offloading to external servers.

2.2. AI Models Running On-Device

We deploy lightweight CNNs such as MobileNet-SSD and a slim OCR pipeline (EAST + CRNN) for real-time inference. By quantizing these models and leveraging depthwise convolutions, we achieve near 30 FPS processing on the AR1, allowing immediate feedback to the wearer. The glasses also integrate a simple face detection model for safety-critical alerts.

3. Real-Time On-Device Edge Compute

Efficient on-device processing is critical for delivering live insights during emergencies, where connectivity to cloud resources may be intermittent or unavailable. In PUSHPA, we implement a 7x optimized convolution pipeline tailored for edge hardware to ensure real-time performance under extreme conditions (e.g., firefighting environments). Our approach leverages several key optimizations:

3.1. Hybrid Convolution

At the core of our optimization is a dual-mode Hybrid Convolution architecture:

• Direct Convolution Path: Used for general kernel sizes (e.g., $1\times 1,5\times 5,7\times 7,$ etc.). This approach follows a straightforward, loop-based method to account for irregular kernel shapes.
• Winograd F(2,3): For the common $3\times 3$ kernel, we embed a specialized Winograd transform. Winograd convolution reduces the number of multiplications by transforming both the kernel and input patches into a smaller domain and then applying an inverse transform post-convolution.

By dynamically selecting between direct or Winograd-based convolution based on the kernel size, we reduce the total number of operations without sacrificing accuracy.

3.2. Blocking and Loop Unrolling

To further improve cache utilization and throughput, we employ:

• 256-Sized Blocks: For channels $\ge 1024$, convolution loops are split into blocks of 256 channels at a time. This ensures that frequently accessed data remains in higher-level caches, minimizing cache misses.
• Partial Loop Unrolling: Inner loops are partially unrolled in chunks of four, reducing the overhead of loop increments and bounds checking.

By batching large channel counts into smaller blocks, we optimize memory locality and reduce redundant instructions, thereby increasing instruction-level parallelism.

3.3. Parallelization

We leverage modern CPU parallel frameworks to distribute workloads across multiple cores:

• at::parallel_for: This function from PyTorch’s backend (ATen) parallelizes operations along the batch and channel dimensions.
• OpenMP Directives: Where appropriate, we insert #pragma omp parallel for to split loops into independent chunks that can be scheduled across CPU threads.

This thread-level parallelism harnesses the full processing power of multicore edge devices, boosting throughput during convolution steps.

3.4. Fused Winograd

A distinctive feature of our pipeline is Fused Winograd, which directly integrates transformation steps into the convolution routine:

• Transform matrices (input and kernel) and the inverse transform are applied in-place, avoiding extra buffer copies or separate memory passes.
• This is especially beneficial for $3\times 3$ kernels, where Winograd’s $F(2,3)$ method offers significant reduction in multiplications with minimal overhead.

By merging these transforms into a single pass, we eliminate unnecessary read/write cycles, leading to faster inference on edge hardware.

3.5. Example Convolution Kernel Pseudocode

3.6. Performance Gains

Empirically, this pipeline achieves up to a 7x increase in throughput on edge devices compared to a naive convolution baseline. In a typical scenario with $K=1024$ output channels and $3\times 3$ kernels, fused Winograd and blocking result in minimal overhead, significantly reducing multiply-accumulate counts and memory transfers. Such improvements are crucial for real-time applications like firefighting or rescue missions, where immediate local inference ensures that life-saving decisions are not delayed by unreliable network connections.

3.7. Image and Video Processing Workflow

• Capture: Frames are read from the glasses’ cameras at 30 FPS.
• Preprocessing: Each frame is resized and normalized for inference (e.g., 300×300 for MobileNet-SSD).
• Inference: The pipeline runs inference on the AI accelerator. Bounding boxes and OCR text regions are produced.
• Post-processing: Non-max suppression is applied; recognized text is transcribed.
• User Feedback: Results are either spoken to the user via the glasses’ audio channel or used by the AI agent to make decisions (e.g., “Obstacle 5m ahead!”).

A simplified on-device loop in Python-style pseudocode:

4. Autonomous Drone Control

4.1. Communication Protocols for Control and Telemetry

PUSHPA creates reliable communication between the drone flight controller and off-board control logic by using VLMs. Telemetry (orientation, GPS, battery) is transmitted at high rate; the system issues high-level commands (e.g., SET_POSITION_TARGET) back to the drone. Video is streamed via RTSP/HTTP for the UI.

4.2. Real-Time Decision-Making Logic

The drone handles low-level stabilization via onboard firmware (PX4 or ArduPilot). A higher-level AI planner—part of the LangChain agent—monitors sensor data and mission goals to decide the next action, e.g., “advance waypoint,” “hover,” or “take a closer look.” This sense-plan-act loop runs at ∼10 Hz for high-level planning, while the flight controller updates at 100 Hz for stability.

Figure 1. Conceptual overview of PUSHPA’s interconnected modules. Each node represents a core subsystem or functionality that communicates with adjacent modules to enable real-time decision-making and AR-based guidance.

Figure 1. Conceptual overview of PUSHPA’s interconnected modules. Each node represents a core subsystem or functionality that communicates with adjacent modules to enable real-time decision-making and AR-based guidance.

4.3. Obstacle Avoidance Algorithms

We employ stereo vision and ultrasonic sensors. A depth CNN generates a short-range obstacle map, triggering real-time rerouting if hazards appear. The drone can use potential fields or A* search in an occupancy grid, providing local path re-planning. Rapid reactive halts prevent collisions if a sudden obstacle is detected within 1–2 m.

5. Real-Life Research Examples

5.1. Case Studies of AI-Driven Drone Automation

AI-based drones have proven effective in agriculture for crop health inspection [2] and in warehouse logistics for inventory checks [5]. We extend these capabilities with a high-level reasoning agent. PUSHPA’s drone can autonomously scan a building for hazards, guided by user commands through a chat interface, or by pre-programmed rescue protocols.

5.2. Benchmarks of Convolution-Based Optimizations

By quantizing YOLO-style detection to 8-bit and pruning less active filters, we achieved a ∼2× speedup on embedded hardware with minimal accuracy loss. For example, MobileNet-SSD inference time dropped from 100 ms to 50 ms per frame on a single-board computer, enabling near real-time object detection at 20+ FPS.

5.3. Performance Improvements with LangChain-Powered Agentic AI

A novel element of PUSHPA is the LangChain-based agent that transforms user commands into orchestrated tasks. In multi-step missions, success rates improved from 60% (rule-based) to 90% (agent-based), and average task time dropped by 25% due to parallelizing sub-tasks (e.g., the drone starts searching while the glasses confirm environment safety).

6. WhatsApp Bot API Integration

6.1. PHP Webhook Handler for WhatsApp Messages

We register a webhook with Meta’s WhatsApp Cloud API, enabling POST requests containing user messages. Below is a condensed example of webhook.php:

6.2. Automated Command Generation via LangChain

The Node.js service runs a LangChain agent that interprets messages from the webhook. If the user says, “Scan the area and find my car,” the agent might:

• Break down the request into actions (take off, search pattern).
• Use the drone camera feed + object detection to find a car.
• Return a structured JSON response or direct textual explanation to WhatsApp once found.

6.3. API Calls for Processing Text and Images

Images sent via WhatsApp are downloaded in the PHP layer, then forwarded to the Node microservice for AI-based analysis (object detection, OCR). Conversely, after the drone captures a photo, we upload it to WhatsApp’s media endpoint, retrieve the media ID, and send it back to the user.

7. UI Integration

7.1. Data Pipeline for Live Drone and Glasses Data

A Node.js WebSocket server streams telemetry to a Next.js frontend in real time. The UI merges:

• Drone feed: Low-latency HLS/RTSP.
• Glasses events: Object detections or user triggers.
• LangChain agent outputs: Summaries or recommended actions.

React components update dynamically, e.g. plotting the drone’s GPS path or overlaying bounding boxes on the video.

7.2. Frameworks and APIs Used

We use:

• Next.js + TypeScript: SPA architecture with server-side rendering.
• Radix UI for accessible and customizable UI primitives.
• Map/Chart libs (Mapbox, Chart.js) for geolocation and sensor trends.
• WebSockets (Socket.IO) for real-time telemetry.

7.3. Frontend Bookmarklet for Messenger Chat Monitoring

A small bookmarklet.js snippet can be injected into Facebook Messenger. It captures specific commands from the chat UI (e.g., “/drone takeoff”), then sends them to our Node API. The system replies by injecting messages back into the DOM. While a hack, it demonstrates flexible cross-platform integration.

Figure 2. PUSHPA’s integrated UI, which provides live telemetry from drones and Meta Glasses, overlays AI-driven insights, and enables real-time user interaction through a streamlined interface.

Figure 2. PUSHPA’s integrated UI, which provides live telemetry from drones and Meta Glasses, overlays AI-driven insights, and enables real-time user interaction through a streamlined interface.

8. Codebase Structure

The repository is divided into modules:

• meta/ Includes webhook.php, handling WhatsApp verification and message callbacks.
• glasses-integration/ A TypeScript/Node service for receiving images from the Meta Glasses, interfacing with GPT-4 Vision, and storing results.
• frontend/ A Next.js project with TypeScript, Radix UI, and the main web dashboard. Contains bookmarklet.js for Messenger.

8.1. Key Functions and Their Roles

• getGPTResponse() (Node): Accepts user input, calls OpenAI, returns either text or an action command.
• processImageFromGlasses() (TypeScript/Python): Conducts object detection/OCR on images from the glasses.
• droneController.ts (Node): Wraps MAVLink or drone SDK calls for commands like takeoff(), gotoLocation(), land().
• webhook.php (PHP): Verifies WhatsApp requests, extracts media, and dispatches them to the Node layer.

8.2. Security Best Practices

• API Key Management: All tokens (OpenAI, WhatsApp, etc.) in .env files, not committed to git.
• Webhook Verification: Checking hub.verify_token for WhatsApp calls.
• CORS and Auth: Strict domain checks, login-protected UI, restricted drone commands.
• Input Validation: Sanitizing user inputs to avoid malicious commands (LLM prompt injection).

9. Example: Natural Language Query and Drone Autonomy

9.1. Forest Counting Scenario

To demonstrate the synergy of edge computing, drone control, and chat-based commands, consider the following real-life scenario:

• The user speaks or texts to the Meta Glasses: “Can you count the number of forests in the surrounding areas?”
• The glasses have limited CPU, so they forward the query to the WhatsApp bot, effectively bridging voice input from the glasses to WhatsApp.
• The webhook.php in meta/ receives the user’s text, “Count the number of forests,” and calls getGPTResponse() in Node.
• The LangChain agent decides that satellite/drone images are needed. It composes a command: {“action”: “DRONE_SURVEY”, “survey_area": “surroundings"}
• Our Node service calls droneController.ts to plan an autonomous flight path. The drone takes off and uses its camera feed to capture wide-area images.
• The localized model (a Vision-Language Model, VLM) on an edge compute device processes these images to identify forested regions. The agent might instruct the drone to adjust altitude or location for better vantage.
• Once enough data is collected, the system aggregates the results and calculates the approximate forest count (or coverage area).
• The agent returns a concise explanation to the user via WhatsApp: “There are three main forested areas within a 2 km radius. The largest is approximately 0.8 km².”
• Simultaneously, the user’s Meta Glasses beep with an AR overlay indicating forest boundaries if the user’s vantage point includes any visible region.

This scenario showcases how a high-level request (“count forests”) triggers multiple system components: glasses (input), WhatsApp (API messaging), AI agent (planning logic), drone autonomy (survey flight), and localized VLM for analysis. The response is seamlessly sent back to the user in natural language.

9.2. Automated Commands and Example “Hacks”

The system uses a backdoor approach via WhatsApp to let the glasses effectively call the AI agent. We can also embed short “sample comments” in code so the drone adjusts flight parameters automatically:

When the user’s text triggers this logic, it returns a JSON describing how the drone should move. The Node agent then executes these commands via droneController.ts, employing MAVLink commands to direct flight. Once the images are captured and processed, the final answer is relayed back through WhatsApp to the glasses.

10. Conclusion

PUSHPA integrates wearable edge computing, autonomous drones, and an agentic AI layer for real-time, user-friendly mission execution. By leveraging on-device CNNs in Meta Glasses, the system achieves robust perception even with limited connectivity. The drone’s autonomous flight, guided by MAVLink and obstacle avoidance algorithms, seamlessly executes complex tasks, while a LangChain-powered agent orchestrates multi-step plans. A WhatsApp chatbot and Next.js UI further simplify interactions, enabling intuitive command-and-monitor workflows.

Through real-life examples, we demonstrated how conversational requests can spawn intricate drone tasks, aided by localized AI inference and secure messaging. Our modular codebase, built in PHP, TypeScript, and Node.js, reflects best practices in security and performance, paving the way for broader adoption. PUSHPA’s design can be extended beyond firefighting and disaster response to industrial inspection, agriculture, and public safety operations. Future work will explore deeper integration of large-scale vision-language models on the edge and advanced AR overlays for the glasses, further pushing the boundaries of human-AI collaboration.