A Multi-Camera Wearable Assistive System for Environmental Awareness in Visually Impaired Users Using Mobile Vision and Real-Time Feedback

1. Introduction

Visually impaired individuals face significant challenges in navigation, especially in unfamiliar or cluttered environments. According to the World Health Organization, approximately 2.2 billion people worldwide live with vision impairment, with mobility and environmental awareness representing fundamental barriers to independence [1]. Existing tools such as white canes are essential but generally restricted to detecting obstacles at ground level via direct contact. They cannot effectively detect “hanging” obstacles (such as tree branches or signage) or sudden drop-offs without physical sweeping [2].

Meanwhile, advancements in computer vision and efficient neural networks (e.g., MobileNet [3], YOLO series [4,5]) have made it possible to run real-time perception tasks on small, portable devices. The goal of this research is to design a multi-camera wearable system that enhances spatial awareness while remaining low-cost, non-intrusive, and easy to use [6,7]. By leveraging modern mobile processors and edge computing capabilities, this system aims to bridge the gap between traditional tactile aids and advanced digital perception, while preserving user privacy through on-device processing [8].

2. Background and Related Work

2.1. Conventional Mobility Aids

Traditional mobility aids include:

• White Canes: Reliable for ground-level texture and immediate obstacles but lack vertical coverage and cannot detect overhead hazards [9].
• Guide Dogs: Highly effective for navigation and safety but are expensive (typically $40,000-60,000 USD), require extensive training (18-24 months), and are limited in availability with only about 10,000 guide dogs serving millions of visually impaired people in the United States alone [10].

2.2. Vision-Based Assistive Technology

Previous solutions have explored smart glasses, ultrasonic sensors, and smartphone applications. Elmannai and Elleithy [11] provided a comprehensive survey of sensor-based assistive devices, highlighting that many current implementations suffer from:

• High Cost: Specialized hardware such as OrCam MyEye ($4,500) and Envision Glasses ($3,500) are often prohibitively expensive for widespread adoption [12].
• Bulkiness: Headsets or large sensor arrays can be socially obtrusive, leading to user reluctance and privacy concerns from bystanders [13,14].
• Computation Heavy: Many systems require connection to cloud servers, introducing latency (typically 200-500ms) and raising privacy concerns regarding continuous video streaming [15].

Recent reviews by Wang et al. [16] and Kuriakose et al. [17] emphasized the importance of form factor in wearable assistive devices, noting that body-centered and discrete embodiments show higher user acceptance compared to head-mounted systems.

2.3. Gaps in Current Research

Current research highlights specific gaps that this paper addresses [18,19]:

• Limited Detection of Overhead Obstacles: Most camera-based systems focus on forward-facing or ground-level detection, missing hazards at head height.
• Insufficient Detection of Negative Obstacles: Steps, holes, curbs, and drop-offs represent significant fall risks but are poorly addressed by current systems [20].
• Form Factor Issues: User studies indicate preferences for discrete wearables over glasses or headsets due to social acceptance and comfort during extended use [21].

2.4. Mobile AI Advancements

Recent developments in mobile artificial intelligence have enabled sophisticated on-device processing:

• Lightweight Neural Networks: Howard et al. [3] introduced MobileNets with depthwise separable convolutions, achieving efficient real-time object detection on smartphones and embedded devices with minimal computational overhead.
• Real-Time Object Detection: The YOLO series [4,5] enables single-shot detection with high accuracy. Recent work by Khurana et al. [22] demonstrated successful integration of MobileNet and YOLO for resource-constrained devices, achieving mean Average Precision (mAP) of 0.61 on COCO dataset.
• Edge Computing: AI runs directly on mobile devices, reducing latency to under 50ms and preserving privacy without relying on cloud processing [23].
• Depth Estimation: Mobile AI can predict distances from single camera streams using monocular depth estimation techniques, improving obstacle awareness [24].
• Energy-Efficient Inference: Quantized and pruned models minimize power consumption to 500-800mW, allowing 6-8 hours of continuous operation on typical smartphone batteries [25].

3. Proposed System

The proposed system is a modular architecture designed to offer flexibility and affordability. It utilizes a central processing unit (smartphone or microcontroller) connected to distinct camera modules to cover the user’s blind spots, employing state-of-the-art mobile AI techniques for real-time obstacle detection and classification.

3.1. Hardware Architecture

The hardware design focuses on off-the-shelf components to ensure affordability (target cost under $150) and replaceability:

• Camera Module A (Forward Camera): A wide-angle sensor (120° field of view) mounted on a smartphone (chest pocket) or a discrete clip-on module. This covers eye-level and mid-level obstacles up to 5 meters distance.
• Camera Module B (Downward Camera): A specialized micro-camera (e.g., ESP32-CAM or similar, ~$10) mounted on the shoe or ankle. This is critical for surface-level detection (puddles, curbs, stairs) within 2 meters.
• Processing Unit: Computing is handled via a standard Android smartphone (Snapdragon 6-series or higher) or a Raspberry Pi Zero 2W (~$15) for a standalone variant.
• Feedback Interface: Bone-conduction headphones (e.g., AfterShokz, ~$80) provide audio cues without blocking ambient environmental sounds, ensuring the user remains aware of traffic and other auditory signals critical for safety [26].

3.2. Software Architecture

The software pipeline is designed for low latency (<100ms end-to-end). The detailed pipeline includes:

• Image Acquisition: Captures video streams at 15-60 FPS depending on lighting conditions and processing load.
• Image Preprocessing: Includes resizing to model input dimensions (typically 320x320 or 416x416), histogram equalization for light correction, and normalization.
• Object Detection and Classification: Utilizes lightweight models like YOLOv8-Nano [5] or MobileNet SSD [3,27] optimized for mobile GPUs using TensorFlow Lite or ONNX Runtime.
• Depth Estimation: Uses monocular depth prediction networks (e.g., MiDaS [24]) to calculate the distance of obstacles in real-time.
• Object Prioritization Module: A logic layer that ranks obstacles based on immediate risk using time-to-collision metrics (e.g., a fast-approaching wall at 1.5 seconds is higher priority than a distant bench at 10 seconds).
• Feedback Module: Converts the highest priority threat into spatial audio (indicating direction using stereo panning) or haptic vibration patterns (frequency indicates urgency, pattern indicates obstacle type).

3.3. Wearable Integration and Novel Design Features

• Shoe-Mounted Advantage: The unique placement on the shoe allows for the detection of “negative obstacles” (drop-offs, stairs, curbs) and low-hanging trip hazards that chest-mounted cameras often miss due to occlusion by the user’s own body. This approach was inspired by terrain analysis systems in robotics [28].
• Clip-on Flexibility: Small camera modules can be rotated or clipped to bags/belts, accommodating different user heights and preferences, addressing form factor concerns identified in user acceptance studies [17,21].

4. Methods

4.1. Experimental Design

The system will be evaluated in controlled test scenarios and real-world environments:

• Indoor Environments: Hallways with varying light conditions (50-1000 lux), cluttered spaces with furniture, and doorways.
• Outdoor Environments: Sidewalks with curbs, pedestrian crossings, parks with natural obstacles, and urban streets with dynamic elements.
• Surface Variations: Concrete, asphalt, grass, tile, and gravel to ensure the shoe camera handles texture variance and surface discontinuities.
• Hazard Scenarios: Controlled tests with overhead signage (1.8-2.2m height), simulated potholes (10-30cm depth), and temporary obstacles.

4.2. Evaluation Metrics

Following established benchmarks in assistive technology research [29,30], the system will be evaluated using:

• Mean Average Precision (mAP): For detection accuracy across different obstacle types, computed at IoU thresholds of 0.5 and 0.75.
• False Positive Rate (FPR): To ensure users are not annoyed by phantom alerts (target <5% FPR).
• False Negative Rate (FNR): Critical safety metric measuring missed obstacles (target <2% FNR for critical hazards).
• System Latency: Time from obstacle detection to user feedback, measured in milliseconds (target <100ms).
• User Confidence Score: Self-reported confidence levels (1-10 scale) before and after system use, using standardized questionnaires.

4.3. Dataset Selection and Training

Training and evaluation will utilize the following publicly available datasets:

• COCO Dataset: For general object detection training with 80 object categories and 200,000+ labeled images [31].
• Cityscapes: For outdoor urban navigation scenarios with 5,000 high-quality annotated images from 50 cities [32].
• NYU Depth Dataset V2: For indoor depth estimation with 1,449 densely labeled RGBD images [33].
• Custom Dataset: Shoe-level perspective images capturing negative obstacles, curbs, and stairs from below will need to be collected and annotated (target: 2,000+ images across diverse conditions).

Transfer learning will be employed, starting with models pre-trained on COCO, then fine-tuning on domain-specific data including the custom shoe-level dataset. Data augmentation techniques (rotation, brightness adjustment, synthetic fog/rain) will improve robustness.

4.4. Ethical Considerations

Note: All human subject research will require proper institutional review board (IRB) approval before commencement. The study will adhere to the Declaration of Helsinki [34] and obtain informed consent from all participants. Special considerations include:

• Participant safety during navigation tests with immediate intervention protocols
• Privacy protection through on-device processing with no data storage or transmission
• Accessibility of consent materials in Braille and audio formats
• Right to withdraw at any time without penalty

5. Discussion

5.1. Performance Considerations

Preliminary simulations suggest the system will be highly effective in well-lit environments (>300 lux) but may struggle in low-light conditions (<50 lux) without auxiliary infrared (IR) illumination. Future iterations will explore low-cost IR LED arrays (~$5) and low-light enhancement algorithms based on recent advances in computational photography [35].

5.2. Privacy and Data Security

The use of cameras in public raises significant privacy concerns, as documented by Denning et al. [13]. The proposed system is designed to process all data locally (“on the edge”) without saving video feeds or transmitting data to cloud servers. All processing occurs on-device with a maximum buffer of 2 seconds (automatically deleted), and no personally identifiable information is collected, stored, or transmitted. This approach directly addresses privacy concerns while maintaining system functionality [36].

5.3. User Experience and Feedback Preferences

Initial informal testing with visually impaired consultants indicates that haptic feedback is preferred for immediate dangers (vibration intensity proportional to urgency), while audio is preferred for navigation information and object identification. The system allows users to customize their feedback preferences through voice commands, balancing information delivery with cognitive load management [37].

5.4. Limitations and Future Directions

Several limitations must be acknowledged:

• Weather Dependency: Heavy rain, snow, or fog significantly degrade camera performance. Weather-resistant enclosures and sensor fusion with ultrasonic sensors may mitigate this limitation.
• Battery Life: Continuous operation at high frame rates drains batteries in 3-5 hours. Adaptive frame rate adjustment and sleep modes during stationary periods can extend usage to 8-10 hours.
• Training Data: The custom shoe-level dataset requires substantial effort to collect and annotate. Synthetic data generation using 3D rendering engines may accelerate this process [38].
• User Acceptance: Long-term user studies are needed to assess real-world adoption, usability, and integration with existing mobility strategies [39].

6. Conclusion

The multi-camera wearable assistive system proposes a practical and potentially transformative solution for enhanced navigation for visually impaired individuals. By combining the processing power of modern smartphones with the unique vantage point of shoe-mounted sensors and state-of-the-art mobile AI techniques, this system addresses critical gaps in current assistive technology. The modular design ensures affordability (estimated $100-150), while edge computing preserves privacy and reduces latency. As hardware becomes more efficient and AI models continue to advance, such systems can become widely available, affordable, and essential tools for independence. Future work will focus on real-world user trials, weather robustness, and integration with existing assistive technologies to create a comprehensive mobility solution.