This project is a core component of the cp_wearable_ai ecosystem.
|
|
Information retention and the cognitive load of tracking professional interactions represent significant barriers to productivity. This paper introduces a necklace-form wearable assistant designed to streamline task management through intelligent conversation capture and context analysis. Utilizing a privacy-centric edge-computing architecture, the device transmits audio to a localized Raspberry Pi server for secure processing. Advanced speech-to-text and small language models synthesize raw transcriptions into structured, context-aware summaries and personalized journals. A core contribution is the automated extraction of "action items" from natural dialogue, which the system uses to notify users and facilitate task execution via a mobile application. Technical evaluations demonstrate a 90% transcription accuracy and optimized power management, achieving an 9.52-hour operational battery life. By automating the transition from verbal commitments to digital tasks, the system reduces the cognitive effort required for recall and interaction review, transforming ambient conversation into organized, actionable productivity insights.
Our goal is to provide a seamless "second brain" that lives as your necklace. By integrating continuous audio capture, the device doesn't just listen—it understands the physical and social context of your day to provide proactive support.
The system firmware is built on FreeRTOS, utilizing binary semaphores to synchronize high-priority hardware tasks and ensure reliable real-time performance.
🔄 Execution Flow
- Initialization: Upon boot, the device configures the digital microphone, SD card interface, and wireless (Wi-Fi/BLE) peripherals.
- Transmission Priority: The system checks the SD card for buffered audio. It attempts to upload files to the central server via Wi-Fi.
- Data Integrity: Files are only purged from local storage after a successful server acknowledgment is received.
- Interrupt Monitoring: The system listens for user input via the tactile button array.
- Configuration Mode: Triggered via button interrupt; allows settings updates from the mobile app via BLE.
- Recording Mode: The default state if no interrupt is detected.
🎙️ Audio Recording Specifications To balance audio clarity with low-power transmission, the device uses the following encoding profile:
| Parameter | Specification |
|---|---|
| Interval | 300 seconds (5 minutes) — Subject to optimization |
| Sample Rate | 16 kHz |
| Channels | Mono |
| Format | AAC |
| Bitrate | 32,000 bps (32 kbps) |
- Low-Power Design: Optimized hardware for all-day wearable use.
- Actionable Insights: Converts hours of conversation into high-value summaries and tasks.
🧠 Processing & Connectivity
- MCU: ESP32-S3 (Xtensa® 32-bit LX7 dual-core processor).
- Memory: 8MB PSRAM and 8MB Flash memory.
- Wireless: Dual-mode support for Wi-Fi and Bluetooth Low Energy (BLE) for seamless data transmission to the Raspberry Pi server.
⚡ Power & Interface
- Battery: 400 mAh Lithium Polymer (LiPo) battery, optimized for all-day efficiency.
- User Input: A tactile button array for manual wireless connectivity toggles and manual recording triggers.
The wearable software is built using the Espressif IoT Development Framework (ESP-IDF), ensuring optimized performance for the ESP32-S3. The system leverages several specialized libraries to manage audio, storage, and networking:
- FatFS: Manages file system operations for reading from and writing audio recordings to the microSD card.
- SNTP (Simple Network Time Protocol): Synchronizes timestamps via the network to ensure accurate, time-stamped file naming for all recordings.
- ESP Audio Codec: Handles the real-time conversion and encoding of Pulse Density Modulation (PDM) data from the digital microphone into Advanced Audio Coding (AAC) format.
- lwIP (Lightweight IP): Manages the TCP/IP stack for robust network connectivity.
- ESP HTTP Client: Facilitates secure and efficient communication with the central Raspberry Pi server.
- Bluedroid: Powers the Bluetooth peripheral stack, enabling the device to communicate with the mobile application for configuration.
- Framework: ESP-IDF (v5.x recommended)
- RTOS: FreeRTOS (Integrated with ESP-IDF)
- Language: C/C++
- A configured Raspberry Pi server (see Server Setup Guide).
- Compatible wearable hardware.
- ESP-IDF v5.x
git clone https://github.com/osugiw/CP_MCU_Codes.git
cd [your-repo-name]Some configuration needs to be changed before the software can run. Change the WiFi SSID and Password to your network, it can be changed in main.h
Make sure to choose ESP32S3 as the target device and click build project.
Upon successful built, plug your Hardware and click flash device.
Click monitor to see device output and see the debug messages.
- Sugiarto Wibowo (sw183@rice.edu)* — Lead Developer
- Jeeven Balasubramaniam (jb310@rice.edu)** — Collaborator
- Nakul Garg (ng83@rice.edu)* — Project Supervisor
Affiliation: Rice University, Houston, Texas, USA
* Electrical and Computer Engineering Department
** Data Science Department