This study aims to construct a lightweight action recognition pipeline for work environments that does not rely on detailed manual labeling to achieve both the reduction of manual labeling load in the offline stage and the lightweight recognition of user actions in the online stage. The proposed method generates pseudo-labels from accelerometer data using time-series clustering. Extending the labeling scheme to other subjects, it integrates data from multiple subjects into a common set of action classes. Furthermore, it trains a lightweight action recognition model using the obtained pseudo-labels, which enables us to evaluate the trade-off between model complexity and recognition performance when the method is deployed on edge devices. The process verifies not only the transferability of action structures using pseudo-labels, but also the applicability of lightweight models to sequential action recognition.