publications

2026

1. Under review

   

   Augmenting Human Performance with an XR Agent Learning from Online Behavior and BCI Evidence

   Ziheng Li, Xichen He, Haoyan Chen, Charlie Zou, Sheng Bai, Benjamin Yang, Mengyuan Wu, Jake Ledner, Yi-Jie Cheng, Akito Yamauchi, Dishita G. Turakhia, Steven Feiner, and Paul Sajda

   In ACM Symposium on User Interface Software and Technology (UIST), 2026

   Abs

   We present OLIVE, a framework for adapting a foundation model to provide real-time assistance in temporally demanding, high-stakes and dynamic tasks. OLIVE learns from both explicit behavioral signals (user actions) and implicit physiological signals (fixation-locked EEG) to provide timely guidance to supplement the user’s perceptual and action bandwidth. By fusing these complementary channels, OLIVE continuously adapts a frozen vision-language model’s belief about which items are currently task-relevant, jointly estimating per-source reliability without manual labels or offline retraining. Through three user studies, where we deploy an assistive agent driven by OLIVE in XR, we show that OLIVE outperforms prior test-time adaptation frameworks in both convergence time and rate. Compared to learning from explicit behaviors alone, the agent combining implicit and explicit learning signals improves user performance, achieving 45% more target coverage. When targets shift silently mid-block, the combined agent re-adapts 2.1x faster than the behavioral-only baseline. These results show OLIVE can be built toward empowering adaptive co-pilot systems that extend human performance under demanding operational conditions.

2. Under review

   

   A Messaging Toolkit for Spatial Hierarchical Routing in Interactive Applications

   Benjamin Yang, Shanying Liu, Charlie Zou, Xichen He, Carmine Elvezio, and Steven Feiner

   In ACM Symposium on User Interface Software and Technology (UIST), 2026

   Abs

   Interactive applications built on hierarchical scene graphs often rely on complex inter-component communication. Many frameworks for developing these applications support message communication up and down the parent–child hierarchy, but require custom code to define non-hierarchical communication graphs. For example, this includes sending messages between objects based on their spatial proximity, independent of the scene-graph hierarchy. Furthermore, limited prior work has empirically evaluated whether frameworks that support non-hierarchical message routing improve developer productivity over custom code. We present Cascade, a messaging toolkit for hierarchical scene graphs that integrates composable spatial filtering primitives (proximity, direction, bounding region) directly into message routing. Unlike previous work, it enables developers to create non-hierarchical communication graphs that express complex routing patterns in a single chained expression. We evaluate Cascade through a controlled within-subjects experiment comparing task completion time, code correctness, cognitive load, and usability for Cascade and event-based messaging across tasks in robot teleoperation and Internet of Things (IoT) domains.

3. Under review

   

   Delegate the Details: Flexible Assembly Constraint Authoring for Collaborative XR

   Benjamin Yang, Xichen He, Charlie Zou, Jen-Shuo Liu, Barbara Tversky, and Steven Feiner

   In IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2026

   Abs

   Extended reality (XR) interfaces for assembly specification typically require users to specify the 6DoF pose of every part. This works well when exact poses are necessary, but can impose unneeded constraints when not. We present a collaborative XR constraint-authoring interface that supports a spectrum of specification authority. In our interface, an author can specify groups of parts and transfer them to a human or robot executor that carries out the author’s instructions in the physical world. In prescribed mode, the author specifies the 6DoF pose of a part or group. In delegated mode, the author entrusts the executor to determine the 6DoF pose of a part or group. Both modes can be mixed within a single task. We conducted a within-subjects study comparing prescribed-only authoring against dual-mode authoring. Results show that dual-mode authoring significantly reduced authoring task completion time and perceived workload. Most participants significantly preferred dual-mode, with delegation rate scaling with the complexity of authored structures. We also evaluated both conditions offline with a robot executor, feeding authored constraints through a workspace optimizer and motion planner. Delegated constraints achieved significantly higher inter-group clearance than prescribed-only specifications.

4. IEEE VR

   

   Bimanual XR Specification of Relative and Absolute Assembly Hierarchies for Teleoperation

   Ben Yang, Xichen He, Charlie Zou, Jen-Shuo Liu, Barbara Tversky, and Steven Feiner

   In 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2026

   Abs DOI Poster

   We present a bimanual XR interaction approach for specifying remote assembly tasks as hierarchies of relative and absolute object constraints that specify high-level teleoperation goals for robots. Grabbing one object in each hand creates a constraint group (visualized as a hull) and groups can be nested into hierarchies. Each group can be relative (with a robot-specifiable 6DoF pose) or absolute (with an author-specified fixed 6DoF pose) in relation to its parent. A relative group specifies a subassembly that can be constructed at a location chosen by the robot software for efficiency rather than mandated by the user.

2025

1. ISMAR

   

   Virtual Reality Task Guidance Through Relative 6DoF Pose Specification

   Ben Yang, Xichen He, Charlie Zou, Jen-Shuo Liu, Barbara Tversky, and Steven Feiner

   In 2025 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2025

   Abs DOI Poster

   Virtual reality (VR) systems for guiding remote physical tasks typically require object poses to be specified in absolute world coordinates. However, many of these tasks only need object poses to be specified relative to each other. Thus, supporting only absolute pose specification can create inefficiencies in giving or following task guidance when unnecessary constraints are imposed. We are developing a VR task-guidance system that avoids this by enabling relative 6DoF poses to be specified within subsets of objects. We describe our user interface, including how geometric relationships are specified and several ways in which they are visualized, and our plans for validating our approach against existing techniques.