About Me

I am Jiatuo “Charlie” Zou, a researcher in Human-Computer Interaction (HCI) and a graduate of Columbia University (M.S.). I recently joined the SPICE Lab at Northwestern University, advised by Karan Ahuja. Previously, I was a research assistant in the CGUI Lab at Columbia University, working with Steven Feiner.

Research interests

Today’s interfaces make people spell out every input they want. This creates two gaps:

  • Effort gap: specifying every detail is possible but slow.
  • Ability gap: people know their goal but not what to specify to reach it.

To close these gaps, I build context-aware interaction systems that adapt to users and their environments based on novel sensing and multimodal AI, with a specific focus on extended reality (XR), wearable computing, and human-AI interaction (HAI). I envision approaching these problems through a human-AI loop that consists of three components: sensing, sensemaking, and interaction. Sensing captures signals, sensemaking understands those signals, and interaction uses that understanding to respond, producing new signals for the next cycle.

Sensing, sensemaking, and interaction form a continuous loop Sensing captures signals from people and their surroundings. Sensemaking understands those signals. Interaction uses that understanding to respond, producing new signals that return to sensing. People and their context remain at the center.
A continuous cycle of sensing, understanding, and responding.

  • Sensing

    I am interested in capturing behavioral, physiological, and environmental signals through wearables and novel sensing. These measurements provide the raw evidence for sensemaking, including cues that are difficult to convey through explicit input alone.

  • Sensemaking

    Sensemaking turns those signals into an understanding of people's activity, intent, and context. I am interested in multimodal AI that combines complementary evidence, accounts for uncertainty and source reliability, and updates its understanding as situations change.

  • Interaction

    I aim to use that understanding to build XR, wearable, and AI-assisted interfaces that adapt what they present and when they respond. As people act and respond, these interactions produce new signals for sensing, helping the system support them while keeping them in control.

News

  • I joined the SPICE Lab at Northwestern University.

  • Submitted 2 papers to UIST 2026.

  • Submitted a paper to ISMAR 2026.

Publications

Under review
A spatial routing demonstration showing objects and routing regions in a 3D scene

Cascade: Spatial Routing Abstractions for Human Developers and AI Coding Agents

Benjamin Yang, Shanying Liu, Charlie Zou, Xichen He, Carmine Elvezio, Steven Feiner

ACM CHI Conference on Human Factors in Computing Systems (CHI), 2027. Submitted to CHI 2027; under review.

Abstract

Interactive applications built on hierarchical scene graphs often rely on complex inter-component communication. Many frameworks for developing these applications support message communication up and down the parent--child hierarchy, but require custom code to define non-hierarchical communication graphs. For example, this includes sending messages between objects based on their spatial proximity, independent of the scene-graph hierarchy. Furthermore, limited prior work has empirically evaluated whether frameworks that support non-hierarchical message routing improve developer productivity over custom code. We present Cascade, a messaging toolkit for hierarchical scene graphs that integrates composable spatial filtering primitives (proximity, direction, bounding region) directly into message routing. Unlike previous work, it enables developers to create non-hierarchical communication graphs that express complex routing patterns in a single chained expression. We evaluate Cascade through a controlled within-subjects experiment comparing task completion time, code correctness, cognitive load, and usability for Cascade and event-based messaging across tasks in robot teleoperation and Internet of Things (IoT) domains.

UIST 2026
An XR task environment with virtual aircraft, EEG signals, and a participant wearing a headset

Augmenting Human Performance with an XR Agent Learning from Online Behavior and BCI Evidence

Ziheng Li*, Xichen He*, Haoyan Chen*, Charlie Zou*, Sheng Bai, Benjamin Yang, Mengyuan Wu, Jake Ledner, Yi-Jie Cheng, Akito Yamauchi, Dishita G. Turakhia, Steven Feiner, Paul Sajda

ACM Symposium on User Interface Software and Technology (UIST), 2026.

* Co-first authors; equal contribution.

[Paper] [arXiv] [Preview] [Video]

An adaptive XR assistant that learns from behavior and brain signals to provide timely guidance in demanding, dynamic tasks.

Abstract

We present OLIVE, a framework for adapting a foundation model to provide real-time assistance in temporally demanding, high-stakes and dynamic tasks. OLIVE learns from both explicit behavioral signals (user actions) and implicit physiological signals (fixation-locked EEG) to provide timely guidance to supplement the user's perceptual and action bandwidth. By fusing these complementary channels, OLIVE continuously adapts a frozen vision-language model's belief about which items are currently task-relevant, jointly estimating per-source reliability without manual labels or offline retraining. Through three user studies, where we deploy an assistive agent driven by OLIVE in XR, we show that OLIVE outperforms prior test-time adaptation frameworks in both convergence time and rate. Compared to learning from explicit behaviors alone, the agent combining implicit and explicit learning signals improves user performance, achieving 45% more target coverage. When targets shift silently mid-block, the combined agent re-adapts 2.1x faster than the behavioral-only baseline. These results show OLIVE can be built toward empowering adaptive co-pilot systems that extend human performance under demanding operational conditions.

ISMAR 2026
An XR assembly interface showing grouped components and a handheld controller

Delegate the Details: Flexible Assembly Constraint Authoring for Collaborative XR

Benjamin Yang, Xichen He, Charlie Zou, Jen-Shuo Liu, Barbara Tversky, Steven Feiner

IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2026.

[Paper] [Preview] [Video]

A collaborative XR interface that lets authors choose what to specify precisely and what to delegate to a human or robot executor.

Abstract

Extended reality (XR) interfaces for assembly specification typically require users to specify the 6DoF pose of every part. This works well when exact poses are necessary, but can impose unneeded constraints when not. We present a collaborative XR constraint-authoring interface that supports a spectrum of specification authority. In our interface, an author can specify groups of parts and transfer them to a human or robot executor that carries out the author's instructions in the physical world. In prescribed mode, the author specifies the 6DoF pose of a part or group. In delegated mode, the author entrusts the executor to determine the 6DoF pose of a part or group. Both modes can be mixed within a single task. We conducted a within-subjects study comparing prescribed-only authoring against dual-mode authoring. Results show that dual-mode authoring significantly reduced authoring task completion time and perceived workload. Most participants significantly preferred dual-mode, with delegation rate scaling with the complexity of authored structures. We also evaluated both conditions offline with a robot executor, feeding authored constraints through a workspace optimizer and motion planner. Delegated constraints achieved significantly higher inter-group clearance than prescribed-only specifications.

IEEE VR 2026
A bimanual XR interface for specifying assembly hierarchies

Bimanual XR Specification of Relative and Absolute Assembly Hierarchies for Teleoperation

Ben Yang, Xichen He, Charlie Zou, Jen-Shuo Liu, Barbara Tversky, Steven Feiner

2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2026.

[DOI] [Poster]

Abstract

We present a bimanual XR interaction approach for specifying remote assembly tasks as hierarchies of relative and absolute object constraints that specify high-level teleoperation goals for robots. Grabbing one object in each hand creates a constraint group (visualized as a hull) and groups can be nested into hierarchies. Each group can be relative (with a robot-specifiable 6DoF pose) or absolute (with an author-specified fixed 6DoF pose) in relation to its parent. A relative group specifies a subassembly that can be constructed at a location chosen by the robot software for efficiency rather than mandated by the user.

ISMAR 2025
A virtual reality visualization of relative object poses for task guidance

Virtual Reality Task Guidance Through Relative 6DoF Pose Specification

Ben Yang, Xichen He, Charlie Zou, Jen-Shuo Liu, Barbara Tversky, Steven Feiner

2025 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2025.

[DOI] [Poster]

Abstract

Virtual reality (VR) systems for guiding remote physical tasks typically require object poses to be specified in absolute world coordinates. However, many of these tasks only need object poses to be specified relative to each other. Thus, supporting only absolute pose specification can create inefficiencies in giving or following task guidance when unnecessary constraints are imposed. We are developing a VR task-guidance system that avoids this by enabling relative 6DoF poses to be specified within subsets of objects. We describe our user interface, including how geometric relationships are specified and several ways in which they are visualized, and our plans for validating our approach against existing techniques.

My Journey

  1. –2017 Hangzhou
  2. 2017 Fort Lauderdale, FL
  3. 2021 Champaign, IL
  4. 2024 Hangzhou
  5. 2024 New York, NY
  6. 2026 Evanston, IL

Hangzhou, China

Before 2017

The Daguan area of Gongshu District, Hangzhou, before leaving for high school in the United States in 2017.

  • place

    Hangzhou, China

    A wide, calm canal runs straight ahead between banks of willows and dense trees under an overcast sky, with apartment towers far in the distance.

    Daguan area, Gongshu District · before 2017

    Home until 2017. Daguan, “the great pass”, is named for Beixinguan, the Ming-dynasty customs pass on the Grand Canal here.

    Photo: 方的1P, CC BY-SA 4.0, cropped

  • photo

    Gongchen Bridge on the Grand Canal

    A long three-arched stone bridge spans the wide canal, with a moored white boat on the left and new apartment towers and cranes behind.

    Gongshu District, Hangzhou

    A three-arched stone bridge, first built in 1631 and rebuilt in 1885; it marks the end of the Grand Canal in Hangzhou.

    Photo: Zhangzhugang, CC BY-SA 4.0

  • photo

    Qiaoxi historic block

    A paved lane runs between low white-walled houses and open timber pavilions, with people sitting on benches and autumn trees behind.

    West of Gongchen Bridge

    Old canal-side streets restored and opened in 2010, part of the Grand Canal World Heritage Site since 2014.

    Photo: Siyuwj, CC BY-SA 4.0

  • photo

    Xiangji Temple by the canal

    A temple hall with red pillars, a grey tiled roof and a gold name plaque stands behind a paved square, flanked by two white stone elephants.

    Dadou Road, Gongshu District

    Founded in 978 and rebuilt in 2009–2010; one of its two Qing-dynasty stone pagodas survives.

    Photo: 猫猫的日记本, CC BY-SA 4.0

  • photo

    The Grand Canal by West Lake Cultural Square

    Seen from high up, a wide canal runs into the distance under a footbridge, between tree-lined banks and apartment towers, with a tall glass tower narrowing at its waist on the right.

    Hangzhou

    The tall tower on the right is the main building of West Lake Cultural Square, on the canal's bank.

    Photo: Jiatuo Zou

Streets of Hangzhou: map data © OpenStreetMap contributors

Fort Lauderdale, Florida

2017 – 2021

Four years of high school at Westminster Academy in Fort Lauderdale, on the school's FIRST Robotics team, graduating as salutatorian in 2021.

  • place

    Fort Lauderdale, FL

    A raised rusty drawbridge and downtown towers under a pink evening sky, with small boats docked on the New River.

    2017 – 2021

    Photo: Ianaré Sévi, CC BY-SA 3.0, cropped

  • education

    Westminster Academy

    High school · Fort Lauderdale, Florida

    2017 – 2021

    Four years of high school. Salutatorian, Class of 2021.

  • project

    FRC Team 744

    FIRST Robotics Competition · 2019 & 2020 seasons

    “Shark Attack”, Westminster Academy's FIRST Robotics Competition team.

    • Team 744's 2019 robot, a square base in blue bumpers numbered 744 under an aluminium A-frame with clear side panels and a black carbon-fibre lift, holding an orange ball between green rollers.

      artifact

      2019 robot · Destination: Deep Space

      Holding a cargo ball

      Image: Jiatuo Zou

  • photo

    FRC Team 744 · Shark Attack

    Pixelated until revealed. Members of FRC Team 744 standing arm in arm; everyone except Jiatuo, far right, has a black bar over their eyes.

    Westminster Academy

    Photo: Jiatuo Zou

    • Team 744's 2020 robot seen from the front, an aluminium frame in blue bumpers numbered 744, with green-wheeled rollers below a hooded shooter at the top and two tall curved uprights either side.

      artifact

      2020 robot · Infinite Recharge

      Season cut short by COVID-19

      Image: Jiatuo Zou

  • photo

    The beach on the Atlantic

    A long sandy beach with palm trees and beachgoers, the blue Atlantic on the right and a line of high-rise buildings along the shore.

    Fort Lauderdale

    Photo: Daniel Dudek, CC BY 2.0

  • photo

    Downtown skyline, 2015

    Downtown Fort Lauderdale towers under scattered clouds, with palm trees, low buildings and a large yacht in the foreground.

    Fort Lauderdale

    Photo: KeanoManu, CC BY-SA 4.0, cropped

Streets of Fort Lauderdale: map data © OpenStreetMap contributors

Champaign, Illinois

Aug 2021 – May 2024

Undergraduate years at Illinois, with research in a data lab, course assisting, and game development, plus a hybrid internship in 2025.

  • place

    Champaign, IL

    A broad green lawn under large shade trees, with the domed Foellinger Auditorium at the far end and students walking in the distance.

    2021 – 2024

    Photo: Amphylite, CC0

  • Jiatuo Zou's diploma from the University of Illinois Urbana-Champaign, Bachelor of Science in Computer Science with Highest Honors, on cream paper; only the university's name, his name, the degree and the honours are legible, and the rest, including the seal and signatures, is blurred.

    education

    University of Illinois Urbana-Champaign

    B.S. Computer Science

    Aug 2021 – May 2024

    Graduated with Highest Honors in Engineering; Dean's List.

  • lab

    Forward Data Lab

    Research Assistant · UIUC

    Aug – Dec 2023

  • Projects (2)

    • project

      LLM Context Augmentation

      Forward Data Lab

      Aug – Dec 2023

      A retrieval-augmented LLM pipeline (LangChain, Chroma) over papers and conversation history, evaluated with BLEU against ChatGPT references.

    • project

      TrippinDark

      Course project · Unreal Engine 5

      Jan – May 2023

      NPC behavior, two game levels, inventory, puzzles, and traps.

  • award

    HackIllinois 2022

    Winner, General Track (Web Development)

  • Work & teaching (2)

    • course

      CS 361 Probability & Statistics

      Course Assistant

      Aug 2023 – May 2024

    • work

      PhinD

      Software Engineer Intern · Hybrid, during my M.S.

      Jan – May 2025

      File management on AWS S3, FastAPI services, and an embedding-based job recommender.

  • photo

    Siebel Center for Computer Science

    A modern brick-and-glass building seen through bare autumn trees along a curving campus path.

    University of Illinois

    Photo: Joe Futrelle, CC BY-SA 2.0

  • photo

    Grainger Engineering Library

    A red-brick library with a grey slate roof and a large arched dormer window, set behind a green lawn.

    University of Illinois

    Photo: Daderot, public domain

Streets of Champaign: map data © OpenStreetMap contributors

Hangzhou, China

May – Aug 2024

A summer leading a team that built a full-stack telemedicine health pod.

  • place

    Hangzhou, China

    A wide expanse of blue lake water with small boats, and a long line of city towers on the far shore.

    Summer 2024

    Summer research at Hangzhou Dianzi University.

    Photo: Windmemories, CC BY-SA 4.0

  • lab

    Lab of Brain-Computer Intelligence

    Research Assistant · Hangzhou Dianzi University

    May – Aug 2024

  • project

    IDTS Healthcare “Pod”

    Intelligent Diagnosis & Telemedicine System

    An onboard React web UI, WeChat mini-program, and MySQL backend, streaming ECG, temperature, blood pressure, SpO₂, and glucose readings over Bluetooth.

    • A small, flat white health monitor with rounded corners, seen from above its front, printed with “LINKTOP” toward the back of its top; a silver-grey ring around a black sensor pad sits in the middle, and a small grey window with a blue light is set in its front edge.

      artifact

      Linktop health monitor

      Readings streamed over Bluetooth (Linktop SDK)

      Image: Linktop (product image, supplied by Jiatuo)

  • note

    Team of 7

    Project lead

    Planned 8-week milestones, weekly check-ins, and one-to-one development help.

  • photo

    Hangzhou Dianzi University, main gate

    A tall red-brick gate tower with a round opening at its top, joined to a long orange steel truss spanning the campus entrance plaza.

    Xiasha campus, Hangzhou

    Photo: CatOnMars, CC BY-SA 4.0, cropped

  • photo

    Xiasha from the air

    An aerial view through haze of a city grid on a curving riverbank, with wide tidal flats along the water.

    Eastern Hangzhou

    Photo: Windmemories, CC BY-SA 4.0

Streets of Hangzhou: map data © OpenStreetMap contributors

New York, New York

Sep 2024 – May 2026

Graduate study at Columbia and research across two labs on XR, robot teleoperation, and brain–computer interfaces.

Streets of New York: map data © OpenStreetMap contributors

Evanston, Illinois

Jun 2026 – present Now

Research at Northwestern's SPICE Lab on wearable sensing and AI agents that adapt to people and places.

  • place

    Evanston, IL

    Dune grass and sand in the foreground give way to the calm, pale water of Lake Michigan under an overcast sky.

    2026 – present

    Where I am now.

    Photo: Paul R. Burley, CC BY-SA 4.0

  • lab

    SPICE Lab

    Research Intern · Northwestern University

    Jun 2026 – present

  • Pixelated until revealed. Head-and-shoulders photo of Karan Ahuja, with glasses, in a blue button-down shirt.

    person

    Karan Ahuja

    Advisor

    Photo: Karan Ahuja (karan-ahuja.com)

  • Projects (3)

    • project

      SensorClaw

      In progress

      Turns a one-sentence request (“Buzz me when I knock the table”) into a personal smartwatch gesture tracker, using a multi-agent LLM pipeline that trains a personal IMU detector.

      • A Pixel Watch on a pale blue-grey band, its screen showing the inrtl Lab app running.

        artifact

        Pixel Watch

        The smartwatch SensorClaw's gesture trackers run on

        Image: Jiatuo Zou

    • project

      Spatial Memory

      Agent-assisted handoff notes anchored to objects, places, and projects in shared workspaces.

    • project

      EIT + IMU Hand Pose

      Research project · Northwestern University

      2026

      Synchronized wrist EIT and IMU data with motion-capture ground truth for wearable hand-pose estimation.

  • photo

    University Hall

    A Victorian Gothic limestone building with pointed towers and a central arched entrance, seen from below against a dramatic sky.

    Northwestern University

    Photo: Madcoverboy, CC BY-SA 3.0

  • photo

    Deering Library

    A Collegiate Gothic stone library with tall arched windows, behind a wide green lawn and spring trees.

    Northwestern University

    Photo: Madcoverboy, CC BY-SA 3.0

  • photo

    The Technological Institute plaza in autumn

    Students cross a brick-paved plaza toward the stone entrance of a large building, framed by orange autumn trees.

    Northwestern University

    Photo: Chris Rycroft, CC BY 2.0

  • note

    Next stop?

    PhD, Fall 2027

    Applying to PhD programs now.

Streets of Evanston: map data © OpenStreetMap contributors