Harry H. Zhang

I am a graduate student in the SPARK Lab of MIT LIDS. I am extremely fortunate to be advised by Prof. Luca Carlone.

Prior to MIT, I was a MS-Research student in the CMU Robotics Institute studying Artificial Intelligence and Robotics, advised by Prof. David Held. I also worked at Amazon as an Applied Scientist II.

Prior to CMU, I earned my B.S. (2017-2021) with Honors from UC Berkeley with a major in EECS and a minor in Mechanical Engineering. During my time at Berkeley, I did research under Prof. Ken Goldberg and Dr. Jeffrey Ichnowski in AUTOLab. I maintain and curate a popular deep reinforcement learning tutorial on my Github.

Outside of school, I do quantitative finance.

Email  /  CV  /  LinkedIn  /  Google Scholar  /  Github  /  Twitter

In reverse chronological order:

• Oct. 2023: DiffCLIP accepted to WACV, see you in Hawaii!
• Aug. 2023: FlowBot++ accepted to CoRL, see you in ATL!
• Sep. 2022: TAX-Pose accepted to CoRL, see you in New Zealand!
• May 2022: I am joining Amazon this summer as an applied research scientist, working on 3D learning problems.
• Apr. 2022: FlowBot3D accepted to RSS, see you in NYC!
• May. 2021: Won the Warren Y. Dere Award from UC Berkeley EECS.
• Mar. 2021: Dynamic cable manipulation paper accepted to ICRA 2021.
• Nov. 2020: Dynamic cable manipulation paper featured at Bay Area Robotics Symposium (BARS) hosted by Stanford.
• Jun. 2020: Dex-Net AR got featured on VentureBeat.
• Jun. 2020: Dex-Net AR got featured on Sohu.

My current research focuses on trustworthy autonomy. Specifically, I wish to make autonomous systems robust, certifiable, and trustworthy.

We propose DiffCLIP, a new pre-training framework that incorporates stable diffusion with ControlNet to minimize the domain gap in the visual branch. Additionally, a style-prompt generation module is introduced for few-shot tasks in the textual branch.

We explore yet another novel method to perceive and manipulate 3D articulated objects that generalizes to enable the robot to articulate unseen classes of objects.

We conjecture that the task-specific pose relationship between relevant parts of interacting objects is a generalizable notion of a manipulation task that can transfer to new objects. We call this task-specific pose relationship "cross-pose". We propose a vision-based system that learns to estimate the cross-pose between two objects for a given manipulation task.

We explore a novel method to perceive and manipulate 3D articulated objects that generalizes to enable the robot to articulate unseen classes of objects.

We present present AVPLUG: Approach Vector PLanning for Unicontact Grasping: an algorithm for efficiently finding the approach vector using an efficient oct-tree occupancy model and Minkowski sum computation to maximize information gain.

We propose a self-supervised learning framework that enables a UR5 robot to perform these three tasks. The framework finds a 3D apex point for the robot arm, which, together with a task-specific trajectory function, defines an arcing motion that dynamically manipulates the cable to perform tasks with varying obstacle and target locations.

We present a distributed pipeline, Dex-Net AR, that allows point clouds to be uploaded to a server in our lab, cleaned, and evaluated by Dex-Net grasp planner to generate a grasp axis that is returned and displayed as an overlay on the object.

We present an algorithm to orient novel objects given a depth image of the object in its current and desired orientation.

We present an algorithm to train a robot to control free-end cables in a self-supervised fashion.

We introduce andexplore a novel method for adding safety constraints for model-based RL during training and policy learning.

10-725: Graduate Convex Optimization
16-385: Computer Vision

CS 189: Introduction to Machine Learning

EE 127: Introduction to Convex Optimization

Website template from Jon Barron