Vision has ImageNet. Language has BERT. Tactile robotics has… twenty incompatible formats.

A GelSight Mini produces 240×320 optical deformation images. Xela uSkin outputs pressure arrays. Sharpa Wave's Dynamic Tactile Array delivers high-resolution vision-based touch plus 6-DoF wrench readings. Train a slip detector on GelSight, deploy on Wave — performance collapses. The sensors measure the same physical phenomenon (contact), but their representations live in different worlds.

This fragmentation blocks the central promise of tactile dexterity: pooling diverse touch data at scale to learn contact-rich manipulation. FTP-1 attacks this at the policy level with morphology-aware tokens across 21 sensors. Heterogeneous Tactile Transformer (HTT) — from National University of Singapore and Carnegie Mellon — attacks it at the representation level: a shared tactile encoder pretrained across four radically different sensors, transferable to new perception tasks and previously unseen hardware.

On real-world experiments with a Sharpa hand on a Franka arm (camera-free), HTT raises toy-screw tightening success from 5% (joint positions only) to 95% — and from 50% (raw wrench) to 95% with zero-shot tactile embeddings.

On real-world experiments with a Sharpa hand on a Franka arm (camera-free), HTT raises toy-screw tightening success from 5% (joint positions only) to 95% — and from 50% (raw wrench) to 95% with zero-shot tactile embeddings.

The Problem: Same Physics, Incompatible Pixels

Tactile sensing is inherently heterogeneous:

Each modality has different resolution, morphology, contact response, and noise profile. Models trained on one sensor do not transfer — forcing every new hand or fingertip to start representation learning from scratch.

This matters because:

1. Data cannot compound — GelSight datasets, array-sensor corpora, and Sharpa teleoperation logs stay siloed
2. Policies overfit to sensor format — not contact physics
3. New hardware requires full retraining — expensive for OEMs integrating Wave alongside legacy tactile stacks

HTT asks: Can we learn a shared tactile latent space — like CLIP for touch — that aligns heterogeneous sensors through synchronized pairing?

The Solution: Sensor-Specific Encoders, Shared Transformer Trunk

HTT follows a proven foundation-model recipe, adapted for touch:

Pretraining objectives (dual):

1. Per-modality masked reconstruction — each sensor's encoder-decoder path learns to reconstruct its own input (self-supervised, sensor-specific structure preserved)
2. Cross-modal alignment — paired frames from different sensors viewing the same contact event are pulled together in the shared latent space

HPT Dataset (Heterogeneous Paired Tactile)

• 1.6M synchronized paired frames
• Four sensors mounted on a UMI gripper shell: GelSight Mini, 9DTact (optical); TAC-02, Xela uSkin (array)
• Force/slip ground truth via probe rig + 6-DoF F/T sensor (up to 40 N normal, 14 N shear)
• Slip labels: 13.6% static, 1.2% incipient, 85.2% slide — rare classes are deliberately hard

After pretraining, the shared trunk produces transferable tactile embeddings usable for:

• Force estimation across all four sensor types
• Slip detection (including incipient slip)
• Downstream manipulation policy learning on unseen sensors

Putting It to the Test: Perception, Simulation, and Sharpa Hardware

Tactile perception (four sensors)

HTT pretrained representations transfer to force estimation and slip detection on each of the four HPT sensors — demonstrating cross-modal alignment works across optical and array modalities.

Simulation — ManiFeel benchmark

HTT embeddings boost policy learning on contact-rich simulated tasks (bulb installation, peg insertion) with both RGB tactile images and force-field (FF) modalities — showing the encoder helps policies, not just classifiers.

Real-world — Sharpa hand on Franka (camera-free)

The most striking results come from deploying HTT on hardware Sharpa's ecosystem cares about:

Setup details from the project page:

• Sharpa dexterous hand on Franka arm
• No cameras — proprioception + tactile only
• Toy Screw: >600° rotation (near-tight) = success; thread engagement demands slip-aware force control
• Grasp Tofu: deformable, slip-prone object; slip is the dominant failure mode
• HTT pretrained on UMI gripper sensors — zero-shot on Sharpa fingertip at deployment

Raw joint positions barely work (5%). Raw wrench readings help but plateau (35–50%). HTT's shared tactile representation — never trained on Sharpa during pretraining — nearly solves screw tightening.

HTT vs FTP-1: Complementary Layers of Tactile Foundation Models

Both HTT and FTP-1 address tactile heterogeneity; they operate at different layers:

HTT answers: "What is a good universal tactile feature?" FTP-1 answers: "What is a good universal tactile policy initialization?"

Together they sketch a tactile foundation stack analogous to vision (DINO encoder + VLA policy) — with Sharpa Wave as the deployment target both papers validate on.

Why This Matters for Sharpa Wave

Zero-shot transfer to Sharpa DTA

HTT's 95% screw success on Sharpa — without Sharpa in the pretraining mixture — demonstrates that optical and array sensor pretraining can bootstrap vision-based tactile hands. OEMs need not collect millions of Sharpa-specific frames before tactile policies become useful; heterogeneous pretraining provides a warm start.

Camera-free contact-rich control

Toy Screw at 95% without any vision input validates a deployment mode relevant to occlusion-heavy manipulation (in-hand rotation, bimanual overlap). Proprio + HTT embeddings may suffice when cameras cannot see the contact.

Complements DTA + wrench API

Sharpa Wave exposes both high-resolution tactile images and 6-DoF force/wrench per fingertip. HTT shows raw wrench (50%) underperforms structured embeddings (95%) — how touch is encoded matters more than whether touch is present. This supports CraftNet's System 0 design: dedicated tactile processing, not bolt-on F/T channels.

Data ecosystem unlock

HPT's paired-collection methodology (same contact, multiple sensors) is a recipe Sharpa can extend: pair Wave DTC with GelSight or Contactile during teleoperation (DexTeleop-0), feeding both FTP-1-scale policy pretraining and HTT-scale encoder pretraining.

Slip detection for deformables

Grasp Tofu highlights slip as the primary failure mode — exactly the regime where Wave's high-frequency DTA and HTT's incipient-slip training (1.2% of HPT labels) target.

Sim-to-real path

ManiFeel simulation gains + real Sharpa deployment mirror Sharpa's Isaac Sim asset strategy — pretrained tactile encoders as portable modules across sim and hardware.

The Takeaway: Unify Touch, Then Scale Touch

Robotics has spent years unifying vision across cameras and action across embodiments (EgoScale, CAIP). Touch remains the fragmented modality — until now.

HTT provides the missing tactile encoder layer: sensor-specific front ends, shared transformer trunk, cross-modal alignment on paired data. On Sharpa Wave, it turns zero-shot embeddings into 95% screw-tightening success without a single training frame from that sensor.

For Sharpa's platform strategy: heterogeneous tactile foundation models are no longer theoretical. HTT (representations) + FTP-1 (policies) + Wave DTA (hardware) form a coherent stack for contact-rich dexterity at scale.

One contact event. Many sensors. One shared understanding of touch.

Status: Under review (dataset/code release upon publication)