USB Passthrough for Keyboard and mouse using two ESP32

Today I'm going to talk to you about a personal project I've been working on. For me, it was the first time I have programmed microcontrollers. Being able to physically program a chip adds a certain charm. I called the project “macroPassthrough”

My original need was based on a problem I have. I wanted to be able to create macros in video games that are protected by anti-cheat software. For example, if you use AutoHotkey for keyboard macros, the anti-cheat software can easily detect that AutoHotkey actions are being performed during gameplay. Unfortunately, since macros are software-based, there is no way to be certain that anti-cheat software will not be able to detect the PC's activity in some way. I would like to be able to create macros that are completely invisible to the computer and the video game. Some Razer or Logitech mouse devices sometimes offer macro solutions that are directly integrated into the device drivers. This is a good solution, but it requires going through their configuration interface to add macros. No programming is possible. This quickly limits the type of macro that can be created, like key-interceptions are not feasible. Options such as Razer Snap Tap (where you prioritizing only the last movement key) are only available on certain keyboards.

Then came the idea of creating a macro tool that could be implemented at the physical hardware level. I quickly came to the conclusion that I needed to create a USB passthrough, meaning intercepting the USB communication of an HID device, in order to implement my own macros. At this point in the article, you may be thinking that my project is very similar to a Cronus Zen hardware. It's true, but Cronus is very console- and controller-oriented. Here, my needs are really keyboard/mouse-oriented. And I want to be able to build an economical solution that doesn't cost hundreds of euros.

Here are my needs:

1. A physical interception device capable of reading HID frames from a USB port.
2. The ability to intercept HID frames in order to add our own macros.
3. High performance in processing HID frames. We're talking about gaming here, so there should be no added latency to the HID device.
4. Compatible with 1000Hz polling rate devices.

I looked into several solutions. I thought I could do it with a single microcontroller:

• An Arduino board that supports USB OTG, to which a USB Host Shield is added. This way, a single device performs all the processing. I didn't go with this solution, mainly because of the Arduino's performance issues. I have serious doubts about its ability to handle a 1000Hz polling rate from a gaming mouse. The processor frequency is quite low. And the latest, more powerful Arduino boards are also more expensive. An Arduino Leonardo is only 16 MHz clock frequency.
• A Raspberry Pi with a USB gadget installed. The advantage of the Raspberry Pi is that everything is built in. It's a real mini-computer. But isn't that a bit overkill for simply creating a USB passthrough? In terms of energy consumption, it's not adequate. When you start up the computer, the Raspberry Pi takes about ten seconds to boot up. During this time, we don't have access to our keyboard/mouse. There are drivers such as Raw Gadget that facilitate the development of custom USB devices. But this means that our code that processes packets runs in user land. This adds a lot of latency in terms of processing. We can also see that when the Raspberry Pi 5 was released, USB gadget mode was simply not supported. This leaves some doubt about how this card can be used. My conclusion was that the Raspberry Pi is simply not designed to create reactive USB devices that need to react from input packet.

GitHub - xairy/raw-gadget: USB Raw Gadget — a low-level interface for the Linux USB Gadget subsystem

USB Raw Gadget — a low-level interface for the Linux USB Gadget subsystem - xairy/raw-gadget

GitHubxairy

• I am writing this after having made good progress on the project, and I think that the ideal board would have been an STM32. This is because the STM32F2xx family has several USB OTG ports on the same board. Unfortunately, I didn't start with that at the beginning. This is still a project I'm working on in my spare time, and I preferred to start with more “consumer-oriented” boards. STM32 remains very professionally oriented; these are very powerful boards with great customization possibilities. But unfortunately, the learning curve was a little too steep for a first attempt.

You need a project with two cards:

• Espressif's ESP32s have the advantage of offering a range of boards with USB OTG ports. I opted for the ESP32-S3. It has a USB OTG port and another USB UART port for debugging. This is perfect for iterating and testing the developed code. There are no two USB Host and Device ports, which means that another ESP32-S3 board must be connected to enable passthrough input and output. The advantage of the Espressif working environment is that it is much easier to access than STM. It is very similar to the Arduino working environment with its IDE. ESP32-S3 is a powerful dual-core 240MHz board. And to meet my real-time processing needs, it comes with the RTOS operating system.
  It's also very cheap at ~5 euros.

In the following picture you can see how I have connected my two ESP32-S3 together:

• At the top, the first ESP32 receiving the USB HID Input (Keyboard or Mouse). You can also connect an USB Hub to connect both Keyboard and Mouse at the same time! Power is delivered by the other USB 5V port. You can use this port to debug ESP32 via UART.
• Below, there is the second ESP32-S3. It is connected directly to the PC. And the ESP emulate an HID peripheral. One HID peripheral can have multiple descriptor, one for Keyboard and one for Mouse. This how the ESP is configured.
• Both microcontroller are connected together using two 4-PIN SPI connection. This is because for Keyboard, you need to send Keyboard keystroke to the PC. But the PC also response with LED lights (caps, lock, scroll lock) report. So, one connection is use to transmit keyboard and mouse report. And one connection is use to transmit to the keyboard LED report from the PC.

The USB Device plugged in the PC is detected as a single USB peripheral. Containing two HID description. This is fairly normal behavior, and most gaming mice/keyboards also implement both descriptions. (This is so that drivers can manage their own macro mechanisms).

The code implementation is available on Github:

GitHub - arfevrier/macroPassthrough: An ESP32 USB HID Passthrough, for Keyboard macro injection.

An ESP32 USB HID Passthrough, for Keyboard macro injection. - arfevrier/macroPassthrough

GitHubarfevrier

I won't go into the details of implementation, as they are already described in the README. But I will go over a few points.

Project Structure

• usb-input/
  ESP32 firmware that acts as a USB host, receives HID reports from a keyboard, and forwards them over SPI.
• usb-ouput/
  ESP32 firmware that acts as a USB device, receives HID/macro reports over SPI, aggregates them, and presents as a USB HID keyboard to the PC. Supports macro injection.

You can build and run both project directory using ESP-IDF framework. To validate the project build, a CI/CD have been implemented on run on each commit.

Development environment

I'm on Windows, but I use WSL to build and flash ESP devices. USBIpd help me to access my USB UART device inside WSL environment. The advantage is that ESP-IDF is easier to install on Linux than on Windows.

During the development of the project, I have performed some test to validate that the ESP is able to reach the desired Gaming performance.

Inside benchmark/ directory contains latency and throughput measurements for different configurations (e.g., ESP-IDF USB host, TinyUSB, SPI). And as you can see, a Capslock ping (pressing capslock and receiving LED report from PC) is taking less than 1.5ms. Considering that the PC has a polling rate of 1000Hz. So on average, the PC will read the report every 1 ms. We can therefore conclude that the ESP32 does not add more than 500 microseconds to the overall latency of the device. Which is very pleasing.

I (2236) spi_hid_sender: CapsLock ping => 1377 us
I (2396) spi_hid_sender: CapsLock ping => 1358 us
I (2536) spi_hid_sender: CapsLock ping => 1388 us
I (2686) spi_hid_sender: CapsLock ping => 1325 us
I (2846) spi_hid_sender: CapsLock ping => 1092 us
I (2986) spi_hid_sender: CapsLock ping => 927 us
I (3126) spi_hid_sender: CapsLock ping => 1280 us
I (3276) spi_hid_sender: CapsLock ping => 1238 us

It should be noted that throughout the code, I took care to correctly configure the priorities of the RTOS tasks running on the ESP32. This was done in order to always prioritize user inputs. This ensures that the user experience is not degraded, even if many macros are defined. Macro management, in terms of priority, is secondary.

And for that reason, I also took care to use only timing via the ESP-timer library. This allows for greater precision than the vTaskDelay() functions, which wait based on tick frequency. Here, the ESP-timer library uses hardware interrupts to wake up tasks. This allows for clock accuracy of 1 microsecond.

And so, in the same way, we can hope to achieve 1 ms precision when sending packets to the PC. Bearing in mind that I am limited by the ESP32-S3's USB 1.1 protocol, which only allows a polling rate of 1000 Hz maximum. And so I have to wait 1 ms between sending each HID report.

On both project, the file config.h contain the main configuration file available to end-user.
Macros are managed by the USB-Output project.

Debug

In the config.h you can configure the level of debug log. This realy impact performance, as printing text through UART takes few milliseconds. But usefull to debug an macro customization.

You have also access to USB PID and VID customization.

Configure macros

Inside the variable macro_sequence, you find the structure to define your custom macro sequences. The list parameter contains the time to wait in microseconds, and the Keyboard / Mouse report to send. We define here real Keyboard / Mouse report with their real report structure. This allows me to send any type of report.

And that's also why you have the ONE_KEYBOARD_KEY() functions, which make it easy to manage the keys to be sent.

Macro interception

Macro interception should be implemented directly inside macpass_macro.c, and the prehook function. Pre-hook and post-hook are called before and after sending the HID report to PC. By manipulating transmissions at the pre-hook level, you can modify or delete them. This allows, for example, the implementation of Razer Snap Tap (here just after the START USER CUSTOM MACRO comment).

Here some example of macro which can be deployed inside the project:

{
    .list = {
        {50000, ONE_KEYBOARD_KEY(HID_KEY_A)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_B)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_C)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_D)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_E)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_F)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_G)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_H)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_I)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_J)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_K)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_L)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_M)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_N)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_O)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_P)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_Q)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_R)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_S)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_T)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_U)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_V)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_W)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_X)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_Y)},
        {50000, ONE_KEYBOARD_KEY(HID_KEY_Z)},
        {0, EMPTY_KEYBOARD}
    },
    .size = 27,
    .event_press = ONE_MOUSE_KEY(MOUSE_BUTTON_RIGHT),
}

#define calc(mouvement) (int)(mouvement*(2.52/__game_sensitivity))
{
    .list = {
        {50*1000, MOUSE_MOUVEMENT(calc(-4), calc(7))},
        {99*1000, MOUSE_MOUVEMENT(calc(4), calc(19))},
        {99*1000, MOUSE_MOUVEMENT(calc(-3), calc(29))},
        {99*1000, MOUSE_MOUVEMENT(calc(-1), calc(31))},
        {99*1000, MOUSE_MOUVEMENT(calc(13), calc(31))},
        {99*1000, MOUSE_MOUVEMENT(calc(8), calc(28))},
        {99*1000, MOUSE_MOUVEMENT(calc(13), calc(21))},
        {99*1000, MOUSE_MOUVEMENT(calc(-17), calc(12))},
        {99*1000, MOUSE_MOUVEMENT(calc(-42), calc(-3))},
        {99*1000, MOUSE_MOUVEMENT(calc(-21), calc(2))},
        {99*1000, MOUSE_MOUVEMENT(calc(12), calc(11))},
        {99*1000, MOUSE_MOUVEMENT(calc(-15), calc(7))},
        {99*1000, MOUSE_MOUVEMENT(calc(-26), calc(-8))},
        {99*1000, MOUSE_MOUVEMENT(calc(-3), calc(4))},
        {99*1000, MOUSE_MOUVEMENT(calc(40), calc(1))},
        {99*1000, MOUSE_MOUVEMENT(calc(19), calc(7))},
        {99*1000, MOUSE_MOUVEMENT(calc(14), calc(10))},
        {99*1000, MOUSE_MOUVEMENT(calc(27), calc(0))},
        {99*1000, MOUSE_MOUVEMENT(calc(33), calc(-10))},
        {99*1000, MOUSE_MOUVEMENT(calc(-21), calc(-2))},
        {99*1000, MOUSE_MOUVEMENT(calc(7), calc(3))},
        {99*1000, MOUSE_MOUVEMENT(calc(-7), calc(9))},
        {99*1000, MOUSE_MOUVEMENT(calc(-8), calc(4))},
        {99*1000, MOUSE_MOUVEMENT(calc(19), calc(-3))},
        {99*1000, MOUSE_MOUVEMENT(calc(5), calc(6))},
        {99*1000, MOUSE_MOUVEMENT(calc(-20), calc(-1))},
        {99*1000, MOUSE_MOUVEMENT(calc(-33), calc(-4))},
        {99*1000, MOUSE_MOUVEMENT(calc(-45), calc(-21))},
        {99*1000, MOUSE_MOUVEMENT(calc(-14), calc(1))},
    },
    .size = 29,
    .loop = true,
    .event_press = ONE_MOUSE_KEY(MOUSE_BUTTON_LEFT),
}