A General Camera in the Gun — Infrared Blobs and a Border on a Microcontroller

An OV2640 on an ESP32-S3 has already been made to find infrared points at 136 frames a second. What that would mean as a second source of camera, what a rolling shutter costs a moving gun, the Himax sensors made for the Pico, and whether the same hardware could track a Sinden-style border inside the gun

Figure 1 — An AI-Thinker ESP32-CAM board: an ESP32 module under a camera socket holding an OV2640 sensor on its flexible cable (the ribbon is printed TY-OV2 640-V2.0), with a microSD socket and a flash LED. T… — Figure 1 — An AI-Thinker ESP32-CAM board: an ESP32 module under a camera socket holding an OV2640 sensor on its flexible cable (the ribbon is printed TY-OV2 640-V2.0), with a microSD socket and a flash LED. The OV2640 is the commonest camera on hobby microcontroller boards and the one the published high-frame-rate infrared blob tracker uses, on the newer ESP32-S3. Photo: Nowforever, ESP32-CAM, CC BY-SA 4.0, via Wikimedia Commons.

5.1 Why Look Past PixArt

The kit camera is the PixArt PAJ7025R3 (decision D18) because it is the only part in production with a datasheet and an authorised distributor. It is also a single-source part sold by quotation. Decision D32 records what that means in practice: PixArt lists one distributor for the Americas, whose R3 page said “Currently Out of Stock”, 8 to 32 weeks, price on request, and the sample kits come only from CODICO in Austria. A kit that may sell in quantity needs to know what it would do if that supply failed.

The alternative is a general-purpose image sensor of the kind sold in millions for webcams and doorbells, read by a microcontroller that does the blob-finding the PixArt chip does on its die. Reference Electronics, Volume 3 considered this and set it aside for the kit camera, for want of measured frame rates and latency. Since then, measurements have been published.

5.2 What Has Been Built

ESP32-IR (GitHub, frosted123456, created 3 August 2026; MIT for its own code). Its README states the result in its first lines: “136 fps at ~7.4 ms glass-to-coordinate latency on an ESP32-S3, and ~90 fps on a plain ESP32-CAM — with blob detection running inside the camera driver while the frame is still arriving.” The sensor is an OV2640 “with the IR-cut filter removed”, behind an infrared-pass filter, read at 240 × 176 (HQVGA) in greyscale. Detection is “run-length connected components + intensity-weighted subpixel centroids, up to 4 blobs.” The author is candid about status: “the sensing stack works on both targets; the aiming stack is not finished” — the four-point sort, homography and USB HID are “written and unit-tested but not yet wired to the live stream.”

The README’s “hard-won findings” are worth any builder’s time, because each is a failure that looks like something else:

The ESP32-S3’s stock camera driver computes its clock divider with integer division, so “Ask for 27 MHz and you silently get 32 MHz”, which overdrives the sensor while “still reporting a plausible frame rate”.
One register bit (REG32 bit 7) wrong “silently delivers half a frame”, which “looks exactly like a field-of-view or aiming bug rather than a register bug”.
Exposure is counted in sensor lines, “and the frame is ~296 lines”; past that, frame rate falls one for one.

All the numbers are the author’s own measurements, not independently checked.

The OpenFIRE OV2640 overlay (frosted123456, LIGHTGUN-STUDIO-ESP32S3-OV2640-…, MIT, created 22 August 2026) puts a camera of this kind behind the OpenFIRE ESP32 port’s aiming code, on a Freenove ESP32-S3-WROOM CAM board with four 850 nm LEDs, “5 W class recommended”, and an 800 nm long-pass filter “recommended (non tested)”. It is pinned to one upstream commit and asks to be tested. Reference Electronics, Volume 3 lists it; it is the closest thing to a whole gun on a general camera, and it publishes no frame rate or latency of its own.

So an in-gun infrared blob tracker on a general camera is a prototype that exists, with its sensing measured by its author and its aiming integration in progress.

5.3 Sensors a Microcontroller Can Read

Figure 2 — Arducam's HM0360 module for the Raspberry Pi Pico: a Himax HM0360 monochrome VGA sensor and lens on a small board with a row of header pins. It is sold for the RP2040 boards that this build uses, w… — Figure 2 — Arducam's HM0360 module for the Raspberry Pi Pico: a Himax HM0360 monochrome VGA sensor and lens on a small board with a row of header pins. It is sold for the RP2040 boards that this build uses, with a one-bit serial video interface. Photo: Arducam, product page, used with credit.

Table 1 — 5.3 Sensors a Microcontroller Can Read

Sensor (board)	Pixels, frame rate	Shutter	Interface to the MCU	Notes
OmniVision OV2640 (ESP32-S3 camera boards; Freenove FNK0085 US$19.99)	240 × 176 at 136 fps, measured by ESP32-IR	Rolling	8-bit parallel into the ESP32-S3’s LCD_CAM	IR-cut filter must be absent; colour sensor read as luma
Himax HM01B0 (Arducam B0315 for the Pico)	320 × 320 at 51 fps; 320 × 240 at 60 fps	”Electronic Rolling Shutter”	1-bit or 4-bit serial with frame and line sync (“8b is not supported”)	Under 4 mW at QVGA 60 fps; monochrome
Himax HM0360 (Arducam B0319 for the Pico, US$19.99)	640 × 480, up to 60 fps	Not stated on the page	The module: “Configurable 1-bit video data serial interface”	Monochrome; “External frame synch”; “<2ms frame trigger”; sensitivity given at 850 nm
OmniVision OV7251, OV9281 class	Global shutter	Global	Usually MIPI CSI-2	No light-gun firmware or paper found (Reference Electronics, Volume 3)

Prices are from the makers’ own pages on 19 October 2026. Two cautions apply to the Pico modules. A one-bit serial interface at the HM01B0’s quoted maximum of 50 MHz carries at most 20 VGA frames a second at 8 bits a pixel, or 81 QVGA frames; whether the HM0360 module’s serial clock goes higher is not on the page. And the RP2040 has no camera peripheral; it reads such a link with PIO and DMA, which Arducam’s examples do, at rates not published.

5.4 What a General Camera Gives Up

The filter. A camera sold for daylight has an infrared-cut filter that must be removed or bought without (“NV” modules), and an infrared-pass filter added. The PixArt modules have the right filter built in (IR Point Tracking, Volume 2).

The global shutter. Both PixArt sensors are global-shutter parts. The OV2640 and HM01B0 are rolling-shutter parts, which read each row at a different moment. In ESP32-IR’s configuration the rows span 7.35 ms from top to bottom. A gun swinging at 180° a second turns 1.3° in that time, about 46 mm at 2 m: the top emitters and the bottom emitters are imaged from different aims, and the quadrilateral the homography sees is sheared. At a gentler 30° a second it is about 8 mm. The error is transient, present only while the gun moves, and it is an error the PixArt camera does not make. It also rules out the synchronised pulsed bar of Volume 2 for these sensors (§2.8 there).

The processor. The PixArt die does labelling and centroiding for free and hands over sixteen objects by SPI. A general camera’s pixels must all pass through the microcontroller: 5.7 million a second in ESP32-IR’s mode, which it handles inside the DMA copy, and 18.4 million a second at VGA and 60 fps, where Border Tracking, Volume 2, §2.9 counts 26 cycles a pixel on an ESP32-S3 and 14 on an RP2040.

What it gains. More pixels: 240 to 640 across against the PixArt’s 98, so each pixel subtends less angle and the centroid can be finer, if the noise allows. And any lens: a general camera takes an M12 or smaller lens of any focal length, so the field of view, which the PixArt range offers in only two sizes, becomes a design choice.

5.5 A Border Tracker in the Gun

Figure 3 — A streaming border tracker for a microcontroller. The camera feeds lines by DMA into a small buffer; each pixel is compared with a threshold as it arrives; for each row and column, only the positio… — Figure 3 — A streaming border tracker for a microcontroller. The camera feeds lines by DMA into a small buffer; each pixel is compared with a threshold as it arrives; for each row and column, only the positions where the image crosses from dark to bright are recorded. At the end of each frame, four straight lines are fitted through those edge points, intersected into four corners, and the homography computed. Memory holds the edge lists, a few kilobytes, never the frame. Own work.

Border Tracking left one question here: could the border method run on the gun’s own microcontroller, so that a border gun is a plain USB pointer with no host driver for the tracking? Its Volume 2 did the processor arithmetic and concluded that a threshold-and-scan design that never stores a frame is feasible at QVGA and plausible at VGA on an ESP32-S3. ESP32-IR shows where such code lives: inside the camera driver’s DMA copy, where its blob detector already runs. A border scan is simpler than blob labelling: one threshold and a few comparisons a pixel.

The harder limit is optical. A Sinden-style border must be several camera pixels wide to be found reliably, and the camera’s field must hold the whole picture from the closest playing distance.

Figure 4 — The width of a 32-pixel border on a 55-inch 1080p picture (20.3 mm), as seen by four cameras, against distance. With a 63° lens, the narrowest that holds the picture from 1 m, a VGA sensor sees the… — Figure 4 — The width of a 32-pixel border on a 55-inch 1080p picture (20.3 mm), as seen by four cameras, against distance. With a 63° lens, the narrowest that holds the picture from 1 m, a VGA sensor sees the border as 6 pixels at 2 m and 4 at 3 m; QVGA 3 and 2; the HQVGA mode of ESP32-IR 2.2 and 1.5. The PixArt R3 sees it as one pixel at 1 m. The shaded band is below 2 pixels. Own work.

The chart sets the requirement: VGA, or a border wider than Sinden’s. The HQVGA mode that gives ESP32-IR its 136 frames a second sees a standard border as barely two pixels at 2 m. A VGA sensor at 50 to 60 frames a second, the rate Border Tracking, Volume 8, §8.5 names as the threshold that would change its verdict, needs the ESP32-S3’s parallel camera port, since the Pico modules’ one-bit links stop near 20 VGA frames a second. Nobody has published this. And it would remove only one of border tracking’s costs: the border must still be drawn by host software on every platform, in every video path.

5.6 The Bench Tests

Test 1: ESP32-IR as published. Build it on a Freenove FNK0085 (US$19.99, store.freenove.com, 19 October 2026) with an OV2640 NV module and an 850 nm long-pass filter, aimed at the kit’s 850 nm bar. Measure the frame rate on the camera’s VSYNC with the ADP3450 logic analyser, and latency by stepping one bar point on from the pattern generator and timing the first serial report that shows it. Then hold the board in the same fixture as the R3 and compare centroid noise in screen millimetres at 2 m and 3 m (IR Point Tracking, Volume 6 has the procedure).

Test 2: rolling-shutter shear. Swing both cameras together on a pivot driven at a known rate (a printed arm on a stepper is enough) past the bar, and log the apparent quadrilateral’s shear against the R3’s. The arithmetic in 5.4 predicts tens of millimetres at fast swings.

Test 3: a border scan. Only if Test 1 is promising: write the streaming scan of the figure above into the same driver hook, run it at the highest VGA rate the OV2640 sustains on the S3, and compare with the host-side cygun trial that decision BT-2 proposes.

5.7 Why Try This, and Why Not the Others

What it buys the build. A second source of camera if PixArt supply fails, built from parts sold in millions, on the ESP32-S3 that the D25 wireless trial puts on the bench anyway. Later, possibly, a border mode in the gun.

What it costs. A US$19.99 board and a filter; for the border, real firmware effort.

The risk. Rolling-shutter shear when the gun moves; the processor cost of every pixel; a filter and lens to source; and the work sits on the ESP32-S3 track, which decision D30 keeps separate from the Pico firmware being fixed. None of it touches the kit gun.

Why try this, and why not the others. Later, not first. The synchronised bar and retroreflectors (Volumes 2 and 3) improve the camera already chosen; this replaces it with a worse-shuttered one, justified only as insurance on supply. Run Test 1 when the ESP32-S3 boards for the D25 trial are on the bench, because it costs one evening there. Leave the border scan until the border trial (BT-2) shows border tracking is wanted at all.

5.8 What Could Not Be Verified

ESP32-IR’s frame rate and latency (author’s figures); its aiming integration is unfinished.
The OV2640’s achievable VGA frame rate on the ESP32-S3 in greyscale.
The HM0360’s shutter type and its module’s maximum serial clock; RP2040 capture rates for either Himax module.
Centroid noise of any general camera in this use.

Sources

frosted123456, ESP32-IR (“ESP32 asnd ESP32S3 High speed blob detection”), GitHub, README, created 2026-08-03, read via the GitHub API 2026-10-19.
frosted123456, LIGHTGUN-STUDIO-ESP32S3-OV2640-New-quad-Pipeline-Calibration-and-RX and openfire-ov2640-overlay (GPL-3.0, 2026-08-13), GitHub, READMEs, read 2026-10-19.
Arducam, HM01B0 Monochrome QVGA SPI Camera Module for Raspberry Pi Pico (B0315) and HM0360 VGA SPI Camera Module for Raspberry Pi Pico (B0319, US$19.99), product pages, read 2026-10-19.
Freenove, ESP32-S3 CAM Board FNK0085, store.freenove.com product data: 8 MB flash US$19.99, 16 MB US$21.99, read 2026-10-19.
Border Tracking, Volumes 2 and 8; Reference Electronics, Volume 3; decision log rows D18, D25, D30, D32, BT-2.
Arithmetic: 04-templates/calc_experimental.py.