Adding support for USB UVC devices

[nickverlinden]

I am thinking about trying to add some basic support for UVC devices to capture video frames from low res web cams or video capture devices over USB. I looked at some example code for libusb and it doesn't look so complicated. Converting YUV frames to RGB is also relatively simple. Recently i did something with an rgb buffer and arm neon instructions using Circle on a pi4 and it seems to be up to speed processing wise. I'm not super familiar with C++, most of the code i write is plain old C. Usually i look at similar examples to try and figure out with I need to do. I briefly looked at some of the usb stuff in Circle, but it does not seem to be that simple (like the libusb functions in the example code below). Do you have a good suggestion to look at inside the Circle source repo to get me started?

This code is untested, but at first glance it seems to be quite simple to set up using libusb.

#include <libusb-1.0/libusb.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define VENDOR_ID 0x0000  // Replace with your camera's vendor ID
#define PRODUCT_ID 0x0000 // Replace with your camera's product ID

#define UVC_VS_FRAME_UNCOMPRESSED 0x05
#define UVC_VS_FORMAT_UNCOMPRESSED 0x04

#define MAX_USB_TRANSFERS 5
#define FRAME_WIDTH 640   // Adjust based on your camera's capabilities
#define FRAME_HEIGHT 480  // Adjust based on your camera's capabilities
#define FRAME_SIZE (FRAME_WIDTH * FRAME_HEIGHT * 3) // RGB format

struct uvc_stream_ctrl {
    uint16_t bmHint;
    uint8_t bFormatIndex;
    uint8_t bFrameIndex;
    uint32_t dwFrameInterval;
    uint16_t wKeyFrameRate;
    uint16_t wPFrameRate;
    uint16_t wCompQuality;
    uint16_t wCompWindowSize;
    uint16_t wDelay;
    uint32_t dwMaxVideoFrameSize;
    uint32_t dwMaxPayloadTransferSize;
};

static libusb_device_handle *dev_handle = NULL;
static uint8_t *rgb_buffer = NULL;
static struct libusb_transfer *transfers[MAX_USB_TRANSFERS];

static void LIBUSB_CALL cb_xfr(struct libusb_transfer *xfr) {
    if (xfr->status != LIBUSB_TRANSFER_COMPLETED) {
        fprintf(stderr, "Transfer failed: %d\n", xfr->status);
        return;
    }

    // Process the received frame (convert from YUY2 to RGB)
    uint8_t *yuy2_frame = xfr->buffer;
    for (int i = 0, j = 0; i < FRAME_SIZE; i += 6, j += 4) {
        int y0 = yuy2_frame[j];
        int u = yuy2_frame[j + 1];
        int y1 = yuy2_frame[j + 2];
        int v = yuy2_frame[j + 3];

        rgb_buffer[i] = y0 + 1.4075 * (v - 128);
        rgb_buffer[i + 1] = y0 - 0.3455 * (u - 128) - 0.7169 * (v - 128);
        rgb_buffer[i + 2] = y0 + 1.7790 * (u - 128);

        rgb_buffer[i + 3] = y1 + 1.4075 * (v - 128);
        rgb_buffer[i + 4] = y1 - 0.3455 * (u - 128) - 0.7169 * (v - 128);
        rgb_buffer[i + 5] = y1 + 1.7790 * (u - 128);
    }

    // Resubmit the transfer
    if (libusb_submit_transfer(xfr) < 0) {
        fprintf(stderr, "Error re-submitting transfer\n");
    }
}

int init_uvc_camera() {
    int r = libusb_init(NULL);
    if (r < 0) {
        fprintf(stderr, "Failed to initialize libusb\n");
        return -1;
    }

    dev_handle = libusb_open_device_with_vid_pid(NULL, VENDOR_ID, PRODUCT_ID);
    if (!dev_handle) {
        fprintf(stderr, "Failed to open device\n");
        return -1;
    }

    if (libusb_claim_interface(dev_handle, 0) < 0) {
        fprintf(stderr, "Failed to claim interface\n");
        return -1;
    }

    // Configure UVC stream
    struct uvc_stream_ctrl ctrl;
    memset(&ctrl, 0, sizeof(ctrl));
    ctrl.bFormatIndex = 1;
    ctrl.bFrameIndex = 1;
    ctrl.dwFrameInterval = 333333; // 30 fps

    // Send UVC probe and commit commands
    uint8_t probe_buf[34];
    memcpy(probe_buf, &ctrl, 26);
    
    r = libusb_control_transfer(dev_handle, 0x21, 0x01, UVC_VS_PROBE_CONTROL << 8,
                                1, probe_buf, sizeof(probe_buf), 5000);
    if (r < 0) {
        fprintf(stderr, "Failed to set probe\n");
        return -1;
    }

    r = libusb_control_transfer(dev_handle, 0x21, 0x01, UVC_VS_COMMIT_CONTROL << 8,
                                1, probe_buf, sizeof(probe_buf), 5000);
    if (r < 0) {
        fprintf(stderr, "Failed to set commit\n");
        return -1;
    }

    // Allocate RGB buffer
    rgb_buffer = malloc(FRAME_SIZE);
    if (!rgb_buffer) {
        fprintf(stderr, "Failed to allocate RGB buffer\n");
        return -1;
    }

    // Set up USB transfers
    for (int i = 0; i < MAX_USB_TRANSFERS; i++) {
        transfers[i] = libusb_alloc_transfer(0);
        if (!transfers[i]) {
            fprintf(stderr, "Failed to allocate transfer %d\n", i);
            return -1;
        }

        uint8_t *buffer = malloc(FRAME_SIZE);
        libusb_fill_bulk_transfer(transfers[i], dev_handle, 0x81, buffer, FRAME_SIZE, cb_xfr, NULL, 5000);

        r = libusb_submit_transfer(transfers[i]);
        if (r < 0) {
            fprintf(stderr, "Failed to submit transfer %d\n", i);
            return -1;
        }
    }

    return 0;
}

void cleanup_uvc_camera() {
    for (int i = 0; i < MAX_USB_TRANSFERS; i++) {
        libusb_cancel_transfer(transfers[i]);
        free(transfers[i]->buffer);
        libusb_free_transfer(transfers[i]);
    }

    free(rgb_buffer);
    libusb_release_interface(dev_handle, 0);
    libusb_close(dev_handle);
    libusb_exit(NULL);
}

uint8_t* get_frame() {
    libusb_handle_events_timeout(NULL, &(struct timeval){0, 100000});
    return rgb_buffer;
}

I can see that the usb device descriptor for a composite video capture device is like this, and it seems that it only returns MJPEG frames, which is fine too. There is source for embedded mjpeg decoders out there.

Bus 000 Device 002: ID 18ec:5555  
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass          239 [unknown]
  bDeviceSubClass         2 [unknown]
  bDeviceProtocol         1 
  bMaxPacketSize0        64
  idVendor           0x18ec 
  idProduct          0x5555 
  bcdDevice            1.00
  iManufacturer           1 
  iProduct                2 
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x01d6
    bNumInterfaces          4
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              200mA
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         0
      bInterfaceCount         2
      bFunctionClass         14 [unknown]
      bFunctionSubClass       3 [unknown]
      bFunctionProtocol       0 
      iFunction               0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      1 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      VideoControl Interface Descriptor:
        bLength                13
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdUVC               1.00
        wTotalLength       0x0033
        dwClockFrequency       48.000000MHz
        bInCollection           1
        baInterfaceNr( 0)       1
      VideoControl Interface Descriptor:
        bLength                18
        bDescriptorType        36
        bDescriptorSubtype      2 (INPUT_TERMINAL)
        bTerminalID             1
        wTerminalType      0x0201 Camera Sensor
        bAssocTerminal          0
        iTerminal               0 
        wObjectiveFocalLengthMin      0
        wObjectiveFocalLengthMax      0
        wOcularFocalLength            0
        bControlSize                  3
        bmControls           0x00000000
      VideoControl Interface Descriptor:
        bLength                11
        bDescriptorType        36
        bDescriptorSubtype      5 (PROCESSING_UNIT)
      Warning: Descriptor too short
        bUnitID                 2
        bSourceID               1
        wMaxMultiplier          0
        bControlSize            2
        bmControls     0x0000063f
          Brightness
          Contrast
          Hue
          Saturation
          Sharpness
          Gamma
          Gain
          Power Line Frequency
        iProcessing             0 
        bmVideoStandards     0x09
          None
          SECAM - 625/50
      VideoControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (OUTPUT_TERMINAL)
        bTerminalID             3
        wTerminalType      0x0101 USB Streaming
        bAssocTerminal          0
        bSourceID               2
        iTerminal               0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0010  1x 16 bytes
        bInterval              10
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      VideoStreaming Interface Descriptor:
        bLength                            14
        bDescriptorType                    36
        bDescriptorSubtype                  1 (INPUT_HEADER)
        bNumFormats                         1
        wTotalLength                   0x0057
        bEndpointAddress                 0x83  EP 3 IN
        bmInfo                              0
        bTerminalLink                       3
        bStillCaptureMethod                 2
        bTriggerSupport                     1
        bTriggerUsage                       0
        bControlSize                        1
        bmaControls( 0)                     0
      VideoStreaming Interface Descriptor:
        bLength                            11
        bDescriptorType                    36
        bDescriptorSubtype                  6 (FORMAT_MJPEG)
        bFormatIndex                        1
        bNumFrameDescriptors                1
        bFlags                              1
          Fixed-size samples: Yes
        bDefaultFrameIndex                  1
        bAspectRatioX                       0
        bAspectRatioY                       0
        bmInterlaceFlags                 0x00
          Interlaced stream or variable: No
          Fields per frame: 1 fields
          Field 1 first: No
          Field pattern: Field 1 only
        bCopyProtect                        0
      VideoStreaming Interface Descriptor:
        bLength                            42
        bDescriptorType                    36
        bDescriptorSubtype                  7 (FRAME_MJPEG)
        bFrameIndex                         1
        bmCapabilities                   0x00
          Still image unsupported
        wWidth                            640
        wHeight                           480
        dwMinBitRate                   128000
        dwMaxBitRate                 11059200
        dwMaxVideoFrameBufferSize      614400
        dwDefaultFrameInterval         333333
        bFrameIntervalType                  4
        dwFrameInterval( 0)            166666
        dwFrameInterval( 1)            333333
        dwFrameInterval( 2)            666666
        dwFrameInterval( 3)           2000000
      VideoStreaming Interface Descriptor:
        bLength                            14
        bDescriptorType                    36
        bDescriptorSubtype                  3 (STILL_IMAGE_FRAME)
        bEndpointAddress                 0x00  EP 0 OUT
        bNumImageSizePatterns               1
        wWidth( 0)                        640
        wHeight( 0)                       480
        bNumCompressionPatterns             4
        bCompression( 0)                    1
        bCompression( 1)                    2
        bCompression( 2)                    5
        bCompression( 3)                   14
      VideoStreaming Interface Descriptor:
        bLength                             6
        bDescriptorType                    36
        bDescriptorSubtype                 13 (COLORFORMAT)
        bColorPrimaries                     0 (Unspecified)
        bTransferCharacteristics            0 (Unspecified)
        bMatrixCoefficients                 0 (Unspecified)
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       1
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x00c0  1x 192 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       2
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x0180  1x 384 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       3
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       4
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x0280  1x 640 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       5
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x0320  1x 800 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       6
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x03c0  1x 960 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       7
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x0a80  2x 640 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       8
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x0b20  2x 800 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       9
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x0be0  2x 992 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting      10
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x1380  3x 896 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting      11
      bNumEndpoints           1
      bInterfaceClass        14 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x13e8  3x 1000 bytes
        bInterval               1
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         2
      bInterfaceCount         2
      bFunctionClass          1 [unknown]
      bFunctionSubClass       1 [unknown]
      bFunctionProtocol       0 
      iFunction               4 USB2.0 MIC
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass         1 [unknown]
      bInterfaceSubClass      1 [unknown]
      bInterfaceProtocol      0 
      iInterface              4 USB2.0 MIC
      AudioControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdADC               1.00
        wTotalLength       0x0028
        bInCollection           1
        baInterfaceNr(0)        3
      AudioControl Interface Descriptor:
        bLength                12
        bDescriptorType        36
        bDescriptorSubtype      2 (INPUT_TERMINAL)
        bTerminalID             1
        wTerminalType      0x0201 Microphone
        bAssocTerminal          0
        bNrChannels             2
        wChannelConfig     0x0003
          Left Front (L)
          Right Front (R)
        iChannelNames           0 
        iTerminal               0 
      AudioControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (OUTPUT_TERMINAL)
        bTerminalID             3
        wTerminalType      0x0101 USB Streaming
        bAssocTerminal          0
        bSourceID               2
        iTerminal               0 
      AudioControl Interface Descriptor:
        bLength                10
        bDescriptorType        36
        bDescriptorSubtype      6 (FEATURE_UNIT)
        bUnitID                 2
        bSourceID               1
        bControlSize            1
        bmaControls(0)       0x00
        bmaControls(1)       0x03
          Mute Control
          Volume Control
        bmaControls(2)       0x03
          Mute Control
          Volume Control
        iFeature                0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        3
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass         1 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              4 USB2.0 MIC
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        3
      bAlternateSetting       1
      bNumEndpoints           1
      bInterfaceClass         1 [unknown]
      bInterfaceSubClass      2 [unknown]
      bInterfaceProtocol      0 
      iInterface              4 USB2.0 MIC
      AudioStreaming Interface Descriptor:
        bLength                 7
        bDescriptorType        36
        bDescriptorSubtype      1 (AS_GENERAL)
        bTerminalLink           3
        bDelay                  1 frames
        wFormatTag         0x0001 PCM
      AudioStreaming Interface Descriptor:
        bLength                11
        bDescriptorType        36
        bDescriptorSubtype      2 (FORMAT_TYPE)
        bFormatType             1 (FORMAT_TYPE_I)
        bNrChannels             1
        bSubframeSize           2
        bBitResolution         16
        bSamFreqType            1 Discrete
        tSamFreq[ 0]        48000
      Endpoint Descriptor:
        bLength                 9
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            5
          Transfer Type            Isochronous
          Synch Type               Asynchronous
          Usage Type               Data
        wMaxPacketSize     0x0060  1x 96 bytes
        bInterval               4
        bRefresh                0
        bSynchAddress           0
        AudioStreaming Endpoint Descriptor:
          bLength                 7
          bDescriptorType        37
          bDescriptorSubtype      1 (EP_GENERAL)
          bmAttributes         0x01
            Sampling Frequency
          bLockDelayUnits         0 Undefined
          wLockDelay         0x0000
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass          239 [unknown]
  bDeviceSubClass         2 [unknown]
  bDeviceProtocol         1 
  bMaxPacketSize0        64
  bNumConfigurations      1

[rsta2]

This looks complicated. Because this device uses isochronous endpoints, there may be some similarity with the USB audio streaming driver.

[nickverlinden]

i'm comfortable with i2c, so i can figure it out to set the device up. I'm not familiar with CSI2, so i wanted to ask you first if there was any reason for it not to work (resolution does not match the camera modules, there is no need for debayering, ...) since it's an expensive module. Ideally i think it would benefit other people more to invest time in implementing UVC, since there are tons of (cheap) usb video capture devices out there (webcams, composite video inputs, hdmi inputs, ...), but looking at the audio streamer code, i think i'm not going to be able to hack my way in. Perhaps i'll give it a shot later this year.

[rsta2]

A question: What do you want to do with the grabbed video data? The problem is, that the processing is not very fast and also the SD card driver can write maybe 1 or 2 MBytes/s, but not probably not continuously. Unfortunately I'm not sure, if Circle is the right platform for video processing. The camera library works well for taking photos, but for video streaming it reaches 12 fps on a Raspberry Pi 3 A+ for an 1280x1024 image. It's also not sure, if the CSI and/or USB driver works without modifications for this purpose.

[nickverlinden]

Well, i dont need much actually. I ported over some code for encoding and decoding video onto an audio signal. It's for an art installation. The max resolution of the video is about 80x80@10fps, or if you want 'high-res' images 240x240@1fps. I already made a standalone decoder using circle on a pi4 with some nice visual fx. I think the decoder part is much more cpu intensive because it has to do some fading effect of scanlines at 60fps, i'm using neon instructions to do that, and for the standard 800x480 resolution screen it just about works (reading the framebuffer, do some fading operation on each pixel, and write back to framebuffer). So i think the power that i need is there. Right now the encoding part is done by a macbook, but like to make it standalone.

[rsta2]

This image size and frame rate sounds doable. I think the Linux driver for this device is here. The RPi cameras use two CSI-2 lanes and 8- or 10-bit Bayer colors. My CSI-2 class is here. I think it needs modifications. Don't blame me, in case you get it not working. ;)

[nickverlinden]

Thanks, i found that kernel driver too. Doesn't look too complicated, but i gave it some thought, and think it would be better to try and see if i can get uvc working. it's much more flexible input wise (composite video, webcam, hdmi, ...) and much cheaper. I'll try to hack around later this year :-).