代码拉取完成，页面将自动刷新
Copyright 2023 Canaan Inc. ©
The products, services or features you purchase should be subject to Canaan Inc. ("Company", hereinafter referred to as "Company") and its affiliates are bound by the commercial contracts and terms and conditions of all or part of the products, services or features described in this document may not be covered by your purchase or use. Unless otherwise agreed in the contract, the Company does not provide any express or implied representations or warranties as to the correctness, reliability, completeness, merchantability, fitness for a particular purpose and non-infringement of any statements, information, or content in this document. Unless otherwise agreed, this document is intended as a guide for use only.
Due to product version upgrades or other reasons, the content of this document may be updated or modified from time to time without any notice.
, "Canaan" and other Canaan trademarks are trademarks of Canaan Inc. and its affiliates. All other trademarks or registered trademarks that may be mentioned in this document are owned by their respective owners.
Copyright 2023 Canaan Inc.. © All Rights Reserved. Without the written permission of the company, no unit or individual may extract or copy part or all of the content of this document without authorization, and shall not disseminate it in any form.
This article explains how to implement the USB camera function on the k230 development board. That is, connect the K230 development board to the computer, and the computer can play the image collected by the K230 camera through the player.
The toolchains are provided in the k230_sdk and are available in the following paths.
k230_sdk/toolchain/riscv64-linux-musleabi_for_x86_64-pc-linux-gnu
k230_sdk/toolchain/Xuantie-900-gcc-linux-5.10.4-glibc-x86_64-V2.6.0
The toolchain can also be downloaded via the link below
wget https://download.rt-thread.org/rt-smart/riscv64/riscv64-unknown-linux-musl-rv64imafdcv-lp64d-20230222.tar.bz2
wget https://occ-oss-prod.oss-cn-hangzhou.aliyuncs.com/resource//1659325511536/Xuantie-900-gcc-linux-5.10.4-glibc-x86_64-V2.6.0-20220715.tar.gz
Refer to section 2/3/4/5 of the K230_SDK_ Instructions for Use
Refer to section 2.9 UVC_demo of the K230_SDK_Demo User Guide
The USB protocol has more content, is a very common interface, and there is a lot of information on the Internet. This article describes only a few points that I think will help in understanding the USB protocol.
USB2.0 has 4 lines, VBUS/GND/D+/D-, differential signal transmission. 3.3V at low speed/full speed, and 400mV at high speed. In addition to the differential 0/1 signal that carries the data, other voltage combinations can be used as speed identification, idle, reset, wake-up signals, and so on.
PHY can be understood as doing a parallel string operation to convert UTMI+ signals into differential signals of D+/D-.
Theoretically, the two data signals can be controlled using GPIO, but the protocol is too complex and GPIO changes too slowly. Whether it is SPI/SDIO/UART/IIC, etc., these interface controllers are like this, the controller provides the necessary interfaces to the software as much as possible, and let the software do as few operations as possible. The USB controller is to let the hardware do the functions of the protocol as much as possible, such as speed negotiation, automatic generation of response packets after receiving data packets, and other functions. A number of register interfaces are provided to inform the software of the current status of USB communication. Of course, the most important thing is to provide an interface for transmitting data, so that the software can send and receive different data, and DMA is generally used to send and receive data.
USB transmission data is based on packets. The following figure shows the composition of the package
PID determines the type of packet, token packet, data packet, handshake packet and special packet, different packet types contain different field components, such as packet only contains PID + data + CRC, handshake package only PID.
Packets make up transactions, token packets + packets (optional) + handshake packets (optional). The following figure uses the USB interaction information captured by Nanjing Qinheng's USB2.0 high-speed bus analyzer, and if you want to capture data on the USB line, this instrument is recommended.
You can see that the front is an enumeration transfer, which contains a SETUP transaction, a data IN/OUT transaction, and a status IN/OUT transaction. This is followed by synchronous transmission, which contains data IN transactions.
Transfers consist of single or multiple transactions and there are 4 transport types:
Control Transfer - Used during the enumeration phase, all USB devices connected to the host require a uniform set of protocols to identify the various USB device types. The USB controller initially makes the 0 endpoint a bidirectional control endpoint.
Interrupt transmission - small data volume and discontinuity, and real-time performance requirements are high. For example, mouse and keyboard.
Synchronous transmission - where the amount of data is large, continuous, and real-time is required. For example, USB camera devices.
Batch transfer - used in situations where the amount of data is large, but there is no real-time requirement. For example, U disk equipment.
No matter what kind of USB device protocol, it is realized through these 4 kinds of transmission. Therefore, the USB protocol stack of operating systems like Linux provides an interface for these four transmission methods.
The USB descriptor is used to let the host know the attribute information of the device. When a device first connects to a host, the host sends a request command that all devices support. Common descriptors include device descriptors, configuration descriptors, interface descriptors, endpoint descriptors, and string descriptors. Different device types may define unique descriptors that extend the description of the device.
Windows can use the UsbTreeView software to view the descriptors of USB devices. The following figure shows the overall layout of a UVC device with its descriptors.
In this layout of the layout, the first item is the device descriptor, followed by the configuration descriptor, which the device possesses. The configuration descriptor is followed by an interface association descriptor IAD, which has a video control interface VC and N video stream interfaces.
The video control interface includes a video control interface header descriptor, an input terminal descriptor, a processing unit descriptor, a coding unit descriptor, an output terminal descriptor, and an interrupt breakpoint descriptor.
The video stream interface includes an interface and a number of corresponding Alternate Setting.
Through the video control interface descriptor, the host side can know the topology of the UVC camera and control it. For example, the processing unit PU, including backlight, contrast, chrominance, etc., the host side first knows which are adjustable items through the descriptor, and then interacts with the UVC device to obtain control range information.
The VS interface contains many formats (YUV/MJPEG/H264, etc.), and each format contains multiple frames (various resolutions). The process of parameter setting requires negotiation between the host and the USB device, and the negotiation process is roughly shown in the following figure:
Process Description:
You can see that the payload data contains a header in front of it, and the payload data contains multiple USB packets, how does the host identify a frame of image data?
The payload header fixes the first 2 bytes and the subsequent extensions. Focus on FID - different formats have different regulations. But all formats use this bit to switch between 0 and 1 to recognize a new frame of image data.
The SDK design of the K230 puts the video and audio functions on the large-core RTT for performance. The function implementation of USB on Linux is very mature, so UVC function is developed based on Linux to obtain large-core video data through inter-core mapi communication.
The K230 integrates a Synopsys USB module, and the linux/drivers/usb/dwc2 directory is the driver for this controller.
The current SDK design UVC uses USB1 fixedly, and otg is recognized as device mode through the ID signal.
platform.c, do some reset, register interrupts, configure parameters and other operations. The USB of the SDK supports buffer DMA mode by default. Scatter Gather DMA mode can improve ISO transmission performance, HOST mode cannot support HUB using this DMA mode.
gadget.c, which focuses on the driver code, dwc2_gadget_init initializes important structs, gadget.ops and ep.ops. usb_ep_ops queue is a request to submit data to send and receive, which is actually put into a linked list for processing. USB will process the requests on these linked lists one by one and then report a completion callback.
//device tree
usbotg1: usb-otg@91540000 {
compatible = "kendryte,k230-otg";
reg = <0x0 0x91540000 0x0 0x10000>;
interrupt-parent = <&intc>;
interrupts = <174>;
g-rx-fifo-size = <512>;
g-np-tx-fifo-size = <64>;
g-tx-fifo-size = <512 1024 64 64 64 64>;
dr_mode = "otg";
otg-rev = <0x200>;
};
The source code is located at linux/drivers/usb/gadget
The UVC of the K230 hardly modifies the gadget driver layer, only porting the H264 format support function and expansion unit.
Linux compiles the kernel to add the USB Gadget framework
-> Device Drivers
-> USB support
-> USB Gadget Support
UVC involves the functionality of the V4L2 module, and the Media framework needs to be added
-> Device Drivers
-> Multimedia support
-> Media core support
-> Media core support
From the SDK design architecture of K230, it can be seen that the difference between the UVC function of K230 and the UVC function on Linux is that the way to obtain video data is obtained from the large core RTT through inter-core communication.
The code location of the K230 UVC application layer: cdk/user/mapi/sample/camera
Source code file description:
application.c - Main function
camera.c - Provides camera object manipulation and can include UVC/UAC controls
frame_cache.c - Management of complex buffers
kstream.c - Implements video streaming operations
kuvc.c - Implements kuvc object operations
sample_venc.c - The operation of acquiring encoded images from large nuclei via MAPI
sample_yuv.c - Operation to acquire YUV images from large nuclei via mapi
UVC-gadget.c - Implements UVC device operation
The debugging step is to first port the common uvc_app on Linux and run through the standalone function on Linux, that is, play fixed image data. Then debug the ability to play back real camera image data.
Below is the design flowchart of the K230 UVC APP
Most of the operations of UVC APP are general operations, and there are more materials on the Internet. Here we talk about the K230 private operation, focusing on the processing of venc_normalp_classic functions.
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
马建仓 AI 助手
