Using ISI (Image Sensor Interface)

Last modified by Microchip on 2025/06/09 11:30

Introduction

This page is mainly about how to enable and configure the ISI in AT91SAM SoCs for different image sensors.

The Image Sensor Interface (ISI) connects a CMOS-type image sensor to the processor and provides image capture in various formats from sensor side.
Use H/VSYNC signal for synchronization or EAV/SAV.
ISI supported sensor input formats
- YCbCr422, RGB565, RGB888 and grayscale raw data.
ISI supported output formats
- ISI has two paths (Preview path and Codec path) for the output.
  - Preview path will output RGB data with different format.
    - Can convert YCbCr or YUV to RGB.
    - Support downscale and decimation.
    - Max output solution is 640x480.
  - Codec path will output YUV data with different orders.
    - Can convert RGB to YCrCb.
    - Max output solution is 2048x2048.

Prerequisites

Linux Kernel v3.6.9 , Linux Kernel v3.10 , Linux Kernel v3.18 in Linux4sam, all support ISI.
Linux4SAM 6.0 and Linux Kernel v4.14 ISI support is explained in
this page
- Supports ISI of the SAM9G45/9M10, SAM9G25 of SAM9x5 series, SAM9N12 and SAMA5 series.
  - SAM9263 ISI is not supported since it use an old ISI IP.
  - In SAM9x5 series only SAM9G25 support ISI. And SAM9G25 has no LCD controller.
- In SAMA5D3x-EK boards:
  - J11's PIN29 need to be disconnected when insert ISI module board. Since PIN29 is connected to ISI_D11 (pin mux as TWD1).
  - TWI0 (i2c0)'s TWD0 & TWCK0 signals are using same pins as ISI_VSYNC & ISI_HSYNC. So need to disable i2c0 when you enable ISI.
  - TWI1 (i2c1)'s TWD1 & TWCK1 signals are using same pins as ISI_D11 & ISI_D10. So it cannot support 12bit data input.
  - Led d3 should be disable as well as it conflict with camera sensor's reset pin PE24.
- Supported CMOS sensors: OV2640, OV2643, OV5640, OV7740 and OV9740.

Warning: Currently the dts file of SAMA5D3x-EK disabled ISI by default. To enable ISI, you need uncomment the dts file to include sama5d3xmb_isi_sensors.dtsi, and disable i2c0 and led D3..

Detail description of software

ISI driver in Linux kernel

ISI driver support the standard v4L2 APIs. And it is in soc-camera framework.
Current supported sensors: OV2640, OV2643, OV5640, OV7740 and OV9740.
It's easy to support a sensor if the sensor is
- Using DVP interface connection. Means it uses H/VSYNC signals.
- In soc-camera framework. You can find all soc-camera supported sensors in Kernel menuconfig:
  - all the items under the menu: "Device Drivers -> Multimedia support -> Sensors used on soc_camera driver"
- In platform camera framework. You can find all camera supported platform sensors in Kernel menuconfig by:
  - select the menu: "Device Drivers -> Multimedia support -> Media Controller API"
  - select the menu: "Device Drivers -> Multimedia support -> V4L2 sub-device userspace API"
  - deselect the menu: "Device Drivers -> Multimedia support -> Autoselect ancillary drivers (tuners, sensors, i2c, frontends)"
  - Then all support sensors can be found in the menu: "Device Drivers -> Multimedia support -> Encoders, decoders, sensors and other helper chips"

Tips: to add a new sensor support, you also need to modify the dts file to add the sensor remote port. Like arch/arm/boot/dts/at91sam9x5ek.dtsi to add the new sensor's i2c information and configure the pin mux.

User Applications support ISI

CameraApp

fswebcam

fswebcam is a neat and simple webcam app. It captures images from a V4L1/V4L2 compatible device or file, averages them to reduce noise and draws a caption using the GD Graphics Library which also handles compressing the image to PNG or JPEG. The resulting image is saved to a file or sent to stdio where it can be piped to something like ncftpput or scp.

Add fswebcam in Buildroot
- Select "Package Selection for the target -> Graphic libraries and applications -> fswebcam".
Use fswebcam to capture a image. <pre>

#/bin/sh

VIDEO_DEV=/dev/video0 SKIP_FRAMES=20

# test preview channel fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p RGB565 -r 640x480 rgb565.jpg fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p RGB565 -r 320x240 rgb565_defactor.jpg

# test codec channel fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p YUYV -r 640x480 yuyv.jpg fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p YUYV -r 800x600 yuyv_800x600.jpg

fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p UYVY -r 640x480 uyvy.jpg fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p UYVY -r 800x600 uyvy_800x600.jpg

# test codec channel, without any processing, GREY, or Bayer RGB. fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p BAYER -r 640x480 bayer_bggr8.jpg fswebcam -S ${SKIP_FRAMES} -d ${VIDEO_DEV} -p SGRBG8 -r 640x480 bayer_grbg8.jpg

</pre>

-S: frames that need to skip.
-d /dev/video0: specify the ISI as the input source.
-p: pixel format, can be RGB565, YUYV, UYVY, BAYER, SGRBGB8 and etc.
-r: resolution.

FFmpeg

FFmpeg is a complete, cross-platform solution to record, convert and stream audio and video. It supports video4linux2 in Linux.

Add FFmpeg in Buildroot (It's already included in the Linux4SAM buildroot demo).
- Select "Package Selection for the target -> Audio and video applications -> ffmpeg".
Use FFmpeg to capture a mpeg4 video clip. <pre> ffmpeg -r 25 -s vga -t 20 -pix_fmt yuyv422 -f video4linux2 -i /dev/video0 video.avi </pre>
- -r: frame rate
- -s: resolution, it can be qcif, cif, qvga, vga, svga, xga, uxga.
- -t: time duration in second.
- -pix_fmt: pixel format, only support yuyv422.
- -f video4linux2: specify the format. Use ffmpeg -formats will show all supported formats.
- -i /dev/video0: specify the ISI as the input source. Run following command to check the source name: <pre>

# cat /sys/class/video4linux/video0/name isi-camera </pre>

video.avi: output video file name

Tips: As no vcodec specified, it use mpeg4 as default.

Tips: Run ffmpeg -pix_fmts can show all the supported pixel formats.

Use FFmpeg application to capture images. <pre> ffmpeg -f video4linux2 -r 1 -s vga -t 4 -i /dev/video0 -pix_fmt yuyv422 -f image2 -vcodec png image%d.png </pre>
- -r: frame rate
- -s: resolution, it can be qcif, cif, qvga, vga, svga, xga, uxga.
- -t: time duration in second.
- -i /dev/video0: specify the ISI as the input source. Run following command to check the source name: <pre>

# cat /sys/class/video4linux/video0/name isi-camera </pre>

-pix_fmt: pixel format, only support yuyv422.
-f image2: image2 sequence format. Use ffmpeg -formats will show all supported formats.
-vcodec png: specify output format is png. Use ffmpeg -codecs will show all supported codec.
image%d.png: output file name is image1.png, image2.png and etc.

GStreamer

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.

GStreamer has been ported to a wide range of operating systems, processors and compilers.

Add GStreamer in Buildroot (It's already included in the Linux4SAM Buildroot demo).
- Select "Package Selection for the target -> Audio and video applications -> gstreamer" and the plugins that you needed.
Use GStreamer to preview on LCD. <pre> # gstreamer 0.10 gst-launch v4l2src device="/dev/video1" ! video/x-raw-yuv,width=640,height=480 ! ffmpegcolorspace ! fbdevsink # gstreamer 1.0 gst-launch-1.0 v4l2src device="/dev/video1" ! video/x-raw,width=640,height=480 ! videoconvert ! fbdevsink </pre>
- v4l2src: a plugin to support v4l2 device as a source
  - device="/dev/video1": specify the ISI as the v4l2 input device.
    - You can check the device name by run command: cat /sys/class/video4linux/video1/name
  - video/x-raw-yuv,width=640,height=480: For gstreamer 0.10, specify the v4l2 output video format and size.
  - video/x-raw,width=640,height=480: For gstreamer 1.0, specify the v4l2 output video format and size.
- ffmpegcolorspace or videoconvert: a plugin to convert from one color space to another
- fbdevsink: a plugin to render to framebuffer device

Tips: gstreamer 1.0 is different from 0.10, more detail can check: http://cgit.freedesktop.org/gstreamer/gstreamer/plain/docs/random/porting-to-1.0.txt

Tips: run gst-inspect will show all installed plugins.

Tips: run gst-inspect [plugin name] will show all supported parameters for this plugins.

ZXing barcode reader

ZXing is an open-source, multi-format 1D/2D barcode image processing library implemented in Java, with ports to other languages. The project also includes a barcode reader example.

Add the ZXing barcode reader application in Buildroot.
- Add ZXing libary in Buildroot (The Linux4SAM buildroot demo already include it).
  - Select "Package Selection for the target -> Libraries -> Graphics -> zxing".
- Apply attached patch on top of the buildroot-2012.11.1-at91.
  - This patch will change the zxing project's Makefile to generate not only zxing library but also the barcode reader example.
- Run command make zxing to generate the barcode application: zxing_barcode.
  - zxing_barcode is located on outpout/target/use/bin/.

Read barcode from the image by using ZXing barcode reader.
- Get an picture which include a barcode.
  - Please refer to FFmpeg section for the image capture.
- Run following command to reader the barcode. <pre>zxing_barcode *.jpg</pre>

CameraApp

FAQ

Q: How to enable ISI support in SAMA5D3x-EK board?

1 Find the dts file according to your board type in: arch/arm/boot/dts/sama5d3xek.dts

x is the number of your board type. 1 Uncomment the line to enable isi and sensors. <pre>#include "sama5d3xmb_isi_sensors.dtsi"</pre> 1 Disable led d3 and i2c0. <pre> i2c0: i2c@f0014000 { status = "disabled"; }; ... ... leds { d3 { label = "d3"; gpios = <&pioE 24 GPIO_ACTIVE_HIGH>; status = "disabled"; }; }; </pre> 1 Run following command to re-compile the dts to dtb file in the top of kernel source directory. <pre>make dtbs # before doing this make sure your config is based on sama5_defconfig</pre> 1 Load your kernel with above correct dtb file, which will enable ISI and disable the audio.

Warning: Enable ISI will disable audio as these two components are conflicted in hardware configuration.

Q: How to check the ISI is probed in my board?

1 Check the boot message whether there is an information about sensor probe. <pre> soc-camera-pdrv soc-camera-pdrv.0: Probing soc-camera-pdrv.0 ov2640 0-0030: ov2640 Product ID 26:42 Manufacturer ID 7f:a2 i2c i2c-0: OV2640 Probed soc-camera-pdrv soc-camera-pdrv.1: Probing soc-camera-pdrv.1 ov5642 0-003c: reg_read: i2c read error, reg: 300a ov5642: probe of 0-003c failed with error -121 </pre> In above example, the message shows an OV2640 sensor is probed. But fail to probe of OV5642 as we only support one camera module slot in the board. 1 Print all the v4L2 device in system to check ISI device is existed? <pre> # ls /sys/class/video4linux/video* /sys/class/video4linux/video0: debug dev index name power subsystem uevent

/sys/class/video4linux/video1: debug device name subsystem dev index power uevent # cat /sys/class/video4linux/video0/name

# cat /sys/class/video4linux/video1/name isi-camera </pre> In above example, we can find the video1 is the isi-camera device.

Q: If the ISI cannot be probed in my board, what should I do?

1 Check the kernel boot message to see if there is any error message about ISI and sensor 1 Following the below check list to troubleshoot above errors.

Check the kernel config file:
- Is the ISI driver and the sensor driver enabled?
Check your board's device tree file(.dts, .dtsi):
- Is the ISI device node is enabled?
- Is the ISI pins are configured correctly?
Check arch/arm/mach-at91/board-dt.c file or .dts, .dtsi (in linux-3.18 and later) files:
- Is the sensor's i2c info correct?
- Is the sensor's power/reset pin correct?
- Is the PCK correct?