Introduction

Starting from kernel version linux 5.15 and mainline 6.2, MPU products Linux software requires media controller paradigm for user space configuration.</br> This page describes how to use and understand the media controller pipeline in SAM MPU products.</br> To understand Media Controller, we take SAMA7G5 as example. SAMA7G5 supports both serial/parallel camera image acquisition.</br> hand You can find out more about SAMA7G5 image acquisition pipeline on this page </br> hand In this page we will use the setup with Sony imx219 sensor connected to the board. It is an example, and if you have a different sensor, the pipeline will be slightly different. </br>

Hardware blocks

Please refer to the product datasheet in the IMAGE SUBSYSTEM chapters, where each hardware block is described.</br>

Software pipeline

• The software pipeline is formed out of specific pipeline _entities_.
  • Entities are defined as software constructs that perform a simple video task. It can be for example acquisition, conversion, scaling or outputting video.
• Each entity can have a variable number of _pads_. Pads are ways to send or receive data to or from an entity. Pads can be either _sink_ pads, or _source_ pads.
  • A sink pad by definition is a pad where you can send data to (_sink_ data to).
  • A source pad by definition is a pad that produces (it's a _source_ of) data.
• Pads can be linked together through a _link_
  • You can imagine links like a flow of data. It is then logical to assume that a link is always between a _source_ pad and a _sink_ pad.

</br> hand Entities may or may not correspond to real hardware blocks that process data. Some of them may be pure software implementations.

SAMA7G5 media graph

IMX219 media graph (MIPI CSI2)

<img src="%PUBURLPATH%/%WEB%/%TOPIC%/graph.png" alt="graph.png" width="120" height="500" />

In the graph above, you can see the entities , pads, and links for the SAMA7G5 media pipeline.

This information can be retrieved using the command line tool media-ctl from the v4l2-utils package.

• In this media-ctl output we can find out all the entities in the pipeline, pads and links between them, together with the current configured format for each pad.</br> hand It is important to notice that each entity has a type.

• We can see that one of them is Sensor , the imx219 entity. This is the Sony imx219 sensor.
• We can see that one of them is V4L , the video4linux2 entity. This corresponds to the /dev/video0 classic v4l2 video node.

Querying subdevice capabilities

• Entities can have a subdevice associated.
  • A subdevice is a /dev/v4l-subdev node that can be queried for information regarding the subdevice capabilities.
  • The media-ctl -p command will show for each entity, if there is a v4l2 subdevice associated or not, and which index each subdevice has.</br>

• Let's query our subdevices for capabilities.

Enum mbus codes

• One important capability that each subdevice must have, is a list of mbus (media bus) codes supported.</br> hand The media bus code is a number that represents the format of the data that is supported by the subdevice.</br> * A media entity associated with a subdevice can have multiple pads, and each pad can have different media buses supported. * For example one entity can be a format converter, then it's only logical that it's a different set of mbus codes is supported on the source pad compared to the sink pad. </br> hand A list of mbus codes can be found here</br>

Query the sensor (subdevice 3 ):

• We queried subdev3 to see the mbus codes supported by this pad (pad number 0).
  • We see that there are two codes supported: 0x300f and 0x3014 . These correspond to SRGGB10_1x10 and SRGGB8_1X8 .

Query the dw_csi entity (subdevice 2):

• We queried both pads of the subdev2 and obtained the possible mbus codes for this subdevice on each of its pads.
  • It's good to see that the source pad of the subdevice3 has codes that are also found in the list of codes for the sink pad of the subdevice2.
    • This means that most likely a common format can be found for the link between them.

Query the csi2dc entity (subdevice 1):

• We queried both pads of the subdev1 and obtained the possible mbus codes for this subdevice on each of its pads.

Query the atmel_isc_scaler (subdevice 0):

• We queried both pads of the subdev0 and obtained the possible mbus codes for this subdevice on each of its pads.

Enum frame sizes

• One other important capability that a subdevice can have, is a list of supported frame sizes, or in other words, resolutions.</br> hand Resolutions can be a fixed discrete size, or a continuous size, in which a minimum and a maximum of a frame are specified.</br>

• A media entity associated with a subdevice can have multiple pads, and each pad can have different frame sizes supported.
  • For example one entity can be a scaler, then it's only logical that it's a different set of frame sizes is supported on the source pad compared to the sink pad. </br>

Query the sensor (subdevice 3):

• We queried the subdevice3 to display the framesizes for the given pad, and given mbus code.
  • According to one mbus code or another, a subdevice can have different possible frame sizes.

If we query the sensor for a mbus that is not supported:

• We queried the subdevice3 to display the framesizes for the given pad, and an unsupported mbus code.
  • No frame size was returned.

hand Some subdevices do not support querying for frame size. This means that the frame size is unknown. When configuring the subdevice, one has to be careful if the configuration with a given frame size was successful or not.

Query the subdevice2 for frame size:

• We queried the subdevice2 to display the framesizes for the given pad.
  • We obtained a range of frame sizes from 16x16 to 4000x3000. This means any frame size that fits is supported.

Get current subdevice format (per pad)

• It is important to distinguish between mbus code and format. A format _includes_ an mbus code, but a format also means a frame size, colorspace, and other type of information.
  • We can query the subdevice to return the current configured format per pad.

Query the sensor (subdevice3) for the current configured format:

• We notice the pad number, the mbus code and the frame size that is currently configured on the subdevice.</br> hand A summary of this format can be seen in media-ctl -p output

Query the dw_csi (subdevice2) for the current configured format:

• We notice that the frame size is not set correctly. This has to be set for this subdevice such that _streaming_ works.

Configuring subdevice format

• Once we know how to query the subdevices for different settings, we need to know also how to change these according to what we would like to do.

To configure a subdevice's pad format, we need to use media-ctl :

The above sequence of commands will configure the 4 given entities with the format specified.</br> hand The format string and specification is identical with the one printed by media-ctl -p

We can observe the difference in media-ctl output:

hand All links now have frame size and mbus code configured.

• We can configure the pipeline with a different frame size and the same mbus code. Here is an example:

• And the corresponding media-ctl -p output:

Configuring top video driver

• In v4l2 , the top video driver is the driver that registers the /dev/video node. * This driver also registers the /dev/media node, which is the main device of the associated media controller * Video device and media device cannot be intertwined, they perform different tasks. * You could consider the media device and video device like two devices that complete each other for full functionality access and tweaking for the video capture pipeline * Video node is used to configure the video device. This is usually the _top driver_ that performs the video capture and includes the way to access the video frame. * Media node is the node that allows to access all entities, pads and links, and informs the userspace about the topology, status, and allows topology configuration and reconfiguration</br> hand Device tree is still needed to describe the hardware on the SoC / board. Device tree is a description of the hardware. The drivers themselves may or may not register media entities for their hardware devices, or, could register entities that do not have any specific hardware device associated (pure software entities), or a piece of hardware could translate into multiple entities. It's the drivers' choice what to _expose_ or _not expose_ to userspace. Thus, the media topology is a view from userspace of the available video pipeline.

• The top video driver, /dev/video can be queried for information and configured using the v4l2-ctl tool.

Querying the top video driver

• We can query the top video driver using v4l2-ctl
  • Let's see what formats the top video driver can output to user space:

• We can query the video node also for possible resolutions:

• We see that the video device can output any resolution from 16x16 up to 3264x2464 . * This means that any frame size that _fits_, it's fine from the video node perspective. * We will see below how to get the current frame size.</br> hand The actual frame size is usually given by the sensor. The top video device cannot create frames, frames are created by sensors. However, entities in the pipeline can scale, adjust, crop and compose the frame into a new frame at a different frame size.

• How to see the complete format that is configured now ?

• This is the format that the video node will output to the userspace.
  • We notice the pixel format (FOURCC codification), resolution, bytes for each line and the total image size.

Configuring the top video driver

• How to change the video format ?

• Format was changed as requested</br> hand A list of possible pixelformat FOURCC values can be obtained from the output of the command v4l2-ctl --list-formats

Capture a frame

• Once the pipeline is configured correctly, we can use different tools to capture a frame from the video device.
  • If one of the elements in the pipeline is not configured correctly, we will have errors.
  • The top video driver will assert the situation at the start streaming step.
    • If the subdevice is not configured to a compatible format, capture will fail.

Using v4l2-ctl to capture a frame

• The raw frame has been captured with v4l2-ctl .

Converting raw frame to png

• v4l2-ctl only captures raw frames
  • We can use another tool like ffmpeg to convert the raw frame into a usual format photo.
• For example, from BAYER 8 bit RGGB raw format :

• Another example, from YUYV packed 16 bits format:

• RGB565 :

hand To see ffmpg supported pixel formats, try this:

warning resolution must match the one configured at the top video driver. ffmpeg cannot guess the resolution from a raw file.

Using fswebcam to capture a frame

• fswebcam can still be used to take a photo, once the pipeline is correctly configured

hand Even if we request 3280x2464 and the maximum resolution is 3264x2464, the media controller will resize the output frame and we will still obtain 3264x2464. It is automatically adjusted to best found resolution

Using GStreamer to capture a frame

• GStreamer can still be used to capture a frame
  • GStreamer _classic_ v4l2src plugin can be used for GStreamer 1.0 .

hand Pipeline must be configured before using GStreamer .

libcamera

• libcamera is a userspace library that configures a media pipeline and allows higher level application to stop hassling with the media controller configuration
  • libcamera is actually a library, so other applications have to link with libcamera in order to use it
  • basically libcamera is a layer between /dev/video and /dev/media and the user
• libcamera also provides few test apps that are linked with libcamera, and can be used to validate the functionality

libcamera cam app

• The cam application is a basic libcamera app that can capture a frame.

Identifying the pipeline using cam

• Using cam application to test if the camera is available:

• cam app has identified the camera and displays it to the user with the name of the video sensor (imx219 in our case)
  • camera number 1 is _imx219_

Using cam to capture a frame

• Example:

• We can then use ffmpeg to convert this raw file to a png file if we wish to.

Video-capture-at91

video-capture-at91_summary

• Video-capture-at91 is a collection of scripts present on Github that is part of the Linux4SAM distribution, and offers examples and scripts that automatically configure the media controller pipeline.

Using video-capture-at91 scripts with sama7g5

• The scripts are available in your home directory, and are organized in directories corresponding to each sensor supported.
  • There is a directory for _imx219_ and another directory for _imx274_

• There is a script that prepares the media controller pipeline for capturing for each supported resolution:

• There are also demo scripts available for each resolution, on how to use fswebcam to capture video in different formats, and how to configure the video format accordingly.

• Example of setting 3264x2464 resolution with imx219

Using video-capture-at91 scripts with sama5d2

• The scripts are available in your home directory, and are organized in directories corresponding to each sensor supported.
  • There is a directory for each of the supported sensors:

• There is a script that prepares the media controller pipeline for capturing for each supported resolution:

• There are also demo scripts available for each resolution, on how to use fswebcam to capture video in different formats, and how to configure the video format accordingly.

• Example on how to use scripts for sama5d2 + mt9v022 sensor:

• Example on how to use scripts for sama5d2 + ov7740 sensor in raw bayer mode:

video-capture-at91_sama5d2

Capturing video

media-controller_ffmpeg

Using ffmpeg

• ffmpeg can be used to capture video from the v4l2 device.

• To list possible pixel formats use :

warning Media controller pipeline must be properly configured before using ffmpeg

Example at 640x480 , RGB565 format

Example at 640x480, YUYV format

media-controller_ffmpeg

Related Topics

Boards

Sam9x75Curiosity
Sama7g5-ek
Sama5d27WLSom1EK
Sama5d27Som1EK
Sama5d2Xplained

Components

linux 5.15
linux 6.1

Summary

What is Media controller and how to use it with SAM products.