{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Basics of Image Formation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Digital images"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A raster image consists of a grid of pixels as shown in {numref}`Pixel-example`. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} images/Pixel-example.png\n",
"---\n",
"name: Pixel-example\n",
"align: center\n",
"---\n",
"An image is made from pixels. Image source [wikipedia](https://commons.wikimedia.org/w/index.php?curid=807503)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To represent an image, we use a three dimensional array with shape (H,W,3). We say that the array has H rows, W columns and 3 color channels (red, green, and blue). Let's load an image with python and have a look!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from pathlib import Path\n",
"import cv2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"from IPython import display\n",
"display.set_matplotlib_formats('svg')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img_fn = str(Path(\"images/carla_scene.png\"))\n",
"img = cv2.imread(img_fn)\n",
"# opencv (cv2) stores colors in the order blue, green, red, but we want red, green, blue\n",
"img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
"\n",
"plt.imshow(img)\n",
"plt.xlabel(\"$u$\") # horizontal pixel coordinate\n",
"plt.ylabel(\"$v$\") # vertical pixel coordinate\n",
"\n",
"print(\"(H,W,3)=\",img.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's inspect the pixel in the 100th row and the 750th column:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"u,v = 750, 100\n",
"img[v,u]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This means that the pixel at `u,v = 750, 100` has a red-intensity 28, green-intensity 59, and blue-intensity 28. Hence, it is greenish. If we have a look at the image, this makes sense, since there is a tree at `u,v = 750, 100`.\n",
"Additionally, the above output \"`dtype=uint8`\" tells us that the red, green, and blue intensities are stored as 8 bit unsigned integers, i.e., `uint8`. Hence, they are integers between 0 and 255."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following sketch summarizes what we have learned about storing digital raster images so far:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} tikz/uv/uv_grid.svg\n",
"---\n",
"name: uv_grid\n",
"align: center\n",
"width: 67%\n",
"---\n",
"Image as an array of pixels. \n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If your digital image was taken by a camera, then there is a direct correspondence between the pixels in the digital image and \"sensor pixels\" in the image sensor of the camera. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} images/photo_sensor.jpeg\n",
"---\n",
"name: photo_sensor\n",
"align: center\n",
"---\n",
"Corner of the photosensor array of a webcam digital camera. Image source: [wikipedia](https://commons.wikimedia.org/w/index.php?curid=24805309)\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An image sensor consists of a two dimensional array of photosensors. Each photosensor converts incoming light to electricity via the [photoelectric effect](https://en.wikipedia.org/wiki/Photoelectric_effect), which is converted into a digital signal by means of an analog-to-digital converter. To obtain color information, one \"sensor pixel\" is divided into a 2 by 2 grid of photosensors, and different color filters are placed in front of these 4 photosensors. One photosensor only receives light through a blue filter, one only through a red filter, and two receive light through a green filter. Combining these 4 measurements gives one color triple: `(red_intensity, green_intensity, blue_intensity)`. This is known as the [Bayer Filter](https://en.wikipedia.org/wiki/Bayer_filter)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} images/Bayer_pattern_on_sensor.svg\n",
"---\n",
"name: BayerPattern\n",
"align: center\n",
"width: 50%\n",
"---\n",
"The Bayer arrangement of color filters on the pixel array of an image sensor. Image source: [wikipedia](https://commons.wikimedia.org/w/index.php?curid=1496858)\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{tip}\n",
"For more details about image sensors, I recommend this [youtube video](https://youtu.be/MytCfECfqWc).\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pinhole Camera Model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Imagine putting an image sensor in front of an object. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} tikz/no_pinhole/no_pinhole.svg\n",
"---\n",
"name: no_pinhole\n",
"align: center\n",
"width: 50%\n",
"---\n",
"Bad idea: Image sensor placed in front of a tree. Sketched in side view. \n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You would not be able to capture a sharp image like this, since a point on the image sensor will get hit by light rays from the whole environment. Now imagine putting the image sensor inside a box with a very tiny pinhole (also known as aperture)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} tikz/pinhole_box/pinhole_box.svg\n",
"---\n",
"name: pinhole_box\n",
"align: center\n",
"width: 50%\n",
"---\n",
"Pinhole camera.\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Most of the rays are blocked now and we get a proper image on the image sensor. The image is flipped upside down, but that should not bother us. In case of an *ideal pinhole*, each point on the image sensor is only hit by one ray of light from the outside. \n",
"```{admonition} Ideal pinhole: A good approximation\n",
"In the real world the size of the hole cannot be too small because not enough light would enter the box. Additionally we would suffer from [diffraction](https://en.wikipedia.org/wiki/Diffraction). The hole can't be too large either, since light rays from different angles will hit the same spot on the image sensor and the image will get blurry. To act against blur, a lens is installed in real cameras. The pinhole camera model we will discuss in this section does not include the effects of a lens. However, it turns out that it is a very good approximation to cameras with lenses, and **the** de facto model for cameras. Hence, in the following we can continue to think of our pinhole as an *ideal pinhole*.\n",
"```\n",
"The following sketch introduces the *image plane*:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} tikz/virtual_pinhole_box/virtual_pinhole_box.svg\n",
"---\n",
"name: virtual_pinhole_box\n",
"align: center\n",
"width: 67%\n",
"---\n",
"Image formation.\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"While the image sensor is a distance $f$ behind the pinhole, the so called *image plane*, which is an imaginary construction, is a distance $f$ in front of the pinhole. We can relate the size $h$ of an object to its size on the image sensor $h'$ via similar triangles\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ \n",
" \\frac{h'}{h} = \\frac{f}{d}\n",
"$$ (eq-hprime-over-h)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This means that the object size $h'$ in the image gets smaller as the distance to the camera $d$ increases: Far away objects appear small in an image. We can generalize Eq. {eq}`eq-hprime-over-h`, if we define coordinates $x,y$ in the image plane as shown in the sketch below."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} tikz/camera_projection/CameraProjection.svg\n",
"---\n",
"name: camera_projection\n",
"width: 67%\n",
"align: center\n",
"---\n",
"Camera projection. Sketch adapted from [stackexchange](https://tex.stackexchange.com/a/323778/56455).\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The origin of the camera coordinate system $(X_c,Y_c,Z_c)$ is at the location of the pinhole. The gray shaded region is the part of the image plane that gets captured on the image sensor. The coordinates $(u,v)$ are the pixel coordinates that were already introduced in {numref}`uv_grid`.\n",
"The sketch allows us to determine the mapping of a three-dimensional point $P=(X_c, Y_c, Z_c)$ in the camera reference frame to a two-dimensional point $p=(x,y)$ in the image plane. We have a look at the above figure in the $Z_c$-$X_c$ plane:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} tikz/camera_projection_side_view/camera_projection_side_view.svg\n",
"---\n",
"name: camera_projection_side_view\n",
"width: 67%\n",
"align: center\n",
"---\n",
"Mapping of the three-dimensional point $P$ to the image point $p$.\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similar triangles in the above figure reveal"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\\frac{x}{f} = \\frac{X_c}{Z_c} ~ \\Rightarrow ~ x = f \\frac{X_c}{Z_c} $$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And similarly we find"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ y = f \\frac{Y_c}{Z_c} $$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we want to establish what pixel coordinates $(u,v)$ correspond to these values of $(x,y)$ (cf. {numref}`camera_projection`). First, we note that the so called *principal point* $(x=0,y=0)$ has the pixel coordinates $(u_0, v_0)=(W/2,H/2)$. Since $x$ and $y$ are measured in meters, we need to know the width and height of one \"sensor pixel\" in the image plane measured in meters. If the pixel width in meters is $k_u$ and the pixel height is $k_v$, then"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
" \\begin{gather} u &= u_0 + \\frac{1}{k_u}x &=u_0 + \\frac{f}{k_u}\\frac{X_c}{Z_c} \\\\ v &= v_0 + \\frac{1}{k_v}y &=v_0 + \\frac{f}{k_v}\\frac{Y_c}{Z_c}\\end{gather}\n",
"$$ (eq-uv-from-XYZ-no-matrix)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we define $\\alpha_u = f/k_u$ and $\\alpha_v = f/k_v$ we can formulate the above equations in matrix form"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ \\begin{pmatrix} u \\\\ v \\\\ 1 \\end{pmatrix} = \\frac{1}{Z_c} \\begin{pmatrix} \\alpha_u & 0 & u_0 \\\\ 0 & \\alpha_v & v_0 \\\\ 0 & 0 & 1\\end{pmatrix} \\begin{pmatrix} X_c \\\\ Y_c \\\\ Z_c \\end{pmatrix} $$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Typically $\\alpha_u = \\alpha_v = \\alpha$.\n",
"To continue our discussion, we need to introduce the concept of homogeneous coordinates\n",
"```{admonition} Homogeneous Coordinates\n",
"We give a short informal introduction to homogeneous coordinates here. For a more detailed discussion see the book by Hartley and Zisserman ({cite}`hartley2003multiple`), or the [short](https://youtu.be/PvEl63t-opM) and [long](https://youtu.be/MQdm0Z_gNcw) videos by Stachniss. We can convert a **Euclidean vector** $(u,v)^T$ into a **homogeneous vector** by appending a $1$ to obtain $(u,v,1)^T$. If we multiply a homogeneous vector by a nonzero real number $\\lambda$, it is considered to still represent the same mathematical object. In other words $(u,v,1)^T$ and $\\lambda (u,v,1)^T$ are the \"same\" homogeneous vector, they are *equivalent*. Similarly $(2,3,7)^T$ and $(4,6,14)$ are equivalent. To express that two homogeneous vectors $\\mathbf{v}$ and $\\mathbf{w}$ are equivalent we write $\\lambda \\mathbf{v} = \\mathbf{w}$, often without exactly specifying what value $\\lambda$ has. Typically, we want to represent a homogeneous vector in its canonical form, for which the last entry is $1$. The canonical form of $(2,3,7)^T$ is $(2/7, 3/7, 1)^T$. To go from a homogeneous vector back to a Euclidean vector, we first bringt it to canonical form, and then take all but the last vector components. In our example this would be $(2/7,3/7)^T$. Note that we can do the same procedure to construct a homogeneous vector with four components by appending a $1$ to a three dimensional Euclidean vector. You might think that this procedure is quite strange, but soon you will see the usefulness of homogeneous coordinates.\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So if we interpret the above equation as an equation for a homogeneous vector, we can absorb the scalar factor $1/Z_c$ and rewrite the equation as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ \n",
" \\lambda \\begin{pmatrix} u \\\\ v \\\\ 1 \\end{pmatrix} = \\begin{pmatrix} \\alpha_u & 0 & u_0 \\\\ 0 & \\alpha_v & v_0 \\\\ 0 & 0 & 1\\end{pmatrix} \\begin{pmatrix} X_c \\\\ Y_c \\\\ Z_c \\end{pmatrix} \n",
"$$ (eq-intrinsic-matrix-multiplication)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we multiply this equation by a scalar factor $s \\neq 0$, we can absorb it into $\\lambda$ on the left hand side and multiply it into the vector $(X_c,Y_c,Z_c)^T$ on the right hand side. What we find is that any point with coordinates $s (X_c,Y_c,Z_c)^T$ will be mapped to the same $(u,v)$ coordinates. Note that all these points define a line $\\mathbf{l}(s) = s (X_c,Y_c,Z_c)^T$ which goes through the pinhole at position $(0,0,0)$. You can also verify this fact with the previous form of our equations: Eq. {eq}`eq-uv-from-XYZ-no-matrix`. But even more importantly, this fact is intuitive if we look at {numref}`camera_projection`: Each point on the red ray from the camera origin to the point $P$ will be mapped into the same image coordinate $(u,v)$."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reference frames"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In our applications, we will deal with with different coordinate systems, so we will not always have our point $P$ given in coordinates of the camera reference frame $(X_c,Y_c,Z_c)^T$. For our lane-detection system the following reference frames are relevant\n",
"* World frame $(X_w, Y_w, Z_w)^T$\n",
"* Camera frame $(X_c, Y_c, Z_c)^T$,\n",
"* Default camera frame $(X_d, Y_d, Z_d)^T$\n",
"* Road frame $(X_r, Y_r, Z_r)^T$\n",
"* Road frame according to [ISO8855 norm](https://www.sis.se/api/document/preview/914200/) $(X_i, Y_i, Z_i)^T$\n",
"\n",
"These frames are defined in the following figure"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} tikz/coordinate_systems/coordinate_systems.svg\n",
"---\n",
"name: coordinate_systems\n",
"width: 67%\n",
"align: center\n",
"---\n",
"Different coordinate systems. Due to the side view, only two of the three coordinate axes are drawn for most of the coordinate systems. The direction of the third coordinate axis can be inferred using the fact that all these coordinate systems are right-handed.\n",
"Note that in general the $X_w$-$Y_w$ plane is not parallel to the road. However, in the Carla simulator this is often the case.\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The axes of the default camera frame correspond to the \"right\", \"down\", and \"forwards\" directions of the vehicle. Even if we wanted to mount our camera in a way that the *camera frame* equals the *default camera frame*, we might make some small errors. In the exercises the *camera frame* is slightly rotated with respect to the *default camera frame*, but only along the $X_d$-axis: The camera has a [pitch angle](https://en.wikipedia.org/wiki/Aircraft_principal_axes) of 15°."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the following we will describe how to transform between world and camera coordinates, but the same mathematics hold for a transformation between any two of the the above reference frames.\n",
"A point $(X_w, Y_w, Z_w)^T$ in the world coordinate system is mapped to a point in the camera coordinate system via a rotation matrix $\\bf{R} \\in \\mathbb{R}^{3 \\times 3}$ (to be more precise [$\\mathbf{R} \\in SO(3)$](https://en.wikipedia.org/wiki/3D_rotation_group)) and a translation vector $\\bf{t}$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ \\begin{pmatrix} X_c \\\\ Y_c \\\\ Z_c \\end{pmatrix} = \\bf{R} \\begin{pmatrix} X_w \\\\ Y_w \\\\ Z_w \\end{pmatrix} + \\bf{t} $$\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can write this transformation law just using matrix multiplication"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ \n",
" \\begin{pmatrix} X_c \\\\ Y_c \\\\ Z_c \\\\ 1 \\end{pmatrix} =\\begin{pmatrix} R_{xx} & R_{xy} & R_{xz} & t_x \\\\ R_{yx} & R_{yy} & R_{yz} & t_y \\\\ R_{zx} & R_{zy} & R_{zz} & t_z \\\\ 0 & 0 & 0 & 1 \\end{pmatrix}= \\mathbf{T}_{cw} \\begin{pmatrix} X_w \\\\ Y_w \\\\ Z_w \\\\ 1\\end{pmatrix} \n",
"$$ (eq-change-coordinate-systems)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where we defined the *transformation matrix* $\\bf{T}_{cw}$ in the last equality. It is great that we can relate the coordinates of different reference frames just by using a matrix. This makes it easy to compose several transformations, for example to go from world to camera coordinates and then from camera to road coordinates. It also enables us to get the inverse transformation using the matrix inverse: The transformation from camera to world coordinates is given by the matrix $\\bf{T}_{wc} = \\bf{T}_{cw}^{-1}$."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To conclude this section, we combine Eq. {eq}`eq-intrinsic-matrix-multiplication` and {eq}`eq-change-coordinate-systems` into one single equation:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ \n",
" \\lambda \\begin{pmatrix} u \\\\ v \\\\ 1 \\end{pmatrix} = \\begin{pmatrix} \\alpha_u & 0 & u_0 \\\\ 0 & \\alpha_v & v_0 \\\\ 0 & 0 & 1\\end{pmatrix} \\begin{pmatrix} R_{xx} & R_{xy} & R_{xz} & t_x \\\\ R_{yx} & R_{yy} & R_{yz} & t_y \\\\ R_{zx} & R_{zy} & R_{zz} & t_z \\end{pmatrix} \\begin{pmatrix} X_w \\\\ Y_w \\\\ Z_w \\\\ 1\\end{pmatrix} \n",
"$$ (eq-intrinsic-and-extrinsic-matrix-multiplication)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or by defining the **intrinsic camera matrix** $\\mathbf{K}$ and the **extrinsic camera matrix** $\\begin{pmatrix} \\mathbf{R} | \\mathbf{t} \\end{pmatrix}$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
" \\lambda \\begin{pmatrix} u \\\\ v \\\\ 1 \\end{pmatrix} = \\mathbf{K} \\begin{pmatrix} \\mathbf{R} | \\mathbf{t} \\end{pmatrix} \\begin{pmatrix} X_w \\\\ Y_w \\\\ Z_w \\\\ 1\\end{pmatrix}\n",
"$$ (eq-world-coords-to-uv)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise: Projecting lane boundaries into an image"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now you should start to work on your first exercise. For this exercise, I have prepared some data for you. I captured an image in the Carla simulator using a camera sensor attached to a vehicle:\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} images/carla_lane_image.svg\n",
"---\n",
"name: carla_lane_image\n",
"width: 67%\n",
"align: center\n",
"---\n",
"Image captured by a camera sensor in a Carla simulation.\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Additionally I created txt files containing\n",
"* world coordinates of the lane boundaries $(X_w,Y_w,Z_w)^T$\n",
"* a transformation matrix $T_{cw}$ that maps world coordinates to coordinates in the camera reference frame\n",
"\n",
"Your job is to write code based on equation {eq}`eq-world-coords-to-uv` that creates a label image like this:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{figure} images/carla_lane_label.svg\n",
"---\n",
"name: carla_lane_label\n",
"width: 67%\n",
"align: center\n",
"---\n",
"Label image.\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First you will need to set up your python environment, install the necessary libraries, and download the code for the exercises. Please visit [the appendix](../Appendix/ExerciseSetup.md) to learn how."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To start working on the exercises, open `code/tests/lane_detection/lane_boundary_projection.ipynb` and follow the instructions in that notebook."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## References\n",
"```{bibliography} \n",
":filter: docname in docnames\n",
"```"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 4
}