\section{The Application of YOLOv3}\label{sec:the-application-of-yolov3}
\ac{YOLO} is a fast convolutional network to detect objects in images and classify them at the same time.
Joseph Redmon and Ali Farhadi developed the network, and their latest version, YOLOv3, was released in 2018.\cite{DBLP:journals/corr/abs-1804-02767}
In spring 2020, Alexey Bochkovskiy released the fourth version, YOLOv4, with further improvements.\cite{DBLP:journals/corr/abs-2004-10934}
This iteration increases the \ac{mAP} while maintaining the speed of the mesh.
Since YOLOv4 does not bring any additional notable enhancements for detecting small objects, YOLOv3 takes application in this work instead.
Further optimizations will be made after a more detailed analysis to specialize YOLOv3 for small items.

To better understand the architecture of YOLOv3, the structure of the previous versions will be explained first, whereby the features that are still used in YOLOv3 will be discussed in detail.

In the following, constants are defined as unchangeable values.
Parameters are freely selectable, but afterward, they cannot be changed.
In contrast, variables can be changed at any time, even after the training of the network.

\subsection{YOLOv1}\label{subsec:yolov1}
The basic functionality of YOLO does not change in the course of the versions.
The input for the network is an image with three dimensions.
In YOLOv1, the size of the image is predetermined due to the \q{Fully Connected Layer} at the end and equals 448 x 448 x 3 pixels.
The output is also a three\textendash dimensional vector that describes the so\textendash called output boxes \textendash  meaning in which part of the image with which size a particular object is located.

\begin{figure}[!htbp]
 \vspace{0cm}
 \minipage{0.0\textwidth}
 \endminipage\hfill
 \minipage{0.85\textwidth}
 \includegraphics[width=\linewidth]{../data/yolov1.png}
 \caption{Architecture of YOLOv1 Taken from the Paper of YOLOv1\cite{redmon2016look}}    %Bildunterschrift
 \label{fig:arcchitecture_v1}       %fig:ID
 \endminipage\hfill
 \minipage{0.0\textwidth}
 \endminipage
\end{figure}

With YOLOv1, the width and height of the image are constants with 448 x 448 pixels.
That is followed by several convolutional and maximum pooling layers, which double the depth of the image while halving the width and height.

In a convolutional neuronal network (CNN), each layer increases the understanding complexity of the image.
The first layer detects edges and surfaces, the second one already abstract shapes and the following parts of objects and objects itself.
Several layers are needed to detect the whole complexity of an object.\cite{DBLP:phd/hal/Cherti18}
In YOLO, the width and height are halved at each step to detect larger, connected objects.
The image depth doubles to detect multiple objects.
Alternating between convolutional filters with a filter size of 1x1 and 3x3 with double depth reduces the \q{feature space} of the previous layers and thus capture their full complexity.
YOLOv3 still uses this technique.

In YOLOv1, the 7x7x1024 tensor resulting from the reduction is converted into a tensor with the dimensions 7x7x30 via two fully connected layers.
These tensors are interpreted as output boxes and are used to locate and classify the bounding boxes of objects in the image.

\subsubsection{Output Boxes in YOLOv1}
YOLOv1 already implements the output as bounding boxes in a tensor.
The concept is modified just a bit in the following versions of YOLO.

\begin{figure}[!htbp]
 \vspace{0cm}
 \minipage{0.0\textwidth}
 \endminipage\hfill
 \minipage{0.85\textwidth}
 \includegraphics[width=\linewidth]{../data/yolov1-bounding.png}
 \caption{Separating the Image into Multiple Boxes Taken from the Paper of YOLOv1\cite{redmon2016look}}    %Bildunterschrift
 \label{fig:boxes_v1}       %fig:ID
 \endminipage\hfill
 \minipage{0.0\textwidth}
 \endminipage
\end{figure}

YOLOv1 has as output a tensor of the size 7x7x30.
The image is divided into 49 equally distributed cells.
Each cell has 30 attributes in the reference implementation.
The attributes in the cell indicate whether an object was found in this cell.
A cell is responsible for an object if the center of the object is located in the respective cell.
Thus in the illustration, cell (1, 4) is responsible for recognizing the dog, and cell (2, 3) to recognize the bicycle, whereas one is the first column from the left and four the fourth row from the top.

In the 30 attributes of a cell, YOLOv1 encodes two bounding boxes (with five attributes each) and a one\textendash hot encoding of 20 classes.
In the five attributes of the bounding box, the position of the center (x, y) is relative to the center of the respective cell, and the width and height (w, h) relative to the total size of the image.
The fifth value specifies the \q{confidence} of how secure the network is that this given Bounding Box is also an object.
The two middle graphs in the figure indicate which class has the highest activation in the respective cell (bottom) and which bounding boxes have been detected (top).
The more secure the network is for a bounding box, the thicker the surrounding frame in the figure.
It can be seen that the dog, bicycle, and car are framed more intensively, and the respective class in the cell of the centers is correctly recognized.

If the confidence for one of the two bounding boxes and a class from the One\textendash Hot\textendash Coding is greater than the threshold value 0.5, then an object with the recognized class exists in the image with the dimensions of the bounding box.

When detecting the traffic light class only, the tensor would have a size of 7x7x11 because just two bounding boxes and one class would be present.

\subsubsection{Problems of YOLOv1}
YOLOv1 has a fast runtime but rather many problems with the exact localization of objects.
Besides, the recall (percentage of actual positive results from all as correctly identified items) is low compared to other neural networks for object detection.\cite{DBLP:conf/cvpr/RedmonF17}

Especially the recall is essential for the recognition of traffic lights because otherwise, many traffic lights are not recognized.
At the expense of precision (proportion of the actual positive results from all identified objects), the recall can be increased.
However, it would lead to worse results overall because the standard threshold value of 0.5 offers the best cost\textendash benefit ratio (see figure).

The comparatively low localization accuracy results mainly from the inaccuracy of the anchor boxes at the end.
 The network predicts several anchor boxes and one class for a single cell.
 If there are centers of several objects (for instance, a human in front of a car) in a cell, the two different anchor boxes both predict both bounding boxes with high certainty, but due to the network design, only one class can be detected.

\subsection{Improvements in YOLOv2 and YOLOv3}\label{subsec:improvements-in-yolov2-and-yolov3}
The transition from first-generation YOLOv1 to YOLOv2 has resulted in significant changes.
However, the basic concept has been maintained.
YOLOv2 and YOLOv3 differ only marginally.
The following concept summarizes both iterations and describes the current and used state of YOLOv3.

\subsubsection{Batch Normalization}
Starting with YOLOv2, batch normalization is performed after each convolutional layer.
The use of Batch Normalization accelerates convergence while eliminating the need for other forms of regularization.
Thus, dropout could be avoided even during training without overfitting the network.
After adding Batch Normalization behind each Convolutional Layer in the network, followed by LeakyReLU activation, YOLO's mAP is increased by over two percent.

The network reuses the combination of the convolutional layer and the batch normalization several times.
It occurs in the following text under the name \q{DarknetConv}\myref{par:darknet-convolutional-layer}.

\subsubsection{Darknet}

\begin{figure}[!htbp]
    \vspace{0cm}
    \minipage{0.0\textwidth}
    \endminipage\hfill
    \minipage{1\textwidth}
    \includegraphics[width=\linewidth]{../data/Darknet.pdf}
    \caption{Network Architecture of Darknet-53}    %Bildunterschrift
    \label{fig:darknet}       %fig:ID
    \endminipage\hfill
    \minipage{0.0\textwidth}
    \endminipage
\end{figure}

The probably most significant change is the introduction of a network for general object recognition (backbone).
Nowadays, it is general practice in image recognition with neural networks to base on an already existing, generally pre-trained backbone.
The most common ones are VGG or ResNet, each in different variants.\cite{DBLP:conf/eccv/2018-11}]
YOLOv2 uses an own backbone called \q{Darknet}.
That is better designed to handle the full utilization of the graphics card and can therefore perform more operations per second.

While the Darknet-19 used in YOLOv2 has a lower accuracy (74.1\%), Darknet-53 from YOLOv3 has a similar accuracy (77.2\%) as ResNet-152 (77.6\%).
Note that ResNet-152 performs 157\% more FLOPs (Floating Point Operations) than Darknet-53.
Because ResNet has more and less performing layers, Darknet achieves more operations per second, and therefore more \ac{FPS}.

\begin{table}[]
    \begin{tabularx}{\textwidth}{@{}llllll@{}}
        \toprule
        \textbf{Backbone} & \textbf{Top-1 in \%} & \textbf{Top-5 in \%} & \textbf{FLOP in $10^9$} & \textbf{GFLOP\/s} & \textbf{\acs{FPS}} \\ \midrule
        Darknet-19        & 74.1                 & 91.8                 & 7.29                    & 1246              & 171          \\
        ResNet-101        & 77.6                 & 93.7                 & 19.7                    & 1039              & 53           \\
        ResNet-152        & 77.6                 & 93.8                 & 29.4                    & 1090              & 37           \\
        Darknet-53        & 77.2                 & 93.8                 & 18.7                    & 1457              & 78           \\ \bottomrule
    \end{tabularx}
    \caption{Performance Comparison of Darknet and ResNet\cite{DBLP:journals/corr/abs-1804-02767}}
    \label{tab:comparison_resnet_darknet}
\end{table}

\paragraph{Darknet Convolutional Layer}\label{par:darknet-convolutional-layer}
The base of Darknet is the already described \q{DarknetConv}, a convolutional layer with subsequent batch normalization and Leaky ReLU as the activation function.

\paragraph{Residual Block}
In general, a deep neural network can understand more complex structures and learn more information.
However, the derivation, the gradient, in the deeper layers becomes more and more marginal, and therefore the training of those deeper layers becomes more and more difficult.
The additional capacity decreases with increasing depth.
To counteract this problem, ResNet was created, which works with 152 layers and has achieved a complexity that was unmatched at the time of development.
ResNet introduces so-called \q{Residual Blocks}.
This block encloses two convolutional layers and adds the result with a shortcut from the source layer.\cite{DBLP:journals/corr/HeZRS15}

By skipping some layers during backpropagation and adjusting weights during training, the derivation does not shrink as much and allows the training of more complex structures.
Additionally, a residual block is a measure to prevent overfitting since adding the block improves and enriches the previously existing information only.

Darknet uses residual blocks in \q{DarknetResidual}, which include two \q{DarknetConv} layers.
The width, height, and depth of the initial layer remain unchanged.

\paragraph{Darknet Block}
A \q{DarknetBlock} consists of a \q{DarknetConv} layer and several \q{DarknetResidual} blocks.
The \q{DarknetConv} layer halves the width and height of the image while doubling its depth so that the capacity of the information remains intact.
A resulting pixel aggregates four pixels into one and thus encompasses the entire structure.
This step is called pooling.
In many neural networks, max pooling is used, which passes the highest activation to the next layer.
The advantage of pooling by convolution as in YOLOv3 is the increased speed.

The resulting reduced block is followed by several \q{DarknetResidual} blocks to improve the context of the new size.
The number of blocks varies, and in the process of developing YOLOv3, it got optimized.

\paragraph{Darknet-53}
Darknet-53 consists of a composition of five Darknet blocks.
Before this, the depth of the image got increased to 32 using a \q{DarknetConv}.

Each Darknet block halves the width and height and doubles the depth of the image.
That preserves the capacity of the information the layers can hold.
The number of \q{DarknetResidual} blocks used in it varies and represents a local maximum in tests by the developers of YOLOv3.

\subsubsection{YOLOv3}
The outputs of the last three \q{DarknetResidual} blocks are then passed from the object detection network Darknet-53 to the network to create the anchor boxes.
Continued use of the last three stages instead of just the last output enables the network to recognize smaller objects.

The final output of the Darknet with the dimensions \q{w/32 x h/32 x 1024}, where w and h are the input sizes, and the depth corresponds to 1024 layers, is processed by a \q{YoloConv} and then processed with a \q{YoloOutput} layer to the dimensions needed for the Anchor Boxes.

\paragraph{YoloConv}
YOLOv1 already uses a technique to reduce the \q{feature space} of the previous layers.
That means that subsequent layers forget details of the previous layer and therefore have more capacity to learn new, more abstract knowledge.
To achieve this goal, 1x1- and 3x3-Convolutional Layers are applied alternately, and the depth is varied, as shown in the diagram.

In this part of the neural network, the network learns to extract relevant objects from the general object recognition of Darknet-53 and to set the parameters for the bounding box (\q{anchor box}) in its center.
The center point is usually the location with the highest activation for the object and is therefore easy for the network to find.

Many objects are located in the center of the image.\cite{DBLP:conf/cvpr/RedmonF17}
Reducing the default resolution from 448x448 (14x14 cells) to 416x416 (13x13 cells) during the transition from YOLOv1 to YOLOv2 ensures the center point is not located on the edge of two cells.
A dedicated cell is now responsible for the image center.
However, this is not relevant for detecting traffic lights since traffic lights are usually located on the side of the road.
When extending the network to other classes, like cars in front, this can be a critical point.

\paragraph{YoloOutput}
The result of the \q{YoloConv} layer gets transformed into anchor boxes.
Only on this last layer it is important how many classes the output should be trained to.

The output tensor is four-dimensional.
Two dimensions indicate the cell;
the third dimension indicates the respective anchor box.
Since usually three anchor boxes are used per level, the length of this dimension is also three.
The fourth dimension contains the properties of an anchor box in a cell.
These consist of the anchor box' width, height, shifting of the center of the cell, and the confidence of having an object in the anchor box.
In addition to these five properties, there are the One-Hot encoded classes.
Currently, only the traffic light class gets trained, and the length of the fourth dimension is, therefore, six.

\begin{figure}[!htbp]
    \vspace{0cm}
    \minipage{0.0\textwidth}
    \endminipage\hfill
    \minipage{0.50\textwidth}
    \includegraphics[width=\linewidth]{../data/bounding-box.png}
    \caption{Properties of an anchor box in YOLOv2 (confidence and classes are omitted)\cite{DBLP:conf/cvpr/RedmonF17}}    %Bildunterschrift
    \label{fig:bounding-box}       %fig:ID
    \endminipage\hfill
    \minipage{0.0\textwidth}
    \endminipage
\end{figure}

In YOLOv2, the offset of an anchor box \q{bx} and \q{by} is relative to the upper left corner of the image.
In contrast, the anchor box' offset in YOLOv3 refers to the upper left corner of the containing cell.

Before training, the sizes of the Anchor Boxes are defined as hyperparameters.
In the reference implementation of YOLOv3 are already nine anchor boxes specified, which are used in this work.\cite{DBLP:conf/cvpr/RedmonF17}
For each of the three resolution levels of YOLOv3 detection, it is suggested to use three anchor boxes.
One anchor box is a horizontal rectangle, one is a vertical rectangle, and one box is approximately one square.
The box that most closely matches the aspect ratio of the object to be recognized is responsible for the recognition.

If an object is now located in a cell, then the \q{Confidence} of the responsible anchor box in the cell exceeds an arbitrary, but previously defined, required threshold, usually 50 percent.
The class with the highest value in the One-Hot encoding is detected.
Additionally, the remaining four properties of the responsible anchor box specify the position and size of the object.

\paragraph{YoloConvConcatenate}
The first YoloConv layer results in a layer with a 32nd of the width and height of the original image (in the following called A).
The depth is 512.
The information about approximate positions of possible objects created after this layer should be helpful for the recognition in the more detailed layer (in the following called B).
However, this layer has a 16th of the original width and height and is twice as wide and twice as high.

The output of the coarse YoloConv layer (A) has to be extrapolated first to connect the two layers.
That allows the detection of smaller objects with the support of already detected large structures.
A crowd of people can lead to a low activation of the class \q{Person}.
This information is passed to the layer with more detailed information.
Based on the already approximate positioning, it is now more accessible for the network to provide more precise positions for the class \q{Person}.
The activation on the layer with more detail is higher and overshadows the activation of the first result.

\begin{figure}[!htbp]
    \vspace{0cm}
    \minipage{0.0\textwidth}
    \endminipage\hfill
    \minipage{0.75\textwidth}
    \includegraphics[width=\linewidth]{../data/YOLOv3.pdf}
    \caption{Network Architecture of Darknet-53}    %Bildunterschrift
    \label{fig:yolov3}       %fig:ID
    \endminipage\hfill
    \minipage{0.0\textwidth}
    \endminipage
\end{figure}

\subsubsection{Flexible Image Size}\label{subsubsec:flexible-image-size}
By using only \q{Convolutional Layers} and no \q{Fully Connected Layer} starting with YOLOv2, the output size of the network is variable depending on the original image size and corresponds to a scalar before 1/32.
It makes Darknet incredibly flexible.
It can be applied to any image size, with the runtime increasing linearly with the number of pixels during training, testing, and execution.
By default, the size of YOLO is 416x416 pixels, which results in a final size of 13x13 pixels.
The size is smaller than in YOLOv1 but has an advantage in the case of an odd number of pixels.
The center of the image belongs to one output box and is not on the edge of several.
In many photographs, the subject is centered.
However, this condition is not essential for the detection of traffic lights.

Another advantage of not using a fully connected layer is that it is independent of location.
Although the network learns the context of an object, for example, a traffic light is usually mounted on a mast, it is irrelevant where the object is located in the image.
In some datasets, the distribution of the positions of the traffic lights diverges between the training and test data.
YOLO can abstract from this and focus on the shape and color of the object and its surroundings.
