The experiments take place parallel to the implementation in order to let those benefit from the results.
This chapter deals with the evaluation phase of the trained networks.

The networks get tested with the test dataset of Cityscapes.
Later on, test data from the DriveU set will be added.

In addition to the hard metric \ac{mAP}, the tested and annotated images and videos are manually examined in detail.
A well\textendash trained network characterizes that for every traffic light, which can be seen immediately by humans, an exact box is found around its corpus.
Traffic lights that are not directly relevant to the driver, such as the backside or traffic lights for other traffic participants, remain undetected.
In case of videos, there is also no jumping of the box between several frames.
Also no dropouts should be present in a video.

\section{YOLOv3 Against YOLOv3\textendash Tiny}\label{sec:yolov3-against-yolov3-tiny}

Training on the training dataset leads to a minimum validation loss of 16.3835  for YOLOv3.
The loss indicates the absolute error of the weights.
On its own, this value is insignificant and is only used for relative comparison of the results of individual experiments within the same architecture.

\begin{table}[]
 \centering
 \begin{tabular}{@{}lrr@{}}
  \toprule
  & \multicolumn{1}{l}{\textbf{YOLOv3}} & \multicolumn{1}{l}{\textbf{YOLOv3\textendash Tiny}} \\ \midrule
  mAP with IoU of 0.25   & 0.88                       & 0.83                            \\
  mAP with IoU of 0.50   & 0.43                       & 0.26                            \\
  mAP with IoU of 0.75   & 0.04                       & 0.03                            \\
  time (min)             & 89                         & 22                              \\
  time (25\% percentile) & 96                         & 24                              \\
  time (50\% percentile) & 98                         & 25                              \\
  time (75\% percentile) & 101                        & 25                              \\
  time (max)             & 135                        & 44                              \\
  avg. \acs{FPS}              & 17                         & 30                              \\
  \# of epochs           & 14                         & 9                               \\
  min. val loss         & 16.3835                    & 12.4516                         \\ \bottomrule
 \end{tabular}
 \caption{Comparison: YOLOv3 Against YOLOv3\textendash Tiny}
 \label{tab:compare_yolov3-against-yolov3-tiny}
\end{table}

When testing the images, YOLOv3\textendash Tiny is about four times faster.
When measuring the time for images, only the time it takes to perform the object recognition is taken into account.
The metric FPS specifies how many frames per second are displayed.
The loading of new images is, therefore, included.
Using this metric, the time advantage of the small version shrinks to half.
The limiting factor is the \ac{HDD} used to store the datasets, which reaches its maximum reading and writing speed in many tests and training sessions.
The 200 \ac{FPS} on a weaker graphics card, as stated on the official YOLO website, supports this argument.\cite{yoloWebsite}
The speed of 30 \ac{FPS} or 17 \ac{FPS} with YOLO achieved under these conditions is still sufficient for traffic light detection.

The mAP of YOLOv3\textendash Tiny is about one third worse than that of the large network.
The low precision is also evident when examining images and videos.
The tiny YOLO variant does not recognize many traffic lights in the pictures.
Especially small traffic lights are not recognized.
When driving, this leads to a reduction in visibility.
Traffic lights are only recognized later.

%TODO IMAGE späte Erkennung

In videos, the inaccuracy of YOLOv3\textendash Tiny becomes noticeable by high instability.
That is also observable in YOLOv3 but not as prominent.

%TODO IMAGE video Instabilität

A fundamental difference between the two versions is the number of levels of the feature pyramid.
YOLOv3 has one additional step with one\textendash eighth of the image width and height for the detection of small objects.
That is entirely missing in YOLOv3\textendash Tiny.
Besides, YOLOv3\textendash Tiny has no Darknet backbone and a lower depth.
The context\textendash sensitivity with which the tiny variant works cannot be as complex as YOLOv3.

\section{Varying the Image Input Size}\label{sec:varying-the-image-input-size2}

The larger the input image during training, the larger the validation loss.
Shifting in a larger input image by a given size in pixels has more relative effect than on a small image.
In contrast more can be captured in more detail on a larger image.

\begin{table}[]
 \centering
 \begin{tabular}{@{}lrrrr@{}}
  \toprule
  & \multicolumn{4}{l}{\textbf{YOLOv3}}    \\ \midrule
  Size                            & 416x416 & 832x832 & 512x1024 & 416x832 \\
  Pixels in $10^5$                & 1.73    & 6.92    & 5.24     & 3.46    \\
  Batch Size                      & 8       & 4       & 4        & 8       \\ \midrule
  mAP with IoU of 0.25            & 0.88    & 0.84    & 0.85     & 0.86    \\
  mAP with IoU of 0.50            & 0.43    & 0.44    & 0.44     & 0.43    \\
  mAP with IoU of 0.75            & 0.04    & 0.07    & 0.07     & 0.06    \\
  time (min)                      & 89      & 99      & 95       & 92      \\
  time (25\% percentile)          & 96      & 112     & 109      & 101     \\
  time (50\% percentile)          & 98      & 125     & 121      & 110     \\
  time (75\% percentile)          & 101     & 137     & 132      & 120     \\
  time (max)                      & 135     & 150     & 143      & 140     \\
  avg. \acs{FPS}                       & 17      & 13      & 14       & 17      \\
  \# of epochs                    & 14      & 6       & 7        & 10      \\
  min. val loss                  & 16.3835 & 21.8095 & 19.4198  & 17.5367 \\ \bottomrule
 \end{tabular}
 \caption{Varying the Image Input Size}
 \label{tab:varying-the-image-input-size}
\end{table}

The time it takes the network to process an image increases linearly with the number of pixels.
As the image size increases, the mAP also increases, but the slope of the mAP is monotonically decreasing.
Since the number of pixels is proportional to the required time, the required computing time increases overproportionally to mAP.

Even if the amount of data to be processed is more massive with a large image as input, the required training time reduces.
The validation loss decreases faster and to a lower value.
The target object is easier to identify with more details.

If the images are too small, the traffic lights are not identifiable, and the mAP is disproportionately bad.
With too large images, the additional gain in knowledge decreases because all relevant details are already present in the image.
Training on large images is also tricky because the batch size has to be reduced to ensure that the graphics card's memory is sufficient.
A smaller batch size makes the network more vulnerable to outliers and overfitting.
The optimal size for our problem is 416x832.
This size has roughly as much pixels as a 588x588 image and is therefore comparatively small.
Nevertheless, the width is doubled and the traffic lights are more clearly visible in the image.

\section{More Data}\label{sec:more-data}

The introduction of the DriveU dataset leads to a reduction of the validation loss.

\begin{table}
 \footnotesize
 \centering
 \begin{tabularx}{\textwidth}{@{}lrrrrrr@{}}
  \toprule
  & \multicolumn{3}{l}{\textbf{YOLOv3}}                                                                                                                                  & \multicolumn{3}{l}{\textbf{YOLOv3-Tiny}}                                                  \\ \toprule
  & \multicolumn{1}{l}{\textbf{Cityscapes}} & \multicolumn{1}{l}{\textbf{CS + DU}}           & \multicolumn{1}{l}{\textbf{DriveU}}      & \multicolumn{1}{l}{\textbf{Cityscapes}} & \multicolumn{1}{l}{\textbf{CS + DU}} & \multicolumn{1}{l}{\textbf{DriveU}} \\ \midrule
  \textbf{mAP with IoU of 0.25}   & 0.88                                    & 0.87                                           & 0.88                                     & 0.83                                   & 0.78                                & 0.81                        \\
  \textbf{mAP with IoU of 0.50}   & 0.43                                    & 0.44                                           & 0.42                                     & 0.26                                   & 0.36                                & 0.30                        \\
  \textbf{mAP with IoU of 0.75}   & 0.04                                    & 0.07                                           & 0.05                                     & 0.03                                   & 0.04                                & 0.03                        \\
  \textbf{time (min)}             & 89 ms                                   & 88 ms                                          & 90 ms                                    & 22 ms                                  & 26 ms                               & 24 ms                       \\
  \textbf{time (25\% percentile)} & 96 ms                                   & 97 ms                                          & 95 ms                                    & 24 ms                                  & 26 ms                               & 26 ms                       \\
  \textbf{time (50\% percentile)} & 98 ms                                   & 101 ms                                         & 98 ms                                    & 25 ms                                  & 27 ms                               & 27 ms                       \\
  \textbf{time (75\% percentile)} & 101 ms                                  & 104 ms                                         &  99 ms                                   & 25 ms                                  & 30 ms                               & 32 ms                       \\
  \textbf{time (max)}             & 135 ms                                  & 132 ms                                         & 133 ms                                   & 43 ms                                  & 42 ms                               & 49 ms                       \\ \bottomrule
 \end{tabularx}
 \caption{Comparison: More Data}
 \label{tab:comparison:_more-data}
\end{table}

This experiment shows that the combination of both datasets leads to a higher precision than when used alone.
Even though Cityscapes contains some irrelevant traffic lights, the precision increases, and the videos become visibly more interference-free.
Both datasets have different white balance, color matching, and distribution of traffic light positions.
It seems likely that this will make the network more robust and give it a better understanding of the concept of traffic lights.

YOLOv3 benefits from additional data more than YOLOv3-Tiny.
The small YOLO variant does not have sufficient capacity to hold all the information from the dataset.

\section{Pre-Trained Weights}\label{sec:pre-trained-weights}

Using the pre-trained weightings the number of epochs required is further reduced.
Now that Darknet, with 18 billion weights, is frozen, only about 71% of the network is trained.

\begin{table}[]
 \centering
 \begin{tabular}{@{}lrr@{}}
  \toprule
  & \multicolumn{1}{l}{\textbf{YOLOv3 (CS+DU)}} & \multicolumn{1}{l}{\textbf{YOLOv3 (CD+DU; Darknet freezed)}} \\ \midrule
  mAP with IoU of 0.25   & 0.87                       & 0.82                            \\
  mAP with IoU of 0.50   & 0.44                       & 0.42                            \\
  mAP with IoU of 0.75   & 0.07                       & 0.08                            \\
  time (min)             & 88 ms                      & 89 ms                           \\
  time (25\% percentile) & 97 ms                      & 98 ms                           \\
  time (50\% percentile) & 99 ms                      & 99 ms                           \\
  time (75\% percentile) & 103 ms                     & 105 ms                          \\
  time (max)             & 127 ms                     & 133 ms                          \\
  avg. \acs{FPS}              & 17                         & 17                              \\
  \# of epochs           & 8                          & 9                               \\
  min. val loss         & 12.6568                    & 12.4516                         \\ \bottomrule
 \end{tabular}
 \caption{Comparison: Pre-Trained Weights}
 \label{tab:compare_pre-trained-weights}
\end{table}

In comparison to previous experiments, the mAP has decreased slightly.
The Darknet-53 now focuses on general object recognition.
The complete information about the structure and context of traffic lights is inside the YOLO part.
The information reduction to the YOLO part makes the mAP decrease, but the task splitting with Darknet-53 makes the network more stable.

When looking at the images, the decreased mAP is not visible.
Especially the good results in the annotation of the videos add value to the action.
 The detection is more stable, and the bounding box sizes between the frames do not fluctuate as much anymore.

\section{Recurrent Layers}\label{sec:recurrent-layers}

The use of recurrent layers cannot be measured with mAP because the datasets used do not contain annotated videos.
When examining the videos manually, one can see that there are fewer, but not statistically significant fewer, micro-failures than before.
Using the last frame alone is not sufficient to compensate for interruptions over several frames.
Some fluctuations in image size have disappeared, but there are still some outliers.

Recurrent layers work in this application case, but the past frames must be more extensive and must consist of the last 16 frames, for example.
