文档yolo学习
Source
Edit
History

YOLOv3 model.py代码的简单分析注解

Catalogue
  1. 1. YOLOv3 model.py代码的简单分析注解
    1. 1.1. > 准备
    2. 1.2. > YOLO网络
      1. 1.2.0.1. 首先要通过特征提取网络对输入的输入图像提取特征，得到不同尺寸的特征图
    3. 1.2.1. 流程图
    4. 1.2.2. 整个yolov3网络共252层，组成如下：
  2. 1.3. > tiny_yolo_body
    1. 1.3.0.1. 轻量版的YOLO,有速度快，占内存少等的优点，视情况选择使用。
  • 1.4. > 预测框的获得
    1. 1.4.1. 求得bx,by,bw,bh的公式
    2. 1.4.2. 对yolo_correct_boxes部分代码的补充说明
  • 1.5. > 寻找最佳的anchor box
    1. 1.5.1. preprocess_true_boxes中涉及到的anchor box和ground truth经过/2和取反等处理后，其相对关系应大概如图所示：
  • 1.6. > LOSS值的计算
    1. 1.6.0.1. （1）计算xy（物体中心坐标）的损失：xyloss=bool*(2-areaPred)*bce
    2. 1.6.0.2. （2）计算wh（anchor长宽回归值）的损失：whloss=bool*(2-areaPred)*bce
    3. 1.6.0.3. （3）计算置信度的损失：whloss=bool*(2-areaPred)*bce
    4. 1.6.0.4. （4）计算类别损失：置信度乘上多分类的交叉熵
  • 1.6.1. 在loss值的计算过程中预测框被分为三类：
    1. 1.6.1.1. 正例：任取一个ground truth，与4032个框全部计算IOU，IOU最大的预测框，即为正例。并且一个预测框，只能分配给一个ground truth。例如第一个ground truth已经匹配了一个正例检测框，那么下一个ground truth，就在余下的4031个检测框中，寻找IOU最大的检测框作为正例。ground truth的先后顺序可忽略。正例产生置信度loss、检测框loss、类别loss。预测框为对应的ground truth box标签（需要反向编码，使用真实的x、y、w、h计算出 [公式] ）；类别标签对应类别为1，其余为0；置信度标签为1。
    2. 1.6.1.2. 忽略样例：正例除外，与任意一个ground truth的IOU大于阈值（论文中使用0.5），则为忽略样例。忽略样例不产生任何loss。（存在意义：由于Yolov3使用了多尺度特征图，不同尺度的特征图之间会有重合检测部分。比如有一个真实物体，在训练时被分配到的检测框是特征图1的第三个box，IOU达0.98，此时恰好特征图2的第一个box与该ground truth的IOU达0.95，也检测到了该ground truth，如果此时给其置信度强行打0的标签，网络学习效果会不理想。）
    3. 1.6.1.3. 负例：正例除外（与ground truth计算后IOU最大的检测框，但是IOU小于阈值，仍为正例），与全部ground truth的IOU都小于阈值（0.5），则为负例。负例只有置信度产生loss，置信度标签为0。
  • 1.7. > 图片预测

    • YOLOv3 model.py代码的简单分析注解

    > 准备

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    from functools import wraps
    import numpy as np
    import tensorflow as tf
    from keras import backend as K
    from keras.layers import Conv2D, Add, ZeroPadding2D, UpSampling2D, Concatenate, MaxPooling2D
    from keras.layers.advanced_activations import LeakyReLU
    from keras.layers.normalization import BatchNormalization
    from keras.models import Model
    from keras.regularizers import l2
    from yolo3.utils import compose

    > YOLO网络

    首先要通过特征提取网络对输入的输入图像提取特征，得到不同尺寸的特征图

    ### 代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    @wraps(Conv2D)#@wraps在此处的作用是为了使DarknetConv2D.__name__和DarknetConv2D.__doc__与Conv2D__name__和Conv2D.__doc__相同。
    def DarknetConv2D(*args, **kwargs):
        """Wrapper to set Darknet parameters for Convolution2D."""
        darknet_conv_kwargs = {'kernel_regularizer': l2(5e-4)}
        darknet_conv_kwargs['padding'] = 'valid' if kwargs.get('strides')==(2,2) else 'same'
        darknet_conv_kwargs.update(kwargs)
        return Conv2D(*args, **darknet_conv_kwargs)

    def DarknetConv2D_BN_Leaky(*args, **kwargs):
        """Darknet Convolution2D followed by BatchNormalization and LeakyReLU."""
        no_bias_kwargs = {'use_bias': False}
        no_bias_kwargs.update(kwargs)
        return compose(
            DarknetConv2D(*args, **no_bias_kwargs),
            BatchNormalization(),
            LeakyReLU(alpha=0.1))

    def resblock_body(x, num_filters, num_blocks):
        '''A series of resblocks starting with a downsampling Convolution2D'''
        # Darknet uses left and top padding instead of 'same' mode
        x = ZeroPadding2D(((1,0),(1,0)))(x)#((top_pad,bottom_pad),(left_pad,right_pad))   这里的ZeroPadding2D是为紧跟着的DBL服务的，使其尺寸正好缩小为原来的一半
        x = DarknetConv2D_BN_Leaky(num_filters, (3,3), strides=(2,2))(x)
        for i in range(num_blocks):
            y = compose(
                    DarknetConv2D_BN_Leaky(num_filters//2, (1,1)),
                    DarknetConv2D_BN_Leaky(num_filters, (3,3)))(x)
            x = Add()([x,y])
        return x

    def darknet_body(x):
        '''Darknent body having 52 Convolution2D layers'''
        x = DarknetConv2D_BN_Leaky(32, (3,3))(x)
        x = resblock_body(x, 64, 1)
        x = resblock_body(x, 128, 2)
        x = resblock_body(x, 256, 8)
        x = resblock_body(x, 512, 8)
        x = resblock_body(x, 1024, 4)
        return x

    def make_last_layers(x, num_filters, out_filters):
        '''6 Conv2D_BN_Leaky layers followed by a Conv2D_linear layer'''
        x = compose(
                DarknetConv2D_BN_Leaky(num_filters, (1,1)),
                DarknetConv2D_BN_Leaky(num_filters*2, (3,3)),
                DarknetConv2D_BN_Leaky(num_filters, (1,1)),
                DarknetConv2D_BN_Leaky(num_filters*2, (3,3)),
                DarknetConv2D_BN_Leaky(num_filters, (1,1)))(x)
        y = compose(
                DarknetConv2D_BN_Leaky(num_filters*2, (3,3)),
                DarknetConv2D(out_filters, (1,1)))(x)
        return x, y


    def yolo_body(inputs, num_anchors, num_classes):
        """Create YOLO_V3 model CNN body in Keras."""
        darknet = Model(inputs, darknet_body(inputs))
        #第一个特征层y1=(batch_size,13,13,3,85)
        x, y1 = make_last_layers(darknet.output, 512, num_anchors*(num_classes+5))

        x = compose(
                DarknetConv2D_BN_Leaky(256, (1,1)),
                UpSampling2D(2))(x)
        x = Concatenate()([x,darknet.layers[152].output])#darknet.layers[152].output]#这里的[152]中的数字为对应层数
        #第二个特征层y2=(batch_size,26,26,3,85)
        x, y2 = make_last_layers(x, 256, num_anchors*(num_classes+5))

        x = compose(
                DarknetConv2D_BN_Leaky(128, (1,1)),
                UpSampling2D(2))(x)
        x = Concatenate()([x,darknet.layers[92].output])
        #第三个特征层y3=(batch_size,52,52,3,85)
        x, y3 = make_last_layers(x, 128, num_anchors*(num_classes+5))

        return Model(inputs, [y1,y2,y3])

    流程图

    o3tSts.png

    o3tphn.png

    整个yolov3网络共252层，组成如下：

    o3amNQ.png

    > tiny_yolo_body

    轻量版的YOLO,有速度快，占内存少等的优点，视情况选择使用。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    def tiny_yolo_body(inputs, num_anchors, num_classes):
        '''Create Tiny YOLO_v3 model CNN body in keras.'''
        x1 = compose(
                DarknetConv2D_BN_Leaky(16, (3,3)),
                MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'),
                DarknetConv2D_BN_Leaky(32, (3,3)),
                MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'),
                DarknetConv2D_BN_Leaky(64, (3,3)),
                MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'),
                DarknetConv2D_BN_Leaky(128, (3,3)),
                MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'),
                DarknetConv2D_BN_Leaky(256, (3,3)))(inputs)
        x2 = compose(
                MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'),
                DarknetConv2D_BN_Leaky(512, (3,3)),
                MaxPooling2D(pool_size=(2,2), strides=(1,1), padding='same'),
                DarknetConv2D_BN_Leaky(1024, (3,3)),
                DarknetConv2D_BN_Leaky(256, (1,1)))(x1)
        y1 = compose(
                DarknetConv2D_BN_Leaky(512, (3,3)),
                DarknetConv2D(num_anchors*(num_classes+5), (1,1)))(x2)

        x2 = compose(
                DarknetConv2D_BN_Leaky(128, (1,1)),
                UpSampling2D(2))(x2)
        y2 = compose(
                Concatenate(),
                DarknetConv2D_BN_Leaky(256, (3,3)),
                DarknetConv2D(num_anchors*(num_classes+5), (1,1)))([x2,x1])

        return Model(inputs, [y1,y2])

    > 预测框的获得

    ​ 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    #将预测值的每个特征层调成真实值
    def yolo_head(feats, anchors, num_classes, input_shape, calc_loss=False):
        """Convert final layer features to bounding box parameters."""
        num_anchors = len(anchors)#num_anchors=3
        # Reshape to batch, height, width, num_anchors, box_params.
        anchors_tensor = K.reshape(K.constant(anchors), [1, 1, 1, num_anchors, 2])
        grid_shape = K.shape(feats)[1:3] # height, width
        grid_y = K.tile(K.reshape(K.arange(0, stop=grid_shape[0]), [-1, 1, 1, 1]),
            [1, grid_shape[1], 1, 1])#将x在各个维度上重复n次，x为张量，n为与x维度数目相同的列表
        grid_x = K.tile(K.reshape(K.arange(0, stop=grid_shape[1]), [1, -1, 1, 1]),
            [grid_shape[0], 1, 1, 1])
        grid = K.concatenate([grid_x, grid_y])#从倒数第1个维度进行拼接,可视为得到每个单元格左上角的坐标
        grid = K.cast(grid, K.dtype(feats))

        feats = K.reshape(
            feats, [-1, grid_shape[0], grid_shape[1], num_anchors, num_classes + 5])#(batch_size,13,13,3,85)

        # Adjust preditions to each spatial grid point and anchor size.
        #box_xy对应框的中心点，box_wh对应框的宽和高
        box_xy = (K.sigmoid(feats[..., :2]) + grid) / K.cast(grid_shape[::-1], K.dtype(feats))#grid 为偏移 ，将x,y相对于featuremap尺寸进行了归一化
        box_wh = K.exp(feats[..., 2:4]) * anchors_tensor / K.cast(input_shape[::-1], K.dtype(feats))#exp() 方法返回x的指数,ex。
        box_confidence = K.sigmoid(feats[..., 4:5])#置信率
        box_class_probs = K.sigmoid(feats[..., 5:])#80个类别
        #在计算loss的时候返回以下参数
        if calc_loss == True:
            return grid, feats, box_xy, box_wh
        return box_xy, box_wh, box_confidence, box_class_probs

    #对box进行调整，使其符合真实图片的亚子
    def yolo_correct_boxes(box_xy, box_wh, input_shape, image_shape):
        '''Get corrected boxes'''
        box_yx = box_xy[..., ::-1]
        box_hw = box_wh[..., ::-1]
        input_shape = K.cast(input_shape, K.dtype(box_yx))
        image_shape = K.cast(image_shape, K.dtype(box_yx))
        #这里K.min取较小值，便于后续步骤调整适应image的尺寸，new是image等比缩小得来
        new_shape = K.round(image_shape * K.min(input_shape/image_shape))#K.round()底层即为tf.round(),Bankers Rounding——四舍六入五取偶,如4.5——>4;3.5——>4
        offset = (input_shape-new_shape)/2./input_shape
        scale = input_shape/new_shape
        box_yx = (box_yx - offset) * scale
        box_hw *= scale

        box_mins = box_yx - (box_hw / 2.)
        box_maxes = box_yx + (box_hw / 2.)
        boxes =  K.concatenate([
            box_mins[..., 0:1],  # y_min
            box_mins[..., 1:2],  # x_min
            box_maxes[..., 0:1],  # y_max
            box_maxes[..., 1:2]  # x_max
        ])#（x_min,y_min),（x_max,y_max)分别为左上，右下点的坐标

        # Scale boxes back to original image shape.
        boxes *= K.concatenate([image_shape, image_shape])
        return boxes

    #获取每个box和它的得分
    def yolo_boxes_and_scores(feats, anchors, num_classes, input_shape, image_shape):
        '''Process Conv layer output'''
        # -1,13,13,3,2;-1,13,13,3,2;-1,13,13,3,1;-1,13,13,3,80;

        box_xy, box_wh, box_confidence, box_class_probs = yolo_head(feats,
            anchors, num_classes, input_shape)
        #将box_xy,box_wh调节成y_min,y_max,x_min,x_max
        boxes = yolo_correct_boxes(box_xy, box_wh, input_shape, image_shape)
        boxes = K.reshape(boxes, [-1, 4])
        box_scores = box_confidence * box_class_probs
        box_scores = K.reshape(box_scores, [-1, num_classes])
        return boxes, box_scores

    求得bx,by,bw,bh的公式

    o3Yj0g.png

    对yolo_correct_boxes部分代码的补充说明

    o3tCpq.jpg

    > 寻找最佳的anchor box

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    def preprocess_true_boxes(true_boxes, input_shape, anchors, num_classes):
        '''Preprocess true boxes to training input format

        Parameters
        ----------
        true_boxes: array, shape=(m, T, 5)
            Absolute x_min, y_min, x_max, y_max, class_id relative to input_shape.
        input_shape: array-like, hw, multiples of 32
        anchors: array, shape=(N, 2), wh
        num_classes: integer

        Returns
        -------
        y_true: list of array, shape like yolo_outputs, xywh are reletive value

        '''
        assert (true_boxes[..., 4]<num_classes).all(), 'class id must be less than num_classes'
        #assert（断言）用于判断一个表达式，在表达式条件为 false 的时候触发异常。
        #.all（）测试沿给定轴的所有数组元素是否都计算为True。元素除了是 0、空、None、False 外都算 True
        num_layers = len(anchors)//3 # default setting
        # 先验框
        # 678为116，90；156，198；373，326
        # 345为30，61；62，45；59，119
        # 012为10，13；16，30；33，23

        anchor_mask = [[6,7,8], [3,4,5], [0,1,2]] if num_layers==3 else [[3,4,5], [1,2,3]]

        true_boxes = np.array(true_boxes, dtype='float32')
        input_shape = np.array(input_shape, dtype='int32')
        #将边框的两点坐标形式，转化成训练的中心加边长形式，并进行归一化
        boxes_xy = (true_boxes[..., 0:2] + true_boxes[..., 2:4]) // 2
        boxes_wh = true_boxes[..., 2:4] - true_boxes[..., 0:2]
        true_boxes[..., 0:2] = boxes_xy/input_shape[::-1]
        true_boxes[..., 2:4] = boxes_wh/input_shape[::-1]

        m = true_boxes.shape[0]#m张图片
        grid_shapes = [input_shape//{0:32, 1:16, 2:8}[l] for l in range(num_layers)]
        #y_true的格式为（m,13,13,3,85),(m,26,26,3,85),(m,52,52,3,85) num_classes是一个one_hot编码
        y_true = [np.zeros((m,grid_shapes[l][0],grid_shapes[l][1],len(anchor_mask[l]),5+num_classes),
            dtype='float32') for l in range(num_layers)]

        # Expand dim to apply broadcasting.增加维度是为了更好的计算
        anchors = np.expand_dims(anchors, 0)#np.expand_dims(input, dim, name=None)函数,将维度加1
        #anchor/2和取反则是为了计算iou 可以看作把anchor的中心移到坐标原点
        anchor_maxes = anchors / 2.
        anchor_mins = -anchor_maxes
        #长度要大于0才有效
        valid_mask = boxes_wh[..., 0]>0

        for b in range(m):
            # Discard zero rows.
            wh = boxes_wh[b, valid_mask[b]]
            if len(wh)==0: continue
            # Expand dim to apply broadcasting.
            wh = np.expand_dims(wh, -2)
            box_maxes = wh / 2.
            box_mins = -box_maxes

            #计算真实框和哪个先验框最契合、对于有效的边框、采用与anchor相似的处理方法
            #INTERSECT（交集) 集合运算
            intersect_mins = np.maximum(box_mins, anchor_mins)
            intersect_maxes = np.minimum(box_maxes, anchor_maxes)
            intersect_wh = np.maximum(intersect_maxes - intersect_mins, 0.)
            intersect_area = intersect_wh[..., 0] * intersect_wh[..., 1]
            box_area = wh[..., 0] * wh[..., 1]
            anchor_area = anchors[..., 0] * anchors[..., 1]
            iou = intersect_area / (box_area + anchor_area - intersect_area)

            # Find best anchor for each true box
            best_anchor = np.argmax(iou, axis=-1)#np.argmax()返回张量沿指定维度最大值的引索

            for t, n in enumerate(best_anchor):#enumerate()是python的内置函数，可遍历每个元素，组合为索引 元素，常在for循环中使用
                for l in range(num_layers):
                    if n in anchor_mask[l]:
                        #np.floor()返回不大于输入参数的最大整数。（向下取整）
                        i = np.floor(true_boxes[b,t,0]*grid_shapes[l][1]).astype('int32')#中心点x在grid中的对应位置
                        j = np.floor(true_boxes[b,t,1]*grid_shapes[l][0]).astype('int32')
                        k = anchor_mask[l](n)#返回真实框对应最契合的先验框中
                        c = true_boxes[b,t, 4].astype('int32')#该框所对应的类别
                        #将真实框的信息放进y_true与之对应的特征层的相对应中心点中
                        y_true[l][b, j, i, k, 0:4] = true_boxes[b,t, 0:4]
                        y_true[l][b, j, i, k, 4] = 1
                        y_true[l][b, j, i, k, 5+c] = 1

        return y_true

    preprocess_true_boxes中涉及到的anchor box和ground truth经过/2和取反等处理后，其相对关系应大概如图所示：

    o3Yv7Q.png

    > LOSS值的计算

    （1）计算xy（物体中心坐标）的损失：xyloss=bool*(2-areaPred)*bce

    （2）计算wh（anchor长宽回归值）的损失：whloss=bool*(2-areaPred)*bce

    （3）计算置信度的损失：whloss=bool*(2-areaPred)*bce

    （4）计算类别损失：置信度乘上多分类的交叉熵

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    #定义一个iou函数
    def box_iou(b1, b2):
        '''Return iou tensor

        Parameters
        ----------
        b1: tensor, shape=(i1,...,iN, 4), xywh
        b2: tensor, shape=(j, 4), xywh

        Returns
        -------
        iou: tensor, shape=(i1,...,iN, j)

        '''

        # Expand dim to apply broadcasting.
        b1 = K.expand_dims(b1, -2)
        b1_xy = b1[..., :2]
        b1_wh = b1[..., 2:4]
        b1_wh_half = b1_wh/2.
        b1_mins = b1_xy - b1_wh_half
        b1_maxes = b1_xy + b1_wh_half

        # Expand dim to apply broadcasting.
        b2 = K.expand_dims(b2, 0)
        b2_xy = b2[..., :2]
        b2_wh = b2[..., 2:4]
        b2_wh_half = b2_wh/2.
        b2_mins = b2_xy - b2_wh_half
        b2_maxes = b2_xy + b2_wh_half

        intersect_mins = K.maximum(b1_mins, b2_mins)
        intersect_maxes = K.minimum(b1_maxes, b2_maxes)
        intersect_wh = K.maximum(intersect_maxes - intersect_mins, 0.)
        intersect_area = intersect_wh[..., 0] * intersect_wh[..., 1]
        b1_area = b1_wh[..., 0] * b1_wh[..., 1]
        b2_area = b2_wh[..., 0] * b2_wh[..., 1]
        iou = intersect_area / (b1_area + b2_area - intersect_area)

        return iou


    def yolo_loss(args, anchors, num_classes, ignore_thresh=.5, print_loss=False):
        '''Return yolo_loss tensor

        Parameters
        ----------
        yolo_outputs: list of tensor, the output of yolo_body or tiny_yolo_body
        y_true: list of array, the output of preprocess_true_boxes
        anchors: array, shape=(N, 2), wh
        num_classes: integer
        ignore_thresh: float, the iou threshold whether to ignore object confidence loss

        Returns
        -------
        loss: tensor, shape=(1,)

        '''
        num_layers = len(anchors)//3 # default setting
        #将预测结果和实际ground truth分开，args是[*model_body.output,*y_true]
        #y_true是一个列表，包含三个特征层，shape分别为(m,13,13,3,85),(m,26,26,3,85),(m,52,52,3,85)
        #yolo_outputs是一个列表，包含三个特征层，shape分别为(m,13,13,3,85),(m,26,26,3,85),(m,52,52,3,85)
        yolo_outputs = args[:num_layers]
        y_true = args[num_layers:]
        anchor_mask = [[6,7,8], [3,4,5], [0,1,2]] if num_layers==3 else [[3,4,5], [1,2,3]]
        input_shape = K.cast(K.shape(yolo_outputs[0])[1:3] * 32, K.dtype(y_true[0]))
        grid_shapes = [K.cast(K.shape(yolo_outputs[l])[1:3], K.dtype(y_true[0])) for l in range(num_layers)]
        loss = 0
        m = K.shape(yolo_outputs[0])[0] # batch size, tensor
        mf = K.cast(m, K.dtype(yolo_outputs[0]))

        for l in range(num_layers):
            #取出其对应的种类(m,X,X,3,1)
            object_mask = y_true[l][..., 4:5]#obiect_mask就是置信度
            #取出其对应的种类(m,X,X,3,80)
            true_class_probs = y_true[l][..., 5:]
            #将yolo_outputs的特征层输出进行处理
            #grid为网格结构(x,x,1,2),raw_pred为尚未处理的预测结果(m,x,x,3,85)
            #还有解码后的xy，wh，(m,13,13,3,2)
            grid, raw_pred, pred_xy, pred_wh = yolo_head(yolo_outputs[l],
                 anchors[anchor_mask[l]], num_classes, input_shape, calc_loss=True)
            #解码后的预测的box的位置
            #(m,13,13,3,4)
            pred_box = K.concatenate([pred_xy, pred_wh])

            # Darknet raw box to calculate loss.
            raw_true_xy = y_true[l][..., :2]*grid_shapes[l][::-1] - grid
            raw_true_wh = K.log(y_true[l][..., 2:4] / anchors[anchor_mask[l]] * input_shape[::-1])
            raw_true_wh = K.switch(object_mask, raw_true_wh, K.zeros_like(raw_true_wh)) # avoid log(0)=-inf#switch接口，就是一个if/else条件判断语句
            box_loss_scale = 2 - y_true[l][...,2:3]*y_true[l][...,3:4]

            # Find ignore mask, iterate over each of batch.
            ignore_mask = tf.TensorArray(K.dtype(y_true[0]), size=1, dynamic_size=True)
            object_mask_bool = K.cast(object_mask, 'bool')
            def loop_body(b, ignore_mask):
                true_box = tf.boolean_mask(y_true[l][b,...,0:4], object_mask_bool[b,...,0])
                #计算预测结果与真实情况的iou
                #pred_box为13，13，3，4
                #计算的结果是每个pred_box和其它所有真实框的iou
                #13，13，3，n
                iou = box_iou(pred_box[b], true_box)
                best_iou = K.max(iou, axis=-1)
                ignore_mask = ignore_mask.write(b, K.cast(best_iou<ignore_thresh, K.dtype(true_box)))
                return b+1, ignore_mask
            #遍历所有图片
            _, ignore_mask = K.control_flow_ops.while_loop(lambda b,*args: b<m, loop_body, [0, ignore_mask])#K.control_flow_ops.while_loop(cond, body, loop_vars)，其中cond是条件判断函数，返回True或False; body是循环体；loop_vars是传入cond和body的参数列表。
            #将每幅图的内容压缩，进行处理
            ignore_mask = ignore_mask.stack()
            #(m,13,13,3,1,1)
            ignore_mask = K.expand_dims(ignore_mask, -1)

            # K.binary_crossentropy is helpful to avoid exp overflow.
            xy_loss = object_mask * box_loss_scale * K.binary_crossentropy(raw_true_xy, raw_pred[...,0:2], from_logits=True)
            wh_loss = object_mask * box_loss_scale * 0.5 * K.square(raw_true_wh-raw_pred[...,2:4])
            #如果该位置本来有框，那么计算1与置信度的交叉熵
            #如果该位置本来没有框，而且满足best_iou<ignore_thresh，则被认定为负样本
            #best_iou<ignore_thresh用于限制负样本
            confidence_loss = object_mask * K.binary_crossentropy(object_mask, raw_pred[...,4:5], from_logits=True)
                (1-object_mask) * K.binary_crossentropy(object_mask, raw_pred[...,4:5], from_logits=True) * ignore_mask
            class_loss = object_mask * K.binary_crossentropy(true_class_probs, raw_pred[...,5:], from_logits=True)

            xy_loss = K.sum(xy_loss) / mf
            wh_loss = K.sum(wh_loss) / mf
            confidence_loss = K.sum(confidence_loss) / mf
            class_loss = K.sum(class_loss) / mf
            loss += xy_loss + wh_loss + confidence_loss + class_loss
            if print_loss:
                loss = tf.Print(loss, [loss, xy_loss, wh_loss, confidence_loss, class_loss, K.sum(ignore_mask)], message='loss: ')
        return loss

    在loss值的计算过程中预测框被分为三类：

    正例：任取一个ground truth，与4032个框全部计算IOU，IOU最大的预测框，即为正例。并且一个预测框，只能分配给一个ground truth。例如第一个ground truth已经匹配了一个正例检测框，那么下一个ground truth，就在余下的4031个检测框中，寻找IOU最大的检测框作为正例。ground truth的先后顺序可忽略。正例产生置信度loss、检测框loss、类别loss。预测框为对应的ground truth box标签（需要反向编码，使用真实的x、y、w、h计算出 [公式] ）；类别标签对应类别为1，其余为0；置信度标签为1。

    忽略样例：正例除外，与任意一个ground truth的IOU大于阈值（论文中使用0.5），则为忽略样例。忽略样例不产生任何loss。（存在意义：由于Yolov3使用了多尺度特征图，不同尺度的特征图之间会有重合检测部分。比如有一个真实物体，在训练时被分配到的检测框是特征图1的第三个box，IOU达0.98，此时恰好特征图2的第一个box与该ground truth的IOU达0.95，也检测到了该ground truth，如果此时给其置信度强行打0的标签，网络学习效果会不理想。）

    负例：正例除外（与ground truth计算后IOU最大的检测框，但是IOU小于阈值，仍为正例），与全部ground truth的IOU都小于阈值（0.5），则为负例。负例只有置信度产生loss，置信度标签为0。

    > 图片预测

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    def yolo_eval(yolo_outputs,
                  anchors,
                  num_classes,
                  image_shape,
                  max_boxes=20,
                  score_threshold=.6,
                  iou_threshold=.5):
        """Evaluate YOLO model on given input and return filtered boxes."""
        #获得特征层的数量
        num_layers = len(yolo_outputs)
        #特征层1对应的anchor是678
        #特征层2对应的anchor是345
        #特征层3对应的anchor是012
        anchor_mask = [[6,7,8], [3,4,5], [0,1,2]] if num_layers==3 else [[3,4,5], [1,2,3]] # default setting
        input_shape = K.shape(yolo_outputs[0])[1:3] * 32
        boxes = []
        box_scores = []
        #对每个特征层进行处理
        for l in range(num_layers):
            _boxes, _box_scores = yolo_boxes_and_scores(yolo_outputs[l],
                anchors[anchor_mask[l]], num_classes, input_shape, image_shape)
            boxes.append(_boxes)
            box_scores.append(_box_scores)
        #将每个特征层结果进行堆叠
        boxes = K.concatenate(boxes, axis=0)
        box_scores = K.concatenate(box_scores, axis=0)

        mask = box_scores >= score_threshold
        max_boxes_tensor = K.constant(max_boxes, dtype='int32')
        boxes_ = []
        scores_ = []
        classes_ = []
        for c in range(num_classes):
            # TODO: use keras backend instead of tf.
            #取出所有box_scores>=score_threshold的框和成绩
            class_boxes = tf.boolean_mask(boxes, mask[:, c])#tf.boolean_mask 的作用是 通过布尔值 过滤元素tensor：被过滤的元素mask：一堆 bool 值，它的维度不一定等于 tensorreturn： mask 为 true 对应的 tensor 的元素当 tensor 与 mask 维度一致时，return 一维
            class_box_scores = tf.boolean_mask(box_scores[:, c], mask[:, c])
            #非极大抑制
            nms_index = tf.image.non_max_suppression(
                class_boxes, class_box_scores, max_boxes_tensor, iou_threshold=iou_threshold)
            #获得非极大值抑制后的结果
            #下列三个分别是：框的位置，得分与种类
            class_boxes = K.gather(class_boxes, nms_index)
            class_box_scores = K.gather(class_box_scores, nms_index)#gather（reference,indices)在给定的张量中搜索给定下标的向量。
            classes = K.ones_like(class_box_scores, 'int32') * c#ones_like给定一个tensor（tensor 参数），该操作返回一个具有和给定tensor相同形状（shape）和相同数据类型（dtype），但是所有的元素都被设置为1的tensor。
            boxes_.append(class_boxes)
            scores_.append(class_box_scores)
            classes_.append(classes)
        boxes_ = K.concatenate(boxes_, axis=0)
        scores_ = K.concatenate(scores_, axis=0)
        classes_ = K.concatenate(classes_, axis=0)

        return boxes_, scores_, classes_