Extract upwards pointing lane lines

Question

We try to detect lane lines on a running track (and with this information the upcoming direction).

We currently use the following (simplified) steps:

1. Binary: transform the input with a binary threshold

2. Cropped: crop the region of interest (currently just the lower half of the image)

3. Canny: detect edges and group them with HoughLinesP

4. Upward Lines & Closing: only keep lines with an extreme slope - ignore horizontal lines

5. Result: find connected components and fit quadratic function through each line

This works in general (examples: straight.png, left.png) but has issues if for example a horizontal connection line wasn't removed (example: problems.png - bottom right & bottom left). In such a case the two lines and the connection get interpreted as one connected component.

As our point of view can tilt a lot from left to right (camera mounted on a running person) it is quite hard to define a slope threshold for upwards pointing lines.

Is there a better way to get rid of non upwards pointing track lines? As the current solution with canny, hough transform and slope filtering is not optimal for curves and sometimes doesn't work at all (as described above).

Would it be possible to get from the cropped image straight to the separated lines through morphology operations? Similar to this example. I know we have no strict horizontal lines which makes any fitting kernel a lot more complex (I assume).

Currently, we try to use perspective transformations to get a bird view on the track. This should help to distinguish between horizontal and upward pointing lines.

Another small issue is too short lines which result in inaccurate approximated quadratic functions (most left line in problems.png and straight.png). This might be easily solvable (by requiring a minimal pixel count for a component to be counted as track line) and should not be part of this question.

EDIT (answer questions from vlad_tepesch)

What is with the curves? should they be part of your lane models or not?

We clearly want to detect curves of the running track - as this is needed for the direction estimation. But we want to ignore (remove) the horizontal connection curve on the bottom of the problems.png example.

4./6. rectification - camera distortion

I will take another look at rectification, until now I postponed it, as I thought it isn't that important.

7.3 how many pixel if overlap between segments is ignored (group pixel count)

Just draw all lines from a group and check the non zero pixel count?

10. lanes normally are not quadratic - look int clothoids instead - this may overkill so may be use a 3rd order polynomial instead

I get your point. Clothoids are out of scope at the moment, but I will keep them in mind. How would you go with the direction estimation if we use more complex fitting? Currently we just take the first coefficient of the 2nd oder polynomial to estimate the bending direction (and the degree of curvature to differentiate between curves and straight parts).

optional more advanced

Good point, we already thought about averaging results over multiple frames. I also keep that proposal in the back of my mind.

Straight

Left

Problems

Answer 1

看来您真正的问题是过滤和连接步骤。因为您过滤掉了很多水平分量，所以为了连接未过滤的近乎垂直的线，只留下次优线段。

曲线是怎么回事？它们是否应该成为您的车道模型的一部分？起初我认为不会，但它只在分组阶段有一些影响。

我建议采取以下步骤

1. 庄稼
2. 二进制（为什么要在被裁剪的图像部分的二值化上浪费处理时间）
3. 精明（也许尝试垂直索贝尔以获得仅水平边缘）
4. 考虑校正以纠正相机畸变
5. 霍夫线P
6. 如果您决定不对输入图像进行校正，那么至少要校正输出线的坐标
   从处理时间的角度来看，这可能是更理想的选择 - 然而，霍夫线检测器将根据输出线的强度产生不太理想的结果你的相机失真
7. 连接线，无论其方向如何
   1. 我会首先尝试以下方法
   2. 为每个线段计算一条线（点，方向）
   3. 查找具有相似参数的线并检查线段的距离
   4. 如果足够近，将它们分成一个线组
   5. 最后，您应该有一些线组，其中包含一些属性，您可以计算这些属性来估计质量
      1. 有多少条线/段属于该组
      2. 有多少个像素属于该组（所有线段的长度之和）
      3. 如果忽略段之间的重叠，则有多少像素（组像素计数）
      4. 线组上最左边/最右边的像素（某种组线段）
      5. 填充率（组线长度/组像素数）
   6. 之后，不再需要对线段进行分组及其对组的分配，但我将保留计算的属性。
8. 如果弯道应该是车道的一部分，那么我会尝试寻找在末尾继续找到的组的组并将它们合并。
   这应该可以防止交叉线的合并。
9. 根据您的需要过滤组
10. 配件。
    1. 车道通常不是二次方的
    2. 改为查看回旋曲线
    3. 这可能太过分了，所以可以使用三阶多项式代替

可选更高级

1. 将所有组保留在最后
2. 仅在输出级别进行过滤
3. 在下一帧中，将旧组与新组进行匹配，并对模型进行一些跟踪和合理性检查。