Automated detection of damages and objects in sewerage systems with artificial intelligence
Fig.1: Automatically detected deposits at the bottom in a sewerage system (image unwinding of the sewerage system).
Optimizing and increasing the efficiency of sewerage system inspection and revision is an important topic for our society. The engineering company Basler & Hofmann AG has taken up the challenge in cooperation with LeanBI AG.
And to anticipate the conclusion of the article: Artificial intelligence (AI) massively increases efficiency, but engineers and experts still don’t have to be cut back. AI helps the experts to perform their work better, which is accompanied by a high cost-saving potential. But more on that later.
The total length of public and private sewerage systems in Switzerland is over 100,000 km. According to “Die Volkswirtschaft” [1], the replacement value is estimated at a total of 114 billion CHF. In principle, the average service life of sewerage pipelines of about 70 years is high [2]. But experts assume an increased need for investment, since many pipelines were built in the 60’s to 80’s. According to “Die Volkswirtschaft” [1], the total value maintenance costs over the next 30 years will be around 130 billion Swiss francs. This means that we will have average annual maintenance costs of 4.3 billion CHF. According to [3], the value maintenance of the sewerage network is currently between 50 and 400 CHF per year and inhabitant and thus shows a high range. With a population of 8.7 million inhabitants in Switzerland, this figure indeed indicates that the total value maintenance will be accompanied by a strong cost increase in the coming years.
At a minimum, public sewerage systems should be inspected every 10 to 15 years to determine their condition. Allegedly, studies show that about 20% of sewerages are renewed unnecessarily and 15% are not rehabilitated in time. We estimate that a qualitative improvement of the current monitoring of sewerages would save 10% of the CHF 4.3 billion, i.e., savings of CHF 430 million per year in Switzerland on average over the next 30 years. We are convinced that these savings are achievable, as shown in the following.
Two system principles are in use today for sewerage inspections:
- Conventional video cameras that can be panned by the camera operator when damage is suspected. The damage is then documented directly by the operator in software.
Fig. 2: Example iPEK Special TV, carriage for pipe diameters 250 – 1000 mm from Wikipedia
- PANO-scan technology with two fisheye cameras that shoot an image every 5 cm, from which a 3D image can be generated along the sewerage.
Fig. 3: Example IBAK PANORAMO 4K, camera system / 3D scanner, application range from DN 200
Most public sewerages have diameters between DN 200 and DN 1800, where the PANO-scan technology can be used. The advantage of the PANO-scan technology is that the damage inspection can be carried out not on site, but afterwards on a “digital twin”, a 3D panoramic image of the sewerage system. In addition to the improved inspection process, this technology provides a digitized and traceable database for future redevelopment.
PANO-scan recording on site in the sewerage takes place without stops and is accordingly recorded quickly and can be planned well in advance. The error rate of damage detection is reduced compared to conventional technology, as a trained expert does not perform the inspection on site, but afterwards on the computer. Various software solutions are available today for this purpose. The KINS solution from the company Kanalinformatik GmbH, which was also used in this project, is widely used in Switzerland.
Despite the great progress of PANO-Scan technology, it remains a feat to inspect the many sewerage kilometers (estimated 7,000 km/year) manually at the office workstation. The inspection takes place in a combination directly on the 3D image and the unwinding of the sewerage system. In the 3D image, the perspective can be freely selected, so that even difficult objects such as root inclusions can be better identified. Despite new camera technology, the quality of the inspection is and remains a question of the experience and concentration of the expert employed. In addition to the manual effort, the evaluation and classification of the damage allows for a great deal of subjective freedom, making the recognition of condition and damage dependent on the analyzing expert. We were also able to determine such subjectivity in our statistical evaluations of various municipalities.
Automated evaluation of the recorded data material, on the other hand, leads to improved and consistent monitoring quality by minimizing errors due to fatigue, ignorance (insufficient training) or overlooking of damage. In other words, subjectivity is largely eliminated.
In 2020, Basler & Hofmann AG commissioned LeanBI AG to develop machine learning models for the automated detection of damage and contamination in sewerages. Today, such models are also referred to as artificial intelligence.
As an internationally active engineering firm headquartered in Switzerland, Basler & Hofmann AG provides services for the planning and maintenance, in particular condition surveys and inspections, of infrastructure structures and is familiar with the needs and requirements of sewerage operators. Basler & Hofmann uses its own engineers to evaluate the damage in the sewerage systems.
For the project, a team of Basler & Hofmann engineers, LeanBI data scientists and subject matter experts was formed to drive the automation of damage detection. Large amounts of image material from large and small rural communities, as well as cities, were statistically analyzed. The approximately 200 damage categories of the VSA (Verband Schweizer Abwasser- und Gewässerschutzfachleute) guideline were clustered in such a way that in the future 95% of damage cases and pollution can be detected automatically with artificial intelligence.
Fig. 4 shows a schematic representation of the manual annotation process of damaged lateral connections: The relevant markings are marked on the images with pixel precision using an annotation tool. These markings are then exported and combined into binary masks on a class-by-class basis. Together with the unwinding images, these masks form the basis for the training data.
Fig. 4: Manual annotation process in the unwinding of a sewerage pipe
The machine learning models were then trained based on the manual annotations. In contrast to existing solutions, the project team decided to apply the so-called “machine learning image segmentation”, i.e., a pixel-precise defect detection of the object or damage. The machine learning models thus detect the exact shape of the defects, in contrast to the few solutions available on the market today, which usually place “bounding boxes” around the objects. Let us illustrate the segmentation with an example based on an incompletely bound connection shown in Fig. 5.
Fig. 5: Image of a lateral connection “incompletely integrated
In the automated detection with artificial intelligence according to Fig. 6, both the connection (green) and the damage (red) are marked. This immediately provides a visual representation of the severity of the damage.
Fig. 6: Lateral connection incompletely integrated
The advantage of “machine learning image segmentation” is that the geometries of the damage and contamination can be evaluated via the pixel information. These evaluations can then be provided as information to the engineer directly or to downstream applications. This makes it much easier to identify the severity of the damage.
Another advantage of machine learning image segmentation is that fewer manual annotations are needed to train the models. In our project, we nevertheless performed far more than 1000 such manual annotations.
On the other hand, the core competence of LeanBI is to generate high-quality machine learning models even with little training data. In addition to the selection of the models, the so-called “augmentation”, i.e., the artificial expansion of the image material for training purposes, plays a decisive role.
However, we do not want to conceal one disadvantage of “machine learning image segmentation”. The manual annotation must be carried out very precisely, otherwise the artificial intelligence will not work well enough. Therefore, we have implemented all annotations internally in the project by specialists and deliberately did not use external forces.
The aim of the project was to achieve a “recall” of almost 100%. This means that we do not miss any damage or contamination within the defined damage classes. This is because not detecting damage is far more problematic than falsely reporting damage from time to time. In this case, the engineer can detect such a “false positive” in the inspection process by simply pressing a button. We have achieved this goal for several damage classes, and the artificial intelligence will learn a few others over time, since the engineers provide continuous feedback to the application.
And this brings us back to the topic of how artificial intelligence supports the engineer. The engineer should concentrate on the difficult and rare cases such as root inclusions or infiltrations, while for the frequent cases such as cracks, deposits, deformations, etc., the engineer is supported by artificial intelligence in monitoring and documentation. This not only makes the engineer more efficient, but also improves the quality of his work. The engineer still has at least the important control function.
The present recognition software is set up in such a way that it can be integrated into an existing inspection tool such as KINS. The image data is exported from the inspection software, runs through the automated recognition and annotation process, and is then imported back into the inspection software. There, the image data can then be checked, post-processed and documented by the engineer.
If damage is detected better and earlier, renewals will take place instead of replacements and fewer redevelopment s that would not have been necessary at all. This leads to a reduction in redevelopment costs. And something is also done for people and the environment: qualitatively better monitoring prevents unintentional sewerage pipeline leaks into the environment and infiltrations in freshwater pipes.
We are convinced that with the help of artificial intelligence in combination with PANO technology, around half to two-thirds of today’s recognition errors can be eliminated. Accompanying this estimate is that visual limitations such as poor lighting, poor image resolution, or poor visibility due to debris will also be reduced. In the coming years, imaging technology should improve with the increased use of PANO technology.
[1] Redevelopment case sewerage system, Max Maurer, Sabine Hoffmann | 21.05.2019
[2] Environmental enforcement, communication on water protection No. 42, costs of sewerage disposal, 2003.
[3] Guideline for calculating the value maintenance costs of sewerage plants, page 70, environment, and energy (uwe), canton of Lucerne, 2019.
