The benefits of production robots, which play a vital role in automotive manufacturing, are undisputed. However, they also require elaborate safety systems to ensure people who might accidentally get in their way are protected from injury. Under everyday production conditions at its Sindelfingen plant, DaimlerChrysler is about to test a new concept: SafetyEYE is designed to monitor a robot’s radius of action — the volume within which it operates. Group researchers in Ulm have provided the algorithms needed to process the images.

Robots can stack and pallet goods, weld, bolt, glue, drill, paint and polish workpieces — and many other tasks besides. What’s more, these immensely versatile pieces of production machinery can accomplish a vast range of operations faster, more precisely and more consistently than even the most conscientious of workers. In fact, there’s only one thing they lack: the ability to perceive workers in their midst. As soon as they are switched on, they will continue carrying out their programmed operation until someone presses the stop button or pulls the plug.

That’s why robot stations must be made safe for workers by means of extensive systems, including barriers to physically prevent access to the danger area, door contact switches, light barriers, laser scanners and pressure mats to immediately activate the emergency stop if someone crosses the barrier or unwittingly sets foot in the wrong place.

The radius of a production robot’s activities can encompass a volume of as much as 70 cubic meters, and workers who penetrate this space run a high risk of injury. The movements of these massive pieces of machinery may be remarkably graceful — despite weighing as much as two tons — but humans are absolutely powerless to prevent or even divert by one millimeter any of a robotic arm’s programmed actions, which are driven by powerful electric motors with ratings as high as ten kilowatts.

Comprehensive safety regulations such as the machinery guideline provide effective protection against such fatal collisions of the human and mechanical worlds. Yet, although such accidents at work are fortunately rare, the safety systems currently in use are still not ideal. Three factors in particular give safety and production experts cause for concern:

Optical devices such as light barriers and laser scanners can’t monitor volumes — at best, they can cover a plane. They erect a kind of optical barrier marking a danger zone’s boundary. Moreover, such systems can only be installed where there is an unobstructed view of the area under surveillance. In instances where this isn’t possible, the work station must be made safe with additional systems, such as pressure mats. The fact remains that today’s technology often cannot provide seamless monitoring of a robot’s radius of action, or this can be achieved only with an extremely complex combination of different sensors.

At the same time, all of the safety systems currently in use are connected to the robot’s emergency stop function. This means that as soon as the alarm is activated, the robot is immediately brought to a halt. Before production can be resumed, however, the robot must be returned to exactly the position it was in immediately before the emergency stop was activated. This consumes valuable time and drives up costs, not least because of the snowballing effect that such downtime has on work stations located farther down the production line.
Depending on the specific type of robot station used, the safety systems needed can also be very expensive — in some cases more costly than the robot itself.

For Anton Hirzle, head of Process Development for Plant and Control Technology at the Sindelfingen plant, and Rainer Ott, former head of Industrial Image Processing at DaimlerChrysler Research, these were good enough reasons to create a new concept. Their solution was to use a combination of cameras to produce a three-dimensional image of the protected area. This way, they reasoned, arithmetical methods could then be applied to the data in order to identify any objects penetrating this space.

It was a very similar story at the Pilz company in Ostfildern, the leading supplier of high-safety automation technology. Engineers at Pilz had started to search for alternative ways to monitor protected zones around production robots at an early stage. This presented an ideal opportunity for cooperation: Pilz would take on responsibility for developing, manufacturing and marketing the new system, and a DaimlerChrysler research team led by Christian Wöhler was charged with supplying the algorithms needed to analyze the three-dimensional image data.

Wöhler works at the Ulm Research Center in the Department of Ambient Interpretation. Areas in which his team has already produced successful results include the design of optical systems to help drivers identify hazards that are either overlooked or not registered in time — a stoplight changing to red, for example, or a ball rolling into the street with a child chasing it.

All of these systems use image-processing technology, whereby video data is analyzed by computer. This in turn requires programs that continuously scan a specific area — for instance the road up ahead — for objects that are in imminent danger of colliding with the vehicle, due to their position, direction of movement, and speed. As soon as the system identifies such an object, it warns the driver.

In line with the cooperation between DaimlerChrysler and Pilz, it was decided back in 2002 that Wöhler’s team should apply this type of image processing to the requirements of automated production and develop the necessary algorithms. The job task assigned to Pilz, on the other hand, was to make these algorithms fit for use in industry and to develop the video-based system — named SafetyEYE — into a marketable product.

In fact, SafetyEYE’s sensor unit, which is currently undergoing tests at the Sindelfingen plant, features a total of three eyes. For humans and technical systems alike, two eyes are required to see in three dimensions. SafetyEYE, however, has been given three optical sensors to enable it to measure horizontal lines within a triangulated system and thereby precisely determine object distances along such a line — a faculty that is underdeveloped in the human sense of sight.

> Sensors that can cope with wide variations in brightness

The members of the project team spent quite a long time looking for just the right sensors before finally deciding in favor of CMOS units, which are also used in commercial surveillance cameras. The three CMOS sensors that are used in the SafetyEYE system have one major advantage, though: They can produce a usable picture of every part of the monitored zone, even when the area encompasses extreme variations in terms of brightness — a range of as much as one million to one. As a result, with the type of lighting conditions that are encountered under everyday manufacturing conditions, there is no danger of producing images that are overexposed or that have dark patches.
Unlike conventional robot stations, which are generally cordoned off with barriers and protected by other visible safety systems, the test bay for SafetyEYE is completely open, looking for all intents and purposes unsecured. Apart from the production robot, the only other piece of equipment to be seen is a metal mast slightly over 2.5 meters high that supports an aluminum plate to which three sensors have been fixed. The plate is positioned in such a way that the three sensors have an unobstructed view of the robot’s complete radius of action.

Only a look at the corresponding monitor shows just how well the danger zone is protected. Superimposed on the black-and-white image of the area around the robot are colored cubes and squares. The latter represent the robot station’s danger areas, which the computer program that controls SafetyEYE defines purely in terms of spatial coordinates. These correspond to all the space segments penetrated by the robot’s moving parts during one complete working cycle. Taken together, these segments form a kind of “safety cocoon” around the robot. Objects outside of this zone are in no danger, but those within it are at risk.

“That’s the essential difference,” explains Lars Krüger, one of Wöhler’s colleagues. “SafetyEYE monitors the entire danger zone, not only the transitional planes that mark the boundary between the inside and outside of the safety area.” If someone were to accidentally move a toolbox into the working area of a robot, for example, a laser scanner would immediately sound the alarm, because the object would have breached an optical barrier. By contrast, SafetyEYE might not necessarily react in certain instances — if the radius of action only begins one meter above the ground at this point, for example, in which case the toolbox would remain outside of the robot’s radius of action.

Together with the Pilz team, Werner Progscha, who works in Hirzle’s department in Sindelfingen, is currently preparing SafetyEYE for its inaugural deployment on several robot stations on the production line for the new Mercedes-Benz C-Class. The system displays an even more crucial advantage here: SafetyEYE has been configured for use at “manual work stations,” where assembly workers might be carrying out a task on the raw body shell in the vicinity of the engine compartment.

> Safety in concealed areas

The danger in this instance is posed by the use of a moveable sledge that transports the body shell into the work station. The sledge stops moving while the assembly procedure is being carried out, and then it continues on to the next station once the worker has pressed the release button. Because it is so crucial to ensure that the sledge will move only when nobody else is working on the vehicle, the work station is equipped with safety systems.

“But let’s say that someone stands on the front members of the sledge in order to work on the engine compartment. That person is then in danger despite the fact that the floor beneath is fitted with a pressure mat,” explains Progscha. An optical safety device is also out of the question here, since the front members of the sledge mask the danger zone behind. However, a safety cocoon specially mapped out by SafetyEYE can be shaped around the body shell in such a way as to include the engine compartment as well.

Back at the test bay, Wöhler provides a graphic demonstration of exactly how the SafetyEYE system works. He first stretches out his hand into the virtual safety zone. Although his hand appears on the monitor as nothing more than a small, red pixilated mass, the robot nonetheless comes to an immediate halt. In a matter of milliseconds, SafetyEYE has analyzed the sensor images — which are generated at a rate of 20 per second — discovered an object (Wöhler’s hand) in one of the defined danger segments, and monitored its presence over the next two images. The latter factor is required because the algorithms are only defined to recognize a violation of the protected area on the basis of three successive images. Within 150 milliseconds after Wöhler’s hand has entered the danger zone, the robot has been entirely immobilized.

To demonstrate yet another advantage of the new system, Krüger places his foot inside a segment of the protected area that the robot will only enter several seconds later. Once again, SafetyEYE recognizes the danger. This time, however, the system doesn’t immediately activate the emergency stop. Instead, it slows the robot down to a snail’s pace and sounds an alarm signal, prompting Krüger to withdraw his foot. The robot then reverts to normal speed once Krüger’s foot is clear of the zone. “SafetyEYE operates more flexibly than just activating the emergency stop button,” he explains. As a result, the robot can continue to operate without delay if the danger is removed.

And the new system offers even more advantages. For a start, it only takes a couple of hours to create a safety cocoon in a work station. By contrast, at least one full day is needed to position, set up and check conventional safety systems. Likewise, the costs of SafetyEYE monitoring are highly competitive — according to Progscha, under 10,000 euros per system. A conventional safety setup, by way of comparison, can cost as much as 35,000 euros.

Apart from the system’s mast and sensors, the only other hardware required by SafetyEYE are two industry-grade PCs. Each computer runs with a different operating system, and each analyzes the video images with a different set of algorithms. This parallelism of two independently operating systems substantially increases the reliability of SafetyEYE. Only one of them has to identify an object in the danger zone in order for the alarm to be raised.

Systems such as SafetyEYE could bring about major changes in the safety philosophy that is applied to manufacturing operations, Progscha explains. At present, the golden rule is always: one safety system for every work station. If it is positioned carefully, however, SafetyEYE would be quite capable of monitoring two adjacent, independently operating robots or production machines.

By the same token, the system also would make it possible to merge neighboring safety zones, which would open up entirely new possibilities for the design of production lines. Still another option might be to utilize SafetyEYE at the same time for auxiliary tasks. Wöhler and Krüger also agree that SafetyEYE has big potential in other areas. At present, Krüger is looking at a method known in the field as “blanking.” This involves the computerized suppression of image data that are irrelevant to, or interfere with, the monitoring of the danger zone.

> Ensuring safe cooperation between man and machine

One example of a possible application for the system is a work station through which a body shell moves slowly while an employee works on it. Because it is an object moving within the danger zone, the body shell itself would normally be expected to activate the alarm. By programming the software to place what is known as an “envelope curve” around the body shell, however, it would be possible to blank out this image, with the result that the monitoring system would only react if another object enters the safety cocoon.

Wöhler is even prepared to take things one step further here: “Our system still isn’t able to distinguish between different objects. All the computer does is to register the presence of a pixel cloud within the danger zone. But it can’t differentiate between a worker’s hand and a falling wrench. If it were able to identify objects and distinguish between them — if it could ‘know’ the difference between a human hand and a tool, for example — it would in principle be feasible to create a safety system that would make it possible for a worker and a robot to work in parallel on the same workpiece.”

> Triangulation: Pinpointing objects

Thanks to double stereoscopic vision — the product of SafetyEYE’s three video sensors — image data can be analyzed to determine the coordinates of any object in the monitored area (below). The robot’s radius of action — and therefore the size and shape of the danger zone in this work station — are visualized by means of colored spatial segments that are imposed on the video image. The algorithms search for any changes in the pixel values from one image to the next, changes that correspond to the movement of an object. If any such movement is detected within the protection zone, the safety system sounds the alarm and stops or slows down the robot.



With the help of three CMOS video sensors, SafetyEYE monitors the danger area of a work station in three dimensions and under all light conditions encountered in production.


There is no light barrier, laser scanner or safety barrier to prevent access to the radius of action of this production robot at the Sindelfingen plant. Instead, the colored bands on the monitor show the virtual protection zones as defined by the SafetyEYE software.


Triangulation: Pinpointing objects


SafetyEYE brings a robot to a standstill within 150 milliseconds of the instant a person penetrates the virtual protection zone.


Triangulation: Pinpointing objects


SafetyEYE brings a robot to a standstill within 150 milliseconds of the instant a person penetrates the virtual protection zone.


SafetyEYE brings a robot to a standstill within 150 milliseconds of the instant a person penetrates the virtual protection zone.


Even laser scanners merely monitor a horizontal or vertical plane rather than a whole space. Above: The emergency stop and restart buttons for the sledge in the manual work station pictured at top left.


SafetyEYE’s vigilant sensors simultaneously monitor two work stations for windshield assembly. This is a breakthrough in safety philosophy, where the golden rule has always been: one safety system for every work station. In this case, the system consists of two vertically mounted light barriers and a laser scanner positioned on the floor. They monitor each work station separately whenever a windshield is being installed.


The body shell enters the manual work station from the right on a sledge. Only a pressure mat protects someone who is working “concealed” in the engine compartment. A worker standing on the members of the sledge or on the body shell itself would not be protected by this conventional safety system. However, SafetyEYE is also able to monitor this critical situation.


Even laser scanners merely monitor a horizontal or vertical plane rather than a whole space. Above: The emergency stop and restart buttons for the sledge in the manual work station pictured at top left.


SafetyEYE’s vigilant sensors simultaneously monitor two work stations for windshield assembly. This is a breakthrough in safety philosophy, where the golden rule has always been: one safety system for every work station. In this case, the system consists of two vertically mounted light barriers and a laser scanner positioned on the floor. They monitor each work station separately whenever a windshield is being installed.


Copyright © 2006, DaimlerChrysler AG