|
|
|
|
|
|
|
|
Linux drives the experimental vehicle of the Università di Parma.
by Massimo Bertozzi, Alberto Broggi and Alessandra Fascioli
The goal of the ARGO Project is to develop active safety systems for vehicles of the future. These systems are able to understand the environmental conditions and, in case of sudden danger, they can warn the driver or even take control of the vehicle. Furthermore, these systems have the capability to fully drive the vehicle without human intervention (Automatic Driving).
Figure 1. The ARGO Prototype Vehicle
ARGO (see Figure 1), the prototype vehicle developed at the Università di Parma, Italy, was demonstrated to the public and the scientific community in June 1998, when it drove autonomously for more than 2000km on the Italian highway network.
The entire real-time data acquisition and processing is performed on a single Pentium MMX-based PC running Linux.
A large number of problems (not just those related to traffic and safety) could be solved by using automatically driven vehicles.
Beside the obvious advantages of increasing safety and reducing road accidents, thereby saving lives, the possibility of having vehicles moving in much closer proximity than they do today would produce an increase in road capacity. A more intelligent modulation of each vehicle's speed would also result in an appreciable reduction in fuel consumption. In other words, automatic vehicle driving has the potential to achieve optimal use of current transportation infrastructures, improve mobility and minimize risks, travel times and energy consumption. Moreover, commercial and industrial vehicles which repeatedly move along a given path would benefit from a stronger control of their routes and would require fewer personnel to manage their moves.
Unfortunately, an automatic vehicle, more than in other applications, needs real-time response, thus requiring smart algorithms and powerful computing engines. At the same time, as far as a commercial product is concerned, the cost of the system must be kept low. Initially, the underlying technology, such as head-up displays, infrared cameras, radars and sonars are derived from expensive military applications. Thanks to increased interest in these applications and the progress of industrial production, today's technology offers sensors, processing systems and output devices at very competitive prices.
This project shows how a very low-cost solution (regarding both sensors and processing engines) was successfully adapted to drive an ``intelligent'' vehicle in real-world conditions. This article describes the underlying architecture, which is based on a standard Pentium MMX 200 MHz and the Linux operating system, and discusses the main advantages and problems encountered during the whole project.
The project was founded by the Italian National Research Council (CNR), and the MilleMiglia in Automatico tour was sponsored by TIM, Telecom Italia Mobile, which provided GSM apparatus and cellular connectivity throughout the whole journey.
The heart of the computing architecture installed on ARGO (shown in Figures 2 and 3) is based on a single Pentium MMX processor (200MHz, 32MB of RAM). The PC was equipped with some additional boards for image acquisition, image visualization, acoustic warnings and data I/O from specific devices.
Figure 2. Internal View of the ARGO Vehicle
Figure 3. The Computing Architecture and Equipment
GOLD (Generic Obstacle and Lane Detection), the software that drives the ARGO vehicle, has been designed to be as portable as possible; therefore, it is independent of specific Linux distributions/libraries and can be compiled against either libc or glibc. Moreover, because of the high stability of the first kernel (2.0.18) we installed, the current distribution used on ARGO is still a Debian 1.2.
The following is a description of the ARGO adapters/software libraries as well as their support for Linux.
Acquisition device: since ARGO relies on a vision-based processing engine, the acquisition device is the most important piece of hardware. GOLD needs a frame grabber capable of simultaneously grabbing two grey-level images from two different cameras. Thanks to the availability of specific mailing lists about Linux and Vision (http://atlantek.com.au/USERS/wes/linux/frame.html), we discovered that only the Matrox Meteor RGB frame grabber had this feature among the Linux-supported hardware. The Matrox Meteor module is not yet video4linux compliant. Nevertheless, the availability of many examples as well as their source code permitted us to rapidly interact with this hardware.
Data I/O: while the frame grabber could be devised as the main input device, autonomous driving also requires an output device for maneuvering the steering wheel. A National Instruments LabPC+ ISA adapter has been installed in the ARGO PC; it provides a number of multi-functional analog, digital and timing I/O ports. Therefore, it is used for driving the steering wheel actuator (a stepping motor), determining the vehicle's speed (through a Hall-effect speedometer) and inputting user commands from the control panel as in Figure 2. The LabPC+ usage is quite simple, since the Linux module supplies a device and simple ioctls for each different I/O port.
Acoustic warnings: acoustical warnings are fed to the driver. A simple interface for a standard sound card has been developed using the OSS free sound module.
Figure 4. A Screenshot of the Output from the Onboard Monitor
Image visualization: the result of the computation is also fed to passengers through a color 6-inch monitor installed onboard the ARGO vehicle (see Figure 4). Development of the software requires the capability to both inspect intermediate results and input debug commands. For these purposes, two interfaces have been developed, one VGA-based and one X11-based. In the first case, the SVGA library is used for showing the (intermediate) results and the ncurses library is used to input user commands. In the second case, the xform widget library is also used (see Figure 5).
Figure 5. The X11-based Control Panel
Programming facilities: the MMX capabilities of the Pentium processor were exploited in order to boost processing speed. A few portions of the GOLD code were directly rewritten using MMX assembly language and compiled using the Netwide Assembler (NASM), a general-purpose x86 assembler that supports Pentium, P6 and MMX opcodes.
Internet connection: during the vehicle's demonstrations, live video shots of the driving cabin are broadcast to the Internet. To accomplish this, a link to the Internet was established. This type of link implies the use of mobile telecommunications facilities, such as GSM or satellite modems and places the following constraints:
In order to not excessively burden the main processing engine, another cheap Linux box (a Compaq laptop) was installed on the vehicle and was equipped with a parallel port Quickcam Color camera (supported by Linux) and two GSM modems able to work up to 9600Kbps using the MNP10-EC protocol. A custom application is used to grab images from the Quickcam, convert them to the JPEG format and send them to our web server at http://MilleMiglia.ce.unipr.it/ (a third Linux machine) through the two GSM modems exploiting EQL. As soon as these images are received, a graphical timestamp is superimposed onto their lower portion and they are made available on the Internet. A simple script running continuously in the background is used to restore the two connections if, for any reason (tunnels, GSM uncovered areas, etc.), they are lost.
In order to test the vehicle extensively under different traffic conditions, road environments and weather, a 2000km journey was undertaken June 1 through June 6, 1998. During this test, ARGO drove autonomously along the Italian highway network, passing through flat areas and hilly regions including viaducts and tunnels. The Italian road network is particularly suited for such an extensive test since it is characterized by quickly varying road scenarios, changing weather conditions and generally a fair amount of traffic. The tour took place on highways and freeways, but the system also proved to work on sufficiently structured rural roads with no intersections.
During the journey, in addition to the normal tasks of data acquisition and processing for automatic driving, the system logged the most significant data, such as speed, position of the steering wheel, lane changes, user interventions and commands, dumping the whole status of the system (images included) whenever the system had difficulties in detecting the road lane reliably.
This data was processed off-line after the end of the tour in order to compute the overall system performance, such as the percentage of automatic driving, and to analyze unexpected situations. The possibility of reprocessing the same images, starting from a given system status, allows the reproduction of conditions in which a fault was detected and a way of solving it found. At the end of the tour, the system log contained more than 1200MB of raw data, while during the whole tour the system processed about 1,500,000 images (each 768 x 288 pixels) totalling about 330GB of input data.
During the tour, the ARGO vehicle broadcast a live video stream to the Internet: two GSM cellular modems were connected to the Vision Laboratory of the Dipartimento di Ingegneria dell'Informazione in Parma, and were used to transfer both up-to-date news on the test's progression and images acquired by a camera installed in the driving cabin to demonstrate automatic driving. Proving there was high interest in the scientific community, mass media and general public, the web site http://MilleMiglia.CE.UniPR.IT/ was visited more than 350,000 times during the tour and more than 3000MB of information was transferred, with a peak of 16,000 visits per hour during the first day of the tour.
The main problem experienced during the tour was due to image acquisition. One aim of the project is the development of a sufficiently low-cost system that will ease its integration into a large number of vehicles, so the use of low-cost acquisition devices was a clear starting point. In particular, videophone cameras (small-sized sensors at an average cost of $100 US each) were installed. Although these cameras have a high sensitivity even in low-light conditions (at night), a sudden change in illumination of the scene (for example, at the entrance or exit from a tunnel) causes a degradation in the image quality. (They were designed for applications characterized by constant illumination, such as video-telephony.) The cameras have a slow automatic gain control to the exit from a tunnel for a period of about 100 to 200ms; thus, the acquired images are completely saturated and their analysis becomes impossible.
Figure 6. Automatic Driving During the MilleMiglia in Automatico Tour
On the other hand, the design of the processing system proved to be appropriate for automatic driving of the vehicle. Moreover, current technology provides processing systems with characteristics that are definitely more powerful than the one installed on ARGO: a commercial PC with a 200MHz Pentium processor and 32MB of memory. On such a system, enhanced by a video frame grabber able to acquire two stereo images simultaneously (with 768x576 pixel resolution), the GOLD system processes up to 25 pairs of stereo frames per second and provides the control signals for autonomous steering every 40ms. (This is equivalent to one refinement on the steering wheel position for every meter when the vehicle is driving at 100kmph.) Obviously, the processing speed influences the maximum safe vehicle speed: the higher the processing speed, the higher the maximum vehicle speed.
Differing weather conditions in particular light conditions demonstrated the robustness of both the approach and the image processing algorithms. In fact, the system was always able to extract the information for the navigation task even in critical light conditions, with the sun in front of the cameras, high or low on the horizon, during the evening as well as during the day, with high or low contrast. At night, the system's behavior is improved by the absence of sunlight reflections and shadows, while the area of interest is constantly illuminated by the vehicle headlights.
Figure 7. The MilleMiglia in Automatico Tour Route
Finally, the system proved to be surprisingly robust despite high temperatures measured during the tour. In some cases, the external temperature reached 35 degrees Celsius and the system continued to work reliably even with no air conditioning.
Table 1. System Performance Statistics
The analysis of the data collected during the tour allowed the computation of a number of statistics regarding system performance (see Table 1). In particular, for each stage of the tour, the average and maximum speeds of the vehicle during automatic driving were computed. Average speed was strongly influenced by the heavy traffic conditions (especially on Torino, Milano and Roma's bypasses) and by the presence of toll stations, junctions and road works.
The automatic driving percentage and maximum distance show high values, despite the presence of many tunnels (particularly during the Appennines routes Ancona to Roma and Firenze to Bologna) and of several stretches of road with absent or worn lane markings (Ferrara to Ancona and Ancona to Roma) or even no lane markings at all (Firenze to Bologna). It is fundamentally important to note that some stages included passing through toll stations and transiting bypasses with heavy traffic and frequent queues, during which the system had to be switched off.
Throughout the entire project, the choice of an Intel-based platform, coupled with the Linux operating system, was shown to be extremely reliable; the number of faults due to these system components was zero over the last two years.
Originally, the main reason behind the choice of the Linux operating system (instead of real-time OS or operating systems for industrial PCs) was the availability of up-to-date developing and debugging tools, drivers and FAQs about specific hardware devices, and the possibility of interacting with a large number of researchers worldwide on the Internet in order to solve problems.
The main topics for future research (ARGO Project, phase II) are related to the development of a new vehicle integrating both road following and platooning (the automatic following of a manually driven vehicle) functionalities. In this next phase, the processing engine will be a higher performance Intel-based architecture, again driven by the Linux operating system.
Comments From Others about ARGO
Massimo Bertozzi holds a Ph.D. in Information Technology and a Master's degree in Electronic Engineering. He has been a Linux enthusiast since kernel 1.0.9. Since 1994, he has been working at the Università di Parma on research topics regarding machine vision and clusters of Linux boxes. When not recompiling strange programs, reading news, surfing the net, playing Quake or some other adventure game, he can be reached via e-mail at bertozzi@ce.unipr.it.
:
:
Alberto Broggi is married to Simona, who gently allows him to work at home and even during the night on topics related to automatic vehicle guidance using artificial vision. In his spare time, he teaches a number of courses at both Università di Pavia and Università di Parma and has still not learned to turn when someone calls him Professor. He is involved in the organization of many international scientific events, but had a bad time when looking for a photo with a tie to include in this biography. Alberto will certainly answer if you contact him at broggi@dis.unipv.it.
Alessandra Fascioli received the Dr.Eng. (Master) degree in Electronic Engineering from the Università di Parma, Italy. Currently, she is a Ph.D. candidate in Information Technology at the Dipartimento di Ingegneria dell'Informazione of the Università di Parma. Her research activities on real-time computer vision for automatic vehicle guidance and on image processing techniques based on the Mathematical Morphology computational model generally produce better results after a good swim or sunbath at the university campus swimming pool. You can contact her at fascal@ce.unipr.it.
