|
|
|
|
|
|
|
|
The story of how Progressive Computer Concepts has turned United Railway into a Linux shop.
by Lester Hightower and Hank Leininger
I remember being amazed and somewhat impressed in September of 1995 when I made my first trip to the main offices of United Railway Signal Group, Inc. in Jacksonville, Florida. I was impressed that their computer network worked at all, much less that they actually got quite a bit of work done using it. United had the longest single run of 10Base-2 I had ever seen, connecting approximately 30 computers to two Novell Netware servers that lived in a halon fire-protected room at one end of the offices.
The primary server was Netware 3.12 on an Intel P90 with 64MB of RAM, three Adaptec 2940 SCSI controllers, two one gigabyte SCSI hard disks and a 20 cartridge, 26GB Maxoptix magneto-optical jukebox. One of the 1GB disks was exported from both IPX/SPX and Netware NFS as ``ursgpub'', a shared network file system containing shared data and a collection of custom FoxPro applications to track such information as project flow and time sheets.
The magneto-optical jukebox was being used at about 80% capacity. Corel SCSI for Netware controlled the jukebox and used one of the 1GB disks as its ``cache volume''. A DOS TSR was needed on the client side for the PCs to see the jukebox as a contiguous 26GB file system. United's Unix CAD stations needed access to data on the jukebox, as did the PC CAD stations and office PCs. Netware NFS had no knowledge of the jukebox, as the jukebox was handled entirely by the Corel SCSI NLMs. In order to allow the Unix CAD stations some limited access to that data, the jukebox's ``cache volume'' had to be NFS exported. NFS exporting of the cache volume from underneath Corel SCSI proved to be the source of many problems and a constant headache.
The other server, an Intel P75 with 32MB of RAM, ran Netware 4.1. Its sole responsibility was to do backups. ArcServe, the package United used for backups, could never be made to run on the Netware 3.12 server. Thus, United's only option was to purchase a 60-user license Netware 4.1 server for ArcServe and to install the WinAgent client software on each PC. ArcServe never seemed to run properly on this backup server either, crashing almost nightly with out of memory errors or hanging when a client PC's WinAgent software hung.
In mid-October of 1995 we, Progressive Computer Concepts, connected United's main office to the Internet via a dedicated ISDN line using an Ascend Pipeline 50. We installed Linux 1.2.13 on an Intel P90 with 32MB of RAM, a BusLogic 956C and a 1GB SCSI disk to handle DNS, e-mail, WWW and FTP service.
From my time spent at United performing this work, it became obvious to me that the Netware solution was falling apart. The Netware server aborted about once per week (and checking the jukebox's file systems on reboot took hours). Nightly backups via ArcServe failed in some manner almost daily, and hours were wasted each day by United's Netware administrator manually reviewing logs and checking the contents of tapes. I began expressing my opinion that Linux would be a better solution to United's CEO, Mike Wilson.
Over time, the reliability of the one Linux server that handled the Internet operations became more and more clear. I would e-mail Mike an uptime report every 30 days or so. The pivotal point in moving United away from Netware to Linux came when United's Netware system administrator resigned. The door was now opened for us to step in.
In June 1996 we began the formidable task of moving United's entire operation from Netware to Linux. The first to go was the backup server--a Linux boot disk and about an hour turned that little P75 into United's first Linux file server named ursgfs2. We immediately installed SAMBA and smbfs and began writing backup scripts. After removing WinAgent and setting up an administrative share on each PC, we did a full floor backup to the 4mm tape drives in the new Linux server that very night; the backups completed through every machine without error.
We purchased three BusLogic 958 SCSI controllers and four 4GB fast wide SCSI II Quantum Atlas drives in external cases and attached them to the P75 Linux server, ursgfs2. Now, we needed a way to move many gigabytes of CAD files and corporate data off the 26GB magneto-optical jukebox connected to the Netware server, at that time accessible only through a DOS machine using Netware drivers plus a TSR program. We tried various unsuccessful methods.
All of our attempts on DOS clients failed with out of memory errors while trying to pkzip or xcopy files from the jukebox cartridges. The Corel TSR would load under NT but would crash and die at random points during the copy process. We never got NT to successfully copy a single cartridge. Using ncpmount we could mount the jukebox from the Netware server under Linux, but without the TSR the Netware server would kick us off within 60 seconds. The solution was DOSEMU. DOSEMU, when installed on ursgfs2 allowed us to run the Corel TSR, attach to the jukebox on the Netware server and then copy directly to the attached 16GB of new disk space using the xcopy command.
Due to an inefficiency in the FAT (finessed automatic transfer) file system, the FAT tables on the jukebox cartridges were filling long before the cartridges were actually full of data. We were able to store all of the jukebox data onto 16GB of disk space. After the transfer of all of the jukebox data was complete, we blew Netware off the larger server, moved ursgfs2's SCSI cards and disks to the new hardware, and renamed the server ursgfs1. A fifth disk was added a short time later.
The new Linux file server used SAMBA for exporting to PCs and NFS for Unix workstations. The server had three 4mm tape drives. Our backup scripts used smbmount to mount each PC in the building and archive it on tape using tar. Soon, an eight-port Lantronix 10/100 switch was installed, and ursgfs1 was moved to a dedicated 100MB port.
United has a number of CAD stations in each of its offices, where CAD operators work each day on various engineering projects. While using the Netware/ArcServe system, each night ArcServe would copy CAD files from a specific directory hierarchy on each CAD station, in-turn, to the read-only CAD file hierarchy on the Netware server. The potential existed for two different projects inside United to involve the same CAD files. This situation is particularly dangerous when the two CAD operators involved are unaware that they are both working on the same set of files. Under the old Netware/ArcServe system the fact that two CAD operators had been working on the same CAD files, simultaneously, could only be detected by a human and was often not discovered for days or weeks. Much redo CAD work (recadding) would have to be done when those situations were discovered.
To solve this problem, our next software project for United was to write a custom file retrieval and archive commit program that would be, in effect, a revision control system. Every night, the working directories of each CAD station are copied by ursgfs1 to scratch space. Files are then put through a number of sanity checks to detect duplicate works-in-progress, verify file revision and time stamps, file sizes, etc. Files passing all criteria are copied into the read-only CAD file hierarchy; currently, existing files replaced by this process are put into a daily incremental backup. Sixty days' worth of these incrementals are kept in mid-line storage, and any version of any file can be rolled back if needed. Summary reports of committed files and rejected files (if any) are e-mailed to the administrators and to a hypermail archive each night. This system is written entirely in Perl 5 and has been in place, working successfully, since October 25, 1996. The system has recently been expanded to include United's Omaha office.
After the back-up work was completed, we began developing custom Intranet applications for United. We replaced most legacy FoxPro LAN applications with more fully featured and more tightly integrated Intranet programs. The Intranet system, the URSG Daily Operations Control System as it is called, is written entirely in Perl 5. URSGDOCS originally used MiniSQL as its back-end database, but has been ported to and using MySQL for many months now. Once all of the legacy applications like time-sheet entry and project management had been replaced by the Intranet system, we upgraded United's main office Internet connection to 1.536MB T1. United's remote offices in San Francisco, Omaha and Jacksonville (the manufacturing facility) immediately began using URSGDOCS via dial-up Internet connections.
Figure 1. Internet Screen for URSGDOCS
Later, we added a ``Fax This Page'' button to the bottom of all the reports that a user might wish to retrieve from United's Daily Operations Control System. A retired 386DX40 was given 32MB of RAM and an eight-port Comtrol Rocketport board. It now runs multiple PPP dial-in sessions, various network sniffers and all of the URSGDOCS faxing subsystem. The URSGDOCS faxing subsystem is a custom Perl script wrapped around the efax07a package, a virtual X server, a few Netscape ``-remote'' commands and Ghostscript. The result is the ability to fax any URSG Daily Operations Control System report directly to any fax machine in the world, just by clicking on that button.
United's Omaha office grew to the point that dedicated T1 connectivity was deemed necessary. A Linux server was installed in that office in December 1996. The custom file retrieval and archive commit program now runs in Omaha as well. Furthermore, ssh (Secure Shell) is used to move those files automatically to the main server in Jacksonville.
We ran across a great deal on some DEC Alpha UDBs (universal desktop box) and initially picked up four of them for United. Red Hat Alpha Linux allowed us to spread some of the server tasks across those boxes. The URSG Daily Operations Control System was moved to alpha2, for example. Alpha4 was assigned the role of running the old 26GB magneto-optical jukebox which had been collecting dust for a few months. By wrapping our own custom backup scripts around Gerd Knorr's jukebox disk-changer package, we have almost eliminated the need for United to perform tape-based backups. Better yet, nightly incrementals of everything imaginable happen automatically to mid-line storage. Detailed reports of what was backed up, what incrementals were pruned due to age, and the disk-usage status of everything are waiting for the system administrators each morning.
As United continues to grow and the employee count rises, additional desktop computers are continuously needed. Having worked at places like the Supercomputer Computations Research Institute on the campus of Florida State University, where four system administrators support hundreds of Unix users (a large percentage through X Terminals), we knew that an X Terminal model would work for United. We began installing Linux-based X Terminals everywhere new desktop computers were needed. We are able to sell United new Alphas with 32MB of RAM, 15 inch monitors and no hard disks for $1000US each. This is a much cheaper alternative to deploying new Windows 95 boxes considering the cost of hardware, software, setup time and recurring maintenance. The hardware was also cheaper, and in our opinion better than even the dedicated X Terminal/Network Computer equipment that we originally investigated for this task.
The Linux/Alpha X Terminals boot over the network, NFS mount their root and usr file systems, and then open an xdm (X window manager) session on an XDMCP server. The window manager chosen for United's X Terminals was FVWM95. The engineers who have received these X Terminals rarely use MS Office type applications--Applixware is used to fill that occasional need. Netscape Navigator is used to access the URSGDOCS and Internet e-mail.
Figure 2. ApplixWord on X Terminal
The X Terminal model is working well. All of URSG's engineers have migrated to X Terminals. This has allowed United to continue to extend the useful lifespan of aging 386/486 class equipment by redeploying it in both X terminal and fat-client capacities in the wiring facilites and branch offices.
Our calculations show significant cost savings through utilizing slightly higher-end Linux-based X Terminals with fast Ethernet in place of the FAT- client CAD stations that United currently uses and deploys. PCC was a beta test site for Bentley's port of Microstation 95 to Linux. The port seemed flawless and was very promising; an academic release version has been shipping for several months. However, Bentley has not yet received the level of demand from Linux users that it deems necessary to support Microstation commercially on the Linux platform. We plan to continue to work with Bentley and to encourage the commercial release and support of Microstation for Linux.
An NCD Wincenter Multiuser NT server has been installed at URSG and more Alpha X Terminals deployed. Even administrative staff now have Linux-based X terminals on their desk. These staffers use Linux Netscape for Inter/Intranet access and e-mail, and they run office productivity applications (accounting, MS Word, MS Excel, etc.) on the Multiuser NT server.
URSG's Jacksonville wiring facility has moved to a new location, tripled in size, and had its connectivity to the main-office LAN upgraded from the 128K ISDN mentioned earlier to a 1.536MB T1.
PCC has added enhanced extranet functionality to URSGDOCS including a system designed specifically for use by CSX Transportation and all of its railway signal design contractors. This particular extranet section of URSGDOCS allows fast and well documented business transactions between CSX Transportation and its signal design contractors. The system allows contractors to make requests for circuit plans and CAD files, CSXT staffers fulfill those requests by uploading CAD files into URSGDOCS, then contractors can download those files. All transactions inside the system generate e-mail notifications to appropriate persons and to an e-mail alias that is used to archive all transactions. The transactions archive is full-text searchable and browseable. The system has been in operation since July 10, 1997 and averages approximately 2000 transactions and 500MB of compressed CAD file transfers per month.
As new systems and functionality have been added to URSGDOCS, it has been migrated to a Dual 200MHz Pentium Pro Linux server with 128MB RAM. The ursgfS1 machine has been upgraded to a uni-processor 200MHz Pentium Pro Linux server with 64MB RAM. The Wincenter server resides on Dual 150MHz Pentium Pro hardware with 128MB RAM.
Unix, in particular Linux, combined with the GNU tools, can change a company's technology spending focus allowing it to move away from a larger, under-skilled IS staff to a smaller, higher-skilled staff or to outsourcing. A move to either results in a company receiving more reliable and more customized solutions that can easily evolve to the company's changing needs over time.
Yes, Linux means business. The work that we at Progressive Computer Concepts have done with United Railway Signal Group is a wonderful example.
Lester Hightower (hightowe@progressive-comp.com) is Vice President of Development and Hank Leininger (hlein@progressive-comp.com) is Senior Systems Integrator at Progressive Computer Concepts, Inc. in Jacksonville, Florida. Both have extensive backgrounds with a wide variety of Unices. They are involved in all aspects of Unix and Internet consulting including installation, administration and systems programming.
