Writing Man Pages in HTML

HTML is a cool way to look at the Linux man pages. Here's how to do it.

by Michael Hamilton

The Web invasion of the Internet continues at a breakneck pace. In the space of just a few years HTML has become one of the most widely supported document formats. Currently, there's gold-fever driven effort by a cast of thousands to reformat older sources of information for presentation on the Web. The required technology is relatively simple to understand, and is inexpensive to assemble. Linux is an ideal development and delivery environment: low cost, reliable, and necessary software is included in several competing Linux distributions.

In this article I'll discuss a solution I've written for serving up old documents in the new medium--automated translation. The documents to be converted are the Unix man pages. Man pages are highly structured documents in which major headings and references to other documents, are easily recognized. The files making up the man page system are already organized into a rigid set of hierarchies using a formalized naming system; thus, it is easy to deliver the entire existing man system and document format via the Web without reorganization of content or overall structure. I've assembled some pieces of software that translate the old Unix manual format into HTML while preserving the old style and organization. The technologies present in the Web allowed further enhancements: documents can be cross-linked; alternate forms of indices can be automatically generated; and full text searches are possible.

I've called the package I've assembled vh-man2html. It is designed to be activated as a set of CGI (Common Gateway Interface) scripts from a web server--the same technology that drives HTML forms. This means that vh-man2html can be used to serve man pages to other hosts on your LAN or on the Internet. The principle component of vh-man2html is Richard Verhoeven's man2html translator, augmented by several scripts to generate indexes and facilitate searches. vh-man2html also has some supporting scripts that allow you to drive Netscape from the Unix command line, so that vh-man2html can actually replace ``man'' on systems that include Netscape.

What Does It Look Like?

If you have access to the Web, you can see vh-man2html in action at:

http://www.caldera.com/cgi-bin/man2html

Figure 1 shows the main vh-man2html web page which provides direct access to individual man pages or to three different kinds of indices: name-description, name-only and full text search. In the main window you can enter the man page name, man page name and section number, or narrow things down even further by specifying the hierarchy or full path name.

Figure 2 shows a man page generated by the man2html converter. The converter has translated the man formatting tags to approximately equivalent HTML tags. It has also created an HTTP reference and links for any references to other man pages. The converter also generates a subject heading index, which is useful when reading larger man pages. Text highlighting and font changes are correctly translated, and if tables are present, they will be translated into HTML tables. At this time, man2html doesn't translate eqn described equations, but since very few man pages use eqn, this is not a major drawback.

Figure 3 shows part of a name-description index for section 1 of the man pages. The index includes an alphabetic sub-index and links into the other sections of the manual. The name-only index in Figure 4 is similar to the name-description index, except that it is more compact.

Figure 5 shows the result of a full text search for pages containing a reference to ``cdrom''. Links are generated to each page matched.

These figures illustrate one of the key advantages of serving up the man pages via the Web: a variety of access paths can be presented in an integrated form--a one-stop-shop interface.

Obtaining and Installing vh-man2html

My development platform is Caldera 1.0, with a Red Hat 3.0.3 upgrade. If you don't have a Red Hat-based system, you can still successfully employ vh-man2html. If your system uses an unformatted man page source and you run an HTTP daemon, vh-man2html should still work; however, you may need to reconfigure it and rebuild the binaries to match your own setup.

Don't let having to run the HTTP daemon dissuade you. You can handle the security aspects of this process by restricting the HTTP daemon to serving your own host. I use the Apache HTTP daemon shipped with Caldera and Red Hat, so I just adjust the appropriate lines of my system's /etc/httpd/conf/access.conf file to prevent access from outside of my home network:

<Limit GET>
order allow,deny
allow from .pac.gen.nz
deny from all
</Limit>

The performance aspects of running an HTTP daemon are minimal. Most of the time it is idle--if other jobs need the memory the daemon is occupying, the kernel just migrates it to swap. To minimize the amount of startup activity and the total memory consumed, I reduced the number of spare daemons to one by editing /etc/httpd/conf/httpd.conf and changing the following:

MinSpareServers 1
MaxSpareServers 1
StartServers 1

vh-man2html is available in Red Hat package format in both source and i386 ELF-binary form from the following locations:

ftp://ftp.caldera.com/pub/contrib/RPMS\
	/vh-man2html-1.5-1.src.rpm
ftp://ftp.redhat.com/pub/contrib/SRPMS\
	/vh-man2html-1.5-1.src.rpm

ftp://ftp.caldera.com/pub/contrib/RPMS\
	/vh-man2html-1.5-2.i386.rpm
ftp://ftp.redhat.com/pub/contrib/RPMS\
	/vh-man2html-1.5-2.i386.rpm

Also, a source tar file with ELF binaries is available from:

ftp://sunsite.unc.edu/pub/Linux/system\
	/Manual-pagers/vh-man2html-1.5.tar.gz

The rpm version will install correctly in any post-2.0.1 Red Hat-based system (including Caldera). Running the following command when logged in as root will install the binary rpm:

rpm -i vh-man2html-1.5-2.i386.rpm

http://localhost/cgi-bin/man2html

You can use you browser to save this page as a bookmark. If you feel comfortable editing HTML files, you can insert it in the master document for your own system. In my case I edited my system's top-level document:

/usr/doc/HTML/calderadoc/caldera.html

<H3>
<A HREF="http://localhost/cgi-bin/man2html">
<IMG SRC="book2.gif">
Linux Manual Pages
</A>
</H3>

/usr/doc/HTML/index.html

<LI><A HREF="http://localhost/cgi-bin/man2html">
Linux Manual Pages</A>
<P>

/usr/sbin/makewhatis -w

ftp://ftp.redhat.com/redhat-3.0.3/i386\
	/updates/RPMS/man-1.4g-1.i386.rpm
ftp://sunsite.unc.edu/pub/Linux/system/\
	Manual-pagers/man-1.4g.tar.gz

To use the Glimpse full text searching, you must install the program Glimpse in /usr/bin. Red Hat rpm users can get Glimpse from:

ftp://ftp.redhat.com/pub/non-free\
	/Glimpse-3.0-1.i386.rpm

ftp://ftp.cs.arizona.edu/Glimpse/

/usr/bin/Glimpseindex -H /var/man2html \
	/usr/man/man* /usr/X11R6/man/man*\
	/usr/local/man/man* /opt/man/man*
chmod +r /var/man2html/.Glimpse*

21 04 * * 1 root /usr/bin/Glimpseindex -H
   /var/man2html /usr/man/man* /usr/X11R6/man/man*
   /usr/local/man/man* /opt/man/man*i;
   chmod +r /var/man2html/.Glimpse*

If you're building vh-man2html from source, you will have to manually un-tar it and change the Makefile to point to your desired installation directories before issuing a make install. You can also use the rpm2cpio utility to extract a CPIO archive from the rpm, in which case you could read the package spec file to figure out where to put things.

If you don't want to use an HTTP daemon and you know a little C, you might consider using the scripts and C program to pre-translate and pre-index all your man pages. Then they can be referred to directly without an HTTP daemon to invoke conversions on demand.

Technical Overview

This section will cover a few of the implementation details of vh-man2html. It's very brief and is really intended to point out that CGI scripting is something that anyone with a little programming knowledge can do with success.

Without getting into a tutorial on CGI scripting, a CGI script is a program executed by the remote HTTP daemon (i.e., web server). A web browser can cause a remote web server to run a CGI program when you follow an HTTP link that matches its name. For example, pointing a web browser at:

http://www.caldera.com/cgi-bin/man2html

The CGI script can return output to the remote caller by writing a document to its standard output. The start of the output document must contain a small text header describing its contents. In the case of man2html the content returned is an HTML page. Listing 1 shows the HTML output from the man page to HTML converter; the header line is:

content-type: text/html

A script may create an HTML page that contains further references to other CGI scripts. In Listing 1 the following reference returns the reader to the main vh-man2html contents page:

<A HREF="http:/cgi-bin/man2html">Return to Main
Contents</A>

<A HREF="http:/cgi-bin/man2html?man1/from.1l">
from</A>

In addition to clicking on references, the user can also enter data into input fields. The simplest way for a CGI program to introduce an input field onto a form is to include the tag <ISINDEX> in the HTML it generates. This results in a single input field, such as in Figure 1. If the user enters anything in the input field and presses return, the server will re-run the CGI program, passing it the input via the parameter passing conventions we've just discussed. You can also create HTML forms, but I'm not going to discuss them here.

By generating the kinds of HTML references presented above, CGI programs can perform complex interactions with the remote user. The beauty of all this is that, to get started, the only skill you need is the ability to write fairly simple code in a language of your choosing. You need to know how to process command-line arguments and write to standard output. The rest of the knowledge you need can be gotten for free from Web documents or from any one of a number of books on HTML and CGI. CGI is a client-server that actually works. Heavy duty CGI programming languages such as Python and Perl have tools and libraries to assist you with the task.

I should also mention the issue of security. If your HTTP daemon is accessible by potentially hostile users, your CGI scripts could provide an avenue for them to attack you. Hostile users might try to supply malicious parameters to your CGI scripts. For example, by using special shell characters such as back quotes and semicolons, they might be able to get the script to execute arbitrary commands. The only way to prevent this is to carefully examine all input parameters for anything suspicious. For example, vh-man2html can be passed the full file name of a man page; however, it doesn't just accept and return any file name it is passed--it accepts only those filenames present within the man hierarchy. The program also makes sure the file name does not contain relative references such as ``..'' (the parent directory), and removes any suspect characters such as back quotes that might be used to embed commands in the parameter list. In languages like C, where memory bounds checking is lacking, the length of the input arguments should be constrained to fit within the space allocated for them. Otherwise the caller may be able to write beyond the allocated space into other data and change the behavior of the program to his/her advantage (e.g., change a command the program executes from gzip to rm). To help check that long input parameters wouldn't threaten vh-man2html's integrity, I borrowed some time on an SGI box and built vh-man2html with Parasoft's Insight bounds checker. Insight pre-processes a C or C++ program adding array bounds checking, memory leak detection and many other checks. One of the reasons I'm mentioning Insight is that Parasoft's Web site, http://www.parasoft.com/, lists Linux as a supported platform.

vh-man2html includes four CGI programs. They all generate interdependent HTTP references to each other.

Man page to HTML translation is handled by the man2html C program. The Unix man pages are marked with man or BSD mandoc tags which are nroff/troff macros. The bulk of the program is a series of large case statements and table lookups that attempt to cope with all the possible macros.

Listing 2 shows a typical nroff/troff marked up manual page that is using the man macro package. The macros use a full-stop, i.e. a period, as a lead-in to a one or two character macro name. troff/nroff uses two character macro names--apparently they fit nicely into the 16-bit word size of the old Unix platforms such as the PDP11 (at least that's what I was told)--a trick which man2html.c still utilizes. Some of the macros can be directly translated to appropriated HTML tags; for example, lines beginning with ``.SH'' Section Headings are directly translated to HTML <H1> headings.

Many troff tags limit their arguments and effects to just one line and have no corresponding end tag--where as many of the equivalent HTML constructs also require an end tag. For example, the text following a troff ``.SH'' section heading tag needs to be enclosed in a pair of HTML heading level 1 tags, e.g., ``<H1>text</H1>''. Other troff tags with a larger scope, such as many kinds of lists, have both begin and end tags, which makes translation to HTML very easy.

One tricky issue is dealing with multiple troff tags on one line; for example, tags that imply bracketing of following text or font changes. In order to correctly place bracketing, the translator can work recursively within a line. For example, the BSD mandoc sequence for an command option called -b with an argument called bcc-addr is expressed in troff as:

 .Op Fl b Ar bcc-addr

[ -

[ -<B>b</B> <I>bcc-addr</I> ]

There are some troff tags whose effect is terminated by tags of equal and higher rank; in these cases, the translator must remember its context and generate any necessary terminating HTML. Nested lists are also possible. In these situations man2html has to maintain a stack of outstanding nestings that have to be completed when a new equal or higher element is encountered.

I admire Richard's dedication in methodically building up translations of all of the tags. Adding in the BSD mandoc tags proved to be a painful experience, and in the end, the only way to get it right was to convert every BSD mandoc page I could find and pipe the output to weblint (an excellent HTML checker). For example, in tcsh/csh:

foreach i ( `egrep -l '^\.Bl' /usr/man/man1/* \
	/usr/man/man8/*` )
/home/httpd/cgi-bin/man2html $i > tmp/`basename $i`
end
weblint tmp/*

man2html.c also has to do minor translation fix ups, such as translating quotation marks and other special punctuation into HTML special characters.

In the end, to test the sturdiness of the translator, I converted every man page I have:

find /usr/man/man* -name '*.[0-9]' \ 
  -printf "echo %p; /home/httpd/cgi-bin/man2html\
	%p | weblint -x netscape -\n" | sh \
	|& tee weblint.log

The program also has to navigate the man directory hierarchies and generate lists of references to pages that might be relevant (e.g. a page with the same name might be present in multiple man hierarchies). The list of man hierarchies to be consulted is read from /etc/man.config, which is the standard configuration file for the man-1.4 package that ships with Redhat and Caldera. This configuration file is also consulted for details on how to process man pages that have been compressed with gzip or other compression programs.

man2html could have easily been written in Python or Perl, but you can't beat C for speed. man2html is fast enough on my 486 that I didn't think caching its output was worthwhile--each page is just regenerated on demand. However, if I was going to provide man pages from a server for a large number of high frequency users, I would probably pre-generate all the man pages as a static document set.

Two awk scripts, manwhatis and mansec, generate name-title and name only indexes for man sections and cache them in /var/man2html. manwhatis locates and translates whatis files into the desired section index, which it caches in /var/man2html. It rebuilds the cache if any whatis file has been updated since the cached version was generated. The script divides the whatis file alphabetically and constructs an alphabetic index to the HTML document, so that the the user can quickly jump to the section of the alphabet they're interested in.

mansec traverses the man hierarchy to build up a list of names; it rebuilds its cache if any of the directories in the hierarchies have been updated. mansec has to use the sort command to get the names it finds into alphabetical order. It also builds an alphabetic quick index just like manwhatis.

Both manwhatis and mansec accept an argument that indicates which section to index. They have to check the argument for anything potentially malicious and return a document containing an error message if they find anything they weren't expecting:

section = ARGV[1];           # must be 0-9 or all.
if (section !~ /^[0-9]$/ && section != "all") {
 print "Content-type: text/html\n\n";
 print "<head>";
 print "<title>Manual - Illegal section</title>";
 print "<body>";
 print "Illegal section number '" section "'." ;
 print "Must be 0..9 or all";
 print "</body>";
 exit;
}

# Substitute "" for any char not in A-Za-z0-9
# space.
string = gsub(/[^A-Za-z0-9 ]/, "", string);
I chose awk over Python and Perl mainly because it is small,
widely available and adequate for the task. Note that I'm using the post
1985 ``new awk''. For larger, more complex CGI scripts I'd probably use
Python (if I had to start again without Richard's work, I think
man2html would be a Python script).
In order to make vh-man2html usable remotely, I changed man2html and my
scripts to generate HTTP references that were relative to the current server. 
For example, I used:

<A HREF="http:/cgi-bin/man2html">Return to Main
Contents</A>

<A HREF="http://localhost/cgi-bin/man2html">Return to Main
Contents</A>

Location: http://sputnik3/cgi-bin\
	/man2html/usr/man/man1/message.1

Interfacing with Netscape

Recently I wanted to be able to issue man page requests from the command line to an already running Netscape browser--I wanted to replace the command-line man with Netscape. As you're probably aware, launching a large application such as Netscape (or emacs) is prohibitively expensive, so my preference is to communicate with an already running Netscape rather than start a new one for each request. Instructions for driving an already running Netscape from the command line can be found at:

http://www.mcom.com/newsref/std/x-remote.html

function nsfunc () {
 # If this user is running netscape - talk to it
 if ( /bin/ps -xc | grep -q 'netscape$' ) ; then
  if [ -x netscape-remote ] ; then
   # Use utility to talk to netscape
   exec netscape-remote -remote "openURL($1,new_window)"
  else
   # Use netscape to talk to netscape
   exec netscape -remote "openURL($1,new_window)"
  fi
 else
  # Start a new netscape.
  netscape $1 &
 fi
}
 

nsfunc "http:/cgi-bin/man2html?who+1"

Summary

Creating vh-man2html was a rewarding spare time effort. The Internet community provided the components and feedback necessary to get it built. It demonstrates one way in which old information can be repackaged for easier access. Linux provided an excellent development platform. All of the necessary components are available in Linux distributions and via ftp.

I'd like to thank Richard Verhoeven for creating his man2html translator and making it available to the Internet community. I'd also like to thank those people who took the time to send me the feedback that assisted with the development of vh-man2html.

Why Would You Want to Do This?

How Did it All Come Together?

References

Michael Hamilton has been working as a freelance Unix C/C++ developer since 1989. He tripped over one of Linus' postings back at the beginning of 1992, and has been hooked ever since. He can be reached at michael@actrix.gen.nz.