"The Linux Gazette...making Linux just a little more fun!"

The Answer Guy

By James T. Dennis, answerguy@ssc.com
LinuxCare, http://www.linuxcare.com/

Console Logins Fail; KDM Okay?

Or: System Integrity Checking

From Ken Keith on Mon, 06 Mar 2000

Hello,

I had a problem recently that I think was caused when I a shutdown from a terminal in KDE windows. I did a shutdown -h now . The system shut down all right but the next time I booted I had some errors that stopped apmd from loading and bringing up NFS failed. Upgrading some packages I was able to fix all that. The system now boots properly. I now find that when I go to a virtual console and try to log in it fails with a 'login incorrect'. I could sure use some help here.

Thanks
Ken Keith

A shutdown -h from within the X Windows System is likely to leave a few stray lock and PID files laying around. However, it should have any effect on your APM (advanced power management daemon) or your NFS (network file system) subsystem.

I suspect that something else was going on here.

You say that you "upgraded some packages" --- though you don't say which ones, nor which package management system you're using (RPM, Debian, Slackware, Stampede?) * (http://www.stampede.org).

Blindly upgrading packages may occasionally solve some problems. It's as likely to exacerbate them or cause others.

One nice feature of the RPM system is the -V (verify) option which allows you to get a handy list of which files differ between the package database records and their actual nodes in the filesystem. Thus you can see size, MD5 checksum, and especially ownership and permission differences which might be causing problems. Naturally it will also warning of any missing files.

Slackware simply uses compressed binary tarballs. So creative use of the 'tar dzf' command can also give you pretty good hints as to what's wrong, corrupted or missing. (I wonder if Patrick every created/added a script to automate that process. I suggested it to him last year at Linuxworld, but he might have been busy. I'll copy him on this note as a reminder).

From what the Stampede folks told me in New York (at the more recent LinuxWorld) the .slp format is a bzip'd tar file with an appended trailer that contains all their package management meta-data. (I just tested that in another 'screen' session --- it works). So a command like:

bunzip2 -c /.../foo-0.1.2.slp | tar df -

... should report differences between this SLP file and the corresponding files on your system. (This would have to be done from the root directory, of course).

We can ignore the error:

bunzip2: foo-0.1.2.slp: trailing garbage after EOF ignored

To make this into a reasonable system integrity system all we have to do is have some way of getting a list of all the packages that are intalled unto the system --- and some way of translating those package names into corresponding package filenames.

A Debian file (.deb) is generally an 'ar' file with two or three members. The 'ar' command is a predecessor to the 'tar' command -- it was primarily used to "archive" object (.o files) into libraries (.a files). These members are usually a control.tar.gz and a data.tar.gz --- and (usually?) a small debian-binary file text file listing some sort of debian dpkg version specification.

So, all of the popular alternatives to the RPM system are built around some sort of tar file core. Thus we can use 'tar --diff' features to verify the basic integrity of individual packages.

Of course this only tells us about differences among regular files (and device nodes). There is a bug in GNU tar which will report errors on UNIX domain sockets (filesystem nodes that are marked in an ls -l with a leading "s" character). We could ignore those.

To make this into a reasonable system integrity system all we have to do is have some way of getting a list of all the packages that are intalled unto the system --- and some way of translating those package names into corresponding package filenames.

The we have to feed those tar files into a command (or series of commands) to do the difference checking.

For Debian the core of this would be something vaguely like:

cd /
for eachfile in .../*.deb; do
	ar p $eachfile data.tar.gz | tar dz
	done

... if we had a single directory that contained links to every .deb file that we had installed.

One of these days all of the distribution maintainers will take my suggestion and provide us with an easy way to pop in our installation CD, boot from it, and give us a report on the integrity of all our packages.

(I'm oversimplifying a bit here. We really want ubiquitous use of digital signature databases for all those cases where we grab our packages and package updates from off the 'net. The technology is there and free --- but we seem to lack the will to employ it.).

I mean I can manage to do this sort of thing. Little scripts like the one above don't even take a moment to formulate --- though they take much longer for me to type. However, in increasing percentage of the Linux using world doesn't know shell magic and the 'ar' 'tar' and related commands like that.

Anyway, enough about how to do package integrity checking.

Back to your question:

You say that you can no longer log in from your virtual consoles (VCs) after you did these upgrades. Presumably you can still log in through kdm, xdm, or some other means.

You don't say whether you are trying to login as 'root' or as some other user.

There are a number of possible problems that can cause this failure. First, if it only happens to your 'root' user (and regular user accounts are not affected) you might have a bogus /etc/securetty file. /etc/securetty is a list of terminals (and VCs) on which 'root' is allowed to log in directly.

It is generally preferable to log in under a normal user account and use the 'su -' command to to assume 'root' privileges. However, it has classically been considered O.K. to log in as root directly from the console.

If your system is PAM (pluggable authentication module) based (like Red Hat since version 5? or so, all versions of Mandrake, and most other RPM based systems) then you can control the login policies on your system by editing the various files under /etc/pam.d/.

These are small text files. One for each "authenticating service." The ones you care about in this case are /etc/pam.d/login and /etc/pam.d/other. By removing or commenting out the lines that refer to "securetty.so" you can over-ride the default policies regarding where root is allowed to log in.

[ I got problems during a PAM upgrade once; thank goodness for ssh, which doesn't use the typical means to authenticate you if its own keys are properly in place. Thus I was able to go in and reset the PAM policies to match local practices. It took less than 5 minutes, but that was 3 minutes too long, because two important people who couldn't login during the change complained!
If root is fine, and some other low numbered accounts are fine but most user accounts aren't, maybe you were editing the shadow file by hand? One colon out of place and nobody after the broken spot will align properly with their password hashes...
-- Heather ]

If this problem is affecting all users, root included, then it could be that your new upgrades have disabled or enabled some alternative mechanism for store your account passwords.

Most likely this would mean that you were set to use shadow with MD5 password hashes rather than the more traditional DES hashes stored in the world readable /etc/passwd file.

Most Linux distributions that support MD5 passwords at all will include 'pwconv' or 'pwconv5' and 'pwunconv' or 'pwunconv5' commands to convert to and from the shadow system.

Even if you're not using a PAM authentication suite it's quite likely that you do have the shadow suite installed. You might look for a file named /etc/login.defs to see if any of the comments or settings in there relate to your problem.

Let me backtrack for a moment and explain.

Under UNIX and Linux the kernel (which enforces all system security) isn't involved in the authentication process at all. This is done purely by user-space utilities (like the 'login' command). The idea is that these programs, running under the auspices of ultimate system privilege (root) perform whatever process of authentication their programmers designed; they set the real UID and then execute the shell, or service as appropriate to how you are connecting.

The oldest traditional form of UNIX login was a program which was called by 'getty'. 'getty' is a program that simply "gets a TTY" --- setting the communications parameters and terminal device driver characteristics (like 'stty'), emmiting the system's /etc/issue message (pre-login banner), and waiting for input. When you entered anything on such a terminal (into the 'getty' program) then 'getty' would 'exec' the 'login' program, passing it whatever you had entered. This was usually a username. The login program would then request a password. You'd type in that, and it would perform some math on it.

(Specifically it was taking the first 8 characters of what you entered, stripping any of the 8th bits that were set, and using the resulting string as a 56 bit DES key to "encrypt" a sequence of ASCII NUL characters. The result of this encryption would then be encoded into a printable format).

The login program would then read the /etc/passwd file, search for an account that matched the username you gave to 'getty' extract the second field (the hash of the password) and compare that to the value it computed from the password you just entered. If the value stored matched the value it computed then it presumed that you know the right password and it lets you in. The login program would then extract the next field of the passwd file entry, set the UID to that value (this is your numeric user ID --- which is what the kernel uses to identify your processes, files, etc), and it would set you primary GID (group membership), change (cd) to your home directory, and exec your login shell (the seventh field in the old /etc/passwd file).

This "exec()" to which I keep alluding is significant. Through the "exec()" family of system calls a UNIX process "becomes" a new program. The PID doesn't change. It doesn't spawn a new one (that's done by "fork()"). It just starts running a different program.

This authentication model was adequate for early UNIX systems. The number of systems out there, they way that they were managed and monitored, the ways in which they where interconnected, the computing horsepower which they had available and the purposes for which they were being used all make this a reasonable system.

However, over the years UNIX system were deployed in more critical environments (growing beyond their research, academic and modest business system roots into mission critical infrastructure roles throughout industry and government. In addition the level of management and monitoring changed (from primarily programmers to wildly varying levels of system administration). Meanwhile the ways in which these systems became interconnected (via phone, leased line, LAN and Internet) and the amount of of computing horsepower, memory and available storage all increased.

So the simple "adequate" authentication system that they started with had become woefully inadequate. After 10 years it was possible for crackers to compute most simple passwords with "dictionary" attacks.

The /etc/passwd file was traditionally where all primary account information was stored on UNIX systems. It store username and UID, password hash, home directory, full name (originally used for interconnections to GE computer system accounts, and thus still called the GECOS field), etc.

Much of that information really needed to be accessed quite frequently and by all users. For example when you do an 'ls -l' command the 'ls' command is referring to the /etc/passwd file to map the file owner UIDs into user names. Thus the /etc/passwd file had to be "world readable."

(Of course, one could argue that this was a bad design decision. They could have had a series of separate files, one for the mapping of username to UID, another for store the username/password hash values, another for mapping shells to UIDs, etc. This would have been somewhat more complicated and less efficient in the early design, though it might have been better in the long run).

The point of this story is that storing a hash of the password in a world readable file was considered safe back then. Knowing that my password can be used as a DES key to encrypt a sequence of NULs into a string like: dV69Gd3wTNex5 was pretty useless 25 years ago. It took a significant amount of computing power to perform that encryption operation just once (over a full second on the common systems of the day). So trying every word in a dictionary on one of them was prohibitive. You'd be eating up so much CPU time that some system administrator was sure to catch you at it.

(Of course the most common 60,000 words in the English language could by tried in about a day or so --- so it's ALWAYS been strongly recommended that you choose passwords that aren't simple, common words).

Anyway, once the situation changed enough for this formerly "adequate" system to be horribly and notoriously vulnerable most UNIX vendors (and Linux distributions) adopted a different set of conventions for storing the password information. In this case they simply extract the password hashes from the /etc/passwd file and store them elsewhere in an "adjunct" or "shadow" file.

The adjunct/shadow file (so called because its contents must shadow or be "kept in sync with" those of /etc/passwd) is not world readable. It only stores the user names and password hashes. Thus all of the programs that need access to this file must run with advanced privileges. The 'login' program is already running as 'root' which it needs to do in order to set the UID to an arbitrary value and "become" some other user.

[ Nowadays it also stores accounting information, like how long before your password expires, which is also considered privileged info, and none of any standard app's business to be poking around in.
-- Heather ]

(Technically, under Linux, deep inside the kernel internals this isn't strictly true. The 'root' user on Linux is really running with a whole set of "capabilities" or "privileges" and it would be possible for those to be individually raised and lowered for programs like 'login' --- in that regard Linux is a bit more like VMS than like traditional forms of UNIX. Notably there are other forms of UNIX that have this sort of feature --- they are not sold for general use but are sold as "Trusted" versions for military and specialized use. Like I said --- that is all deep under the hood behind many tangles of other issues; don't worry about it here).

Of course this change from traditional to shadow password systems also required that all of the software that check and change passwords be updated to use the extra file. It also entailed the creation of new software to create maintain and verify the consistency of the new file to the old one. A whole suite of utilities was affected.

One problem with this very simple authentication model (where the system administrators, and system programmers can rip out the authentication system an replace it) is that you end up with lots of different programs all co-ordinating on a common set of conventions. Those conventions then become written in stone --- they become STANDARDS.

If you renamed /etc/passwd to /etc/accounts (which would be an obviously more apt name) then you'd break all of the software that relied on using /etc/passwd to perform UID to name mappings. Suddenly 'ls -l' wouldn't work!

(Actually all the programs that care are SUPPOSED to use a set of well-defined library functions to get this information. That abstraction is SUPPOSED to allow one to change underlying implementation details in a way that is transparent to the programs that are linked to those libraries. This is SUPPOSED to be rendered even easier with the nearly universal adoption of shared libraries. et cetera! However, these sorts of things often don't go quite as they're SUPPOSED to).

Now, this brings us back to your problem. I've described quite a bit about how UNIX and Linux perform authentication because I can't pinpoint what the problem is. Now that you understand a little about what's going on under the hood -- you might be able find out what's happening by looking at your /var/log/messages, reading any /etc/pam.d files you find, tracking down any packages that report differences (through whatever verification tricks you try), and look at the /etc/passwd and the /etc/shadow files to see if they don't match up somehow.