Software

Managing large numbers of Linux and UNIX systems takes a lot oforganization, automation and careful use of technology. A significantchunk of one's time as a system administrator is spent buildinginfrastructure to make managing those systems easier. Doing so improvesflexibility, recoverability and reduces downtime.All of this hopefully results in less stress and longer vacations. Thisarticle discussesone of those simple technologies that helps accomplish all of the above,serial consoles.

Figure 1. Managing Many Servers from a Console Server

Serial consoles always have been a standard feature ofenterprise UNIX hardware.Modern high-density server and cluster configurationssometimes can squeeze more than 50 servers in a 19" rack, so havingmonitors and keyboards for each one is unthinkable. AlthoughKVM (keyboard/video/mouse) switches can connect many servers to a smallnumber of monitors and keyboards, they are expensive and even more sowith remote access features. Serial consoles allow you to take racks orshelves of servers and have all their consoles available all the time,from anywhere.

The console is a simple I/O device, initialized early in the kernelboot process to convey informational messages as the system comes up. Once theoperating system starts running startup scripts, the console canbe used to recover an ailing system or to get input from the systemadministrator interactively, like Red Hat's Kudzu does. One compelling feature ofserial consoles is never having to drive in to work because a systemis hung up at reboot asking for input for fsck. After the system isup completely, the console usually becomes a login terminal, sometimesa graphical one. The console also can be used as a last resortmethod of reporting problems inside the kernel. Under these panic conditions it isnot possible always to write to log files or network log servers, somessages are reported to the console. For this reason and many more,consoles on servers should be a simple device, and a serial port is thesimplest device included on standard system configurations. For thoselast-minute panic messages, one could add a console device that supportsbuffering and logging, so you never miss a moment of the excitement.

What we are talking about here is booting up a system without using anattached keyboard, mouse or video monitor. Some motherboards may complainwithout a directly attached keyboard, but this requirement usually can be changed inthe BIOS configuration. In fact, with the recent popularity of USB keyboards,most BIOS versions do not care about missing keyboards. If you areusing a system that was designed to be a server, you may be even morefortunate. Several vendors have started adding extra functionality totheir BIOS versions to better support serial consoles from power-on.These features sometimes include power-on system test (POST)output and BIOS configuration access over the serial port. Depending onyour needs, you can select your hardware accordingly by checking thevendor specifications. Even without BIOS support, you stillcan use serial consoles quite effectively on almost any PC system.

This is not a perfect solution, though, and your average PC hardware does notprovide all of the features available in typical enterprise-class UNIXhardware. PC BIOS versions do not have the concept of a boot monitor(see the “What Is a Boot Monitor?” sidebar), nor can you perform a hardwarehalt of the OS as you can in enterprise UNIX hardware. For many applicationsthis is okay, but when more functionality is needed add-onhardware options are available, and I discuss them later.

Most PC hardware BIOS versions can be configured only with a directlyattached keyboard and video monitor. Luckily most come with usable defaultsettings, so this is not normally an issue. If it is, you may need to havethe system initially configured with direct attached video and keyboardand then switch to serial console. In my experience, I have rarely needed todo this; it needs to be done only once during initial hardware setup.

As packaged by most distributions, the Linux kernel and bootloader selectthe directly attached video controller and keyboard as console, but thisis easily changed. When a PC-based system boots, the bootloader is thefirst program to be loaded off the disk. The three major bootloadersin popular use on Linux systems are GRUB, LILO and SYSLINUX (usedon boot floppies); all of them support serial consoles. Next,the Linux kernel needs to be told to use a serial port for its console,which can be handled at compile time or by passing kernel command-lineoptions from the bootloader configuration. Finally, if you want to beable to log in on the console, you need to configure a getty processto run after the system is up.

We discuss the kernel configuration next because it is a prerequisiteto understanding the bootloader config later on. The most flexibleway to configure the kernel console is with the options passed on thekernel command line. You can append arguments to the command line fromthe bootloader. Here is an example of the kernel command-line syntax:

console=ttyS0,9600n8

This tells the kernel to use ttyS0 (the first serial port discovered bythe kernel), running at 9,600 baud, no parity and 8 bits. The kerneldefaults to one stop bit. This is the most common speed and configurationfor a serial console, which is why most serial ports and terminalsdefault to 9600n8. It is possible to append more than oneconsole=argument to the command line; kernel messages then are output to all ofthem, but only the last one is used for input.

GRUB is a flexible bootloader with excellent support for serialconsoles. When properly configured, GRUB allows multiple devicesto be used as a console. Listing 1 shows an example grub.conf file (usually/boot/grub/grub.conf and symlinked to /etc/grub.conf) as configuredby the Red Hat/Fedora Core installer. Yours may be slightly different.

# grub.conf generated by anaconda## Note that you do not have to rerun grub# after making changes to this file# NOTICE:You have a /boot partition.#        This means that all kernel#        and initrd paths are relative#        to /boot/, eg.#        root (hd0,1)#        kernel /vmlinuz-version ro root=/dev/hda6#        initrd /initrd-version.img#boot=/dev/hdadefault=0timeout=10splashimage=(hd0,1)/grub/splash.xpm.gztitle Red Hat Linux (2.4.20-8)        root (hd0,1)        kernel /vmlinuz-2.4.20-8 ro root=LABEL=/        initrd /initrd-2.4.20-8.img

The first thing to do is remove all splashimage directives. In someearly versions, these directives confuse GRUB and make it default to the video console.Then add a serial and terminal line. The serial line initializes theserial port to the proper baud and settings. In the terminal line, weconfigure GRUB to send prompts to both the serial port and to thekeyboard and monitor. You can press any key on either,and it becomes the default console. The--timeout=10 argument tellsGRUB to default to the first device listed in the terminal line afterten seconds. We also modified the kernel command line to include theoption that tells the Linux kernel to use the serial port as console.Listing 2 shows the complete modified grub.conf file.

The LILO bootloader, although much more mature than GRUB, is lessfeature-rich.We must configure LILO and pass options to the kernel to use aserial port. To do this, we add:

serial=<port>,<bps><parity><bits>

where port0 is the first serial port detected by LILO. Also, theappend=line is modified to include the kernel options. After modifying the/etc/lilo.conf file, be sure to run LILO to update the bootloader.The completed lilo.conf file is shown in Listing 3.

SYSLINUX is a bootloader designed for use with DOS/FAT formattedbootable floppies. Red Hat/Fedora Core Linux uses SYSLINUX for both theinstall boot and rescue floppies. In order to install orrecover from boot floppies over serial console, the floppies needto be modified. We have added theconsole= andtextdirective to the append line, and we have removed the extraboot selections present in Red Hat's original file. The firstline initializes and directs SYSLINUX to use serial port 0(aka /dev/ttyS0) and defaults to 9600n8. Using this modifiedboot floppy, we can install the OS over the serial console. Red Hat's textinstallation option works quite nicely this way. Using theabove modifications, you can convert any SYSLINUX boot floppy touse serial consoles. This procedure also works for ISOLINUX,which is a spinoff of SYSLINUX used on bootable CD-ROMs.

As stated before, the console can becomea login terminal after the system is up. For this to happen, the getty entries in/etc/inittab must be modified. The standard /etc/inittabstarts mingetty on virtual consoles only. Because mingetty isnot suitable for serial terminals, we must use something else.Many getty-type programs are available, but agetty is includedwith almost every Linux distribution, so weuse it. Also, make sure the systemboots to nongraphical mode, normally runlevel 3.Some Linux distributions default to an X login, usually atrunlevel 5, if any X packages were installed.The default runlevel is determined on the initdefault line.To enable agetty on serial lines, you can modify the initdefaultline in /etc/inittab:

id:3:initdefault:

and add a line for agetty:

co:2345:respawn:/sbin/agetty ttyS0 9600 vt100

This tells agetty to start waiting for logins on /dev/ttyS0 at 9,600bps, using vt100 terminal emulation. You may want to keep the originalmingetty entries to allow a directly attached keyboard and monitor tobe used for logins. If not, simply comment them out. Where root canlog in from is controlled strictly; in order for root tolog in from ttyS0, you must add the device to the /etc/securettyfile.

Finally, if your system has created a /etc/ioctl.save file, deleteor rename it. This file is used to save console settings betweenreboots. If the system was booted using a directly attached keyboard andmonitor, this file attempts to restore improper settings. A new oneis created when you reboot using the serial console.

Red Hat's bootup scripts use escape sequences, so the OK, PASS and FAILmessages show up in color. This can confuse serial consoles, so it is bestto disable it. Simply modify /etc/sysconfig/init, and change theBOOTUP=line to sayBOOTUP=serial. This will prevent the use of color messages.

Serial cabling can cause some confusion. Basically, there are two kinds ofserial ports, DCE (Data Communication Equipment) and DTE (Data TerminalEquipment). The ports differ in how specific signals are connected to pinson the connector. Data communication with serial ports uses separatetransmit and receive wires, so when connecting two pieces of equipmenttogether, one must make sure the transmit wire on one side connectsto the receive wire on the other side. As long as you are connectinga DCE device to a DTE device you can use a regular straight-throughcable, where each pin is connected to the same pin on the other side ofthe cable. If you are connecting devices of the same type, however,you must use a special cable or adapter, called a null modem, sothe signals are swapped properly. DTE devices usually are terminals,computers and printers. DCE devices are designed to connectdirectly to computers, such as modems and serial mice.

In addition to thedata transmit and receive wires, a number of handshakingsignals are used to control the flow of data, so one side is not talking toofast for the other to understand. These signals also must be swapped by the null modem.To add to the confusion, two popular connectors are in use forserial ports, the 9-pin DB9 and the 25-pin DB25. These can comein both male and female varieties. In almost every case, the devices usedfor serial consoles (terminals, computers and console servers) are all DTE,which means you need a null modem of some sort. These areavailable in the form of adapters and cables. Most off-the-shelf unitswork fine, but if you want to solder your own, check the on-lineResources section for links to pinouts and cable diagrams.

At this point, we have described a Linux system that can boot up withouta directly attached keyboard and monitor. It uses the first serialport for all informational messages as the system boots and acceptslogins from that console once the system is up. But to what shouldyou connect that console port? There is a world of possibilities.If you have no particular need for remote-console access, you simplycan leave the port unconnected until you need to maintain the system.You can use a computer or laptop connected over a null modem with theminicom program to access your system's console. Simply configureminicom to speak to an unused serial port, set the speed to 9,600 baud, 8bits, no parity and 1-stop bit (aka 9600-8n1). Cable the systems together,then watch the system boot and eventually ask you to log in.

For remote access to a server's console, you can set up a consoleconcentrator, which is a lot like a terminal server. It can be a homegrown Linux box with multiport serial cards, giving you as many portsas you have servers. With this kind of setup, you can access all yourservers' consoles by logging in to a single dedicated Linux box.

If you like the idea of remote access to your consoles but want more ofan appliance, a number of products can help. Cyclades(www.cyclades.com) makes a console concentratorcalled AlterPath;it is reasonably priced and comes in 1, 4, 8, 16, 32 and 48-port models.The AlterPath units run Linux internally from Flash memory. AWeb interface is used for configuration, or you can modify the configuration filesdirectly through a shell login.

The most flexible way to configure the Cyclades unit isto present the consoles using Cyclades' modified SSH dæmon.This way you can SSH directly to eachconnected server's console port, which is identified by a textual nameyou choose. So, to connect to a server identified as server hooked to aCyclades unit with a hostname of cyclades as the user matt, the commandwould look like:ssh matt:server@cyclades. (Thecolon syntax is a Cyclades modification to sshd, allowing you to pass a portname.) Thissetup is easy to use,and you even can set up SSH private key authentication.

Other vendors make console concentrators or servers,including Digi (www.digi.com), Equinox (www.equinox.com)and Raritan (www.raritan.com). All of these vendorsoffer network-attached serialconsole products.

As mentioned earlier, serial consoles on standard PC hardware lack someof the features available on enterprise UNIX hardware.One solution is PC Weasel (www.realweasel.com), which comes in theform of a PCI or ISA card. This device emulates a video cardand translates all output to the serial port as normal terminal escapesequences. Input from the serial port is translated into PC keyboardscan codes. Because it looks like a video card to the system, the systemallows itfull access to BIOS and POST. Additional features allow you to do aremote hard reset. The PC Weasel also has its own processor, so itis available even if the host into which it is plugged crashes.

If you would like to build your own console concentrator,some options are available to make it a little better than a simple box with a lot ofserial ports. Conserver (www.conserver.com) is anopen-sourcesoftware package for managing systems connected to serial consoles.It supports SSL encryption and is highly configurable.

Resources for this article:/article/7507.