Problem

You want to read in records separated by a pattern, but Perl doesn't allow its input record separator variable to be a regular expression.

Many problems, most obviously those involving the parsing of complex file formats, become a lot simpler when you are easily able to extract records that might be separated by a number of different strings.

Solution

Read the whole file and use split:

undef $/;
@chunks = split(/pattern/, <FILEHANDLE>);

Discussion

Perl's record separator must be a fixed string, not a pattern. (After all, awk has to be better at something.) To sidestep this limitation, undefine the input record separator entirely so that the next line-read operation gets the rest of the file. This is sometimes called slurp mode, because it slurps in the whole file as one big string. Then split that huge string using the record separating pattern as the first argument.

Here's an example, where the input stream is a text file that includes lines consisting of ".Se", ".Ch", and ".Ss", which are special codes in the troff macro set that this book was developed under. These lines are the separators, and we want to find text that falls between them.

# .Ch, .Se and .Ss divide chunks of STDIN
{
    local $/ = undef;
    @chunks = split(/^\.(Ch|Se|Ss)$/m, <>);
}
print "I read ", scalar(@chunks), " chunks.\n";

We create a localized version of $/ so its previous value gets restored after the block finishes. By using split with parentheses in the pattern, captured separators are also returned. This way the data elements in the return list alternate with elements containing "Se", "Ch", or "Ss".

If you didn't want delimiters returned but still needed parentheses, you could use non-capturing parentheses in the pattern: /^\.(?:Ch|Se|Ss)$/m .

If you just want to split before a pattern but include the pattern in the return, use a look-ahead assertion: /^(?=\.(?:Ch|Se|Ss))/m . That way each chunk starts with the pattern.

Be aware that this uses a lot of memory if the file is large. However, with today's machines and your typical text files, this is less often an issue now than it once was. Just don't try it on a 200-MB logfile unless you have plenty of virtual memory to use to swap out to disk with! Even if you do have enough swap space, you'll likely end up thrashing.

See Also

The $/ variable in perlvar (1) and in the "Special Variables" section of Chapter 2 of Programming Perl; the split function in perlfunc (1) and Chapter 3 of Programming Perl; we talk more about the special variable $/ in Chapter 8