man Lintian::Util (3): Lintian utility functions

SYNOPSIS

use Lintian::Util qw(slurp_entire_file normalize_pkg_path);

my $text = slurp_entire_file('some-file');
if ($text =~ m/regex/) {
# ...
}
my $path = normalize_pkg_path('usr/bin/', '../lib/git-core/git-pull');
if (defined $path) {
# ...
}

my (@paragraphs);
eval { @paragraphs = read_dpkg_control_utf8('some/debian/ctrl/file'); };
if ($@) {
# syntax error etc.
die "ctrl/file: $@";
}

foreach my $para (@paragraphs) {
my $value = $para->{'some-field'};
if (defined $value) {
# ...
}
}

DESCRIPTION

This module contains a number of utility subs that are nice to have, but on their own did not warrant their own module.

Most subs are imported only on request.

Debian control parsers

At first glance, this module appears to contain several debian control parsers. In practise, there is only one real parser (``visit_dpkg_paragraph'') - the rest are convenience functions around it.

If you have very large files (e.g. Packages_amd64), you almost certainly want ``visit_dpkg_paragraph''. Otherwise, one of the convenience functions are probably what you are looking for.

Use "get_deb_info" when: You have a .deb (or .udeb) file and you want the control file from it.
Use "get_dsc_info" when: You have a .dsc (or .changes) file. Alternative, it is also useful if you have a control file and only care about the first paragraph.
Use "read_dpkg_control_utf8" or "read_dpkg_control" when: You have a debian control file (such debian/control) and you want a number of paragraphs from it.
Use "parse_dpkg_control" when: When you would have used ``read_dpkg_control_utf8'', except you have an open filehandle rather than a file name.

CONSTANTS

The following constants can be passed to the Debian control file parser functions to alter their parsing flag.

DCTRL_DEBCONF_TEMPLATE: The file should be parsed as debconf template. These have slightly syntax rules for whitespace in some cases.
DCTRL_NO_COMMENTS: The file do not allow comments. With this flag, any comment in the file is considered a syntax error.

VARIABLES

$PKGNAME_REGEX: Regular expression that matches valid package names. The expression is not anchored and does not enforce any ``boundary'' characters.

FUNCTIONS

parse_dpkg_control(HANDLE[, FLAGS[, LINES]])

Reads a debian control file from HANDLE and returns a list of paragraphs in it. A paragraph is represented via a hashref, which maps (lower cased) field names to their values.

FLAGS (if given) is a bitmask of the DCTRL_* constants. Please refer to ``CONSTANTS'' for the list of constants and their meaning. The default value for FLAGS is 0.

If LINES is given, it should be a reference to an empty list. On return, LINES will be populated with a hashref for each paragraph (in the same order as the returned list). Each hashref will also have a special key "START-OF-PARAGRAPH" that gives the line number of the first field in that paragraph. These hashrefs will map the field name of the given paragraph to the line number where the field name appeared.

This is a convenience sub around ``visit_dpkg_paragraph'' and can therefore produce the same errors as it. Please see ``visit_dpkg_paragraph'' for the finer semantics of how the control file is parsed.

NB: parse_dpkg_control does not close the handle for the caller.

visit_dpkg_paragraph (CODE, HANDLE[, FLAGS])

Reads a debian control file from HANDLE and passes each paragraph to CODE. A paragraph is represented via a hashref, which maps (lower cased) field names to their values.

FLAGS (if given) is a bitmask of the DCTRL_* constants. Please refer to ``CONSTANTS'' for the list of constants and their meaning. The default value for FLAGS is 0.

If the file is empty (i.e. it contains no paragraphs), the method will contain an empty list. The deb822 contents may be inside a signed PGP message with a signature.

visit_dpkg_paragraph will require the PGP headers to be correct (if present) and require that the entire file is covered by the signature. However, it will not validate the signature (in fact, the contents of the PGP SIGNATURE part can be empty). The signature should be validated separately.

visit_dpkg_paragraph will pass paragraphs to CODE as they are completed. If CODE can process the paragraphs as they are seen, very large control files can be processed without keeping all the paragraphs in memory.

As a consequence of how the file is parsed, CODE may be passed a number of (valid) paragraphs before parsing is stopped due to a syntax error.

NB: visit_dpkg_paragraph does not close the handle for the caller.

CODE is expected to be a callable reference (e.g. a sub) and will be invoked as the following:

CODE->(PARA, LINE_NUMBERS)

The first argument, PARA, is a hashref to the most recent paragraph parsed. The second argument, LINE_NUMBERS, is a hashref mapping each of the field names to the line number where the field name appeared. LINE_NUMBERS will also have a special key "START-OF-PARAGRAPH" that gives the line number of the first field in that paragraph.

The return value of CODE is ignored.

If the CODE invokes die (or similar) the error is propagated to the caller.

On syntax errors, visit_dpkg_paragraph will call die with the following string:

  "syntax error at line %d: %s\n"

Where %d is the line number of the issue and %s is one of:

Duplicate field %s: The field appeared twice in the paragraph.
Continuation line outside a paragraph (maybe line %d should be " ."): A continuation line appears outside a paragraph - usually caused by an unintended empty line before it.
Whitespace line not allowed (possibly missing a "."): An empty continuation line was found. This usually means that a period is missing to denote an ``empty line'' in (e.g.) the long description of a package.
Cannot parse line "%s": Generic error containing the text of the line that confused the parser. Note that all non-printables in %s will be replaced by underscores.
Comments are not allowed: A comment line appeared and FLAGS contained DCTRL_NO_COMMENTS.
PGP signature seen before start of signed message: A ``BEGIN PGP SIGNATURE'' header is seen and a ``BEGIN PGP MESSAGE'' has not been seen yet.
Two PGP signatures (first one at line %d): Two ``BEGIN PGP SIGNATURE'' headers are seen in the same file.
Unexpected %s header: A valid PGP header appears (e.g. ``BEGIN PUBLIC KEY BLOCK'').
Malformed PGP header: An invalid or malformed PGP header appears.
Expected at most one signed message (previous at line %d): Two ``BEGIN PGP MESSAGE'' headers appears in the same message.
End of file but expected a "END PGP SIGNATURE" header: The file ended after a ``BEGIN PGP SIGNATURE'' header without being followed by a ``END PGP SIGNATURE''.
PGP MESSAGE header must be first content if present: The file had content before PGP MESSAGE.
Data after the PGP SIGNATURE: The file had data after the PGP SIGNATURE block ended.
End of file before "BEGIN PGP SIGNATURE": The file had a ``BEGIN PGP MESSAGE'' header, but no signature was present.

read_dpkg_control_utf8(FILE[, FLAGS[, LINES]])

read_dpkg_control(FILE[, FLAGS[, LINES]])

This is a convenience function to ease using ``parse_dpkg_control'' with paths to files (rather than open handles). The first argument must be the path to a FILE, which should be read as a debian control file. If the file is empty, an empty list is returned.

Otherwise, this behaves like:

 use autodie;
 
 open(my $fd, '<:encoding(UTF-8)', FILE); # or '<'
 my @p = parse_dpkg_control($fd, FLAGS, LINES);
 close($fd);
 return @p;

This goes without saying that may fail with any of the messages that ``parse_dpkg_control(HANDLE[, FLAGS[, LINES]])'' do. It can also emit autodie exceptions if open or close fails.

dpkg_deb_has_ctrl_tarfile()

Check if lintian could use dpkg-deb instead of ar and tar

get_deb_info(DEBFILE)

Extracts the control file from DEBFILE and returns it as a hashref.

Basically, this is a fancy convenience for setting up an ar + tar pipe and passing said pipe to ``parse_dpkg_control(HANDLE[, FLAGS[, LINES]])''.

DEBFILE must be an ar file containing a ``control.tar.gz'' member, which in turn should contain a ``control'' file. If the ``control'' file is empty this will return an empty list.

Note: the control file is only expected to have a single paragraph and thus only the first is returned (in the unlikely case that there are more than one).

This function may fail with any of the messages that ``parse_dpkg_control'' do. It can also emit:

 "cannot fork to unpack %s: %s\n"

get_dsc_control (DSCFILE)

Convenience function for reading dsc files. It will read the DSCFILE using ``read_dpkg_control(FILE[, FLAGS[, LINES]])'' and then return the first paragraph. If the file has no paragraphs, "undef" is returned instead.

Note: the control file is only expected to have a single paragraph and thus only the first is returned (in the unlikely case that there are more than one).

This function may fail with any of the messages that ``read_dpkg_control(FILE[, FLAGS[, LINES]])'' do.

slurp_entire_file (FOH[, NOCLOSE])

Reads the contents of FOH into memory and return it as a scalar. FOH can be either the path to a file or an open file handle.

If it is a handle, the optional NOCLOSE parameter can be used to prevent the sub from closing the handle. The NOCLOSE parameter has no effect if FOH is not a handle.

drain_pipe(FD)

Reads and discards any remaining contents from FD, which is assumed to be a pipe. This is mostly done to avoid having the ``write''-end die with a SIGPIPE due to a ``broken pipe'' (which can happen if you just close the pipe).

May cause an exception if there are issues reading from the pipe.

Caveat: This will block until the pipe is closed from the ``write''-end, so only use it with pipes where the ``write''-end will eventually close their end by themselves (or something else will make them close it).

get_file_checksum (ALGO, FILE)

Returns a hexadecimal string of the message digest checksum generated by the algorithm ALGO on FILE.

ALGO can be 'md5' or shaX, where X is any number supported by Digest::SHA (e.g. 'sha256').

This sub is a convenience wrapper around Digest::{MD5,SHA}.

is_string_utf8_encoded(STRING)

Returns a truth value if STRING can be decoded as valid UTF-8.

file_is_encoded_in_non_utf8 (...)

Undocumented

system_env (CMD)

Behaves like system (CMD) except that the environment of CMD is cleaned (as defined by ``clean_env''(1)).

clean_env ([CLOC])

Destructively cleans %ENV - removes all variables %ENV except a selected few whitelisted variables.

The list of whitelisted %ENV variables are:

 PATH
 LC_ALL (*)
 TMPDIR

(*) LC_ALL is a special case as clean_env will change its value to either ``C.UTF-8'' or ``C'' (if CLOC is given and a truth value).

perm2oct(PERM)

Translates PERM to an octal permission. PERM should be a string describing the permissions as done by tar t or ls -l. That is, it should be a string like ``-rw-r---r--''.

If the string does not appear to be a valid permission, it will cause a trappable error.

Examples:

 # Good
 perm2oct('-rw-r--r--') == 0644
 perm2oct('-rwxr-xr-x') == 0755
 # Bad
 perm2oct('broken')      # too short to be recognised
 perm2oct('-resurunet')  # contains unknown permissions

delete_dir (ARGS)

Convenient way of calling rm -fr ARGS.

copy_dir (ARGS)

Convenient way of calling cp -a ARGS.

gunzip_file (IN, OUT)

Decompresses contents of the file IN and stores the contents in the file OUT. IN is not removed by this call. On error, this function will cause a trappable error.

open_gz (FILE)

Opens a handle that reads from the GZip compressed FILE.

On failure, this sub emits a trappable error.

Note: The handle may be a pipe from an external processes.

touch_file(FILE)

Updates the ``mtime'' of FILE. If FILE does not exist, it will be created.

On failure, this sub will emit a trappable error.

fail (MSG[, ...])

Use to signal an internal error. The argument(s) will used to print a diagnostic message to the user.

If multiple arguments are given, they will be merged into a single string (by join (' ', @_)). If only one argument is given it will be stringified and used directly.

locate_helper_tool(TOOLNAME)

Given the name of a helper tool, returns the path to it. The tool must be available in the ``helpers'' subdir of one of the ``lintian root'' directories used by Lintian.

The tool name should follow the same rules as check names. Particularly, third-party checks should namespace their tools in the same way they namespace their checks. E.g. ``python/some-helper''.

If the tool cannot be found, this sub will cause a trappable error.

strip ([LINE])

Strips whitespace from the beginning and the end of LINE and returns it. If LINE is omitted, $_ will be used instead. Example

 @lines = map { strip } <$fd>;

In void context, the input argument will be modified so it can be used as a replacement for chomp in some cases:

  while ( my $line = <$fd> ) {
    strip ($line);
    # $line no longer has any leading or trailing whitespace
  }

Otherwise, a copy of the string is returned:

  while ( my $orig = <$fd> ) {
    my $stripped = strip ($orig);
    if ($stripped ne $orig) {
        # $orig had leading or/and trailing whitespace
    }
  }

lstrip ([LINE])

Like strip but only strip leading whitespace.

rstrip ([LINE])

Like strip but only strip trailing whitespace.

check_path (CMD)

Returns 1 if CMD can be found in PATH (i.e. $ENV{PATH}) and is executable. Otherwise, the function return 0.

dequote_name(STR, REMOVESLASH)

Strip an extra layer quoting in index file names and optionally remove an initial ``./'' if any.

Remove initial ./ by default

signal_number2name(NUM)

Given a number, returns the name of the signal (without leading ``SIG''). Example:

    signal_number2name(2) eq 'INT'

normalize_pkg_path(PATH)

Normalize PATH by removing superfluous path segments. PATH is assumed to be relative the package root. Note that the result will never start nor end with a slash, even if PATH does.

As the name suggests, this is a path ``normalization'' rather than a true path resolution (for that use Cwd::realpath). Particularly, it assumes none of the path segments are symlinks.

normalize_pkg_path will return "q{}" (i.e. the empty string) if PATH is normalized to the root dir and "undef" if the path cannot be normalized without escaping the package root.

Examples:
  normalize_pkg_path('usr/share/java/../../../usr/share/ant/file')
    eq 'usr/share/ant/file'
  normalize_pkg_path('usr/..') eq q{};

 The following will return C<undef>:
  normalize_pkg_path('usr/bin/../../../../etc/passwd')

normalize_pkg_path(CURDIR, LINK_TARGET)

Normalize the path obtained by following a link with LINK_TARGET as its target from CURDIR as the current directory. CURDIR is assumed to be relative to the package root. Note that the result will never start nor end with a slash, even if CURDIR or DEST does.

normalize_pkg_path will return "q{}" (i.e. the empty string) if the target is the root dir and "undef" if the path cannot be normalized without escaping the package root.

CAVEAT: This function is not always sufficient to test if it is safe to open a given symlink. Use is_ancestor_of for that. If you must use this function, remember to check that the target is not a symlink (or if it is, that it can be resolved safely).

Examples:

  normalize_pkg_path('usr/share/java', '../ant/file') eq 'usr/share/ant/file'
  normalize_pkg_path('usr/share/java', '../../../usr/share/ant/file')
  normalize_pkg_path('usr/share/java', '/usr/share/ant/file')
    eq 'usr/share/ant/file'
  normalize_pkg_path('/usr/share/java', '/') eq q{};
  normalize_pkg_path('/', 'usr/..') eq q{};
 The following will return C<undef>:
  normalize_pkg_path('usr/bin', '../../../../etc/passwd')
  normalize_pkg_path('usr/bin', '/../etc/passwd')

parse_boolean (STR)

Attempt to parse STR as a boolean and return its value. If STR is not a valid/recognised boolean, the sub will invoke croak.

The following values recognised (string checks are not case sensitive):

The integer 0 is considered false
Any non-zero integer is considered true
"true", "y" and "yes" are considered true
"false", "n" and "no" are considered false

is_ancestor_of(PARENTDIR, PATH)

Returns true if and only if PATH is PARENTDIR or a path stored somewhere within PARENTDIR (or its subdirs).

This function will resolve the paths; any failure to resolve the path will cause a trappable error.

unix_locale_split(STR)

Read STR as a locale code (e.g. en_GB.UTF-8) and return a list of locale codes ordered by preference. As an example, en_GB.UTF-8 might return

  en_GB
  en

Note encoding is ignored as all Lintian files are always encoded in UTF-8.

Special cases: The ``C'' or ``POSIX'' locale returns the empty list. Other strings that do not match the expected format causes a trappable error.

pipe_tee(INHANDLE, OUTHANDLES[, OPTS])

Read bytes from INHANDLE and copy them into all of the handles in the listref OUTHANDLES. The optional OPTS argument is a hashref of options, see below.

The subroutine will continue to read from INHANDLE until it is exhausted or an error occurs (either during read or write). In case of errors, a trappable error will be raised. The handles are left open when the subroutine returns, caller must close them afterwards.

Caller should ensure that handles are using ``blocking'' I/O. The subroutine will use sysread and syswrite when reading and writing.

OPTS, if given, may contain the following key-value pairs:

chunk_size: A suggested buffer size for read/write. If given, it will be to sysread as LENGTH argument when reading from INHANDLE.

load_state_cache(STATE_DIR)

[Reporting tools only] Load the state cache from STATE_DIR.

save_state_cache(STATE_DIR, STATE)

[Reporting tools only] Save the STATE cache to STATE_DIR.

find_backlog(LINTIAN_VERSION, STATE)

[Reporting tools only] Given the current lintian version and the harness state, return a list of group ids that are part of the backlog. The list is sorted based on what version of Lintian processed the package.

untaint(VALUE)

Untaint VALUE