Table of Contents
storage.conf - configuration file for storage manager
The
storage manager is a unified interface between INN and a variety of different
storage method, allowing the news administrator to choose between different
storage method with different tradeoffs (or even use several at the same
time for different newsgroups or articles of different sizes). The rest
of INN need not care what type of storage method was used for a given article;
the storage manager will figure this out automatically when that article
is retrieved via the storage API.
<pathetc in inn.conf>/storage.conf file contains
the rules to be used in assigning articles to different storage methods.
The file consists of a series of storage method entries. Blank lines and
lines beginning with a number sign (``#'') are ignored. The maximum number of
character in each line is 255. The order of entries in this file is important.
Each entry specifies a storage method and a set of rules. Articles that
match all of the rules of a storage method entry will be stored using that
storage method. If an article matches multiple storage method entries,
the first one will be used. Each entry is formatted as follows:
method <methodname> {
class: <storage_class>
newsgroups: <wildmat>
size: <minsize>[,<maxsize>]
expires: <mintime>[,<maxtime>]
options: <options>
}
If spaces or tabs are included in a value, that value must be quoted with
``"''. If either ``#'' or ``"'' are meant to be included verbatim in a value, they should
be escaped with ``\''.
<methodname> is the name of a storage method to use for
articles that match the rules of this entry. The currently available storage
methods are ``timecaf'', ``timehash'', ``cnfs'', ``tradspool'' and ``trash''. See the STORAGE
METHODS section below for more details.
The meanings of the keys in each
entry are as follows:
- class
- An identifier for this storage method entry.
<storage_class> should be a number and should be unique across all of the
entries in this file. It's used mainly by expire.ctl(5)
for specifying expiration
times by storage class.
- newsgroups
- What newsgroups are stored using this
storage method. <wildmat> is a wildmat(3)
pattern that is matched against
the newsgroups an article is posted to. If ``storeonxref'' in inn.conf is ``true'',
this pattern will be matched against the newsgroup names in the ``Xref'' header;
otherwise, it will be matched against newsgroup names in the ``Newsgroups''
header. Poison wildmat expressions (expressions starting with ``@'') are allowed
and can be used to exclude certain group patterns. ``!'' cannot be used, however.
The <wildmat> pattern is matched in order. There is no default newsgroups
pattern; if an entry should match all newsgroups, use an explicit ``newsgroups:
*''.
- size
- A range of article sizes (in bytes) that should be stored using
this storage method. If <maxsize> is ``0'' or not given, the upper size of articles
is limited only by ``maxartsize'' in inn.conf(5)
. The ``size'' field is optional
and may be omitted entirely if you want articles of any size (that otherwise
fulfill the requirements of this storage method entry) to be stored in
this storage method.
- expires
- A range of article expiration times that should
be stored using this storage method. Be careful; this is less useful than
it may appear at first. This is based only on the ``Expires'' header of the
article, not on any local expiration policies or anything in expire.ctl(5)
!
If <mintime> is non-zero, then this entry will not match any article without
an ``Expires'' header. This key is therefore only really useful for assigning
articles with requested longer expire times to a separate storage method.
<mintime> and <maxtime> are boundaries on the amount of time into the future
the ``Expires'' header of the article requests that it remain around, and are
formatted 0d0h0m0s (days, hours, minutes, and seconds into the future).
If <maxtime> is ``0s'' or is not specified, there is no upper bound on expire
times falling into this entry (note that this key has no effect on when
the article will actually be expired, only on whether or not the article
will be stored using this storage method). This field is also optional and
may be omitted entirely if all articles with or without an ``Expires'' header
(that otherwise fulfill the requirements of this storage method entry)
should be stored according to it.
- options
- This key is for passing special
options to storage methods that require them (currently only ``cnfs''). See
the STORAGE METHODS section below for a description of its use.
If an article
matches all of the constraints of an entry, it is stored via that storage
method and is associated with that <storage_class>. This file is scanned
in order and the first matching entry is used to store the article.
If an
article doesn't match any entry, either by being posted to a newsgroup that
doesn't match any of the <wildmat> patterns or by being outside the size and
expires ranges of all entries whose newsgroups pattern it does match, the
article is not stored and is rejected by innd(8)
. When this happens, the
error message
cant store article: no matching entry in storage.conf
is logged to syslog. If you want to silently drop articles matching certain
newsgroup patterns or size or expires ranges, assign them to the ``trash''
storage method rather than having them not match any storage method entry.
Currently, there are four storage methods available. Each
method has its characteristics. You can choose any of them to be suitable
for your environment. Note that each method has an attribute ``EXPENSIVESTAT''
which means whether checking existense of article is expensive or not.
This is used to run expireover(8)
.
- cnfs
- The ``cnfs'' storage method stores articles
in large cyclic buffers (CNFS stands for Cyclic News File System). It's
by far the fastest of all storage methods (except for ``trash''), since it
eliminates the overhead of dealing with a file system and creating new
files. Articles are stored in CNFS buffers in arrival order, and when the
buffer fills, it wraps around to the beginning and stores new articles
over top of the oldest articles in the buffer. The expire time of articles
stored in CNFS buffers is therefore entirely determined by how long it
takes the buffer to wrap around, which depends on how much data is being
stored in it. (This method is therefore said to have self-expire functionality.)
``EXPENSIVESTAT'' is ``FALSE'' for this method. CNFS has its own configuration file,
cycbuff.conf(5)
. Storage method entries for the ``cnfs'' storage method must
have an ``options'' field specifying the metacycbuff into which articles matching
that entry should be stored. See cycbuff.conf(5)
for details on metacycbuffs.
- timecaf
- This method stores multiple articles in one file, whose name is
based on the article's arrival time and the storage class. The file name
will be <patharticles in inn.conf>/timecaf-nn/bb/aacc.CF, where ``nn'' is the hexadecimal
value of <storage_class> and ``bb'' and ``aacc'' are hexadecimal components of the
arrival time. (The arrival time in seconds since epoch is converted to
hex and interpreted as 0xaabbccFF, with ``aa'', ``bb'', and ``cc'' used to build the
path.) This method does not have self-expire functionality (meaning expire(8)
has to run periodically to delete old articles). ``EXPENSIVESTAT'' is ``FALSE''
for this method.
- timehash
- This method is very similar to ``timecaf'' except
that each article is stored in a separate file. The name of the file for
a given article will be <patharticles in inn.conf>/time-nn/bb/cc/yyyy-aadd, where
``nn'' is the hexadecimal value of <storage_class>, ``yyyy'' is a hexadecimal sequence
number, and ``bb'', ``cc'', and ``aadd'' are components of the arrival time in hexadecimal
(the arrival time is converted to hex and interpreted as 0xaabbccdd). This
method does not have self-expire functionality. ``EXPENSIVESTAT'' is ``TRUE'' for
this method.
- tradspool
- Traditional spool, or ``tradspool'', is the traditional
news article storage format. Each article is stored in a file named: <patharticles in inn.conf>/news/group/name/nnnnn,
where ``news/group/name'' is the name of the newsgroup to which the article
was posted with each period changed to a slash, and ``nnnnn'' is the sequence
number of the article in that newsgroup. For crossposted articles, the
article is linked into each newsgroup to which it is crossposted (using
either hard or symbolic links). This is the way all versions of INN prior
to 2.0 stored all articles, as well as being the article storage format
used by C News and earlier news systems. This method does not have self-expire
functionality. ``EXPENSIVESTAT'' is ``TRUE'' for this method.
- trash
- This method silently
discards all articles stored in it. Its only real uses are for testing
and for silently discarding articles matching a particular storage method
entry (for whatever reason). Articles stored in this method take up no
disk space and can never be retrieved, so this method has self-expire functionality
of a sort. ``EXPENSIVESTAT'' is ``FALSE'' for this method.
The following sample
storage.conf file would store all articles posted to alt.binaries.* in the
``BINARIES'' CNFS metacycbuff, all articles over roughly 50KB in any other
hierarchy in the ``LARGE'' CNFS metacycbuff, all other articles in alt.* in
one timehash class, and all other articles in any newsgroups in a second
timehash class, except for the internal.* hierarchy which is stored in traditional
spool format.
method tradspool {
class: 1
newsgroups: internal.*
}
method cnfs {
class: 2
newsgroups: alt.binaries.*
options: BINARIES
}
method cnfs {
class: 3
newsgroups: *
size: 50000
options: LARGE
}
method timehash {
class: 4
newsgroups: alt.*
}
method timehash {
class: 5
newsgroups: *
}
Notice that the last storage method entry will catch everything. This is
a good habit to get into; make sure that you have at least one catch-all
entry just in case something you didn't expect falls through the cracks.
Notice also that the special rule for the internal.* hierarchy is first,
so it will catch even articles crossposted to alt.binaries.* or over 50KB
in size.
Written by Katsuhiro Kondou <kondou@nec.co.jp> for InterNetNews.
This is revision 1.7.2.2, dated 2001/04/17.
cycbuff.conf(5)
, expire.ctl(5)
,
inn.conf(5)
, innd(8)
, newsfeeds(5)
, wildmat(3)
.
Table of Contents