Table of Contents
innfeed.conf - configuration file for innfeed
This
man page describes the configuration file for version 1.0 of innfeed. This
format has changed dramatically since version 0.9.3.
The file innfeed.conf
is used to control the innfeed(1)
program. It is a fairly free-format file
that consists of three types of entries: key/value, peer and group. Comments
are taken from the hash character ``#'' to the end of the line.
Key/value entries
are a keyword and a value separated by a colon (which can itself be surrounded
by whitespace). For example:
max-connections: 10
A legal key starts with a letter and contains only letters, numbers and
``_'', ``-''.
There are 5 different type of values: integers, floating-point numbers,
characters, booleans, and strings. Integer and floating point numbers are
as to be expected except that exponents in floating point numbers are not
supported. A boolean value is either ``true'' or ``false'' (case is not significant).
A character value is a single-quoted character as defined by the C-language.
A string value is any other sequence of characters. If the string needs
to contain whitespace, then it must be quoted with double quotes, and uses
the same format for embedding non-printing characters as normal C-language
string.
Peer entries look like:
peer <name> {
# body ...
}
The word ``peer'' is required. The ``<name>'' is the same as the site name in INN's
newsfeeds file. The body of a peer entry contains some number (possibly
zero) of key/value entries.
Group entries look like:
group <name> {
# body
}
The word ``group'' is required. The ``<name>'' is any string valid as a key. The body
of a group entry contains any number of the three types of entries. So key/value
pairs can be defined inside a group, and peers can be nested inside a group,
and other groups can be nested inside a group.
Key/value entries that are
defined outside of all peer and group entries are said to be at ``global
scope''. There are global key/value entries that apply to the process as a
whole (for example the location of the backlog file directory), and there
are global key/value entries that act as defaults for peers. When innfeed
looks for a specific value in a peer entry (for example, the maximum number
of connections to set up), if the value is not defined in the peer entry,
then the enclosing groups are examined for the entry (starting at the closest
enclosing group). If there are no enclosing groups, or the enclosing groups
don't define the key/value, then the value at global scope is used.
A small
example could be:
# Global value applied to all peers that have
# no value of their own.
max-connections: 5
# A peer definition. ``uunet'' is the name used by innd in
# the newsfeeds file.
peer uunet {
ip-name: usenet1.uu.net
}
peer vixie {
ip-name: gw.home.vix.com
max-connections: 10 # override global value.
}
# A group of two peers who can handle more connections
# than normal
group fast-sites {
max-connections: 15
# Another peer. The ``max-connections'' value from the
# ``fast-sites'' group scope is used. The ``ip-name'' value
# defaults to the peer's name.
peer data.ramona.vix.com {
}
peer bb.home.vix.com {
max-connections: 20 # he can really cook.
}
}
Given the above configuration file, the defined peers would have the following
values for the ``max-connections'' key.
uunet 5
vixie 10
data.ramona.vix.com 15
bb.home.vix.com 20
Innfeed ignores key/value pairs it is not interested in. Any config file
value that can be set via a command line option, is not used if the command-line
option is given.
Config files can be included in other config files via
the syntax:
$INCLUDE filename
There is a maximum nesting depth of 10.
For a fuller example config file,
see the supplied innfeed.conf.
The following listing show all
the keys that apply to the process as whole. These are not required (compiled-in
defaults are used where needed).
- news-spool
- This key requires a pathname
value. It specifies where the top of the article spool is. This corresponds
to the ``-a'' command-line option.
- input-file
- This key requires a pathname value.
It specifies the pathname (relative to the backlog-directory) that should
be read in funnel-file mode. This corresponds to giving a filename as an
argument on the command-line (i.e. its presence also implies that funnel-file
mode should be used).
- pid-file
- This key requires a pathname value. It specifies
the pathname (relative to the backlog-directory) where the pid of the innfeed
process should be stored. This corresponds to the ``-p'' command-line option.
- debug-level
- This key defines the debug level for the process. A non-zero number generates
a lot of messages to stderr, or to the config-defined ``log-file''. This corresponds
to the ``-d'' command-line option.
- use-mmap
- This key requires a boolean value. It
specifies whether mmaping should be used if innfeed has been built with
mmap support. If article data on disk is not in NNTP-ready format (CR/LF
at the end of each line), then after mmaping the article is read into memory
and fixed up, so mmaping has no positive effect (and possibly some negative
effect depending on your system), and so in such a case this value should
be false. This corresponds to the ``-M'' command-line option.
- log-file
- This key
requires a pathname value. It specifies where any logging messages that
couldn't be sent via syslog(3)
should go (such as those generated when a
positive value for ``debug-value'', is used). This corresponds to the ``-l'' command-line
option. A relative pathname is relative to the ``backlog-directory'' value.
- backlog-directory
- This key requires a pathname value. It specifies
where the current innfeed process should store backlog files. This corresponds
to the ``-b'' command-line option.
- backlog-highwater
- This key requires a positive
integer value. It specifies how many articles should be kept on the backlog
file queue before starting to write new entries to disk.
- backlog-ckpt-period
- This key requires a positive integer value. It specifies how many seconds
between checkpoints of the input backlog file. Too small a number will mean
frequent disk accesses, too large a number will mean after a crash innfeed
will re-offer more already-processed articles than necessary.
- backlog-newfile-period
- This key requires a positive integer value. It specifies how many seconds
between checks for externally generated backlog files that are to be picked
up and processed.
- dns-retry
- The key requires a positive integer value. It
defines the number of seconds between attempts to re-lookup host information
that previous failed to be resolved.
- dns-expire
- The key requires a positive
integer value. It defines the number of seconds between refreshes of name
to address DNS translation. This is so long running processes don't get stuck
with stale data, should peer ip addresses change..
- close-period
- The key requires
a positive integer value. It is the maximum number of seconds a connection
should be kept open. Some NNTP servers don't deal well with connections being
held open for long periods.
- gen-html
- This key requires a boolean value. It
specifies whether the status-file should be HTML-ified.
- status-file
- This key
requires a pathname value. It specifies the pathname (relative to the backlog-directory)
where the periodic status of the innfeed process should be stored. This
corresponds to the ``-S'' command-line option.
- connection-stats
- This key requires
a boolean value. If the value is true, then whenever the transmission statistics
for a peer are logged, then each active connection logs its own statistics.
This corresponds to the ``-z'' command-line option.
- host-queue-highwater
- This key
requires a positive integer value. It defines how many articles will be
held internally for a peer before new arrivals cause article information
to be spooled to the backlog file.
- stats-period
- This key requires a positive
integer value. It defines how many seconds innfeed waits between generating
statistics on transfer rates.
- stats-reset
- This key requires a positive integer
value. It defines how many seconds innfeed waits before resetting all internal
transfer counters back to zero (after logging one final time). This is so
a innfeed-process running more than a day will generate ``final'' stats that
will be picked up by logfile processing scripts.
- initial-reconnect-time
- This
key requires a positive integer value. It defines how many seconds to first
wait before retrying to reconnect after a connection failure. If the next
attempt fails too, then the reconnect time is approximately doubled until
the connection succeeds, or max-reconnection-time is reached.
- max-reconnect-time
- This key requires an integer value. It defines the maximum number of seconds
to wait between attempt to reconnect to a peer. The initial value for reconnection
attempts is defined by initial-reconnect-time, and it is doubled after each
failure, up to this value.
- stdio-fdmax
- This key requires a non-negative integer
value. If the value is greater than zero, then whenever a network socket
file descriptor is created and it has a value less than this, the file
descriptor will be dup'ed to bring the value up greater than this. This is
to leave lower numbered file descriptors free for stdio. Certain systems,
Sun's in particular, require this. SunOS 4.1.x usually requires a value of
128 and Solaris requires a value of 256. The default if this is not specified,
is 0.
- bindaddress
- Which interface IP address innfeed should bind the local
end of its connections to. Must be in dotted-quad format (nnn.nnn.nnn.nnn). If
not set, innfeed defaults to letting the kernel choose this address. The
default value is unset.
All the key/value pairs mentioned
in this section must be specified at global scope. They may also be specified
inside a group or peer definition. Note that when peers are added dynamically
(i.e. when innfeed receives an article for an unspecified peer), it will
add the peer site using the parameters specified at global scope.
- article-timeout
- This key requires a non-negative integer value. If no articles need to be
sent to the peer for this many seconds, then the peer is considered idle
and all its active connections are torn down.
- response-timeout
- This key requires
a non-negative integer value. It defines the maximum amount of time to wait
for a response from the peer after issuing a command.
- initial-connections
- This key requires a non-negative integer value. It defines the number of
connections to be opened immediately when setting up a peer binding. A value
of 0 means no connections will be created until an article needs to be
sent.
- max-connections
- This key requires positive integer value. It defines
the maximum number of connections to run in parallel to the peer. A value
of zero specifies an unlimited number of maximum connections. In general
use of an unlimited number of maximum connections is not recommended. Do
not ever set max-connections to zero with dynamic-method 0 set, as this will
saturate peer hosts with connections. [ Note that in previous versions of
innfeed, a value of 1 had a special meaning. This is no longer the case,
1 means a maximum of 1 connection ].
- dynamic-method
- This key requires a value
between 0 and 3. It controls how connections (up to max-connections) are
opened up to the maximum specified by max-connections. In general (and specifically,
with dynamic-method 0) a new connection is opened when the current number
of connections is below max-connections, and an article is to be sent whilst
no current connections are idle. Without further restraint (i.e. using dynamic-method
0), in practice this means that max-connections connections are established
whilst articles are being sent. Use of other dynamic-method settings imposes
a further limit on the amount of connections opened below that specified
by max-connections. This limit is calculated in different ways, depending
of the value of dynamic-method. Users should note that adding additional
connections is not always productive - just because opening twice as many
connections results in a small percentage increase of articles accepted
by the remote peer, this may be at considerable resource cost both locally
and at the remote site, whereas the remote site might well have received
the extra articles sent from another peer a fraction of a second later.
Opening large numbers of connections is considered antisocial. The meanings
of the various settings are:
- 0 no method
- Increase of connections up to
max-connections is unrestrained.
- 1 maximize articles per second
- Connections
are increased (up to max-connections) and decreased so as to maximize the
number of articles per second sent, whilst using the fewest connections
to do this.
- 2 set target queue length
- Connections are increased (up to max-connections)
and decreased so as to keep the queue of articles to be sent within the
bounds set by dynamic-backlog-low and dynamic-backlog-high, whilst using the
minimum resource possible. As the queue will tend to fill if the site is
not keeping up, this method ensures that the maximum number of articles
are offered to the peer whilst using the minimum number of connections
to achieve this.
- 3 combination
- This method uses a combination of methods
1 and 2 above. For sites accepting a large percentage of articles, method
2 will be used to ensure these sites are offered as complete a feed as
possible. For sites accepting a small percentage of articles, method 1 is
used, to minimize remote resource usage. For intermediate sites, an appropriate
combination is used.
- dynamic-backlog-low
- This key takes a value between 0
and 100 and represents (as a percentage) the low water mark for the host
queue. When the host queue falls below this level, when using dynamic-method
2 or 3, if 2 or more connections are open, innfeed will attempt to drop
connections to the host. An IIR filter is applied to the value to prevent
connection flap (see dynamic-filter). A value of 25.0 is recommended. This
value must be smaller than dynamic-backlog-high.
- dynamic-backlog-high
- This key
takes a value between 0 and 100 and represents (as a percentage) the high
water mark for the host queue. When the host queue rises above this level,
when using dynamic-method 2 or 3, if less than max-connections are open to
the host, innfeed will attempt to open further connections to the host.
An IIR filter is applied to the value to prevent connection flap (see dynamic-filter).
A value of 50.0 is recommended. This value must be larger than dynamic-backlog-low.
- dynamic-backlog-filter
- This key takes a floating-point value between 0 and
1 which represents the filter coefficient used by the IIR filter used to
implement dynamic-method 2 and 3. The recommended value of this filter is
0.7, giving a time constant of 1/(1-0.7) articles. Higher values will result
in slower response to queue fullness changes, lower values with faster
response.
- max-queue-size
- This key requires a positive integer value. It defines
the maximum number of articles to process at one time when using streaming
to transmit to a peer. Larger numbers mean more memory consumed as articles
usually get pulled into memory (see the description of use-mmap).
- streaming
- This key requires a boolean value. It defines whether streaming commands
are used to transmit articles to the peers.
- no-check-high
- This key requires
a floating-point number which must be in the range [0.0, 100.0]. When running
transmitting with the streaming commands, innfeed attempts an optimization
called ``no-CHECK'' mode. This involves not asking the peer if it wants the article,
but just sending it. This optimization occurs when the percentage of the
articles the peer has accepted gets larger than this number. If this value
is set to 100.0, then this effectively turns off no-CHECK mode, as the percentage
can never get above 100.0. If this value is too small, then the number of
articles the peer rejects will get bigger (and your bandwidth will be wasted).
A value of 95.0 is usually pretty good. NOTE: In innfeed 0.9.3 and earlier
this value was in the range [0.0, 9.0].
- no-check-low:
- This key requires a floating-point
number which must be in the range [0.0, 100.0), and it must be smaller that
the value for no-check-high. When running in no-CHECK mode, as described above,
if the percentage of articles the remote accepts drops below this number,
then the no-CHECK optimization is turned off until the percentage gets above
the no-check-high value again. If there is small difference between this and
the no-check-high value (less than about 5.0), then innfeed may frequently
go in and out of no-CHECK mode. If the difference is too big, then it will
make it harder to get out of no-CHECK mode when necessary (wasting bandwidth).
Keeping this to between 5.0 and 10.0 less than no-check-high is usually pretty
good.
- no-check-filter
- This is a floating point value representing the time
constant, in articles, over which the CHECK / no-CHECK calculations are
done. The recommended value is 50.0 which will implement an IIR filter of
time constant 50. This roughly equates to making a decision about the mode
over the previous 50 articles. A higher number will result in a slower response
to changing percentages of articles accepted; a lower number will result
in a faster response.
- port-number
- This key requires a positive integer value.
It defines the tcp/ip port number to use when connecting to the remote.
- drop-deferred
- This key requires a boolean value. By default it is set to
false. When set to true, and a peer replies with code 431 or 436 (try again
later) just drop the article and don't try to re-send it. This is useful for
some peers that keep on deferring articles for a long time to prevent innfeed
from trying to offer the same article over and over again.
- min-queue-connection
- This key requires a boolean value. By default it is set to false. When set
to true, innfeed will attempt to use a connection with the least queue
size (or the first empty connection). If this key is set to true, it is
recommended that dynamic-method be set to 0. This allows for article propagation
with the least delay.
- no-backlog
- This key requires a boolean value. It specifies
whether spooling should be enabled (false, the default) or disabled (true).
Note that when no-backlog is set, articles reported as "spooled" are actually
silently discarded.
- backlog-limit
- This key requires a non-negative integer
value. If the number is 0 then backlog files are allowed to grown without
bound when the peer is unable to keep up with the article flow. If this
number is greater than 0 then it specifies the size (in bytes) the backlog
file should get truncated to when the backlog file reaches a certain limit.
The limit depends on whether backlog-factor or backlog-limit-high is used.
- backlog-factor
- This key requires a floating point value, which must be larger
than 1.0. It is used in conjunction with the peer key backlog-limit. If backlog-limit
has a value greater than zero, then when the backlog file gets larger than
the value backlog-limit * backlog-factor, then the backlog file will be truncated
to the size backlog-limit. For example if backlog-limit has a value of 1000000,
and backlog-factor has a value of 2.0, then when the backlogfile gets to
be larger than 2000000 bytes in size, it will be truncated to 1000000 bytes.
The front portion of the file is removed, and the trimming happens on line
boundaries, so the final size may be a bit less than this number. If backlog-limit-highwater
is defined too, then backlog-factor takes precedence.
- backlog-limit-highwater
- This key requires a positive integer value that must be larger than the
value for backlog-limit. If the size of the backlog file gets larger than
this value (in bytes), then the backlog file will be shrunk down to the
size of backlog-limit. If both backlog-factor and backlog-limit-highwater are
defined, then the value of backlog-factor is used.
As previously
explained, the peer definitions can contain redefinitions of any of the
key/value pairs described in the GLOBAL PEER DEFAULTS section above. There
is one key/value pair that is specific to a peer definition.
- ip-name
- This
key requires a word value. The word is the host's FQDN, or the dotted quad
ip-address. If this value is not specified then the name of the peer is taken
to also be its ip-name. See the entry for data.ramona.vix.com in the example
below.
If innfeed gets a SIGHUP signal, then it will reread the
config file. All values at global scope except for ``backlog-directory'' can
be changed. Any new peers are added and any missing peers have their connections
closed.
Below is the sample innfeed.conf file.
#
# innfeed.conf file. See the comment block at the
# end for a fuller description.
#
##
## Global values. Not specific to any peer. These
## are optional, but if used will override the
## compiled in values. Command-line options used
## will override these values.
##
pid-file: innfeed.pid
debug-level: 0
use-mmap: false
log-file: innfeed.log
stdio-fdmax: 0
backlog-directory: innfeed
backlog-rotate-period: 60
backlog-ckpt-period: 30
backlog-newfile-period: 600
dns-retry: 900
dns-expire: 86400
close-period: 3600
gen-html: false
status-file: innfeed.status
connection-stats: false
host-queue-highwater: 200
stats-period: 600
stats-reset: 43200
max-reconnect-time: 3600
initial-reconnect-time: 30
##
## Defaults for all peers. These must all exist at
## global scope. Any of them can be redefined
## inside a peer or group definition.
##
article-timeout: 600
response-timeout: 300
initial-connections: 1
max-connections: 5
max-queue-size: 25
streaming: true
no-check-high: 95.0
no-check-low: 90.0
no-check-filter: 50.0
port-number: 119
backlog-limit: 0
backlog-factor: 1.10
backlog-limit-highwater:0
dynamic-method: 3
dynamic-backlog-filter: 0.7
dynamic-backlog-low: 25.0
dynamic-backlog-high: 50.0
no-backlog: false
##
## Peers.
##
peer decwrl {
ip-name: news1.pa.dec.com
}
peer uunet {
ip-name: news.uunet.uu.net
max-connections: 10
}
peer data.ramona.vix.com {
# ip-name defaults to data.ramona.vix.com
streaming: false
}
peer bb.home.vix.com {
ip-name: 192.5.5.33
}
# Blank lines are ignored. Everything after a '#'
# is ignored too.
#
# Format is:
# key : value
#
# See innfeed.conf(5) for a description of
# necessary & useful keys. Unknown keys and their
# values are ignored.
#
# Values may be a integer, floating-point, c-style
# single-quoted characters, boolean, and strings.
#
# If a string value contains whitespace, or
# embedded quotes, or the comment character
# (``#''), then the whole string must be quoted
# with double quotes. Inside the quotes, you may
# use the standard c-escape sequence
# (\t,\n,\r,\f,\v,\",\').
#
# Examples:
# eg-string: "NewtConfigtfile0
# eg-long-string: "A long string that goes
# over multiple lines. The
# newline is kept in the
# string except when quoted
# with a backslash
# as here."
# eg-simple-string: A-no-quote-string
# eg-integer: 10
# eg-boolean: true
# eg-char: 'a'
# eg-ctrl-g: ' 07'
Written by James Brister <brister@vix.com> for InterNetNews. This is
revision 1.9.2.1, dated 2001/03/11.
innfeed(1)
, newsfeeds(5)
Table of Contents