=head1 INN Python Filtering and Authentication Support

This is $Revision: 1.6 $, dated $Date: 2003/04/04 16:47:22 $.

This file documents INN's built-in optional support for Python article
filtering.  It is patterned after the TCL and Perl hooks previously added
by Bob Heiney and Christophe Wolfhugel.

For this filter to work successfully, you will need to have Python 1.5.2
(the latest at this writing) installed.  You can obtain it from
<http://www.python.org>.

The innd Python interface and the original Python filtering documentation
was written by Greg Andruk (nee Fluffy) <gerglery@usa.net>.  The Python
authentication and authorization support for nnrpd and the original
documentation for it was written by Ilya Etingof <ilya@glas.net>, 12/1999.

=head1 NOTE TO RED HAT LINUX USERS

Python will be preinstalled, but it may not include all the headers and
libraries required for embedding into INN.  You will need to add the
development package.  Better yet, get the source kit from the above URL
and build it yourself.  Be sure when installing Python on Red Hat, to run
configure with C<--prefix=/usr> so that there are no version conflicts
with the "factory" installation.  You can also find a selection of well
made RPMs at <ftp://starship.python.net/pub/crew/andrich/>.

=head1 INSTALLATION

Once you have built and installed Python, you can cause INN to use it by
adding the C<--with-python> switch to your configure command.

See the ctlinnd(8) manual page to learn how to enable, disable and reload
Python filters on a running server (C<ctlinnd mode>, C<ctlinnd python
y|n>, C<ctlinnd reload filter.python>).

Also, see the example F<filter_innd.py> script in your filters directory
for a demonstration of how to get all this working.

=head1 WRITING AN INND FILTER

You need to create a F<filter_innd.py> module in INN's filter directory
(see the pathfilter setting in F<inn.conf>).  A heavily-commented sample
is provided that you can use as a template for your own filter.  There is
also an F<INN.py> module there which is not actually used by INN; it is
there so you can test your module interactively.

First, define a class containing the methods you want to provide to innd.
Methods innd will use if present are:

=over 4

=item __init__(self)

Not explicitly called by innd, but will run whenever the filter module is
(re)loaded.  This is a good place to initialize constants or pick up where
filter_before_reload or filter_close left off.

=item filter_before_reload(self)

This will execute any time a C<ctlinnd reload all> or C<ctlinnd reload
filter.python> command is issued.  You can use it to save statistics or
reports for use after reloading.

=item filter_close(self)

This will run when a C<ctlinnd shutdown> command is received.

=item filter_art(self, art)

art is a dictionary containing an article's headers and body.  This method
is called every time innd receives an article.  The following can be
defined:
            
    Approved, Control, Date, Distribution, Expires, From, Lines,
    Message-ID, Newsgroups, Path, Reply-To, Sender, Subject,
    Supersedes, Bytes, Also-Control, References, Xref, Keywords,
    X-Trace, NNTP-Posting-Host, Followup-To, Organization,
    Content-Type, Content-Base, Content-Disposition, X-Newsreader,
    X-Mailer, X-Newsposter, X-Cancelled-By, X-Canceled-By, Cancel-Key,
    __LINES__, __BODY__

All the above values will be buffer objects holding the contents of the
same named article headers, except for the special __BODY__ and __LINES__
items.  Items not present in the article will contain None.

__BODY__ is a buffer object containing the article's entire body, and
__LINES__ is an int holding innd's reckoning of the number of lines in the
article.  All the other elements will be buffers with the contents of the
same-named article headers.

If you want to accept an article, return None or an empty string.  To
reject, return a non-empty string.  The rejection strings will be shown to
local clients and your peers, so keep that in mind when phrasing your
rejection responses.

=item filter_messageid(self, msgid)

msgid is a buffer object containing the ID of an article being offered by
IHAVE or CHECK.  Like with filter_art(), the message will be refused if
you return a non-empty string.  If you use this feature, keep it light
because it is called at a rather busy place in innd's main loop.  Also, do
not rely on this function alone to reject by ID; you should repeat the
tests in filter_art() to catch articles sent with TAKETHIS but no CHECK.

=item filter_mode(self, oldmode, newmode, reason)

When the operator issues a ctlinnd pause, throttle or go command, this
function can be used to do something sensible in accordance with the state
change.  Stamp a log file, save your state on throttle, etc.  oldmode and
newmode will be strings containing one of the values in ('running',
'throttled', 'paused', 'unknown') -- oldmode is the state innd was in
before ctlinnd was run, newmode is the state innd will be in after the
command finishes.  reason is the comment string provided on the ctlinnd
command line.

=back

To register your methods with innd, you need to create an instance of your
class, import the built-in INN module, and pass the instance to
INN.set_filter_hook().  For example:

    class Filter:
        def filter_art(self, art):
            ...
            blah blah
            ...

        def filter_messageid(self, id):
            ...
            yadda yadda
            ...

    import INN
    myfilter = Filter()
    INN.set_filter_hook(myfilter)

When writing and testing your Python filter, don't be afraid to make use
of try:/except: and the provided INN.syslog() function.  stdout and stderr
will be disabled, so your filter will die silently otherwise.

Also, remember to try importing your module interactively before loading
it, to ensure there are no obvious errors.  One typo can ruin your whole
filter.  A dummy INND.py module is provided to facilitate testing outside
the server.  To test, change into your filter directory and use a command
like:

    python -ic 'import INN, filter_innd'

You can define as many or few of the methods listed above as you want in
your filter class (it's fine to define more methods for your own use; innd
won't use them but your filter can).  If you I<do> define the above
methods, GET THE PARAMETER COUNTS RIGHT.  There are checks in innd to see
if the methods exist and are callable, but if you define one and get the
parameter counts wrong, INND WILL DIE.  You have been warned.  Be careful
with your return values, too.  The filter_art() and filter_messageid()
methods have to return strings, or None.  If you return something like an
int, innd will B<not> be happy.

=head1 WHAT'S THE DEAL WITH THESE BUFFER OBJECTS?

Buffer objects are cousins of strings, new in Python 1.5.2.  They are
supported, but at this writing you won't yet find much about them in the
Python documentation.  Using buffer objects may take some getting used to,
but we can create buffers much faster and with less memory than strings.

For most of the operations you will perform in filters (like re.search,
string.find, md5.digest) you can treat buffers just like strings, but
there are a few important differences you should know about:

    # Make a string and a two buffers.
    s = "abc"
    b = buffer("def")
    bs = buffer("abc")

    s == bs          # - This is false because the types differ...
    buffer(s) == bs  # - ...but this is true, the types now agree.
    s == str(bs)     # - This is also true, but buffer() is faster.
    s[:2] == bs[:2]  # - True.  Buffer slices are strings.

    # While most string methods will take either a buffer or string,
    # string.join insists on using only strings.
    string.join([str(b), s], '.')   # returns 'def.abc'

    e = s + b        # This raises a TypeError, but...

    # ...these two both return the string 'abcdef'. The first one
    # is faster -- choose buffer() over str() whenever you can.
    e = buffer(s) + b
    f = s + str(b)

    g = b + '>'      # This is legal, returns the string 'def>'.

=head1 FUNCTIONS SUPPLIED BY THE BUILT-IN INND MODULE

Not only can innd use Python, but your filter can use some of innd's
features too.  Here is some sample Python code to show what you get:

    import INN

    # Python's native syslog module isn't compiled in by default,
    # so the INN module provides a replacement.  The first parameter
    # tells the Unix syslogger what severity to use; you can
    # abbreviate down to one letter and it's case insensitive.
    # Available levels are (in increasing levels of seriousness)
    # Debug, Info, Notice, Warning, Err, Crit, and Alert. (If you
    # provide any other string, it will be defaulted to Notice.)  The
    # second parameter is the message text.  The syslog entries will
    # go to the same log files innd itself uses, with a 'python:'
    # prefix.
    syslog('warning', 'I will not buy this record.  It is scratched.')
    animals = 'eels'
    vehicle = 'hovercraft'
    syslog('N', 'My %s is full of %s.' % (vehicle, animals))

    # Let's cancel an article!  This only deletes the message on the
    # local server; it doesn't send out a control message or anything
    # scary like that.  Returns 1 if successful, else 0.
    if INN.cancel('<meow$123.456@solvangpastries.edu>'):
        canceled = "yup"
    else:
        canceled = "nope"

    # Check if a given message is in history. This doesn't
    # necessarily mean the article is on your spool; canceled and
    # expired articles hang around in history for a while, and
    # rejected articles will be in there if you have enabled
    # remember_trash in inn.conf. Returns 1 if found, else 0.
    if INN.havehist('<z456$789.abc@isc.org>'):
        comment = "*yawn* I've already seen this article."
    else:
        comment = 'Mmm, fresh news.'

    # Here we are running a local spam filter, so why eat all those
    # cancels?  We can add fake entries to history so they'll get
    # refused.  Returns 1 on success, 0 on failure.
    canceled_id = buffer('<meow$123.456@isc.org>')
    if INN.addhist("<cancel." + canceled_id[1:]):
        thought = "Eat my dust, roadkill!"
    else:
        thought = "Darn, someone beat me to it."

    # We can look at the header or all of an article already on spool,
    # too.  Might be useful for long-memory despamming or
    # authentication things.  Each is returned (if present) as a
    # string object; otherwise you'll end up with an empty string.
    artbody = INN.article('<foo$bar.baz@bungmunch.edu>')
    artheader = INN.head('<foo$bar.baz@bungmunch.edu>')

    # Finally, do you want to see if a given newsgroup is moderated or
    # whatever?  INN.newsgroup returns the last field of a group's
    # entry in active as a string.
    froupflag = INN.newsgroup('alt.fan.karl-malden.nose')
    if froupflag == '':
        moderated = 'no such newsgroup'
    elif froupflag == 'y':
        moderated = "nope"
    elif froupflag == 'm':
        moderated = "yep"
    else:
        moderated = "something else"

=head1 CHANGES TO PYTHON AUTHENTICATION AND ACCESS CONTROL SUPPORT FOR
NNRPD

The old authentication and access control functionality has been
combined with the new readers.conf mechanism by Erik Klavon
<erik@eriq.org>; bug reports should however go to inn-bugs@isc.org,
not Erik.

The remainder of this section is an introduction to the new mechanism
(which uses the python_auth, python_access, and python_dynamic
F<readers.conf> parameters) with porting/migration suggestions for
people familiar with the old mechanism (identifiable by the now
deprecated nnrpperlauth: parameter in F<inn.conf>).

Other people should skip this section.

The python_auth parameter allows the use of Python to authenticate a
user. Authentication scripts (like those from the old mechanism) are
listed in F<readers.conf> using python_auth in the same manner other
authenticators are using auth:

        python_auth: "auth1.py"

Scripts should be placed as before in the filter directory (see the
I<pathfilter> setting in F<inn.conf>). The new hook method authen_init
takes no arguments and its return value is ignored; its purpose is to
provide a means for authentication specific initialization. The hook
method authen_close is the more specific analogue to the old close
method. These method hooks are not required.

The argument dictionary passed to authenticate remains the same,
except for the removal of the C<type> entry which is no longer needed
in this modification and the addition of several new entries (port,
intipaddr, intport) described below. The return tuple now only
contains either two or three elements, the first of which is the NNTP
response code. The second is an error string which is passed to the
client if the response code indicates that the authentication attempt
has failed. This allows a specific error message to be generated by
the Python script in place of the generic message "Authentication
failed". An optional third return element, if present, will be used to
match the connection with the user: parameter in access groups and
will also be the username logged. If this element is absent, the
username supplied by the client during authentication will be used, as
was the previous behavior.

The python_access parameter (described below) is new; it allows the
dynamic generation of an access group of an incoming connection using
a Python script. If a connection matches an auth group which has a
python_access parameter, all access groups in readers.conf are
ignored; instead the procedure described below is used to generate an
access group. This concept is due to Jeffery M. Vinocur.

In the old implementation, the authorization method allowed for access
control on a per-group basis. That functionality is preserved in the
new implementation by the inclusion of the python_dynamic parameter in
F<readers.conf>. The only change is the corresponding method name of
C<dynamic> as opposed to C<authorize>; C<domain> and C<range> are the
same as before except for the addition of several new entries to the
argument dictionary (port, intipaddr, intport) described
below. Additionally, the associated optional housekeeping methods
dynamic_init and dynamic_close may be implemented if needed.

This new implementation should provide all of the previous
capabilities of the Python hooks, in combination with the flexibility
of readers.conf and the use of other authentication and resolving
programs (including the Perl hooks!). To use Python code that predates
the new mechanism, you would need to modify the code slightly (see
below for the new specification) and supply a simple readers.conf
file. If you don't want to modify your code, the sample directory has
F<nnrpd_auth_wrapper.py>, F<nnrpd_access_wrapper.py>, and
F<nnrpd_dynamic_wrapper.py> which should allow you to use your old
code without needing to change it.

However, before trying to use your old Python code, you may want to
consider replacing it entirely with non-Python authentication. (With
readers.conf and the regular authenticator and resolver programs, much
of what once required Python can be done directly.)  Even if the
functionality is not available directly, you may wish to write a new
authenticator or resolver (which can be done in whatever language you
prefer).

=head1 PYTHON AUTHENTICATION SUPPORT FOR NNRPD

Python authentication, dynamic access group generation and dynamic
access control support in nnrpd along with filtering support in innd
may be compiled in by passing C<--with-python> C<configure>.

Support for authentication via Python is provided in nnrpd by the
inclusion of a python_auth: parameter in a F<readers.conf> auth
group. python_auth: works exactly like the auth: parameter in
F<readers.conf>, except that it calls the script given as argument
using the Python hook rather then treating it as an external
program. Multiple, mixed use of python_auth: with other auth:
statements including perl_auth: is permitted. Each auth: statement
will be tried in the order they appear in the auth group until either
one succeeds or all are exhausted.

If the processing of readers.conf requires that a python_auth:
statement be used for authentication, Python is loaded (if it has yet
to be) and the file given as argument to the python_auth: parameter is
loaded as well. If a Python object with a method authen_init is hooked in
during the loading of that file, then that method is called
immediately after the file is loaded. If no errors have occurred, the
method authenticate is called. Depending on the NNTP response code
returned by authenticate, the authentication hook either succeeds or
fails, after which the processing of the auth group continues as
usual. When the connection with the client is closed, the method
authen_close is called if it exists.

=head1 DYNAMIC GENERATION OF ACCESS GROUPS

A Python script may be used to dynamically generate an access group
which is then used to determine the access rights of the client. This
occurs whenever the python_access: parameter is specified in an auth group
which has successfully matched the client. Only one python_access:
statement is allowed in an auth group. This parameter should not be
mixed with a perl_access: statement in the same auth group.

When a python_access: parameter is encountered, Python is loaded (if
it has yet to be) and the file given as argument is loaded as well. If
a Python object with a method C<access_init> is hooked in during the
loading of that file, then that method is called immediately after the
file is loaded. If no errors have occurred, the method C<access> is
called. The dictionary returned by C<access> is used to generate an
access group that is then used to determine the access rights of the
client. When the connection with the client is closed, the method
C<access_close> is called, if it exists.

While you may include the users: parameter in a dynamically generated
access group, some care should be taken (unless your pattern is just
* which is equivalent to leaving the parameter out). The group created
with the values returned from the Python script is the only one
considered when nnrpd attempts to find an access group matching the
connection. If a users: parameter is included and it doesn't match the
connection, then the client will be denied access since there are no
other access groups which could match the connection.

=head1 DYNAMIC ACCESS CONTROL

If you need to have access control rules applied immediately without
having to restart all the nnrpd processes, you may apply access
control on a per newsgroup basis using the Python dynamic hooks (as
opposed to F<readers.conf>, which does the same on per user
basis). These hooks are activated through the inclusion of the
python_dynamic: parameter in a F<readers.conf> auth group. Only one
python_dynamic: statement is allowed in an auth group.

When a python_dynamic: parameter is encountered, Python is loaded (if
it has yet to be) and the file given as argument is loaded as well. If
a Python object with a method C<dynamic_init> is hooked in during the
loading of that file, then that method is called immediately after the
file is loaded. Every time a reader asks nnrpd to read or post an
article, the Python method C<dynamic> is invoked before proceeding with
the requested operation. Based on the value returned by C<dynamic>, the
operation is either permitted or denied. When the connection with the
client is closed, the method C<access_close> is called if it exists.

=head1 WRITING A PYTHON NNRPD AUTHENTICATION MODULE

You need to create a F<nnrpd_auth.py> module in INN's filter directory
(see the I<pathfilter> setting in F<inn.conf>) where you should define a
class holding certain methods depending on which hooks you want to use.

The following methods are known to nnrpd:

=over 4

=item __init__(self)

Not explicitly called by nnrpd, but will run whenever the auth module is
loaded. Use this method to initialize any general variables or open
a common database connection. This method may be omitted.

=item authen_init(self)

Initialization function specific to authentication. This method may be
omitted.

=item authenticate(self, attributes)

Called when a python_auth statement is reached in the processing of
readers.conf. Connection attributes are passed in the C<attributes>
dictionary. Returns a response code, an error string, and an optional
string to be used in place of the client-supplied username (both for
logging and for matching the connection with an access group).

=item authen_close(self)

This method is invoked on nnrpd termination.  You can use it to save
state information or close a database connection.

=item access_init(self)

Initialization function specific to generation of an access group. This
method may be omitted.

=item access(self, attributes)

Called when a python_access statement is reached in the processing of
readers.conf. Connection attributes are passed in the C<attributes>
dictionary. Returns a dictionary of values representing statements to
be included in an access group.

=item access_close(self)

This method is invoked on nnrpd termination. You can use it to save
state information or close a database connection.

=item dynamic_init(self)

Initialization function specific to dynamic access control. This
method may be omitted.

=item dynamic(self, attributes)

Called when a client requests a newsgroup, an article or attempts to
post. Connection attributes are passed in the C<attributes> dictionary.
Returns C<None> to grant access, or a non-empty string (which will be
reported back to the client) otherwise.

=item dynamic_close(self)

This method is invoked on nnrpd termination.  You can use it to save
state information or close a database connection.

=item attributes dictionary

The keys and associated values of the C<attributes> dictionary are
described below.

=over 4

=item type

C<read> or C<post> values specify the authentication
type; only valid for the C<dynamic> method

=item hostname

resolved hostname (or IP address if resolution fails) of connected
reader

=item ipaddress

IP address of connected reader

=item port

port of connected reader

=item interface

hostname of the local endpoint of the NNTP connection

=item intipaddr

IP address of the local endpoint of the NNTP connection

=item intport

port of the local endpoint of the NNTP connection

=item user

username as reader passed with AUTHINFO command, or None if not
applicable

=item pass

password as reader passed with AUTHINFO command, or None if not
applicible

=item newsgroup

name of the newsgroup reader requests read or post access to; only
valid for the C<dynamic> method

=back

All the above values are buffer objects (see the notes above on what
buffer objects are).

=back

To register your methods with nnrpd, you need to create an instance of
your class, import the built-in nnrpd module, and pass the instance to
nnrpd.set_auth_hook().  For example:

    class AUTH:
        def authenticate(self, attributes):
            ...

        def authorize(self, attributes):
            ...

    import nnrpd
    myauth = AUTH()
    nnrpd.set_auth_hook(myauth)

There is also an F<nnrpd.py> module there that is not actually used by
nnrpd but provides the same set of functions as built-in nnrpd module.
This stub module may be used when debugging your own module.

=head1 FUNCTIONS SUPPLIED BY THE BUILT-IN NNRPD MODULE

As of this writing, nnrpd built-in module exports the following functions:

=over 4

=item set_auth_hook()

used to pass a reference to the instance of authentication and
authorization class to nnrpd

=item syslog()

intended to be a replacement for a Python native syslog

=back

=cut
