I Developed this commandline tool to make it easier to maintain stuff on a groing amount of servers. nexec is a tool that can help you gather ad-hoc info when it’s needed quickly. nexec is written in python, it depends on codespeak’s pylib, and does all it’s communication over SSH. The only thing you need on ‘the other side’ is a python interpreter. (that’s right, no further libs or modules). Oh, and you need your SSH key pairs in place, otherwise it’s ‘password galore’.

Of course, the bash-ers out there would say that the same functionality can be achieved with a for loop. True! But that’s exactly what nexec does for you so you can concentrate on what you’re doing remote. And ofcourse you can have nightly cron-ed scripts in place that do this every day. But nexec aids in all those things that are not ‘in place’.

Examples

Before getting into any details, first some tricks of the ‘nexec’ trade to get you warmed up …

Are my server systems running the proper time:

$ ./nexec.py -n 'server1 server2  server3' -c 'date'
server2: Fri May 23 16:21:18 CEST 2008
server1: Fri May 23 16:21:17 CEST 2008
server3: Fri May 23 16:21:19 CEST 2008

See if all your servers are running ssh protocol 2 only:

$ ./nexec.py -n 'server1 server2  server3' -c 'cat /etc/ssh/sshd_config | grep -v ^# | grep Protocol' | sort
server1: Protocol 2
server2: Protocol 2
server3: Protocol 2

(You could also do a “-n ALL”, then nexec reads your server list from a config file. More on that in a minute)

Get a list with all different kernel versions that are running on your systems:

$ ./nexec.py -n ALL -q -c 'uname -r' | sort -u
2.6.18-5-686
2.6.18.8-xen

(The “-q” parameter suppresses the ‘per line’ hostname prefix).

See if ‘john’ is a member of the sudo group on servers 1 and 2:

./nexec.py -v -q -n 'server1 server2' -c 'cat /etc/group | grep sudo | grep -v john'
Host:  server1.example.com
sudo:x:27:jane,jerry
Host:  server2.example.com
sudo:x:27:jane

(The”-v” adds the “Host: ….” header, per server output. Note that “-v” and “-q” are interchangeable)

As you can see, by combining local and remote greps and using the -v and/or -q parameters there are some interesting ways to obtain live data from your server.

Lets look at the configuration.

Configuration

Run without parms, nexec shows help output. Furthermore the only two files nexec depends on (and not even necessarily) are a “~/.nexec.conf” file and a possible “~/.ssh/config”.

The “~/.nexec.conf” does not contain much, just “key: value” pairs under a hosts section. The actual ‘value’ is only obtained when using the “-v” option to display the header above each server output “Host: server1.example.com“.

$ cat .nexec.conf
[hosts]
server1:   server1.example.com
server2:   server2.example.com
#server3:   server3.example.com

Other then that, nexec simply expands “-c ALL” to all available keys under the “[hosts]” section, skipping the usual hashsign ‘#’ being a comment. In this case the “ALL” parameter given to “-c” would be expanded to “server1 server2″ in the previous example. After that the full name is taken from the “~.ssh/config” file.

$ cat .ssh/config
# server1
Host server1
Hostname server1.example.com
User john
Port 4321
# server2
Host server2
Hostname server2.example.com
User john
Port 4321

I deliberately left all that in there so other usernames and port numbers can be specified. Besides, that’s all access related and shouldn’t be anywhere else anyway.

Then one more note on the “-q” and “-v” option. What they basically do is mark (or suppress mark’s) on the output that get’s returned. So you can mark output per server adding the “-v”, and output get’s marked per line (default) if not suppressed with “-q”.

Download

The script can be downloaded here: nexec-10.tgz

Set the ‘execute’ bit with chmod and put it in your $PATH somewhere. And for the “ALL” parameter, set up the config file as the example shows in the Configuration section.

Gotcha’s

Escape your backticks with with a backslash if you want remote expansion.

This runs the uname part locally and then sends the command to the other side.

$ nexec.py -n ALL -c "date | mailx -s `uname -n` root"

This also executes the uname command on the other side.

$ nexec.py -n ALL -c "date | mailx -s \`uname -n\` root"

Todo

There’s two things still on my list.

• Catch a non-existing hostname when taken from the “.ssh/config” file. The script breaks if a host is not resolvable. (Being a sysadmin you should know what your doing ..
• Make the “Host: server1.example.com” header not show when no output is returned from a host. That is when using the “-c ALL”.

New tricks might be added in the future, so if there’s stuff that you think should be added drop me a reply.

That’s all folks …

You can of course create distributable modules or packages but with simple local workstation like maintenance this seems overkill … yep, you are allowed to disagree

So, we need a place to put code that is easy to import in our used scripts and it should be easy to maintain. Dropping it in my homedir in a directory called “.python” seems well suited for this.

The Setup

Create the directory “.python” in your $HOMEDIR. Then add a line to your “.bashrc” to export $PYTHONPATH so when you do an import in python it is actually found.

PYTHONPATH=~/.python
export PYTHONPATH

Python picks up the paths from this variable and adds them to “sys.path” so there’s no need to add the path in every script you make. In individual cases you can of course do it like it is explained here.

Then create a file in “~/.python/” for instance “userconfig.py”.

To make it available in your script simply import it and execute one of your functions.

import userconfig as mystuff
mystuff.function(with, parms)

I’ll give an example of a userconfig.py function to easily send out emails.

import os
import sys
import pynotify
import smtplib
from email.mime.text import MIMEText

# Settings << These are the "Sensible defaults" 
MAILSERVER = 'localhost'
MAILFROM = 'somebody'
MAILTO = 'you@yourdomain.com'

def send_mail(subject, body, fro=MAILFROM, to=MAILTO):
    """
    Stripped function to send mail from local code
    """
    msg = MIMEText(body)
    msg['Subject'] = subject
    msg['From'] = fro
    msg['To'] = to

    s = smtplib.SMTP(MAILSERVER)
    s.sendmail(msg['From'], msg['To'], msg.as_string())
    s.close()

The code above composes and sends an email. Doing all this in the script you are writing results in quite a bunch of these lines ending up in every one of your scripts.

Sensible defaults

How about not having to enter those ‘always the same defaults’? Well … When using the setup described in this article most can be omitted and parameters only need to be added when you deviate from your sensible defaults.

For argument sake I’ll give an example script that just mails something and calls the “send_mail()” function.

import userconfig as user

body = """Some multiline triple quoted body mass\n\nwith lot's of

funky stuff in it."""

fish = 'whale'
subject = 'something fishy: %s' % fish

user.send_mail(subject, body)

So that’s only 1 line for the actual ‘send my mail’ command. Should you want a different FROM address then you just change the function call to this.

user.send_mail(subject, body, 'notme')

Thus, add parms when you need them or rely on your “Sensible Defaults” from your userconfig.py.

You can build a world of functionality in your “~/.python”. Import and use system wide installed packages in there, test if they are present on your system, etc.

Gotcha’s

When using scripts under CRON you need to add the PYTHONPATH to the definition like below because CRON does not read your “.bashrc”.

*/1 * * * *  (PYTHONPATH=~/.python; /path/to/your/script.py)

NB: Mind the parenthesis and semicolon on the line above.

Besides the fact that you should watch for name clashes, I can’t think of any other gotcha’s. If you do, then please teach me something and drop me a line

GrtzG

SELECT id, name FROM myapp_model WHERE owner_id='1' GROUP BY name;

This would have been (easily) possible except for the WHERE clause on ownership that screwed me. Since the SQL statement was that simple I decided to write a bypass function that get’s the data straight from the database. It’s readonly access and I dont do anything more with it then create a list of links. Even more so, from a KISS point of view this code is easier to read back after several months.

And for reusability I turned it in to a function. It takes the model and the distinct field, and for my purpose the needed ownership as parameter:

from django.db import connection

def get_latest_objects(model_name=None, distinct_field=None, user_id=None):
    """ Get lastest distinct selection (as tuples) of a given model
    """
    model_name = model_name.lower()
    query = ("select id,%(distinct_field)s from myapp_%(model_name)s "
                "where owner_id='%(user_id)s' "
                "group by %(distinct_field)s;") % {
                    'model_name': model_name,
                    'distinct_field': distinct_field,
                    'user_id': user_id,
                    }
    cursor = connection.cursor()
    cursor.execute(query)

There’s not much to it, I just hope it helps you to avoid ‘the hard way’. And it helps to keeps your view methods more readable because you dont need anything more then this:

latest_objects = get_latest_objects(model_name='MyModel',
                        distinct_field='name',
                        user_id=request.user.id)

What is returned (latest_objects) is a list of tuples that you can unpack in your templates straight away. Note that there’s no error checking in there, because I know what I’m doing Besides, worst case you get back an empty list and possible errors are caught when you write your tests. You do write those don’t you?

Grtz Gerard.

But since wrongfull data disclosure is absolutely unacceptable I was still afraid that I would miss something somewhere. A nice example I ran into was populating a dropdown list in a form, where all records were visible instead of only those owned by the logged in user.

That got me thinking and eventually I wrote this small but sweet piece of middleware. Further elaboration below the code.

from django.db import connection
import re

"""
QueryScreener is a middleware development tool. This tool helps to avoid
unwanted data disclosure once you go into production.

It monitors queries to the models in your model_list and warns you when queries
are executed that do not contain a ownership where clause. And thus can be a
potential data disclosure hazard.

It requires a owner attribute in your model definition, e.g:

    owner = models.ForeignKey(User, editable=False)

Edit the 'model_list' below for what models should be monitored. And add
QueryScreener to MIDDLEWARE_CLASSES in you settings.py

Note: This can/should only be used while running Django's testserver command
with e.g: ./manage.py runserver 192.168.1.81:8000
"""

class QueryScreener(object):

    model_list = ['myapp_customer', 'myapp_order', 'myapp_product']

    def process_view(self, request, view_func, view_args, view_kwargs):
        if len(connection.queries) > 0:
            query_parse(connection.queries, self.model_list, 'process_view')

    def process_response(self, request, response):
        if len(connection.queries) > 0:
            query_parse(connection.queries, self.model_list, 'process_response')
        return response

def query_parse(self, model_list, caller_process):

    for query in connection.queries:
        for modelname in model_list:
            modelstring = 'FROM `'+modelname

            if re.search(modelstring, query['sql']) and not \
                re.search(r'^SELECT.\(1\).AS', query['sql']):

                reg = re.compile(r'^SELECT.*WHERE.*owner.*(ORDER BY.*)?$',
                                    re.DOTALL)

                if not reg.search(query['sql']):
                    print ('<<< WARNING >>> Query execution without ownership '
                            'clause, called from "' + caller_process + '"')
                    print query['sql']

            if re.search(r'^SELECT.\(1\).AS.`a`.FROM.*WHERE.*$', query['sql']):
                print ('<<< Django Farted >>>')
#                print query['sql']

Update1: The ‘ORDER BY’ in the regex needs to be optional.
Update2: Django does a ‘try update’ in save_base() without owner (seperated the select statement)

The comment in the code above sums up how to get it working. What it does is print a warning and the query in question that does not respect ownership. If enabled while developing just keep track of your console output for:

<<< WARNING >>> Query execution without ownership clause, called from "process_response"

Should you  have suggestion, criticism, or words of admiration then please, do tell me

GrtzG

Today I ‘moved’ the first version of my application from the Django development setup (run via “./manage runserver”) to a lighttpd/fcgi production setup. Been cleaning up code and concluded I wanted the app straight on the ‘/’ or docroot with a simple config like this:

HTTP["host"] == "example.int" {
    fastcgi.server = (
        "/" => (
            "main" => (
                "host" => "192.168.1.2",
                "port" => 8000,
                "check-local" => "disable",
                #"fix-root-scriptname" => "enable",
            )
        ),
    )
}

After a lot of messing around and not getting the aformetioned setup to work I found the ‘PATH_INFO’ problem. The easiest way is to have lighttpd version 1.4.23 running, because then you can enable:

    "fix-root-scriptname" => "enable",

And therewith actually acces your app on “/” instead of “/dd” as shown in the below example under the line with “fastcgi.server (“.

The problem however is that lighttpd version 1.4.23 is not apt-get-able yet and I like my systems clean. So here’s the somewhat cumbersome but functional example that works:

HTTP["host"] == "example.int" {
    fastcgi.server = (
        "/dd" => (
            "main" => (
                "host" => "192.168.1.2",
                "port" => 8000,
                "check-local" => "disable",
            )
        ),
    )

    url.rewrite-once = (
        "^(/.*)$" => "/dd/$1",
    )
}

What happens is that lighttpd passes an incoming request like “/url_to_something” to your fcgi proces, but fcgi is on “/dd/url_to_something”. So with the rewrite rule we make sure the “/dd” prefix is added, other wise it would not be passed to the fcgi handler.

The problem however is that you should modify all your in-app links so they would be prefixed with /dd otherwise you end up with a 404. This can be globally resolved by adding the line below to your Django settings.py

FORCE_SCRIPT_NAME = ""

Django then strips of the “/dd” prefix just before it gets parsed by your django app URL Dispatcher.

So when lighttpd version 1.4.23 is in the debian and/or Ubuntu repository, you apt get it and adjust your setup in 3 places.

Remove the rewrite rule and the “dd” part from your lighttpd host config and disable the FORCE_SCRIPT_NAME in your settings.py. Doing it like this keeps everything running with the least adjustments now and only a little work to be done in the near future.

I found a workaround for python bug 1519638. It most definitely will not solve all of the puzzles out there but it stops breaking the sub method for replacing with the use of backrefs.

The problem

If you would like to replace this:

<label for="author"><small>Name

With this:

<label for="author"><small>Naam

And you’re not sure if the <small> tags is there, you would group the chars “<small>” and use a question mark for making them optional. BTW, running a replace on just “Name” is not allowed because they would mess up other parts of the file in question.

Example updated. Thanx dbr!

The solution

Using a compiled pattern and thus a regex to replace this, a solution might look like this:

reg = re.compile(r'(<label for="author">)(<small>)?(Name)', \
    re.VERBOSE | re.MULTILINE | re.DOTALL)
replace = r'\g<1>\g<2>\g<3>'
search = reg.sub(replace, data)

In this case the replacement string uses backreferences to the groups being the sub expressions within the parenthesis in the search pattern.

The oops

However, if the “<small>” tag is not there the search command raises an exception.

$ python regex.py
Traceback (most recent call last):
  File "regex.py", line 14, in <module>
    search = reg.sub(replace, data)
  File "/usr/lib/python2.5/re.py", line 274, in filter
    return sre_parse.expand_template(template, match)
  File "/usr/lib/python2.5/sre_parse.py", line 793, in expand_template
    raise error, "unmatched group"
sre_constants.error: unmatched group

This happens because the second group represented with “\g<2>” in the replacement string returns a “None” instead of an empty string. That is (seems) the bug.

Solving the oops

This can be resolved by replacing the optional notation “(<small>)?” with an alternation “(|<small>)” because with the “<small>” tag being absent it matches on the empty subexpression. And then it actually returns an empty string so the search command won’t raise the exception.

In other words …

Brief explanation

When doing a search and replace with sub, replace the group represented as optional for a group represented as an alternation with one empty subexpression. So instead of this “(.+?)?” use this “(|.+?)” (without the double quotes).

If there’s nothing matched by this group the empty subexpression matches. Then an empty string is returned instead of a None and the sub method is executed normally instead of raising the “unmatched group” error.

That’s all folks …

Validating an IP address or range with just a regex seems like self castigation. I looked at the source code of iptables and it check’s whether or not 1 octet fits in a byte. With your octet being only valid from 0 up to and 255 it must fit in 1 byte. That method seems ok but when you’re writing interpretable code the interpreter most likely does a better job then you in checking byte lenght’s. If it doesn’t already do it like that when checking int’s like this:

if not (0 <= int(octet) <= 255):

Digesting all that information I wrote the function below that takes an IP address or range and simply returns ‘True’ or ‘False’.

def check_address(address):

    if not (re.search('^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}(|\/\d{1,2})$', address)):
        return False

    if (address.count('/') == 1):
        (ip, mask) = address.split('/')
        if not (0 <= int(mask) <= 32):
            return False
    else:
        ip = address

    for octet in ip.split('.'):
        if not (0 <= int(octet) <= 255):
            return False

    return True

IMHO, it’s very safe and very readable. You know … KISS.

If you have suggestions on how to do this more pythonesque I’m very curious to here them so please drop me a line.

GrtzG

If your coding in python and trying to write utf-8 encoded files, save yourself some time. The ‘watch’ command on Ubuntu Hardy does not display utf-8 characters when watching the contents of a file that contains them.

The file contents with the utf-8 character:

“This line shows a ‘»’ (Guillemot right) and a ‘®’ (Trademark sign).”

The command that doesn’t work:

watch cat /path/to/some/file.txt

Shows this:

Every 5.0s: cat file.txt 						Thu May 22 07:36:22 2008

"This line shows a '' (Guillemot right) and a '' (Trademark sign)."

The command that does work:

while true ; do cat /path/to/some/file.txt ; sleep 2 ; clear ; done

Shows this:

"This line shows a '»' (Guillemot right) and a '®' (Trademark sign)."

So if your learning proper encoding of utf-8 in python this does save you time when you’re troubeshooting your code ..

GrtzG

The script: chkdir.py

Description
The script emails differences on a directory’s listing. So if you’re monitoring changes in a specific directory you can use this script. Run it under CRON every 5 minutes and on any changes you’ll get an email looking like this:

Configuration
Not much. Change the email related variables in the config section in the header of the script. Example:

# Config section
MAILSERVER = 'localhost'
MAILFROM = 'somebody'
MAILTO = 'gerard'

Explanation
The script takes the absolute path to the directory that needs monitoring as the first argument. And an optional second argument called ‘verbose’ if you want specifics. If all goes well during execution, and run without ‘verbose’, the script only outputs via email. The reason? This precludes your CRON daemon from doing unnecessary logging.

The previous listing of a directory, to which the new listing is compared, is stored in a file. The file is given the same name as the monitored directory. It has a suffix called ‘.dirlist’ and the file is stored in the parent directory. See ‘Issues’ at the end for more details.

The script only sends output to STDOUT during the initial stage when you start to monitor a directory. The script cops out when: the parent directory is not writable, the given path does not exist, creating the first snapshot (dirlist) of a directory, etc. Since this is all done from the commandline at first, knowing what’s going at this stage is normal. After that the script turns quiet.

Examples
Some startup failure messages:

$ ./chkdir.py /tmp/
Sorry, I do not have permission to write in parent directory

$ ./chkdir.py /home/gerard/temp/testdir/
No previous listfile, created new one, will do diff on next run

As said, after that it turns quiet, except when using ‘verbose’. Then your commandline output looks like this:

./chkdir.py /home/gerard/temp/testdir verbose
New list
['Screenshot-GP-net \xe2\x80\xba Dashboard.png', 'GP-net \xe2\x80\xba Dashboard boe.png']

Old list
['Screenshot-GP-net \xe2\x80\xba Dashboard.png', 'GP-net \xe2\x80\xba Dashboard boe.png', 'Screenshot-Google.png']

Removed:
set(['Screenshot-Google.png'])

The funky escaped values are unicode utf-8 characters. Like I said if you use verbose you get specifics. Not to worry though, any characters allowed in your file or directory names are correctly displayed when you get the output emailed (assuming your mail client supports them).

Issues
Due to storing the file with the previous list in its parent directory it currently is not possible to monitor the ‘root’ of you file system. A different approach would be to generate an md5 hash of the path and put the ‘.dirlist’ file in “/tmp”. But that would result in people being able to obtain info on file system parts where they might not be allowed themselves.

Any suggestions on a better approach to store the previously made listing are welcome.

Regards,

Gerard.