<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Web-In-Sight &#187; Technical</title>
	<atom:link href="http://web-in-sight.nl/category/technical/feed/" rel="self" type="application/rss+xml" />
	<link>http://web-in-sight.nl</link>
	<description>Inzicht in internet en werken</description>
	<lastBuildDate>Mon, 30 Jan 2012 09:00:02 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.1</generator>
		<item>
		<title>Dirty hack for WordPress to realize a hard return</title>
		<link>http://web-in-sight.nl/2012/01/10/dirty-hack-for-wordpress-to-realize-a-hard-return/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=dirty-hack-for-wordpress-to-realize-a-hard-return</link>
		<comments>http://web-in-sight.nl/2012/01/10/dirty-hack-for-wordpress-to-realize-a-hard-return/#comments</comments>
		<pubDate>Tue, 10 Jan 2012 11:33:19 +0000</pubDate>
		<dc:creator>Gerard</dc:creator>
				<category><![CDATA[All ENGLISH articles]]></category>
		<category><![CDATA[Technical]]></category>
		<category><![CDATA[hack]]></category>
		<category><![CDATA[post]]></category>
		<category><![CDATA[wordpress]]></category>

		<guid isPermaLink="false">http://www.gp-net.nl/?p=78</guid>
		<description><![CDATA[Update: As long as this post hast lasted and thanking everybody for their reply&#8217;s &#8230; In WordPress 3.3.1 the extra &#8216;return&#8217; now stays in your text after saving your pages and posts. &#160; Ever wanted to have an extra white &#8230; <a href="http://web-in-sight.nl/2012/01/10/dirty-hack-for-wordpress-to-realize-a-hard-return/">Lees verder <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p><strong>Update</strong>: As long as this post hast lasted and thanking everybody for their reply&#8217;s &#8230;</p>
<p>In WordPress 3.3.1 the extra &#8216;return&#8217; now stays in your text after saving your pages and posts.</p>
<hr />
<p>&nbsp;</p>
<p>Ever wanted to have an extra white line in your post just because it looks nice? Searched the forums and all you got was replies to go straight into the CSS code? &#8230; But I just want that 1 line d*mned &#8230; <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_wink.gif' alt=';-)' class='wp-smiley' /> </p>
<p>Mind you, It&#8217;s beyond dirty but very effective. Here&#8217;s how:</p>
<p>Put a &#8220;.&#8221; (dot) on a new line and give it the same color as the background.</p>
<p><span style="color: #ffffff;">.</span></p>
<p>Regards,</p>
<p>Gerard.</p>
<p>P.S. noticed the space above &#8220;Regards,&#8221; &#8230; <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> </p>
<div class="AWD_like_button "><iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fweb-in-sight.nl%2F2012%2F01%2F10%2Fdirty-hack-for-wordpress-to-realize-a-hard-return%2F&amp;send=false&amp;layout=button_count&amp;width=&amp;show_faces=false&amp;action=recommend&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:px; height:21px;" allowTransparency="true"></iframe></div>]]></content:encoded>
			<wfw:commentRss>http://web-in-sight.nl/2012/01/10/dirty-hack-for-wordpress-to-realize-a-hard-return/feed/</wfw:commentRss>
		<slash:comments>13</slash:comments>
		</item>
		<item>
		<title>Refactoring apache vhosts</title>
		<link>http://web-in-sight.nl/2011/06/25/refactoring-apache-vhosts/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=refactoring-apache-vhosts</link>
		<comments>http://web-in-sight.nl/2011/06/25/refactoring-apache-vhosts/#comments</comments>
		<pubDate>Sat, 25 Jun 2011 09:17:05 +0000</pubDate>
		<dc:creator>Gerard</dc:creator>
				<category><![CDATA[All ENGLISH articles]]></category>
		<category><![CDATA[Technical]]></category>
		<category><![CDATA[apache]]></category>
		<category><![CDATA[deployment]]></category>
		<category><![CDATA[server]]></category>
		<category><![CDATA[vhost]]></category>

		<guid isPermaLink="false">http://www.gerardjp.com/?p=1623</guid>
		<description><![CDATA[Apache VirtualHost: Seperating network and application data with mod_define for optimal deployment <a href="http://web-in-sight.nl/2011/06/25/refactoring-apache-vhosts/">Lees verder <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p><em>- Apache VirtualHost: Separating network and application data with mod_define for clean deployment -</em></p>
<p>Ever had the pleasure of supporting a development team in their efforts to reach deadlines and therewith smoothing out the deployment process? It can be a complex task when the deployment covers multiple servers at once. From a sysadmin point of view you are concerned with network and naming conventions of your web-, database- and possibly other servers.</p>
<h1>The context</h1>
<p>Nevertheless, you need to communicate with the developers about this without overloading them with network details. Practically a set of server names should be enough for their deployment configuration. Of course I&#8217;m primarily talking about framework based application deployment running on Django, Rails and so on.</p>
<p>When handling SSL only websites there&#8217;s the IP address per website restriction (not taking <a title="Server Name Indication" href="http://en.wikipedia.org/wiki/Server_Name_Indication" target="_blank">SNI</a> into account). Then there&#8217;s the gray area; The configuration parts that are both server and application related. Doing an article on Apache 2.x trickery I&#8217;m talking about the VirtualHost setup, but there are others to consider like logrotation, cron vs. application framework jobs, etc.</p>
<p>Having peeked, this article indeed describes a setup that involves Apache and how to ease the work of the development team. Before taking of I&#8217;d like to add one more detail into the mix. For design reasons I consider it good practice to have a seperate IP to address a server besides the ones you use for your services as web, database, file uploads or any other. This separates traffic to your server for e.g. monitoring, backups, abuse filtering, security, etc. next to your regular service traffic.</p>
<p>Having such a clean setup, but also for security reasons, setting up Apache with 0.0.0.0:443 or *:443 doesn&#8217;t fit in my opinion (better explicit then implicit).</p>
<p>Having set the context I&#8217;ll get into the technical details.</p>
<h1>The setup</h1>
<p>As was written we need a &#8216;VirtualHost&#8217; setup that is transparent to the developers but should contain all information for Apache to operate properly.  Besides necessary modules and what not the snippet below displays the minimal required configuration directives.</p>
<pre>NameVirtualhost 1.2.3.4:443
Listen 1.2.3.4:443

&lt;Virtualhost 1.2.3.4:443&gt;

    # Your vhost config

&lt;/Virtualhost&gt;</pre>
<p><strong>Note</strong>: I&#8217;m putting the listen statement in the virtualhost config file and not in Apache&#8217;s ports.conf to keep things contained with the application&#8217;s configuration. Here&#8217;s the first conflict because this would mean different configs in de developers codebase for every server they deploy on. We&#8217;ll resolve that in a minute.</p>
<p>&nbsp;</p>
<h1>Apache symlink intermezzo</h1>
<p>More on keeping things contained along side your application, mainly for upscale portability, we put the file with Apache&#8217;s vhost config in an etc directory under the application codebase. E.g: [app basedir]/etc/apache.conf</p>
<p>Usually, with Apache, site configurations are stored in /etc/apache2/sites-available and they are enabled when symlinked to from /etc/apache2/sites-enabled.</p>
<p>When putting the vhost configuration under your app codebase you simply link from &#8216;sites-enabled&#8217; to that location.</p>
<pre>$ cd /etc/apache2/sites-enabled
$ ls -l
total 0
lrwxrwxrwx 1 root root   74 2011-05-04 19:47 www.example.com -&gt; /home/appuser1/sites/www.example.com/config/etc/apache.conf
lrwxrwxrwx 1 root root   79 2011-05-04 19:48 www.other.com -&gt; /home/appuser2/sites/www.other.com/config/etc/apache.conf
lrwxrwxrwx 1 root root   71 2011-06-10 15:22 another.other.com -&gt; /home/appuser2/sites/another.other.com/config/etc/apache.conf</pre>
<p>The thing to lookout for when running multiple apps: the symlink sourcename must be set different from the name &#8216;apache.conf&#8217; otherwise you get a name conflict in the /etc/apache2/sites-enabled directory. This is done with the &#8216;ln&#8217; command in de following manor:</p>
<pre>$ cd /etc/apache2/sites-enabled
$ ln -s /home/appuser1/sites/www.example.com/config/etc/apache.conf www.example.com</pre>
<p>Now that we know what goes where we can work on the relation between the IP adresses and the different applications or vhosts.</p>
<h1>Installing the glue</h1>
<p>We need mod_define for this, found here: <a title="mod_define" href="http://people.apache.org/~rjung/mod_define/" target="_blank">mod_define</a>. With this module we have the ability to set $variables within the Apache configuration files.</p>
<p>Installation is rather simple. Download it and run (as root):</p>
<pre>$ apxs2 -i -a -c mod_define.c</pre>
<p>Good chance the module is automatically enabled using the apxs2 command but that might depend on your distro. Otherwise you can either use a2enmod or simply symlink between /etc/apache2/mods-available and /etc/apache2/mods-enabled. The file your looking (linking) for contains the following:</p>
<pre>$ cat /etc/apache2/mods-enabled/define.load
LoadModule define_module      /usr/lib/apache2/modules/mod_define.so</pre>
<p>Enable the module linking it like so:</p>
<pre>$ cd /etc/apache2/mods-enabled
$ ln -s ../mods-available/define.load</pre>
<p>Note that your installed module, as configured in &#8216;define.load&#8217;, might end up elsewhere under &#8216;/usr/&#8230;. &#8216; depending on your distro.</p>
<h1>The glue in effect</h1>
<p>Besides the seemingly complicated context it simply comes down to a list of variable names containing IP adress/port number combinations. The below example should explain this.</p>
<pre>&lt;IfModule define_module&gt;
    # IP's are example public addresses
    Define bind_address_www_example_com 1.2.3.4:443
    Define bind_address_www_other_com 2.3.4.5:443
&lt;/IfModule&gt;</pre>
<p>After making sure the module is enabled we place the &#8216;vhost_bind_adresses.conf&#8217; file in &#8216;/etc/apache2/conf.d&#8217;. After this file has been set up we can use the defined variables in our VirtualHost configuration. These are about the only thing that the development team should stick to (assuming the vhost file is included in their codebase). Considering the file wherein we defined the variables a vhost file looks like this:</p>
<pre>Listen $bind_address_www_example_com
NameVirtualHost $bind_address_www_example_com

&lt;VirtualHost $bind_address_www_example_com&gt;

    ServerName www.example.com
    ServerAdmin webmaster@example.com

    LogLevel warn
    CustomLog /home/appuser1/sites/www.example.com/log/apache.access.log combined
    ErrorLog /home/appuser1/sites/www.example.com/log/apache.error.log

    DocumentRoot /home/appuser1/sites/www.example.com/

    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/www.example.com.cert
    SSLCertificateKeyFile /etc/ssl/certs/www.example.com.key

    # &lt;-- SNIP --&gt;

&lt;/VirtualHost&gt;</pre>
<p><strong>Note</strong>: As said, the Listen and NameVirtualHost directives are in de vhost file instead of the ports.conf.</p>
<p>The beauty of this setup is that we can shuffle our server configuration without any changes to the developers code base. Creating/deploying new apps simply means adding a variable to &#8216;vhost_bind_adresses.conf&#8217; and updating the development team about the name.</p>
<h1>Gotcha&#8217;s</h1>
<p>Not much I can think of. If you agree on a variable name with the development team that is unambiguous, for instance: $bind_addr_www_domain_com there&#8217;s not much that can go wrong.</p>
<p>One more tip: Learn yourself to use apache2ctl -t after changing stuff because restarting Apache while having syntax errors results in only stopping Apache. Which in it&#8217;s turn causes already running websites being unavailable for your audience.</p>
<p>Any comments or questions are welcome.</p>
<p>GrtzG</p>
<div class="AWD_like_button "><iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fweb-in-sight.nl%2F2011%2F06%2F25%2Frefactoring-apache-vhosts%2F&amp;send=false&amp;layout=button_count&amp;width=&amp;show_faces=false&amp;action=recommend&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:px; height:21px;" allowTransparency="true"></iframe></div>]]></content:encoded>
			<wfw:commentRss>http://web-in-sight.nl/2011/06/25/refactoring-apache-vhosts/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Virtualbox: virtual networking anywhere</title>
		<link>http://web-in-sight.nl/2011/03/21/virtualbox-virtual-networking-anywhere/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=virtualbox-virtual-networking-anywhere</link>
		<comments>http://web-in-sight.nl/2011/03/21/virtualbox-virtual-networking-anywhere/#comments</comments>
		<pubDate>Mon, 21 Mar 2011 12:19:16 +0000</pubDate>
		<dc:creator>Gerard</dc:creator>
				<category><![CDATA[All ENGLISH articles]]></category>
		<category><![CDATA[Technical]]></category>
		<category><![CDATA[network]]></category>
		<category><![CDATA[osx]]></category>
		<category><![CDATA[ubuntu]]></category>
		<category><![CDATA[virtual]]></category>
		<category><![CDATA[virtualbox]]></category>

		<guid isPermaLink="false">http://www.gerardjp.com/?p=1590</guid>
		<description><![CDATA[For lack of a better title, ever had the problem with Virtualbox where you&#8217;re working happily in the guest OS, unplug your laptop, get underway, and see the end of your guest OS access. It&#8217;s always either Internet access or &#8230; <a href="http://web-in-sight.nl/2011/03/21/virtualbox-virtual-networking-anywhere/">Lees verder <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>For lack of a better title, ever had the problem with Virtualbox where you&#8217;re working happily in the guest OS, unplug your laptop, get underway, and see the end of your guest OS access. It&#8217;s always either Internet access or local access for your VM. Well, that&#8217;s no longer an issue.</p>
<h1>Case exploration</h1>
<p>In this example I use my own setup but it should be doable on any host/guest OS combination. My setup is a Macbook running OSX snow leopard with a guest OS running Ubuntu server. Before getting into the details; The trick is to hook up Ubuntu&#8217;s eth0 to one of your normal interfaces (physical or wifi) and link Ubuntu&#8217;s eth1 to a virtual network device  in your host OS. In this setup you always have local (back-end) access to your virtual guest OS and whenever you have Internet access so does your guest OS.</p>
<h1>The details</h1>
<p>I&#8217;m assuming you&#8217;re already running Virtualbox on OS X and an Ubuntu installation as guest OS. First we need to create a virtual network interface to which we can &#8216;link&#8217; our eth1. For this we need a driver that emulates a network device. Typically we need a <a href="http://en.wikipedia.org/wiki/TUN/TAP">tun/tap driver</a>. Therefore download and install <a href="http://tuntaposx.sourceforge.net/">Tuntap for OS X</a>. In the Virtualbox network configuration for the guest OS we need to configure (and enable) the second ethernet adapter and set it as &#8216;host only&#8217;.</p>
<p><a href="http://www.gerardjp.com/wp-content/uploads/2011/03/enable_2nd_nic.png"><img class="size-full wp-image-1596 aligncenter" title="Enable second NIC" src="http://www.gerardjp.com/wp-content/uploads/2011/03/enable_2nd_nic.png" alt="Enable second NIC" width="648" height="520" /><br />
</a>NB: the name vboxnet0 will be changed to be tap0 but that will be handled in a minute.</p>
<p>To setup the virtual interface you need to download and run the following script: <a href="../wp-content/uploads/2011/03/tap.sh.zip">tap.sh</a>. Running tap.sh (as root) will create a virtual network device that can be used for Ubuntu&#8217;s eth1 to hook into. Whilst having the tap0 device operational with IP address 10.10.10.1 the following commands will set the Virtualbox settings for Ubuntu to bind to it.</p>
<p>List your VM&#8217;s and pick the right one:</p>
<pre>$ VBoxManage list vms</pre>
<p>Enable the 2nd network interface of your VM</p>
<pre>$ VBoxManage modifyvm [VM name] --intnet2 hostif</pre>
<p>Set the interface to &#8216;host only&#8217;:</p>
<pre>$ VBoxManage modifyvm [VM name] --hostonlyadapter2  tap0</pre>
<p>Note:  The last command(s) seem obsolete since the release of Virtualbox version 4.0.4 r70112 or later. In older versions the tap0 used to fallback to vboxnet0 after closing Virtualbox. But the network configuration is now persistent when set to device tap0.  So, depending on the version you use your mileage may vary. Test it with restarting Virtualbox and see if your VM network config looks similar to this:</p>
<p><a href="http://www.gerardjp.com/wp-content/uploads/2011/03/network_settings.png"><img class="alignnone size-full wp-image-1597 aligncenter" title="Network settings" src="http://www.gerardjp.com/wp-content/uploads/2011/03/network_settings.png" alt="Network settings" width="571" height="75" /></a></p>
<p>Running that last command once will suffice for the long run instead of between every Virtualbox or workstation startup (besides the tap.sh script that is).</p>
<p>That being said, you&#8217;re now ready to start the Ubuntu VM. You should probably access Ubuntu via the front-end nic the first time since the back-end nic in the guest OS isn&#8217;t configured yet. Having logged into Ubuntu you should setup your interfaces as follows. For eth0 you can either choose static or dhcp, whatever runs easiest on your network. Setup Eth1 as static in network segment 10.10.10.0/24, e.g. 10.10.10.2 with 10.10.10.1 being your tap0 hosts OS interface.</p>
<p>Either reboot your Ubuntu VM, or nicer, restart the network with:</p>
<pre>$ /etc/init.d/networking restart</pre>
<p>You should now be able to login on Ubuntu with:</p>
<pre>$ ssh you@10.10.10.2</pre>
<p>Now unplug your physical network connection in your host OS (that&#8217;s Ubuntu&#8217;s eth0) and you can still login on your VM via the 10.10.10.0/24 network.</p>
<h1>Gotcha&#8217;s</h1>
<p>The only one I can think of is the need to manually add the default route to the outside world over etho. This happens when the host OS&#8217;s physical network has been disconnected. I assume it&#8217;s because Ubuntu&#8217;s  eth0 has been down with which the route is dropped. You can easily resolve this with the following command:</p>
<pre>$ ip route add default via [your gateway's IP]</pre>
<p>If you have any questions or comments please feel free to drop me a line.</p>
<h1>References</h1>
<p>- The inspiration for this article: <a href="http://tinyurl.com/6z29w45">http://tinyurl.com/6z29w45</a><br />
- OSX&#8217;s TunTap driver: <a href="http://tuntaposx.sourceforge.net/">http://tuntaposx.sourceforge.net/</a><br />
- Tun/Tap info on wikipedia: <a href="http://en.wikipedia.org/wiki/TUN/TAP">http://en.wikipedia.org/wiki/TUN/TAP</a><br />
- The tap.sh script: <a href="http://www.gerardjp.com/wp-content/uploads/2011/03/tap.sh.zip">tap.sh</a></p>
<div class="AWD_like_button "><iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fweb-in-sight.nl%2F2011%2F03%2F21%2Fvirtualbox-virtual-networking-anywhere%2F&amp;send=false&amp;layout=button_count&amp;width=&amp;show_faces=false&amp;action=recommend&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:px; height:21px;" allowTransparency="true"></iframe></div>]]></content:encoded>
			<wfw:commentRss>http://web-in-sight.nl/2011/03/21/virtualbox-virtual-networking-anywhere/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>nexec, distributed terminal commands</title>
		<link>http://web-in-sight.nl/2011/03/20/nexec-distributed-terminal-commands/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=nexec-distributed-terminal-commands</link>
		<comments>http://web-in-sight.nl/2011/03/20/nexec-distributed-terminal-commands/#comments</comments>
		<pubDate>Sun, 20 Mar 2011 09:01:19 +0000</pubDate>
		<dc:creator>Gerard</dc:creator>
				<category><![CDATA[All ENGLISH articles]]></category>
		<category><![CDATA[Technical]]></category>
		<category><![CDATA[CLI]]></category>
		<category><![CDATA[python]]></category>
		<category><![CDATA[sysadmin]]></category>

		<guid isPermaLink="false">http://www.gp-net.nl/?p=45</guid>
		<description><![CDATA[Update: since there&#8217;s been a change in pylib&#8217;s netexec that breaks nexec.py I&#8217;ve updated the script. Click here to download the newest version! I Developed this commandline tool to make it easier to maintain stuff on a groing amount of &#8230; <a href="http://web-in-sight.nl/2011/03/20/nexec-distributed-terminal-commands/">Lees verder <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p><!--TOC--><em><strong>Update</strong>: since there&#8217;s been a change in <a href="http://codespeak.net/execnet/">pylib&#8217;s netexec</a> that breaks nexec.py I&#8217;ve updated the script. <a href="http://www.gerardjp.com/wp-content/uploads/2008/06/nexec-1.1.zip">Click here to download the newest version</a>!</em></p>
<p>I Developed <a href="http://www.gerardjp.com/wp-content/uploads/2008/06/nexec-10.tgz">this</a> commandline tool to make it easier to maintain stuff on a groing amount of servers. <strong>nexec </strong>is a tool that can help you gather ad-hoc info when it&#8217;s needed quickly. <strong>nexec</strong> is written in python, it depends on <a title="Py lib from codespeak" href="http://codespeak.net/py/dist/" target="_blank">codespeak&#8217;s pylib</a>, and does all it&#8217;s communication over SSH. The only thing you need on &#8216;the other side&#8217; is a python interpreter. (that&#8217;s right, no further libs or modules). Oh, and you need your SSH key pairs in place, otherwise it&#8217;s &#8216;password galore&#8217;.</p>
<p>Of course, the bash-ers out there would say that the same functionality can be achieved with a for loop. True! But that&#8217;s exactly what <strong>nexec</strong> does for you so you can concentrate on what you&#8217;re doing remote. And ofcourse you can have nightly cron-ed scripts in place that do this every day. But <strong>nexec</strong> aids in all those things that are not &#8216;in place&#8217;.</p>
<h2>Examples</h2>
<p>Before getting into any details, first some tricks of the &#8216;nexec&#8217; trade to get you warmed up &#8230; <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> </p>
<p>Are my server systems running the proper time:</p>
<pre>$ ./nexec.py -n 'server1 server2  server3' -c 'date'
server2: Fri May 23 16:21:18 CEST 2008
server1: Fri May 23 16:21:17 CEST 2008
server3: Fri May 23 16:21:19 CEST 2008</pre>
<p>See if all your servers are running ssh protocol 2 only:</p>
<pre>$ ./nexec.py -n 'server1 server2  server3' -c 'cat /etc/ssh/sshd_config | grep -v ^# | grep Protocol' | sort
server1: Protocol 2
server2: Protocol 2
server3: Protocol 2</pre>
<p>(You could also do a &#8220;-n ALL&#8221;, then nexec reads your server list from a config file. More on that in a minute)</p>
<p>Get a list with all different kernel versions that are running on your systems:</p>
<pre>$ ./nexec.py -n ALL -q -c 'uname -r' | sort -u
2.6.18-5-686
2.6.18.8-xen</pre>
<p>(The &#8220;-q&#8221; parameter suppresses the &#8216;per line&#8217; hostname prefix).</p>
<p>See if &#8216;john&#8217; is a member of the sudo group on servers 1 and 2:</p>
<pre>./nexec.py -v -q -n 'server1 server2' -c 'cat /etc/group | grep sudo | grep -v john'
Host:  server1.example.com
sudo:x:27:jane,jerry
Host:  server2.example.com
sudo:x:27:jane</pre>
<p>(The&#8221;-v&#8221; adds the &#8220;Host: &#8230;.&#8221; header, per server output. Note that &#8220;-v&#8221; and &#8220;-q&#8221; are interchangeable)</p>
<p>As you can see, by combining local and remote greps and using the -v and/or -q parameters there are some interesting ways to obtain live data from your server.</p>
<p>Lets look at the configuration.</p>
<h2>Configuration</h2>
<p>Run without parms, <strong>nexec</strong> shows help output. Furthermore the only two files <strong>nexec</strong> depends on (and not even necessarily) are a &#8220;~/.nexec.conf&#8221; file and a possible &#8220;~/.ssh/config&#8221;.</p>
<p>The &#8220;~/.nexec.conf&#8221; does not contain much, just &#8220;key: value&#8221; pairs under a hosts section. The actual &#8216;value&#8217; is only obtained when using the &#8220;-v&#8221; option to display the header above each server output &#8220;<em>Host: server1.example.com</em>&#8220;.</p>
<pre>$ cat .nexec.conf
[hosts]
server1:   server1.example.com
server2:   server2.example.com
#server3:   server3.example.com</pre>
<p>Other then that, <strong>nexec</strong> simply expands &#8220;-c ALL&#8221; to all available keys under the &#8220;[hosts]&#8221; section, skipping the usual hashsign &#8216;#&#8217; being a comment. In this case the &#8220;ALL&#8221; parameter given to &#8220;-c&#8221; would be expanded to &#8220;server1 server2&#8243; in the previous example. After that the full name is taken from the &#8220;~.ssh/config&#8221; file.</p>
<pre>$ cat .ssh/config
# server1
Host server1
Hostname server1.example.com
User john
Port 4321
# server2
Host server2
Hostname server2.example.com
User john
<a href="http://www.gerardjp.com/wp-content/uploads/2008/06/nexec-1.0.tgz"></a>Port 4321</pre>
<p>I deliberately left all that in there so other usernames and port numbers can be specified. Besides, that&#8217;s all access related and shouldn&#8217;t be anywhere else anyway.</p>
<p>Then one more note on the &#8220;-q&#8221; and &#8220;-v&#8221; option. What they basically do is mark (or suppress mark&#8217;s) on the output that get&#8217;s returned. So you can mark output per server adding the &#8220;-v&#8221;, and output get&#8217;s marked per line (default) if not suppressed with &#8220;-q&#8221;.</p>
<h2>Download</h2>
<p>The script can be downloaded here: <a href="http://www.gerardjp.com/wp-content/uploads/2008/06/nexec-10.tgz">nexec-10.tgz</a></p>
<p>Set the &#8216;execute&#8217; bit with chmod and put it in your $PATH somewhere. And for the &#8220;ALL&#8221; parameter, set up the config file as the example shows in the <a title="TOC: Configuration" href="#toc-configuration">Configuration</a> section.</p>
<h3>Gotcha&#8217;s</h3>
<p>Escape your backticks with with a backslash if you want remote expansion.</p>
<p>This runs the uname part locally and then sends the command to the other side.</p>
<pre>$ nexec.py -n ALL -c "date | mailx -s `uname -n` root"</pre>
<p>This also executes the uname command on the other side.</p>
<pre>$ nexec.py -n ALL -c "date | mailx -s \`uname -n\` root"</pre>
<h2>Todo</h2>
<p>There&#8217;s two things still on my list.</p>
<ul>
<li>Catch a non-existing hostname when taken from the &#8220;.ssh/config&#8221; file. The script breaks if a host is not resolvable. (Being a sysadmin you should know what your doing .. <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> </li>
<li>Make the &#8220;<em>Host: server1.example.com</em>&#8221; header not show when no output is returned from a host. That is when using the &#8220;-c ALL&#8221;.</li>
</ul>
<p>New tricks might be added in the future, so if there&#8217;s stuff that you think should be added drop me a reply.</p>
<p>That&#8217;s all folks &#8230;</p>
<div class="AWD_like_button "><iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fweb-in-sight.nl%2F2011%2F03%2F20%2Fnexec-distributed-terminal-commands%2F&amp;send=false&amp;layout=button_count&amp;width=&amp;show_faces=false&amp;action=recommend&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:px; height:21px;" allowTransparency="true"></iframe></div>]]></content:encoded>
			<wfw:commentRss>http://web-in-sight.nl/2011/03/20/nexec-distributed-terminal-commands/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Downsize an image while keeping aspect ratio</title>
		<link>http://web-in-sight.nl/2010/11/02/downsize-an-image-while-keeping-aspect-ratio/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=downsize-an-image-while-keeping-aspect-ratio</link>
		<comments>http://web-in-sight.nl/2010/11/02/downsize-an-image-while-keeping-aspect-ratio/#comments</comments>
		<pubDate>Tue, 02 Nov 2010 14:53:18 +0000</pubDate>
		<dc:creator>Gerard</dc:creator>
				<category><![CDATA[All ENGLISH articles]]></category>
		<category><![CDATA[Technical]]></category>

		<guid isPermaLink="false">http://www.gerardjp.com/?p=1478</guid>
		<description><![CDATA[A simple python function to downsize an image and calculate the new dimensions while keeping aspect ratio, whether it's in portrait or landscape <a href="http://web-in-sight.nl/2010/11/02/downsize-an-image-while-keeping-aspect-ratio/">Lees verder <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p><em>A simple python function to downsize an image and calculate the new dimensions while keeping aspect ratio.</em></p>
<p>I needed this functionality for a web application. There are some variations found on the net from which I constructed this simple but effective python function to get &#8216;the other&#8217; value.</p>
<p>Based on the original image dimensions &#8220;i_&#8230;&#8221; it figures out whether the image is portrait or landscape and calculates either the width or height.</p>
<pre>def calculate_image_resize(i_width, i_height, max=0):

    i_width = float(i_width)
    i_height = float(i_height)
    ratio = i_width / i_height

    print 'w:', i_width, 'h:', i_height, 'r:', ratio

    if i_width &lt; i_height:
        print 'portrait'
        c_height = max
        c_width = max * ratio
    else:
        print 'landscape'
        c_width = max
        c_height = max / ratio

    c_width = int(c_width)
    c_height = int(c_height)

    return c_width, c_height
</pre>
<p>Dropping the function in a python shell and calling it results in this output:</p>
<pre>In [2]: calculate_image_resize(1700, 330, max=150)
w: 1700.0 h: 330.0 r: 5.15151515152
landscape
Out[2]: (150, 29)

In [3]: calculate_image_resize(170, 330, max=150)
w: 170.0 h: 330.0 r: 0.515151515152
portrait
Out[3]: (77, 150)
</pre>
<p>I deliberately left in the obvious 150 so you know instantly which dimension was calculated &#8220;c_&#8230;&#8221;.</p>
<p>Hope this saves people the trouble to figure it out by them selves.</p>
<p>GrtzG</p>
<div class="AWD_like_button "><iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fweb-in-sight.nl%2F2010%2F11%2F02%2Fdownsize-an-image-while-keeping-aspect-ratio%2F&amp;send=false&amp;layout=button_count&amp;width=&amp;show_faces=false&amp;action=recommend&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:px; height:21px;" allowTransparency="true"></iframe></div>]]></content:encoded>
			<wfw:commentRss>http://web-in-sight.nl/2010/11/02/downsize-an-image-while-keeping-aspect-ratio/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Setting up your own custom python userlib</title>
		<link>http://web-in-sight.nl/2010/09/28/setting-up-your-own-custom-python-userlib/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=setting-up-your-own-custom-python-userlib</link>
		<comments>http://web-in-sight.nl/2010/09/28/setting-up-your-own-custom-python-userlib/#comments</comments>
		<pubDate>Tue, 28 Sep 2010 11:29:04 +0000</pubDate>
		<dc:creator>Gerard</dc:creator>
				<category><![CDATA[All ENGLISH articles]]></category>
		<category><![CDATA[Technical]]></category>
		<category><![CDATA[code]]></category>
		<category><![CDATA[python]]></category>
		<category><![CDATA[script]]></category>

		<guid isPermaLink="false">http://www.gerardjp.com/?p=1222</guid>
		<description><![CDATA[Never were able, when in need of a simple script, to do a quick finish. Always end up copying code from your other scripts for such a simple thing as sending an email? Then this might be something for you. <a href="http://web-in-sight.nl/2010/09/28/setting-up-your-own-custom-python-userlib/">Lees verder <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>I recently wrote a a script that monitors file/dir changes, acts on a change and notifies me via <a title="Notify OSD" href="https://wiki.ubuntu.com/NotifyOSD" target="_blank">notify-osd</a> or email. While writing this I came to the conclusion that a simple thing as sending an email easily kicks up to at least 10 lines of code and some settings. So it was time to find an easier way for this.</p>
<p>You can of course create distributable modules or packages but with simple local workstation like maintenance this seems overkill &#8230; yep, you are allowed to disagree <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_wink.gif' alt=';)' class='wp-smiley' /> </p>
<p>So, we need a place to put code that is easy to import in our used scripts and it should be easy to maintain. Dropping it in my homedir in a directory called &#8220;.python&#8221; seems well suited for this.</p>
<h2>The Setup</h2>
<p>Create the directory &#8220;.python&#8221; in your $HOMEDIR. Then add a line to your &#8220;.bashrc&#8221; to export $PYTHONPATH so when you do an import in python it is actually found.</p>
<pre>PYTHONPATH=~/.python
export PYTHONPATH</pre>
<p>Python picks up the paths from this variable and adds them to &#8220;sys.path&#8221; so there&#8217;s no need to add the path in every script you make. In individual cases you can of course do it like it is <a title="Appending to Your Python Path" href="http://www.johnny-lin.com/cdat_tips/tips_pylang/path.html" target="_blank">explained here</a>.</p>
<p>Then create a file in &#8220;~/.python/&#8221; for instance &#8220;userconfig.py&#8221;.</p>
<p>To make it available in your script simply import it and execute one of your functions.</p>
<pre>import userconfig as mystuff
mystuff.function(with, parms)</pre>
<p>I&#8217;ll give an example of a userconfig.py function to easily send out emails.</p>
<pre>import os
import sys
import pynotify
import smtplib
from email.mime.text import MIMEText

# Settings &lt;&lt; These are the "Sensible defaults" <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' />
MAILSERVER = 'localhost'
MAILFROM = 'somebody'
MAILTO = 'you@yourdomain.com'

def send_mail(subject, body, fro=MAILFROM, to=MAILTO):
    """
    Stripped function to send mail from local code
    """
    msg = MIMEText(body)
    msg['Subject'] = subject
    msg['From'] = fro
    msg['To'] = to

    s = smtplib.SMTP(MAILSERVER)
    s.sendmail(msg['From'], msg['To'], msg.as_string())
    s.close()</pre>
<p>The code above composes and sends an email. Doing all this in the script you are writing results in quite a bunch of these lines ending up in every one of your scripts.</p>
<h2>Sensible defaults</h2>
<p>How about not having to enter those &#8216;always the same defaults&#8217;? Well &#8230; When using the setup described in this article most can be omitted and parameters only need to be added when you deviate from your sensible defaults.</p>
<p>For argument sake I&#8217;ll give an example script that just mails something and calls the &#8220;send_mail()&#8221; function.</p>
<pre>import userconfig as user

body = """Some multiline triple quoted body mass\n\nwith lot's of

funky stuff in it."""

fish = 'whale'
subject = 'something fishy: %s' % fish

user.send_mail(subject, body)</pre>
<p>So that&#8217;s only 1 line for the actual &#8216;send my mail&#8217; command. Should you want a different FROM address then you just change the function call to this.</p>
<pre>user.send_mail(subject, body, 'notme')</pre>
<p>Thus, add parms when you need them or rely on your &#8220;Sensible Defaults&#8221; from your userconfig.py.</p>
<p>You can build a world of functionality in your &#8220;~/.python&#8221;. Import and use system wide installed packages in there, test if they are present on your system, etc.</p>
<h2>Gotcha&#8217;s</h2>
<p>When using scripts under CRON you need to add the PYTHONPATH to the  definition like below because CRON does not read your  &#8220;.bashrc&#8221;.</p>
<pre>*/1 * * * *  (PYTHONPATH=~/.python; /path/to/your/script.py)</pre>
<p>NB: Mind the parenthesis and semicolon on the line above.</p>
<p>Besides the fact that you should watch for name clashes, I can&#8217;t think of any other gotcha&#8217;s. If you do, then please teach me something and drop me a line <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </p>
<p>GrtzG</p>
<div class="AWD_like_button "><iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fweb-in-sight.nl%2F2010%2F09%2F28%2Fsetting-up-your-own-custom-python-userlib%2F&amp;send=false&amp;layout=button_count&amp;width=&amp;show_faces=false&amp;action=recommend&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:px; height:21px;" allowTransparency="true"></iframe></div>]]></content:encoded>
			<wfw:commentRss>http://web-in-sight.nl/2010/09/28/setting-up-your-own-custom-python-userlib/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>HTTP over SSH &#8230; The full Monty!</title>
		<link>http://web-in-sight.nl/2010/06/07/http-over-ssh-the-full-monty/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=http-over-ssh-the-full-monty</link>
		<comments>http://web-in-sight.nl/2010/06/07/http-over-ssh-the-full-monty/#comments</comments>
		<pubDate>Mon, 07 Jun 2010 12:29:53 +0000</pubDate>
		<dc:creator>Gerard</dc:creator>
				<category><![CDATA[All ENGLISH articles]]></category>
		<category><![CDATA[Technical]]></category>

		<guid isPermaLink="false">http://www.gerardjp.com/?p=1301</guid>
		<description><![CDATA[A utterly safe way to access websites over SSH <a href="http://web-in-sight.nl/2010/06/07/http-over-ssh-the-full-monty/">Lees verder <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve been looking into this because I did not want to spend more money on extra IP addresses for SSL connects. But I do have several websites I want to access encrypted (that is .. not over plain HTTP).</p>
<p>There definitely are some articles (and forum threads) out there that come close, but this is how it&#8217;s done <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </p>
<h1>Some predefs</h1>
<p>The goal is to be able to connect to multiple VirtualHost&#8217;s on an Apache webserver.</p>
<p>We have a localpc behind a NAT-ed internet connection (yourpc.locality.lan) and the aforementioned publicly accessible webserver on the net somewhere (yourserver.example.com).</p>
<p>I&#8217;m assuming you can already connect to your server via SSH, based on a pub/priv keypair.</p>
<h1>Local to remote</h1>
<p>There are some things that need to be set local AND remote to make sure that Apache knows to which (not publicly accessible) VirtualHost it must redirect your HTTP request. That is, make up a name and put it in your /etc/hosts file and have it resolve to 127.0.0.1. E.g:</p>
<pre>$ cat /etc/hosts
127.0.0.1 localhost

# For ssh tunneling
127.0.0.1 mysuperadminsite.locality.lan</pre>
<p><strong>Important! </strong>You need to put the &#8216;mysuperadminsite&#8217;  entry in your /etc/hosts file on your server as well. Otherwise it won&#8217;t work because Apache can&#8217;t figure out where this needs to go (more on apache config in a sec).</p>
<p>Then, the actual SSH tunnel command is this:</p>
<pre>ssh  -L 9002:mysuperadminsite.locality.lan:80 yourname@yourserver.example.com</pre>
<p>There are ofcourse nicer ways to fully hide/automate the SSH tunnel, but by doing it manually like this you are explicitly remembered that you&#8217;re in &#8216;admin-mode&#8217; <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </p>
<h1>Apache trickery</h1>
<p>Ok, so how does Apache know where to go when your HTTP request hits the server side? As your mysuperadminsite request is resolved to 127.0.0.1 (on the server), apache needs this as a NameVirtualHost. So put this in your global Apache config somewhere.</p>
<pre># For admin work over SSH
NameVirtualHost 127.0.0.1:80</pre>
<p>Then you create an apache VirtualHost config with a setup like this.</p>
<pre>&lt;VirtualHost 127.0.0.1:80&gt;
	ServerName mysuperadminsite.locality.lan

	# Here you put your docroot, includes, logfile entries and whatever else you want.

&lt;/VirtualHost&gt;</pre>
<p>And so, apache resolves the request to 127.0.0.1 where, based on the X-Header (containing the mysuperadminsite name) sent by your browser, it actually finds a vhost to handle your request.</p>
<p>This is a very safe way to handle your secure access because the VirtualHost is only handled by apache when it comes in on your loopback interface. Moreover FQDN&#8217;s based on &#8220;locality.lan&#8221; are not resolvable in the outside world.</p>
<p>It does however require a second website instance. You could combine an externally accesible VirtualHost by starting your config with this.</p>
<pre>&lt;VirtualHost [external_ip]:80, 127.0.0.1:80&gt;
</pre>
<p>This way you don&#8217;t have to have a second VirtualHost config (and thus website instance) for management. But it&#8217;s easier to overlook going secure and thus have your passwords or whatever sniffed.</p>
<h1>Gotcha&#8217;s</h1>
<p>None that I know of, but feel free to point them out <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </p>
<p>Safe computing!</p>
<div class="AWD_like_button "><iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fweb-in-sight.nl%2F2010%2F06%2F07%2Fhttp-over-ssh-the-full-monty%2F&amp;send=false&amp;layout=button_count&amp;width=&amp;show_faces=false&amp;action=recommend&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:px; height:21px;" allowTransparency="true"></iframe></div>]]></content:encoded>
			<wfw:commentRss>http://web-in-sight.nl/2010/06/07/http-over-ssh-the-full-monty/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Django FormWizard best .. well .. useful practices</title>
		<link>http://web-in-sight.nl/2010/02/24/django-formwizard-best-well-useful-practices/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=django-formwizard-best-well-useful-practices</link>
		<comments>http://web-in-sight.nl/2010/02/24/django-formwizard-best-well-useful-practices/#comments</comments>
		<pubDate>Wed, 24 Feb 2010 11:10:56 +0000</pubDate>
		<dc:creator>Gerard</dc:creator>
				<category><![CDATA[All ENGLISH articles]]></category>
		<category><![CDATA[Technical]]></category>
		<category><![CDATA[django]]></category>
		<category><![CDATA[form]]></category>
		<category><![CDATA[wizard]]></category>

		<guid isPermaLink="false">http://www.gerardjp.com/?p=1231</guid>
		<description><![CDATA[Django FormWizard usefulness: emulating decorators .. more to come <a href="http://web-in-sight.nl/2010/02/24/django-formwizard-best-well-useful-practices/">Lees verder <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p><em>Edit</em>: After some re-writes of this post it is finally setup in a manor that is clean and failsafe.</p>
<p>After researching and setting up two FormWizards in Django I concluded some things that might be useful to others. Hence this post.</p>
<h2>Having userdata available throughout the wizard</h2>
<p>I&#8217;ve seen some solutions on the net for this but when you handle data from several models over different steps the best way to handle this is using Django&#8217;s session framework. In parse_params() in the beginning and end we repectively get/set data and store our data.</p>
<pre>def parse_params(self, request, *args, **kwargs):
    current_step = self.determine_step(request, *args, **kwargs)
    form = self.get_form(current_step, request.POST)

    # Pickup the dict from the users session or create a new one
    wizdata = request.session.get('wizdata') or {}

    # Your code goes here. E.g. set a value in the dict
    wizdata['some_key'] = 'Some value'

    # Store the possibly changed dict back in the users session.
    request.session['wizdata'] = wizdata
</pre>
<p>So whatever you do with the user data, it&#8217;s safe, and we can access it in any step throughout the wizardry.</p>
<pre>def render_template(self, request, form, previous_fields, step, context=None):
    wizdata = request.session.get('wizdata')

    # Your other code goes here. E.g. a conditional on previously set data.
    if wizdata.get('some_key'):
        # Do something with your data
        wizdata['some_key'] = 'Some other value'

    # And again, simply store the dict in the users session.
    request.session['wizdata'] = wizdata
</pre>
<p>You can imagine that in the parse_params() and the render_template() method you respectively have the &#8216;if current_step == &#8216; or &#8216;if step == &#8216; statements along the way. All having the wizdata dict available.</p>
<p>This also works in the done() method. Simply get the dict one last time from the user session.</p>
<pre>def done(self, request, form_list):
    wizdata = request.session.get('wizdata')

    # Other code here

    # Cleanup
    del request.session['wizdata']
</pre>
<p>Then, for tidyness, at the end of done() we remove the dict from the user session.</p>
<h3>Gotchas</h3>
<p>There can be stale data, in the form of our stored dicts, left behind when wizard sessions are not finished by the user. This also is true for the sessions itself. As stated in the <a title="The Django docs about sessions" href="http://docs.djangoproject.com/en/dev/topics/http/sessions/#clearing-the-session-table" target="_blank">The Django docs about sessions</a> you should clean up the sessions from time to time.</p>
<p>Anybody that has improvements, questions or compliments please drop me a line <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </p>
<p>GrtzG</p>
<div class="AWD_like_button "><iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fweb-in-sight.nl%2F2010%2F02%2F24%2Fdjango-formwizard-best-well-useful-practices%2F&amp;send=false&amp;layout=button_count&amp;width=&amp;show_faces=false&amp;action=recommend&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:px; height:21px;" allowTransparency="true"></iframe></div>]]></content:encoded>
			<wfw:commentRss>http://web-in-sight.nl/2010/02/24/django-formwizard-best-well-useful-practices/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Django: get distinct field selection</title>
		<link>http://web-in-sight.nl/2009/12/16/django-get-distinct-field-selection/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=django-get-distinct-field-selection</link>
		<comments>http://web-in-sight.nl/2009/12/16/django-get-distinct-field-selection/#comments</comments>
		<pubDate>Wed, 16 Dec 2009 13:11:55 +0000</pubDate>
		<dc:creator>Gerard</dc:creator>
				<category><![CDATA[All ENGLISH articles]]></category>
		<category><![CDATA[Technical]]></category>
		<category><![CDATA[django]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.gerardjp.com/?p=1131</guid>
		<description><![CDATA[A function for Django that is a simple and safe solution to get a distinct (group by) field selection from a model. <a href="http://web-in-sight.nl/2009/12/16/django-get-distinct-field-selection/">Lees verder <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>I recently have been dinging around in <a href="http://www.djangoproject.com/" target="_blank">Django</a>&#8216;s <a href="http://en.wikipedia.org/wiki/Object-relational_mapping" target="_blank">ORM</a> because I needed a certain &#8216;fields &#8230; group by&#8217; selection. I was very impressed by the logic that the Django developers put in there. Especially the joins and aggregations that have been released with version 1.1. However, after a hours of testing and googling, I was not able to solve my problem with it. I needed a distinct selection like this:</p>
<pre>SELECT id, name FROM myapp_model WHERE owner_id='1' GROUP BY name;</pre>
<p>This would have been (easily) possible except for the WHERE clause on ownership that screwed me. Since the SQL statement was that simple I decided to write a bypass function that get&#8217;s the data straight from the database. It&#8217;s readonly access and I dont do anything more with it then create a list of links. Even more so, from a <a href="http://en.wikipedia.org/wiki/KISS_principle" target="_blank">KISS</a> point of view this code is easier to read back after several months.</p>
<p>And for reusability I turned it in to a function. It takes the model and the distinct field, and for my purpose the needed ownership as parameter:</p>
<pre>from django.db import connection

def get_latest_objects(model_name=None, distinct_field=None, user_id=None):
    """ Get lastest distinct selection (as tuples) of a given model
    """
    model_name = model_name.lower()
    query = ("select id,%(distinct_field)s from myapp_%(model_name)s "
                "where owner_id='%(user_id)s' "
                "group by %(distinct_field)s;") % {
                    'model_name': model_name,
                    'distinct_field': distinct_field,
                    'user_id': user_id,
                    }
    cursor = connection.cursor()
    cursor.execute(query)</pre>
<p>There&#8217;s not much to it, I just hope it helps you to avoid &#8216;the hard way&#8217;. And it helps to keeps your view methods more readable because you dont need anything more then this:</p>
<pre>latest_objects = get_latest_objects(model_name='MyModel',
                        distinct_field='name',
                        user_id=request.user.id)</pre>
<p>What is returned (latest_objects) is a list of tuples that you can unpack in your templates straight away. Note that there&#8217;s no error checking in there, because I know what I&#8217;m doing <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_wink.gif' alt=';)' class='wp-smiley' />  Besides, worst case you get back an empty list and possible errors are caught when you write your tests. You do write those don&#8217;t you?</p>
<p>Grtz Gerard.</p>
<div class="AWD_like_button "><iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fweb-in-sight.nl%2F2009%2F12%2F16%2Fdjango-get-distinct-field-selection%2F&amp;send=false&amp;layout=button_count&amp;width=&amp;show_faces=false&amp;action=recommend&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:px; height:21px;" allowTransparency="true"></iframe></div>]]></content:encoded>
			<wfw:commentRss>http://web-in-sight.nl/2009/12/16/django-get-distinct-field-selection/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Django: QueryScreener, a record level ownership development auditor</title>
		<link>http://web-in-sight.nl/2009/10/26/django-queryscreener-a-record-level-ownership-development-auditor/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=django-queryscreener-a-record-level-ownership-development-auditor</link>
		<comments>http://web-in-sight.nl/2009/10/26/django-queryscreener-a-record-level-ownership-development-auditor/#comments</comments>
		<pubDate>Mon, 26 Oct 2009 16:41:12 +0000</pubDate>
		<dc:creator>Gerard</dc:creator>
				<category><![CDATA[All ENGLISH articles]]></category>
		<category><![CDATA[Technical]]></category>
		<category><![CDATA[development]]></category>
		<category><![CDATA[django]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.gerardjp.com/?p=1049</guid>
		<description><![CDATA[QueryScreener is a middleware development tool that helps to avoid unwanted data disclosure once you go into production. It monitors queries to the models and warns you when queries are executed that lack a ownership clause.  <a href="http://web-in-sight.nl/2009/10/26/django-queryscreener-a-record-level-ownership-development-auditor/">Lees verder <span class="meta-nav">&#8594;</span></a>]]></description>
			<content:encoded><![CDATA[<p>During the development stage of a Django app I&#8217;m working on I was exploring how to best implement rowlevel user ownerships. There are several ways to overwrite methods on <a title="Object Managers" href="http://docs.djangoproject.com/en/dev/topics/db/managers/" target="_blank">object managers</a> and even the Django admin interface is properly configurable to take a ownership from &#8220;request.user&#8221;.</p>
<p>But since wrongfull data disclosure is absolutely unacceptable I was still afraid that I would miss something somewhere. A nice example I ran into was populating a dropdown list in a form, where all records were visible instead of only those owned by the logged in user.</p>
<p>That got me thinking and eventually I wrote this small but sweet piece of <a title="Middleware" href="http://docs.djangoproject.com/en/dev/topics/http/middleware/" target="_blank">middleware</a>. Further elaboration below the code.</p>
<pre class="python" name="code">from django.db import connection
import re

"""
QueryScreener is a middleware development tool. This tool helps to avoid
unwanted data disclosure once you go into production.

It monitors queries to the models in your model_list and warns you when queries
are executed that do not contain a ownership where clause. And thus can be a
potential data disclosure hazard.

It requires a owner attribute in your model definition, e.g:

    owner = models.ForeignKey(User, editable=False)

Edit the 'model_list' below for what models should be monitored. And add
QueryScreener to MIDDLEWARE_CLASSES in you settings.py

Note: This can/should only be used while running Django's testserver command
with e.g: ./manage.py runserver 192.168.1.81:8000
"""

class QueryScreener(object):

    model_list = ['myapp_customer', 'myapp_order', 'myapp_product']

    def process_view(self, request, view_func, view_args, view_kwargs):
        if len(connection.queries) &gt; 0:
            query_parse(connection.queries, self.model_list, 'process_view')

    def process_response(self, request, response):
        if len(connection.queries) &gt; 0:
            query_parse(connection.queries, self.model_list, 'process_response')
        return response

def query_parse(self, model_list, caller_process):

    for query in connection.queries:
        for modelname in model_list:
            modelstring = 'FROM `'+modelname

            if re.search(modelstring, query['sql']) and not \
                re.search(r'^SELECT.\(1\).AS', query['sql']):

                reg = re.compile(r'^SELECT.*WHERE.*owner.*(ORDER BY.*)?$',
                                    re.DOTALL)

                if not reg.search(query['sql']):
                    print ('&lt;&lt;&lt; WARNING &gt;&gt;&gt; Query execution without ownership '
                            'clause, called from "' + caller_process + '"')
                    print query['sql']

            if re.search(r'^SELECT.\(1\).AS.`a`.FROM.*WHERE.*$', query['sql']):
                print ('&lt;&lt;&lt; Django Farted &gt;&gt;&gt;')
#                print query['sql']</pre>
<p><em>Update1: The &#8216;ORDER BY&#8217; in the regex needs to be optional.</em><br />
<em>Update2: Django does a &#8216;try update&#8217; in save_base() without owner (seperated the select statement)<br />
</em></p>
<p>The comment in the code above sums up how to get it working. What it does is print a warning and the query in question that does not respect ownership. If enabled while developing just keep track of your console output for:</p>
<pre>&lt;&lt;&lt; WARNING &gt;&gt;&gt; Query execution without ownership clause, called from "process_response"</pre>
<p>Should you  have suggestion, criticism, or words of admiration then please, do tell me <img src='http://web-in-sight.nl/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> </p>
<p>GrtzG</p>
<div class="AWD_like_button "><iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fweb-in-sight.nl%2F2009%2F10%2F26%2Fdjango-queryscreener-a-record-level-ownership-development-auditor%2F&amp;send=false&amp;layout=button_count&amp;width=&amp;show_faces=false&amp;action=recommend&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:px; height:21px;" allowTransparency="true"></iframe></div>]]></content:encoded>
			<wfw:commentRss>http://web-in-sight.nl/2009/10/26/django-queryscreener-a-record-level-ownership-development-auditor/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

