1
2
3
4
5
6
7
8

---
layout: post
title: Devops w/ Perl @ Linode PPW Talk Slides
wordpress_id: 330
wordpress_url: http://michael.thegrebs.com/?p=330
date: 2011-10-27 14:46:31 -04:00
---
Earlier this month I gave a talk about...

Having the wordpress post id means extracting the post tags from the db should be quite easy. First we define our desired output:

1
2
3
4
5
6
7
8
9

---
layout: post
title: Devops w/ Perl @ Linode PPW Talk Slides
wordpress_id: 330
wordpress_url: http://michael.thegrebs.com/?p=330
date: 2011-10-27 14:46:31 -04:00
tags: geek perl slides
---
Earlier this month I gave a talk about ...

The tags field really can appear anywhere in this YAML fragment but I chose to throw it at the end. With 103 posts to loop over, run a query and insert a new line a short script makes sense. The real win for our script though is using the Tie::File module which presents each file as an array with an element for each line.

From there it’s simply a matter of reverse engineering the Wordpress schema to come up with a query that will return a space delimited list of tags for a given post id.

1
2
3
4
5

SELECT GROUP_CONCAT(t.`name` SEPARATOR " ") FROM `wp_term_relationships` r
INNER JOIN `wp_term_taxonomy` tax ON r.`term_taxonomy_id` = tax.`term_taxonomy_id`
INNER JOIN `wp_terms` t ON tax.`term_id` = t.`term_id`
WHERE r.`object_id` = ?
AND tax.`taxonomy` = "post_tag";

Throw that query in a script that iterates over the list of files and uses Tie::File to add a line to the post and we get:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

#!/usr/bin/perl 

use 5.010;
use strict;
use warnings;

use DBI;
use Tie::File;

my $path_to_posts = 'source/_posts';

my $dbh = DBI->connect('DBI:mysql:db_name:db_host','db_user','db_pass');

my $get_tags = $dbh->prepare(q{
    SELECT GROUP_CONCAT(t.`name` SEPARATOR " ") FROM `wp_term_relationships` r
    INNER JOIN `wp_term_taxonomy` tax ON r.`term_taxonomy_id` = tax.`term_taxonomy_id`
    INNER JOIN `wp_terms` t ON tax.`term_id` = t.`term_id`
    WHERE r.`object_id` = ?
    AND tax.`taxonomy` = "post_tag";
    });


for my $file (glob "$path_to_posts/*.markdown") {
    say $file;
    add_tags_to_file($file);
}

sub add_tags_to_file {
    my $filename = shift;
    tie my @file, 'Tie::File', $filename or die "No tie: $!";
    my $tags;
    while (my ($rec, $data) = each @file) {
        if ($data =~ m/^wordpress_id: (\d+)$/) {
            $tags = get_tag($1);
        }
        if( $data =~ /^date:/) {
            splice @file, $rec + 1, 0, $tags if $tags;
            return;
        }

    }
}

sub get_tag {
    my $post = shift;
    $get_tags->execute($post);
    my ($tags) = $get_tags->fetchrow_array();
    return "tags: " . $tags if $tags;
}

DevOps w/ Perl @ Linode.pdf (2.1MB)

Skip to the next section if you just care about the end result. In the past, I’ve had my remote Irssi client send UDP Growl packets to my static home IP that were then forwarded to the broadcast address on the local network. This worked great and allowed me to get Growl notifications on which ever computer I happened to be sitting at. These days, I’m always on my laptop, even when sitting at another computer (Teleport for the mega win).

I wanted a solution that would allow me to receive Growl notifications without having to forward any ports and no matter what the public IP for my laptop may be. I use App::PersistentSSH to maintain a persistent SSH control master connection to my Linode so the ideal solution uses a reverse tunnel over this connection to get the notifications too my laptop.

My Current Solution

• Added --ssh_opts -R 127.0.0.1:22992:127.0.0.1:22 to my App::PersistentSSH command line. This connects localhost:22992 on my Linode to port 22 on my laptop.
• local-growl.pl – an Irssi script that uses growlnotify to send growl notifications
• growlnotify.pl – an ssh proxy for growlnotify, passes it’s arguments to growlnotify on a remote system via ssh

Simple push growl notifications from my Linode -> my MacBook Air whenever it is online. The scripts and an irssi.png to use in the notifications are available on GitHub.

Configuration for local-growl.pl is via Irssi’s in built settings, type /set growl to see a list of possibilites:

1
2
3
4
5
6
7

[local-growl]
growl_enabled = ON
growl_growlnotifypath = /home/michael/bin/growlnotify
growl_image_path = /Users/mgreb/Documents/irssi.png
growl_show_hilights = ON
growl_show_private_messages = ON
growl_sticky = OFF

Configuration for growlnotify.pl is via a set of config variables near the top:

1
2
3
4

my $ssh_host    = 'localhost';
my $ssh_port    = 22992;
my $ssh_user    = 'mgreb';
my $growlnotify = '/usr/local/bin/growlnotify';

Hopefully you will find this useful.

The wattvision (~$250) most closely resembles the TED in terms of price and features. The wattvision uses an optical sensor that counts the rotation of the disk in the utility meter. This sensor is connected to the device, which is powered by an AC adapter and sends the data to wattvision’s servers via your home wifi network. This is the easiest to install, but would require running the sensor cable from the outside to inside. Additionally, all data is sent to wattvision’s servers. I’d also need to subscribe to wattvision’s $8.99/month service for the other features I want (like API access.)

The eMonitor (starting around $688 for 12 circuits) is something I just recently discovered. Really, this is my ideal solution: for $1,277, you get the necessary equipment for monitoring the power feed plus 44 individual breakers. This gives a much better view of where your power consumption is going. Per-circuit monitoring also enables nifty alerts – an SMS for power draw in the kids room after school starts on a week day, or if the compressor on the fridge is running longer than normal (time to clean the coils, etc.) I just can’t justify the expense at $1,277. The price is certainly reasonable for what you are getting, but there is no way the wife would approve the purchase :/

The Energy Detective (TED, ~$200) from Energy Inc provides full house consumption monitoring like the wattvision. Unlike the wattvision, the data is stored and served from a gateway device on your home network. Two current transformers go around the two lines for the incoming split phase feed. These are connected to a box, which is installed next-to-or-inside the breaker panel. The breaker panel is powered and communicates via a connection to a breaker on each phase. The gateway is installed elsewhere in the house and plugs into an outlet for power and communication with the current sensor, and via Ethernet to serve its precious data up.

So, February 10th I finally got approval from the wife to order the TED and purchased the TED 5000-G with overnight shipping. It arrived as expected on the 11th and I was eager to install it. A coworker (also interested in the TED,) came over for dinner and to help out with the installation. I was too excited to wait for his arrival and finished the install before he got there ;-).

It’s now February 27th and I still don’t have a working TED system. The gateway I was originally shipped has a defective Ethernet port. It’s already been to Energy Inc and back and is still defective. They finally agreed to ship me a replacement gateway on the 25th but sent it with 2nd day shipping, which has the same transit time as ground (just more expensive.) The new unit should arrive March 1st.

I’m quite eager to get things up and running, and anticipate no problems once I have a working gateway. The most common issue people run into is excessive noise or attenuation of the power line communications (PLC) signal between the sensor and gateway. The forums seem to be full of people with lots of grief around this issue. The gateway does contain a ZigBee transceiver but that is for communication with the optional wireless display. Some have questioned switching from PLC to ZigBee for the current data but a device inside the metal breaker box would have a hard time getting any sort of RF signal out, and PLC seems the most logical solution (even with it’s inherent issues.)  Of course, this is easy for me to say since I don’t anticipate having any issue with PLC. I can see a clear ~3-4Vpk signal on the zero crossing of the A/C at the outlet I plan to install the gateway:

I have another blog post in the works documenting my seemingly unending issues with the other half of the TED solution: the support staff and engineers backing the product. To be fair, the support staff is 1:2 – the second guy I spoke to there was great, but I’m pretty sure their engineers shouldn’t be allowed to know customers even exist, let alone talk to them on the phone. But, this is a story for my next blog post.

I composed my first recipe the other night and figured I’d share it with you in the hopes that your doctor might hate me as much as mine does.

Bacon Salad

Ingredients

• 1lb Pancetta
• 1lb Virginia Bacon
• 1lb Peppered Bacon
• 1lb Canadian Bacon
• 1lb Apple-Wood Smoked Bacon
• 1lb Irish Bacon
• 1lb Slab Bacon
• 2lb Hog Jowl
• 1 Bottle reduced fat Bacon Ranch Salad Dressing

Directions

1. Fry each, crispy.
2. Cut into 1x2 inch rectangles.
3. Toss in bowl with salad dressing.

Makes 2, snack sized, servings.

At Linode we have a wiki page where we list 5 or more tasks we wish to accomplish during the week. There is a heading for each employee and below the heading we place our tasks. Throughout the week we can add additional tasks or mark existing ones as done.

I’ve used Hiveminder for some time. When we started the weekly task lists at Linode I found that taking a few minutes to figure out which tasks I wish to complete in the coming week works quite well for me. I started marking these tasks with the ‘week’ tag in Hiveminder. I quickly ended up writing a perl script, week.pl, to help me manage hem.

My Weekly Workflow

First thing Monday morning I run:

$ week.pl report

This prints a report with two sections. The first section lists tasks that currently have the ‘week’ tag with a line through the task ID if it is completed. This gives me a nice summary of what I planned on accomplishing the previous week and how I did. The second section lists all of my tasks currently visible in Hiveminder. I hide tasks that I know I’m not going to work on in the next few weeks so this list is usually no more than 20 or 30 items.

I take this report into the Monday morning meeting with me. During the meeting, I’ll glance over this list and select items for the upcoming week. I also use this page to take notes on during the meeting, writing down any new tasks that come up in the meeting.

After the meeting, I add any new tasks generated in the meeting that I won’t be working on this week to Hiveminder.

$ todo.pl braindump

`todo.pl` comes from App::Todo, a command line Hiveminder interface. The braindump command launches $EDITOR where I add new tasks, one per line. The braindump syntax allows for specifying tags, setting priorities, and other things as well.

Next, I prepare the task list for the upcoming week:

$ week.pl edit
Carry over the following tasks?
bring about world peace (y or n) [default y] y
write some awesome pre
do some other cool stuff
create practical cold fusion (y or n) [default y] n
return library book

The `edit` command iterates over each task tagged with ‘week’ if the task is not marked completed. It prompts whether or not I wish to carry the task over to this week (leave the tag). Any tasks marked completed have the tag removed automatically.

$ week.pl add
Created:
        #YRVK write an awesome report for Tom [week dev]
        #YRVL test some new stuff for deployment [week admin]

The `add

$ week.pl update https://path.to.trac/wiki/Tasks/2010-03-15
Sup dawg, I heard you like tasks so I did ur shit for you

The `update` command grabs my tasks tagged with ‘week’ and formats them one per line started with ’ * ‘, a wiki bullet list. It grabs the current wiki page, finds my heading, substitutes the formatted task list under the heading, and submits the change. It also stores the path given in the YAML config file.

$week.pl go

This opens the stored URL for this week’s tasks wiki page in my default browser, allowing me to confirm week.pl did what it’s supposed to.

Later in the week once I’ve done something:

$ week.pl done tom
#YH7T bring about world peace [dev week]
#YRVL test some new stuff for deployment [week admin]
[DONE] #YRVK write an awesome report for Tom [week dev]

This retrieves the task(s) tagged ‘week’ that are not marked completed containing the given string. If there was only one match, the script marks it as done and then outputs the current state of tasks tagged ‘week’.

$ week.pl update
Sup dawg, I heard you like tasks so I did ur shit for you

Same as `update` before, except when no URL is given, the URL is read from the configuration file. This way I only need to worry about the URL once per week, the first time I update for the week.

Conclusion

So there you have it. If you are also using Hiveminder, maybe aspects of my work flow will make sense for you and pieces of the Perl script may be useful. If you aren’t using Hiveminder, maybe you will be inspired to check it out. I use michael@thegrebs.com on Hiveminder in case you feel the need to assign me a task or wish to gift me another year of Hiveminder Pro ;)

week.pl

Usage looks like:

1
2
3
4
5
6
7
8
9
10
11

use WebService::MobileMe;

my $mme = WebService::MobileMe->new(
    username => 'yaakov', password => 'HUGELOVE' );
my $location = $mme->locate;
print < <"EOT";
    As of $location->{date}, $location->{time}, Yaakov was at
    $location->{latitude}, $location->{longitude} (plus or minus
    $location->{accuracy} meters).
EOT
$mme->sendMessage( message => 'Hi Yaakov!', alarm => 1 );

The online documentation is available via search.cpan.org.

• Git Web: http://git.thegrebs.com/?p=WebService-MobileMe
• Git Clone: http://git.thegrebs.com/git/WebService-MobileMe

Old Google Calendar embed removed.

This wasn’t quite what I wanted and the playTone function expects the notes and timing to be in global variables which makes the baby jesus cry. The premise was simple though. Take the period and divide by two then bring the output high then low for this amount of time, repeat to make up the desired duration.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

int pinSpeaker = 10;

void setup () {
    pinMode(pinSpeaker, OUTPUT);
}

void loop () {
    playTone(750, 500);
    delay(750);
}

// duration in mSecs, frequency in hertz
void playTone(long duration, int freq) {
    duration *= 1000;
    int period = (1.0 / freq) * 1000000;
    long elapsed_time = 0;
    while (elapsed_time < duration) {
        digitalWrite(pinSpeaker,HIGH);
        delayMicroseconds(period / 2);
        digitalWrite(pinSpeaker, LOW);
        delayMicroseconds(period / 2);
        elapsed_time += (period);
    }
}

This simple sketch plays a 500Hz tone for 750 mSec, pauses 750 mSec and repeats. To use it in your sketch simply define pinSpeaker and set it for output, copy playTone to your sketch and call it with the duration in milliseconds and the frequency in hertz.

1
2
3
4
5
6
7

RT 30
REPEAT 3 [REPEAT 3 [FD 100 RT 120] RT 120]
PENUP
FD 100
PENDOWN
RT 90
REPEAT 360 [FD 1.75 RT 1]

The output:

Reading my RSS feeds before bed, I found “My new ticket tracking system is now vaporware” from Matt Wilson’s blog talking about Pitz, his soon to exist Python implementation of Ditz.

Please take a few moments to read over the feature set and example commands on the Pitz site and let Matt know about any ideas/suggestions you may have.

$ perl -MPOE::Filter -E 'say $POE::Filter::VERSION'
1.2357

So I added a quick function to .bashrc:

1
2
3

pm-vers () {
    perl -M$1 -e "print \$$1::VERSION, \"\n\""
}

Now I just run:

$ pm-vers POE::Filter
1.2357

If you haven’t heard of this module before, see this online example of some of the data that can be returned. Online docs are available as well at search.cpan.org.

From the Changes file:

• New get_domain function that converts host name to domain name
• Memoize support though commented out by default, enable by uncommenting the following two lines in `lib/Net/Abuse/Utils.pm`
  

  1 2	# use Memoize; # memoize('_return_rr');

  A future version will likely allow Memoization via an export tag.

The get_domain function is fairly spiffy taking a host name like ‘michael.thegrebs.com’ and turning it into the domain name, ‘thegrebs.com’, but with proper recognition of second level zones like .co.uk.

The Memoization support is weak and I almost removed it but decided to leave it in but commented out for this release. It speeds up batch processing of large quantities of requests fairly well. Processing a days worth of spam used to take an average of 5 minutes and is now down to a minute and a half. This is quite a bit more increase than I expected as I am using bind with heavy local caching.

There were a few more changes mostly related to distribution related stuff, see git for the full gory details.

Not a horribly significant release but it’s been a while since the last one and I wanted to get get_domain out there.

I recently received a nice Honeywell weather station. It turns out they are actually manufactured by a company called Irox which also sells stations under it’s own name but they seem to be more popular in Europe. Being a geek, one of the first things I did was check out Linux or OS X support. Unfortunately only one piece of software supports this weather station under Linux, Weather Display. There Linux software is freeware but not open source and leaves quite a bit to be desired but at least it would talk to the weather station and collect data from it.

With a bit of Googling I found Saratoga Weather which seems to be based on a template from Carter Lakes. Unfortunately the PHP code is a bit of a mess with 43 separate php files all in a top level directory and including each other. With a bit of work I was able to bend it to my will but every little change was a pain to accomplish.

I started slowly re-implementing things in Perl. I found an online WD data-file parser which greatly helped in figuring out what the fields in Weather Displays native log files meant. Fast-forward a couple of weeks and I think things are pretty much done feature wise. I lack NWS warning/watch/advisory notification, the thermometer image with today’s information is still PHP generated, and I don’t yet have a wind graph. These last few details shouldn’t take too much work when I get around to it. I also have a few features the original lacks, graphs, a daily tweet with today’s high/low, and sane code :p.

The information from my weather station is available online at http://weather.thegrebs.com/ and the source is available via git clone from http://git.thegrebs.com/git/wxperl. You may also browse the repo via Git Web at http://git.thegrebs.com/?p=wxperl.

Ruby’s not Ready at glyphobet

Typical voltage divider, matched would have made better sense but the thermistor was a random one salvaged from consumer electronics so I didn’t know the value in advance.

Would work just fine like this with TP1 connected to an Arduino analog input and Vcc and GND connected. Energy savings for the win though, connected the top of the divider (Vcc) to a digital pin on the Arduino. Set the pin HIGH when you want to take a reading, LOW the rest of the time. Now the divider is not consuming power when a reading isn’t being taken.

Arduino Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

int pinDivEn   = 4;
int pinDivRead = 1;
int pinLED     = 13;

void setup() {
    Serial.begin(9600);
    pinMode(pinLED, OUTPUT);
    pinMode(pinDivEn, OUTPUT);
    Serial.println("READY");
}

void loop() {
    if (Serial.available() > 0) {
        digitalWrite(pinDivEn, HIGH);
        digitalWrite(pinLED, HIGH);
        delay(100);
        while (Serial.available() > 0)
            int serByte = Serial.read();
        Serial.println(analogRead(pinDivRead));
        digitalWrite(pinDivEn, LOW);
        digitalWrite(pinLED, LOW);
    }
}

pinDivEn is the divider enable pin (Vcc), pinDivRead is the analog input connected to TP1 in the divider, pinLED is is a digital pin with an LED that is lit while a reading is taken.

Once the Arduino boots it sends READY on the serial port at 9600 bps. It then waits for any data to be available on the serial port, when bytes are available, the voltage divider is enabled, the LED is lit, the bytes on the serial port are consumed, the value of TP1 is read and then written to the serial port, the divider enable pin is brought back low and the led is extinguished.

Perl Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

#!/usr/bin/perl

use strict;
use warnings;

use IO::Handle;
use Device::SerialPort;

my $dev = tie (*FH, 'Device::SerialPort', "/dev/tty.usbserial-A4001JwW")
    || die "Can't tie: $!";

$dev->baudrate(9600);
$dev->databits(8);
$dev->parity("none");
$dev->stopbits(1);

open (my $log, '>>', 'ohms.log') || die "can't open: $!";
$log->autoflush(1);

# wait for arduino to boot
while (1) {
    my $val = <fh>;
    last if $val;
}

print FH "1\n";
while (1) {
    my $val = </fh><fh>;
    next unless $val;
    chomp $val;

    my $vR2 = $val / 1023 * 5;
    my $vR1 = 5 - $vR2;
    my $i   = $vR1 / 1_000 * 1_000;
    my $R2  = $vR2 / $i;

    print $log time . ' ' . $R2 . "\n";
    # printf("V_R2=%.2fV, i=%.2fmA  R2=%.2fK&#8486;\n", $vR2, $i, $R2);

    sleep 10;
    print FH "1\n";
}

Opens the serial port, waits for something (the Arduino sending ‘READY’ but it pays no attention to what) on the serial port then every 10 seconds sends a 1\n to the Arduino prompting it to send a value. The result of analogRead is an integer from 0 - 1023 so this is converted to a voltage. The current through the divider is found allowing the resistance of the thermistor to be solved for. The resistance and time are written to a space separated log file. If the thermistor wasn’t unknown it would make more sense to solve for the actual temp using the Steinhart–Hart equation or similar and logging that rather than the resistance.

Sample log file entries

1224349429 10.1195652173913
1224349439 10.3666666666667
1224349449 10.3666666666667
1224349459 10.3666666666667
1224349469 10.4943820224719
1224349479 10.3666666666667
1224349489 10.2417582417582

A few lines in a text file gets us a graph from the log file by way of GnuPlot:

set term png transparent nocrop enhanced font "./arial.ttf" 9 size 800, 600
set xdata time
set timefmt "%s"
set format x "%m/%d %H:%M"
set xtics  rotate by -45
set title "Thermistor Value"
set xlabel "Date/Time"
set ylabel "Resistance (kOhms)"
set grid
set output "ohms.png"
plot \
    "ohms.log" using 1:2 title "10 sec interval readings" linecolor 2, \
    "ohms.log" using 1:2 title "Bezier fit curve" smooth bezier linecolor 4 lw 2, \
    10 title "77 F" lw 2 linecolor 1

Sorry check back later for pics, Steve Jobs says I don’t need copy & paste as I’m sending this from the wordpress app on my iPhone.

Update: One note of clearification, as crazy as the feature set on these machines is, I still wouldn’t mind having one to use whenever I wanted, imagine what you could do with it! The commercial I saw is available in flash form at http://www.sharpusa.com/files/workwithoutlimits.swf.