Finding Stuff In Your Local CPAN Mirror

So you’ve got this great new local CPAN you can use to install stuff you know about but how do you find stuff you don’t know about? There are two main ways I inspect the contents of my local CPAN mirror. The first is a simple shell function for searching package names, the second has a bit more to it but we will save that for later.

grep to the Rescue

One of the metadata files in our local mirror isn/modules/02packages.details.txt.gz. Let’s take a look inside.

/modules/02packages.details.txt.gz

File:         02packages.details.txtnURL:          http://www.perl.com/CPAN/modules/02packages.details.txtnDescription:  Package names found in directory $CPAN/authors/id/nColumns:      package name, version, pathnIntended-For: Automated fetch routines, namespace documentation.nWritten-By:   CPAN::Mini::Inject 0.33nLine-Count:   145223nLast-Updated: Tue, 12 Aug 2014 15:01:40 GMTnnAAA::Demo                         undef  J/JW/JWACH/Apache-FastForward-1.1.tar.gznAAA::eBay                         undef  J/JW/JWACH/Apache-FastForward-1.1.tar.gznAAAA::Crypt::DH                    0.04  B/BI/BINGOS/AAAA-Crypt-DH-0.04.tar.gznAAAA::Mail::SpamAssassin          0.002  S/SC/SCHWIGON/AAAA-Mail-SpamAssassin-0.002.tar.gznAAAAAAAAA                          1.01  M/MS/MSCHWERN/AAAAAAAAA-1.01.tar.gznAAC::Pvoice                        0.91  J/JO/JOUKE/AAC-Pvoice-0.91.tar.gznAAC::Pvoice::Bitmap                1.12  J/JO/JOUKE/AAC-Pvoice-0.91.tar.gznAAC::Pvoice::Dialog                1.01  J/JO/JOUKE/AAC-Pvoice-0.91.tar.gzn...

nnn

The format is pretty straightforward. After a few header lines, we have onenline per package seen by the indexer. The lines consist of three whitespacenseparated columns of a package name, current package version (if the indexer found one),nand latest distribution containing it. The AAC-Provoice and Apache-FastForwardndistributions show us that each individual package within the distribution isnlisted. We could easily use zgrep to find packages that match a pattern.

nn

/modules/02packages.details.txt.gz
1n2n3n4n5n
$ zgrep -i tiny ~/Dropbox/minicpan/modules/02packages.details.txt.gznAcme::Has::Tiny                   0.001  T/TO/TOBYINK/Acme-Has-Tiny-0.001.tar.gznAcme::Module::Build::Tiny          0.0   D/DA/DAGOLDEN/Acme-Acme-Module-Build-Tiny-0.06.tar.gznAcme::Tiny                        0.003  E/ET/ETHER/Acme-Tiny-0.003.tar.gzn...

nnn

This is a little long to type regularly. A shell alias isn’t appropriate as wenwant to sandwich the provided input between zgrep and the path to the file.nFortunately, a shell function will allow us to do this without much morenverbosity.

nn

1n2n3n
function cpangrep {n    zgrep $* /home/yourusername/Dropbox/minicpan/modules/02packages.details.txt.gzn}n

nnn

Dropping this line in your .zshrc or .bashrc will allow you to searchn02packages.details.txt.gz with the cpangrep command.

nn

1n2n3n4n5n6n7n8n
$ cpangrep tiny | wc -ln21n$ cpangrep -i tiny | wc -ln344n$ cpangrep -i tinynAcme::Has::Tiny                   0.001  T/TO/TOBYINK/Acme-Has-Tiny-0.001.tar.gznAcme::Module::Build::Tiny          0.06  D/DA/DAGOLDEN/Acme-Acme-Module-Buildn...n

nnn

Since we use $* instead of $1, all arguments we give to the function arenpassed on to zgrep. This allows us to do things like search with or withoutnignoring case on demand.

nn

CPAN::Mini::Webserver

nn

CPAN::Mini::Webserver provides anscript that launches a web server providing an interface to your local minicpan.nUsage couldn’t be easier, install CPAN::Mini::Webserver then runnminicpan_webserver. By default the server binds to port 2963 but you cannchange this with the --port argument.

nn

If you’ve used search.cpan.org ornmetacpan, there should be no surprises in how to navigatenaround the site. Thendocumentation has information onna few things you can plug in to your .minicpanrc to enable things like fullntext indexing.

nn

Here are a few screenshots of CPAN::Mini::Webserver to wet your appetite.

nn

nnnn

n

Taking CPAN With You

nn

CPAN is THE killer feature for Perl. Onenproblem you may have from time to time due to it’s online nature is availabilitynissues. How do you install CPAN modules from a plane, train, or automobilenwithout wifi? Or from the middle of nowhere in Oregon (Ione, OR)? Or timely install Moose andnit’s dependencies from a dial-up connection in the middle of the desertednOklahoma plains (Leedey, OK).nWell, my fellow perler, I have been to these places forgotten by the Internets,nand I have installed CPAN modules with impunity.

nn

CPAN::Mini

nn

CPAN::Mini, driven by the includednminicpan,ncreates a local mirror of CPAN containing only the latest non-developer releasenof every distribution on CPAN. With CPAN::Mini yountoo can install CPAN modules with impunity from exotic locations. But just hownmuch disk space must you dedicate for such awesomeness? My mirror is currentlynjust under 3 GB.

nn

Start by installing CPAN::Mini with your CPAN client of choice:

nn

1n2n3n4n5n
$ cpanm CPAN::Minin$ # orn$ cpanp install CPAN::Minin$ # orn$ cpan CPAN::Minin

nnn

Now you need a config file in ~/.minicpan.

nn

~/.minicpanrc
1n2n3n4n
local:   ~/Dropbox/minicpannremote: http://cpan.hexten.net/nalso_mirror: indices/ls-lR.gznskip_perl: 1

nnn

These options should be fairly self-explanatory. Check out thenCPAN Mirrors site to find a healthy mirror in your regionnfor the second line. Now, just run minicpan and wait. Periodically,nyou’ll want to run this command again to download new dists and delete old ones.nThat’s all it takes to be the owner of a shiny tiny local CPAN mirror.

nn

Using Your Shiny New CPAN Mirror

nn

To use your new mirror, just specify the path to your CPAN::Mini mirror in yournCPAN client of choice.

nn

cpanm

nn

For cpanm Inhave an alias in my .zshrc:

nn

1n
minicpanm='cpanm --mirror ~/Dropbox/minicpan --mirror-only'n

nnn

The same line should work in a .bashrc as is. Then, to install a new CPANnmodule, just run minicpanm ACME::urmom.

nn

cpanp

nn

Run cpanp s edit and your editor will launch. If the file you are innisn’t empty, skip to the next step. Now run cpanp s save to copy thencurrent configuration to the user config file and then run cpanp s editnagain.

nn

Find the hosts config and add your new entry at the beginning.

nn

1n2n3n4n5n6n7n8n9n10n11n12n13n14n15n16n17n
$conf->set_conf( hosts => [n      {n        'host' => '',n        'path' => '/Users/yourname/Dropbox/minicpan',n        'scheme' => 'file'n      }n      {n        'host' => 'ftp.cpan.org',n        'path' => '/pub/CPAN/',n        'scheme' => 'ftp'n      },n      {n        'host' => 'www.cpan.org',n        'path' => '/',n        'scheme' => 'http'n      },n# ...n

nnn

Save and exit your editor.

nn

cpan

nn

For cpan, you use the interactive shell to update the configuration:

nn

1n2n3n4n5n6n7n8n9n10n11n12n13n14n15n16n17n18n19n
$ cpanncpan shell -- CPAN exploration and modules installation (v2.00)nEnter 'h' for help.nncpan[1]>  o conf urllist unshift file:///Users/yourname/Dropbox/minicpamnPlease use 'o conf commit' to make the config permanent!nnncpan[2]> o conf urllistn    urllistn    0 [file:///Users/yourname/Dropbox/minicpam]n    1 [http://cpan.erlbaum.net/]n    2 [http://cpan.mirror.clemson.edu/]n    3 [http://cpan.develooper.com/]nType 'o conf' to view all configuration itemsnnncpan[3]> o conf commitncommit: wrote '/Users/yourname/.local/share/.cpan/CPAN/MyConfig.pm'n

nnn

Enjoy (Faster) CPAN Installs From Anywhere

nn

That’s it! From here on out your disk speed and not your Internet connection,nor lack thereof, will be your throttle in your CPAN client of choice acquiringndistribution packages.

n

Listening to Two Icecast Streams at Once

I run a site with two scanner feeds for my local town, Gallowaynow.com, one for EMS & Fire and another less publicized one for PD. I regularly listen to both of these feeds when I’m not around the scanners supplying them. How do you listen to two feeds that may be talking at the same time and make sense of anything? With sox the Swiss Army knife of sound processing. By playing the two mono feeds out each channel of a stereo device, the brain can easily deferenciate between the two sources. This may not work very well from laptop speakers but works great from speakers with a few feet of seperation or headphones. sox can easily handle modifying the streams likes to work with files or piped streams so first we need to grab the icecast stream.

nn

fIcy

nn

fIcy is an open source suit of tools for grabbing icecast streams. It can do things like grab a stream and save it to mp3s when metadata changes, checkout the examples on its site for some of the other neat things it can do. Rather than invoke fIcy directly, I use fPls, included in the suite which can retry if the connection drops. Invocation is fairly straight forward:

nn

1n
fPls -T 10 -L-1  http://gallowaynow.com:8000/stream.m3u -t

nnn

The -T 10 waits ten seconds between reconnection attempts, -L-1 tells it to try indefenitly, next is the stream URL, and -t outputs the stream metadata to STDERR when it changes.

nn

sox

nn

sox is crazy flexible and can manipulate audio files in almost any way imaginable. See the examples in the man page or this blog post for just a sampling (heh, sampling) of the sort of things sox can do. In our case we want to take a mono mp3 on STDIN and place that audio in just one channel of a stereo stream and then play it. We use the remix effect to accomplish this. Since transforming and then playing the audio is common, sox includes a play binary that takes all the same commands sox does but then plays the output.

nn

1n
play -q -t mp3 - remix 1 0n

nnn

Remix takes a space separated list of input channels to be output on the positional output channel. In this case, input channel 1 is output on the first channel, the second output channel uses the special input channel 0, silence. For our second stream, we flip the numbers around to specify that the input be played on the second output channel. You can also adjust the volume of the input channel in the output. My actual command for my first stream is:

nn

1n
play -qV1 -t mp3 - remix 1v0.3 0

nnn

In this case we are attenuating the voltage of the signal by 70{7e898290d39a0d5231795fca98f9d583658ec52d7dd28242e01977702bb4ac4e}. 2 instead of 0.3 would double volume. This lets me adjust the streams for equivalent volume.

nn

play can take any filter chain you can pass to sox, another useful one here is silence trimming. By adding silence -l 1 0.1 1{7e898290d39a0d5231795fca98f9d583658ec52d7dd28242e01977702bb4ac4e} -1 2.0 1{7e898290d39a0d5231795fca98f9d583658ec52d7dd28242e01977702bb4ac4e} to the end of our play command line, we can ask sox to trim any silence longer than 2 seconds to 2 seconds. This may seem odd since we are listening in real time, not recording to a file, but it can help with minimizing delay. Due to packet loss, slow connections, congested links, etc, the stream may stutter at times. Since a large portion of the time the stream is silent, these stutters aren’t noticable to the ear but it means playback was silenced for a few seconds and then resumed. Any of these delays add up and eventually the audio you are listening to could be as much as 5 or 10 seconds behind the metadata printed by fIcy. By telling sox it’s okay to throw away silence, we can keep the audio caught up, minimizing delays.

nn

We can put the commands together in simple shell scripts to simplify calling them.

nn

ems.sh
1n2n3n4n
#!/bin/shnnfPls -v -T 10 -L-1  http://gallowaynow.com:8000/stream.m3u -t n    | play -qV1 -t mp3 - remix 1v0.3 0 silence -l 1 0.0 1{7e898290d39a0d5231795fca98f9d583658ec52d7dd28242e01977702bb4ac4e} -1 2.0 1n

nnnnn

pd.sh
1n2n3n4n
#!/bin/shnnfPls -v -T 10 -L-1  http://gallowaynow.com:8000/gtpd.m3u -t n    | play -qV1 -t mp3 - remix 0 1 silence -l 1 0.0 1{7e898290d39a0d5231795fca98f9d583658ec52d7dd28242e01977702bb4ac4e} -1 2.0 1n

nnn

tmux

nn

Since the metadata is visible, it shows exactly which unit is talking and on which channel, running our two commands in a split tmux window makes since. Especially if you have screen real estate to put this terminal out of the way somewhere. We can script the creation of this window with a simple shell script.

nn

tmux-streams.sh
1n2n3n4n5n6n7n8n
#!/bin/shnntmux new-session -d -s streamsntmux send-keys './ems.sh' 'C-m'ntmux select-window -t streams:0ntmux split-window -vntmux send-keys './pd.sh' 'C-m'ntmux -2 attach-session -t streamsn

nnn

I like send-keys instead of specifying the command to run directly as it lets us ctrl+c the script and then rerun it with up-arrow enter. Handy if I tweak the script or want to launch a fresh copy of stuff instead of having to destroy the tmux session.

nn

tmux screenshot

n