Scripting Git With Perl

Doug Bell

Bank of America

Double Cluepon Software

Git Porcelain and Plumbing

• Porcelain commands
  • User Interface
  • git status
  • git log
  • git diff
• Plumbing commands
  • App Interface
  • git show-ref
  • git name-rev
  • git ls-remote

Git::Repository

• Object interface to running Git commands
• Performs the work using System::Command
  • Runs git from the command-line
• Gets output from stdout

Running Git Commands: git tag

$ cd ~/git/My-Project
$ git tag
v0.6
v0.6.1
v0.6.2
v0.6.3
v0.6.4
v0.6.5

my $git = Git::Repository->new(
    work_tree => "$ENV{HOME}/git/My-Project",
);
my @tags = $git->run( "tag" );
# ("v0.6", "v0.6.1", "v0.6.2", "v0.6.3", 
#  "v0.6.4", "v0.6.5")

Running Git Commands: git log

$ cd ~/git/My-Project
$ git log --oneline -n3
b23123a Release v0.6.12
6762cba do not try to fetch items that don't exist
d045083 errors from the server must extend ServerError

my $git = Git::Repository->new(
    work_tree => "$ENV{HOME}/git/My-Project",
);
my %commits = map { split /\s+/, $_, 2 }
              $git->run( log => '--oneline-', '-n3' );
# (
#   b23123a => "Release v0.6.12",
#   6762cba => "do not try to fetch items ...",
#   d045083 => "errors from the server ...",
# )

Automatic Release via Git Tag

• Release new video game client and world editor
• Release in under 10 minutes
• Perl script builds installer package and XML release metadata
  • git tag
  • git log -n1
• Jenkins CI runs Perl script and syncs with remote server
• Client programs auto-update based on XML release metadata

update.pl - Build all tagged commits

my $git  = Git::Repository->new( work_tree => '.' );
my @tags = $git->run( 'tag' );

# Make sure each v-tag is in the update.xml file
for my $tag ( grep { /^v/ } @tags ) {

    # Get the information from git
    $git->run( checkout => $tag );
    my @log = $git->run( log => '-n1' );
    my $info = parse_git_log( @log );

    # ... build and add to update.xml
}

# Restore ourselves to master
$git->run( checkout => 'master' );

update.pl - Parse git log message

commit 88ecb78596865b6c1303816dc9051e8c62463993
Author: Doug Bell 
Date:   Tue Sep 18 22:28:51 2012 -0500

    Release v0.6.5

    Changes:
    * Add: Schema changes to support new item and 
      anim development
    * Add: Loading images. Scenes can specify an 
      image to show while they are being loaded. A 
      default image can be set in the World dialog.
    * Fix: Some of the loading problems in the 
      client are fixed

update.pl - parse_git_log(@lines)

sub parse_git_log {
    my @lines = @_;
    my $info = { }; my $in_msg = 0;
    for my $line ( @lines ) {
        # First empty line starts the commit message
        if ( $line =~ /^\s*$/ ) {
            $in_msg = 1;
        }
        elsif ( $in_msg ) {
            $info->{description} .= $line . "\n";
        }
        # Extract the date
        elsif ( $line =~ /^Date:\s+(.+)\s*$/ ) {
            my $parser = DateTime::Format::Strptime->new(
                pattern => '%a %b %d %T %Y %z',
            );
            my $dt = $parser->parse_datetime( $1 );
            $info->{date} = $dt->strftime('%F');
        }
    }
    return $info;
}

update.pl - End Result

• New package file
• Version data added to update.xml
• Client checks for updates next time it starts
• Shows the user the git log message containing the release notes
• Update installs automatically

Releasing with Submodules

• Lots of individual, CPAN-style dists
• Any dist changed should go into release
• Changed dists should be labelled with a release number
• The entire set of dists should be labelled with a release number

I Heard You Like Git Repos...

• git submodule
• Put a repo inside a repo
• Reference to the inner repo is stored
• Reference contains a commit ID
• git submodule add <repo>

The Setup

$ mkdir release && cd release
$ git init
Initialized empty Git repository in
/Users/doug/release/.git/

$ git submodule add ~/storyteller
Cloning into 'storyteller'
done.

$ git submodule add ~/swfconduit
Cloning into 'swfconduit'
done.

$ ls
storyteller
swfconduit

The Review

• Check each submodule to see if it can be updated
• git submodule lists the submodules and their commit
• git ls-remote lists all the branches for a remote
• We compare our submodule commit with the submodule's remote 'master' branch

Parse Submodule Refs

$ git submodule
 4abfbe96d4e7ba5821eb1adb99bb5856c3e0422e storyteller (v0.6.13)
 ef95f4294f4dc7554be523bd6cbd573c8c0ae594 swfconduit (v1.0.0)

• First character: status
• Ref ID
• Submodule name (directory)
• If available, remote ref

Parse Submodule Refs

sub git_submodule {
    my ( $git ) = @_;
    my %submodules;
    for $line ( $git->run( 'submodule' ) ) {
        # <status><SHA1 hash> <submodule> (ref name)
        $line =~ m{^.(\S+)\s(\S+)};
        $submodules{ $2 } = $1;
    }
    return %submodules;
}

Parse Remote Refs

$ cd storyteller
$ git ls-remote origin
b72ea4150cf59957dd9d2a847c3cb7b695b68e7b    HEAD
4abfbe96d4e7ba5821eb1adb99bb5856c3e0422e    refs/heads/master
f9d19573a65a1eb8c4caa6d6aaab00cafc7ce047    refs/heads/prop_iso_view
4abfbe96d4e7ba5821eb1adb99bb5856c3e0422e    refs/remotes/origin/HEAD
4abfbe96d4e7ba5821eb1adb99bb5856c3e0422e    refs/remotes/origin/master
a91fd98facea44769481f0256057dd47a2e2f934    refs/stash
efeade8251c5f46b58196624d21b110ac6dc9db0    refs/tags/v0.5.0
b72ea4150cf59957dd9d2a847c3cb7b695b68e7b    refs/tags/v0.6
08060acac4dc32b2577be0523b527187681aa851    refs/tags/v0.6.1

• Submodule repo is just like any other repo
• HEAD may not always be pointing to master

Parse Remote Refs

sub git_ls_remote {
    my ( $git ) = @_;
    my %refs;
    my @lines = $git->run( 'ls-remote', 'origin' );
    for my $line ( @lines ) {
        # <SHA1 hash> <symbolic ref>
        my ( $ref_id, $ref_name ) = split $line;
        $refs{ $ref_name } = $ref_id;
    }
    return %refs;
}

Compare the Two

use feature qw( say );

my $git = Git::Repository->new( work_tree => '.' );
my %submod_refs = git_submodule( $git );

for my $submod ( keys %submod_refs ) {
    my $subgit = Git::Repository->new(
                    work_tree => $submod,
                );
    my %remote = git_ls_remote( $subgit );
    if ( $submod_refs{ $submod } 
        ne $remote{'refs/heads/master'} )
    {
        say "$submod out of date";
    }
}

Update A Submodule

• Checkout the revision in the submodule
• Commit the submodule in the parent repository

$ cd storyteller
$ git checkout refs/remote/origin/master
Note: checking out 'refs/remote/origin/master'.

# ... some noise about 'detached HEAD'

HEAD is now at 4abfbe9... Release v0.6.13
$ cd ..
$ git commit storyteller -m'updated storyteller'
[master 8bae1c3] update storyteller
 1 file changed, 1 insertion(+)

Update A Submodule

my $git = Git::Repository->new( work_tree => '.' );
my %submod_refs = git_submodule( $git );

for my $submod ( keys %submod_refs ) {
    my $subgit = Git::Repository->new(
                    work_tree => $submod,
                );
    $subgit->run(
        checkout => 'refs/remotes/origin/master',
    );
}

$git->run( 'commit' => '-m' => 'Update submodules' );
$git->run( 'push', 'origin', 'master' );

A Release

• Update any submodules
• Tag submodule with release version
• Push submodule tag to remote repository
• Commit parent repository
• Branch parent repository for point releases
• Push parent repository to remote

A Release

my $release_version = 'v1.00';
my $git = Git::Repository->new( work_tree => '.' );
my %submod_refs = git_submodule( $git );

for my $submod ( keys %submod_refs ) {
    my $subgit = Git::Repository->new(
                    work_tree => $submod,
                );
    my %remote = git_ls_remote( $subgit );
    if ( $submod_refs{ $submod } 
        ne $remote{'refs/heads/master'} )
    {
        $subgit->run(
            checkout => 'refs/remotes/origin/master',
        );
    }

A Release (continued)

    $subgit->run( tag => $release_version );
    $subgit->run( push => '--tags', 'origin' );
}

$git->run(
    'commit' => '-m' => "Release $release_version"
);
$git->run( 'push' => 'origin', 'master' );
# Create release branch
$git->run( 'branch' => $release_version );
$git->run(
    'push' => 'origin', 
    "$release_version:$release_version"
);

The End!

• Slides: http://preaction.github.com/Perl/Scripting-Git.html
• Full update.pl code: https://gist.github.com/3943302 .

Slides are licensed under a CC-BY-SA 3.0 license.

Code is licensed under the Artistic License or GNU GPL v1.0 or later (the same terms as Perl itself).