Merge ~sudeephb/charm-nrpe:bug/1988360 into charm-nrpe:master

[Sudeep Bhandari]

Sync check_mem.pl with upstream

https://github.com/justintime/nagios-plugins/blob/master/check_mem/check_mem.pl

We might need to test and reopen the bug - https://bugs.launchpad.net/charm-nrpe/+bug/1930307

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:00f9d066fafbd9d9252cc92075d30be1c8e6c81d
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/130/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/940/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1363/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/130//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:00f9d066fafbd9d9252cc92075d30be1c8e6c81d
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/131/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/943/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1366/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/131//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:00f9d066fafbd9d9252cc92075d30be1c8e6c81d
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/132/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/952/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1375/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/132//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:12a20d58f8b4a699b92b4330db4e6ddaec040319
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/133/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/954/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1377/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/133//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:00f9d066fafbd9d9252cc92075d30be1c8e6c81d
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/134/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/956/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1379/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/134//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:12a20d58f8b4a699b92b4330db4e6ddaec040319
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/135/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/959/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1382/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/135//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:a0c4816ad62f8c372c314106dc9eac7fa3b2b316
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/139/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/977/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1399/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/139//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:a0c4816ad62f8c372c314106dc9eac7fa3b2b316
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/142/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1004/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1419/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/142//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:6e79a3f6531dc388c68d529f0e5ecbef107e4f44
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/143/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1005/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1423/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/143//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:a0c4816ad62f8c372c314106dc9eac7fa3b2b316
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/144/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1008/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1425/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/144//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:6e79a3f6531dc388c68d529f0e5ecbef107e4f44
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/145/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1012/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1430/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/145//rebuild

review: Needs Fixing (continuous-integration)

[Erhan Sunar (esunar)]

check_mem.pl results are not consistent with my checks. Approved referring to discussion(https://chat.canonical.com/canonical/pl/5n4sak7j57bqigyuugnffbsfmy)
Other parts are ok.

review: Approve

[Eric Chen (eric-chen)]

Functional test is solved in
https://code.launchpad.net/~sudeephb/charm-nrpe/+git/charm-nrpe/+merge/430781
So approve it.

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Failed to merge change (unable to merge source repository due to conflicts), setting status to needs review.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:4cda43666fee0c9561cb817607be8f3a2c21a5ef
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/148/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1044/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1456/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/148//rebuild

review: Needs Fixing (continuous-integration)

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision c42aebca6bb2c90fa958316a2d6eb851bbecd39a

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:4cda43666fee0c9561cb817607be8f3a2c21a5ef
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/149/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1087/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1503/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-nrpe-ci/149//rebuild

review: Approve (continuous-integration)

diff --git a/files/plugins/check_mem.pl b/files/plugins/check_mem.pl
index a1cd87c..7ec79fe 100755
--- a/files/plugins/check_mem.pl
+++ b/files/plugins/check_mem.pl
@@ -32,7 +32,7 @@ use Getopt::Std;
 #TODO - Use an alarm
 
 # Predefined exit codes for Nagios
-use vars qw($opt_c $opt_f $opt_u $opt_w $opt_C $opt_v $opt_h %exit_codes);
+use vars qw($opt_c $opt_f $opt_u $opt_a $opt_w $opt_C $opt_v $opt_h %exit_codes);
 %exit_codes   = ('UNKNOWN' , 3,
                  'OK'      , 0,
                  'WARNING' , 1,
@@ -43,8 +43,9 @@ use vars qw($opt_c $opt_f $opt_u $opt_w $opt_C $opt_v $opt_h %exit_codes);
 init();
 
 # Get the numbers:
-my ($free_memory_kb,$used_memory_kb,$caches_kb,$hugepages_kb) = get_memory_info();
+my ($free_memory_kb,$used_memory_kb,$caches_kb,$available_memory_kb,$hugepages_kb) = get_memory_info();
 print "$free_memory_kb Free\n$used_memory_kb Used\n$caches_kb Cache\n" if ($opt_v);
+print "$available_memory_kb Available\n" if ($opt_v and $opt_a);
 print "$hugepages_kb Hugepages\n" if ($opt_v and $opt_h);
 
 if ($opt_C) { #Do we count caches as free?
@@ -59,56 +60,112 @@ if ($opt_h) {
 print "$used_memory_kb Used (after Hugepages)\n" if ($opt_v);
 
 # Round to the nearest KB
-$free_memory_kb = sprintf('%d',$free_memory_kb);
-$used_memory_kb = sprintf('%d',$used_memory_kb);
-$caches_kb = sprintf('%d',$caches_kb);
+$free_memory_kb = sprintf('%.0f',$free_memory_kb);
+$used_memory_kb = sprintf('%.0f',$used_memory_kb);
+$caches_kb = sprintf('%.0f',$caches_kb);
 
 # Tell Nagios what we came up with
-tell_nagios($used_memory_kb,$free_memory_kb,$caches_kb,$hugepages_kb);
+tell_nagios($used_memory_kb,$free_memory_kb,$caches_kb,$available_memory_kb,$hugepages_kb);
 
 
 sub tell_nagios {
-    my ($used,$free,$caches,$hugepages) = @_;
+    my ($used,$free,$caches,$available,$hugepages) = @_; # KB
 
     # Calculate Total Memory
     my $total = $free + $used;
     print "$total Total\n" if ($opt_v);
 
+    # Absolute limits in KB
+    my $limit_warn;
+    my $limit_crit;
+
+    # is level absolute
+    my $is_abs_warn;
+    my $is_abs_crit;
+    my $type_check = "percentage";
+
+
+    # WARN
+    ($limit_warn, $is_abs_warn) = parameterize_thresholds($opt_w, "WARNING", $total);
+
+    # CRITICAL
+    ($limit_crit, $is_abs_crit) = parameterize_thresholds($opt_c, "CRITICAL", $total);
+
+    # Check if both levels are absolute or percentage
+    if ($is_abs_crit != $is_abs_warn) {
+        print "WARNING and CRITICAL should be both absolute or percentage";
+        &usage;
+    }
+    elsif ($is_abs_crit){
+        $type_check = "absolute";
+    }
+
     my $perf_warn;
     my $perf_crit;
-    if ( $opt_u ) {
-      $perf_warn = int(${total} * $opt_w / 100);
-      $perf_crit = int(${total} * $opt_c / 100);
-    } else {
-      $perf_warn = int(${total} * ( 100 - $opt_w ) / 100);
-      $perf_crit = int(${total} * ( 100 - $opt_c ) / 100);
+    if ( $opt_u ) { # used
+        $perf_warn = $limit_warn;
+        $perf_crit = $limit_crit;
+    } else { # free and available
+        $perf_warn = $total - $limit_warn;
+        $perf_crit = $total - $limit_crit;
     }
 
-    my $perfdata = "|TOTAL=${total}KB;;;; USED=${used}KB;${perf_warn};${perf_crit};; FREE=${free}KB;;;; CACHES=${caches}KB;;;;";
+    # Check if levels are sane
+    if ($limit_warn <= $limit_crit and ($opt_f or $opt_a)) {
+        my $opt = ($opt_f eq 1 ? "FREE" : "AVAILABLE");
+        print "*** WARNING level must not be less than CRITICAL when checking $opt memory!\n";
+        &usage;
+    }
+    elsif ($limit_warn >= $limit_crit and $opt_u) {
+        print "*** WARNING level must not be greater than CRITICAL when checking USED memory!\n";
+        &usage;
+    }
+
+    my $perfdata = "|TOTAL=${total}KB;;;;";
+    if ( !$opt_a ) {
+      $perfdata .= " USED=${used}KB;${perf_warn};${perf_crit};;";
+    } else {
+      $perfdata .= " USED=${used}KB;;;;";
+    }
+    $perfdata .= " FREE=${free}KB;;;;";
+    $perfdata .= " CACHES=${caches}KB;;;;";
+    $perfdata .= " AVAILABLE=${available}KB;${perf_warn};${perf_crit};;" if ($opt_a);
     $perfdata .= " HUGEPAGES=${hugepages}KB;;;;" if ($opt_h);
 
-    if ($opt_f) {
+    if ($opt_f) { # free
       my $percent    = sprintf "%.1f", ($free / $total * 100);
-      if ($percent <= $opt_c) {
-          finish("CRITICAL - $percent% ($free kB) free!$perfdata",$exit_codes{'CRITICAL'});
+      if ($free < $limit_crit) {
+          finish("CRITICAL - $percent% ($free kB) free-$type_check!$perfdata",$exit_codes{'CRITICAL'});
       }
-      elsif ($percent <= $opt_w) {
-          finish("WARNING - $percent% ($free kB) free!$perfdata",$exit_codes{'WARNING'});
+      elsif ($free < $limit_warn) {
+          finish("WARNING - $percent% ($free kB) free-$type_check!$perfdata",$exit_codes{'WARNING'});
       }
       else {
-          finish("OK - $percent% ($free kB) free.$perfdata",$exit_codes{'OK'});
+          finish("OK - $percent% ($free kB) free-$type_check.$perfdata",$exit_codes{'OK'});
       }
     }
-    elsif ($opt_u) {
+    elsif ($opt_a) { # available
+      my $percent    = sprintf "%.1f", ($available / $total * 100);
+      if ($available  <= $limit_crit) {
+          finish("CRITICAL - $percent% ($available kB) available-$type_check!$perfdata",$exit_codes{'CRITICAL'});
+      }
+      elsif ($available  <= $limit_warn) {
+          finish("WARNING - $percent% ($available kB) available-$type_check!$perfdata",$exit_codes{'WARNING'});
+      }
+      else {
+          finish("OK - $percent% ($available kB) available-$type_check.$perfdata",$exit_codes{'OK'});
+      }
+    }
+    elsif ($opt_u) {  # used
       my $percent    = sprintf "%.1f", ($used / $total * 100);
-      if ($percent >= $opt_c) {
-          finish("CRITICAL - $percent% ($used kB) used!$perfdata",$exit_codes{'CRITICAL'});
+      if ($used > $limit_crit) {
+          finish("CRITICAL - $percent% ($used kB) used-$type_check!$perfdata",$exit_codes{'CRITICAL'});
       }
-      elsif ($percent >= $opt_w) {
-          finish("WARNING - $percent% ($used kB) used!$perfdata",$exit_codes{'WARNING'});
+      elsif ($used > $limit_warn) {
+          finish("WARNING - $percent% ($used kB) used-$type_check!$perfdata",$exit_codes{'WARNING'});
       }
       else {
-          finish("OK - $percent% ($used kB) used.$perfdata",$exit_codes{'OK'});
+          finish("OK - $percent% ($used kB) used-$type_check.$perfdata",$exit_codes{'OK'});
       }
     }
 }
@@ -117,14 +174,29 @@ sub tell_nagios {
 sub usage() {
   print "\ncheck_mem.pl v1.0 - Nagios Plugin\n\n";
   print "usage:\n";
-  print " check_mem.pl -<f|u> -w <warnlevel> -c <critlevel>\n\n";
+  print " check_mem.pl -<f|u|a> -w <warnlevel> -c <critlevel>\n\n";
   print "options:\n";
-  print " -f           Check FREE memory\n";
-  print " -u           Check USED memory\n";
-  print " -C           Count OS caches as FREE memory\n";
-  print " -h           Remove hugepages from the total memory count\n";
-  print " -w PERCENT   Percent free/used when to warn\n";
-  print " -c PERCENT   Percent free/used when critical\n";
+  print " -a             Check AVAILABLE memory\n";
+  print " -f             Check FREE memory\n";
+  print " -u             Check USED memory\n";
+  print " -C             Count OS caches as FREE memory\n";
+  print " -w PERCENTAGE  Percent free/used/available when to warn\n";
+  print " -w SIZE K/M/G  Absolute size free/used/available when to warn\n";
+  print " -c PERCENTAGE  Percent free/used/available when critical\n";
+  print " -c SIZE K/M/G  Absolute size free/used/available when critical\n";
+  print " -v             Show verbose output\n";
+  print "\nexample:\n";
+  print "check_mem.pl -C -f -w 20 -c .5\n";
+  print "\tReturns 1 (WARNING) if less than 20% free memory.\n";
+  print "\tReturns 2 (CRITICAL) if less than 0.5% free memory.\n";
+  print "\tTakes caches into account.\n";
+  print "check_mem.pl -u -w 80 -c 95\n";
+  print "\tReturns 1 (WARNING) if more than 80% memory in use.\n";
+  print "\tReturns 2 (CRITICAL) if more than 95% memory in use.\n";
+  print "check_mem.pl -C -f -w 2G -c 500M\n";
+  print "\tReturns 1 (WARNING) if less than 2G free memory.\n";
+  print "\tReturns 2 (CRITICAL) if less than 500M free memory.\n";
+  print "\tTakes caches into account.\n";
   print "\nCopyright (C) 2000 Dan Larsson <dl\@tyfon.net>\n";
   print "check_mem.pl comes with absolutely NO WARRANTY either implied or explicit\n";
   print "This program is licensed under the terms of the\n";
@@ -166,15 +238,15 @@ sub get_memory_info {
                     $total_memory_kb = $2;
                 }
             }
-            elsif (/^MemAvailable:\s+(\d+) kB/) {
-                $available_memory_kb = $1;
-            }
             elsif (/^(Buffers|Cached|SReclaimable):\s+(\d+) kB/) {
                 $caches_kb += $2;
             }
             elsif (/^Shmem:\s+(\d+) kB/) {
                 $caches_kb -= $1;
             }
+            elsif (/^MemAvailable:\s+(\d+) kB/) {
+                $available_memory_kb = $1;
+            }
             # These variables will most likely be overwritten once we look into
             # /sys/kernel/mm/hugepages, unless we are running on linux <2.6.27
             # and have to rely on them
@@ -186,7 +258,7 @@ sub get_memory_info {
             }
         }
         $hugepages_kb = $hugepages_nr * $hugepages_size;
-        $used_memory_kb = $total_memory_kb - $available_memory_kb;
+        $used_memory_kb = $total_memory_kb - $free_memory_kb;
 
         # Read hugepages info from the newer sysfs interface if available
         my $hugepages_sysfs_dir = '/sys/kernel/mm/hugepages';
@@ -373,7 +445,39 @@ sub get_memory_info {
         $free_memory_kb = $memlist[1]/1024;
         $total_memory_kb = $used_memory_kb + $free_memory_kb;
     }
-    return ($free_memory_kb,$used_memory_kb,$caches_kb,$hugepages_kb);
+    return ($free_memory_kb,$used_memory_kb,$caches_kb,$available_memory_kb,$hugepages_kb);
+}
+
+sub parameterize_thresholds{
+  my $limit;
+  my $is_abs = 0;
+  my ($opt, $level, $total) = @_;
+
+  if ($opt =~ /^((\d+)\s*([KMG]))$/) {
+      # SIZE INTEGER K|M|G
+      $is_abs = 1;
+      $limit = $2;
+      $3 eq 'K' ? $limit *= 1 :
+      $3 eq 'M' ? $limit *= 1024 :
+      $3 eq 'G' ? $limit *= 1024 * 1024 : die ;
+      if ($limit > $total) {
+          print "*** $level limit: $limit level is bigger than the total of the system memory: $total";
+          &usage;
+      }
+  }
+  elsif ($opt =~ /^(?=.)(([0-9]*)(\.([0-9]+))?)$/) {
+      # PERCENTAGE (1, 95, 0.5)
+      if ($1 > 100) {
+          print "*** $level percentage > 100%!\n";
+          &usage;
+      }
+      $limit = int(${total} * $1 / 100);
+  }
+  else {
+      print "*** $level value not recognized!\n";
+      &usage;
+  }
+  return ($limit, $is_abs);
 }
 
 sub init {
@@ -382,26 +486,30 @@ sub init {
       &usage;
     }
     else {
-      getopts('c:fuChvw:');
+      getopts('c:fuaChvw:');
     }
 
     # Shortcircuit the switches
-    if (!$opt_w or $opt_w == 0 or !$opt_c or $opt_c == 0) {
+    if (! defined $opt_w or ! defined $opt_c) {
       print "*** You must define WARN and CRITICAL levels!\n";
       &usage;
     }
-    elsif (!$opt_f and !$opt_u) {
-      print "*** You must select to monitor either USED or FREE memory!\n";
+    elsif (!$opt_f and !$opt_u and !$opt_a) {
+      print "*** You must select to monitor USED, FREE or AVAILABLE memory!\n";
       &usage;
     }
-
-    # Check if levels are sane
-    if ($opt_w <= $opt_c and $opt_f) {
-      print "*** WARN level must not be less than CRITICAL when checking FREE memory!\n";
+    elsif ($opt_f and $opt_u or $opt_f and $opt_a or $opt_u and $opt_a) {
+      print "*** You must select to monitor either USED, FREE or AVAILABLE memory!\n";
+      &usage;
+    }
+    elsif ($opt_w !~ /^((\d+)\s*([KMG])|(?=.)(([0-9]*)(\.([0-9]+))?))$/) {
+      # SIZE INTEGER K|M|G OR PERCENTAGE INTEGER|FLOAT
+      print "*** WARN level must be defined as PERCENTAGE (1 - 99) or SIZE K/M/G!\n";
       &usage;
     }
-    elsif ($opt_w >= $opt_c and $opt_u) {
-      print "*** WARN level must not be greater than CRITICAL when checking USED memory!\n";
+    elsif ($opt_c !~ /^((\d+)\s*([KMG])|(?=.)(([0-9]*)(\.([0-9]+))?))$/) {
+      # SIZE INTEGER K|M|G OR PERCENTAGE INTEGER|FLOAT
+      print "*** CRITICAL level must be defined as PERCENTAGE (1 - 99) or SIZE K/M/G!\n";
       &usage;
     }
 }
@@ -410,4 +518,4 @@ sub finish {
     my ($msg,$state) = @_;
     print "$msg\n";
     exit $state;
-}
+}
\ No newline at end of file