Showing posts with label cli. Show all posts
Showing posts with label cli. Show all posts

Sunday, April 14, 2019

Comparing Published and Unpublished Documents in Confluence

Introduction

I recently had a challenge to upload over a thousand HTML documents to Confluence. I won't go into the details of what scripts I created using various Node.js modules, but I did want to share with you how I maintained a list of documents that were or were not published to Confluence.

Requirements

You should be comfortable with a terminal interface, managing documents in Confluence, and Confluence CLI plugin.

Using the Confluence CLI

I wrote a script that generates a list of documents and media files from a specified directory (I'll share that script and it's processes another time). From there I used the Confluence CLI plugin to report back a list of files that have already been uploaded to Confluence. The command was pretty simple:

confluence --action getPageList --id "<parent page id>" --descendents > uploaded_docs.txt

Note: the Confluence command itself needs to be setup as an alias in your Bash profile. The instructions for setting up the Confluence CLI plugin mentions how do some of this. My Bash alias looks something like this:

alias confluence="<path to confluence script>./confluence.sh --server <base Confluence URL> --user <user> --password <pasword>"

With that alias setup and a little forward thinking about how the space was going to be structured under a single document, I saved myself some time by parenting all the documents under this one ultimate parent document. (I wrote a script that handles that task as well which I'll share another time.) Having a single parent document, the CLI command reported back all the documents I needed to work with in one single execution of this command. Otherwise, I would have had to identify each parent document, execute this command on parent document, and tally up all the uploaded documents.

From here, with the two lists in hand, it was now a simple matter of finding the differences. There are several options out there to accomplish this but in the end, I just used Excel and used the conditional formatting feature to highlight the duplicates and the ones that weren't highlighted were the ones that needed to uploaded.

Maybe in the future I'll write a script that does this automatically from the two lists and share that process as well.

Thursday, February 14, 2019

Creating a CLI For a Node.js Script

Introduction

I've been noodling around with allowing my Node.js script accept arguments and decided it was time to document some of the basics of using a library to give my scripts the flexibility of a CLI flags.

The npm package commander allows any Node.js script to accept flags (unordered arguments) and display usage information or warnings. This tutorial will walk you through the basics of setting up a CLI, confirm that a required flag was submitted, and display any errors with the CLI arguments.

Requirements

You should be fairly comfortable with JavaScript in general and have some working knowledge of how Node.js and command line interfaces works prior to digging into this tutorial.

Required npm packages

In this tutorial, we'll need to ensure the following packages have been install in your project directory:
Note: This tutorial was written with Node.js (version 10.11.0).

Setting Up a Node.js Script With a CLI

Let's start out by requiring the commander module:

const program = require('commander');

Set the flags

Next, we need to set up what options our CLI will have. In this case, we'll set the usage info and foo and bar flags making the foo flag required. The version method can be any number you wish and is entirely optional but it's nice to let your users know how many iterations this script has, right? The usage method tells the users how to use this script as a CLI.

program
  .version('0.0.1')
  .usage('-f <foo> -b <bar>')
  .option('-f, --foo', '*Required* foo')
  .option('-b, --bar', 'bar')
  .parse(process.argv);

Getting the arguments

With the flags set up, the script needs to be able to get at the arguments. We'll use the arguments object to hold those values entered by the user in the terminal. The program object has a nested object called rawArgs that we can iterate through looking for matches to the flags we want to associate with the process argument input. It should be noted that rawArgs escape some special characters like single and double quotes, exclamation points, dollar signs, and so on.

var arguments = {};

for (var i = 0; i < program.rawArgs.length; i++) {
  if (program.rawArgs[i] == '--foo' || program.rawArgs[i] == '-f') {
    arguments.user = program.rawArgs[i + 1];
  }
  if (program.rawArgs[i] == '--bar' || program.rawArgs[i] == '-b') {
    arguments.pass = program.rawArgs[i + 1];
  }
}

Fail or success

With the arguments properly stored, we can now either fail or allow the script to continue with the main routine. Since we are only requiring the foo flag, we'll set the script to fail it is not supplied by the user. Otherwise, the script will continue onto the main routine.

if (!arguments.foo) {
  if (!arguments.foo) {
    console.log('Foo is required.');
  }
} else {

  console.log(arguments.foo, arguments.bar);
  ..main routine...
}

Friday, July 14, 2017

Inserting A Date Stamp into HTML Documents

Introduction

In Using Node.js for Text Processing, I showed you how a simple method to use Node.js processing content of HTML files. In this tutorial, I'll show you another technique for adding content to HTML files using Node.js as a command line interface (CLI) by supplying a script with arguments (flags), set up a basic help message, and append a date stamp to an user supplied HTML file. If this is your tune, let's jam!

Requirements

You should be fairly comfortable with JavaScript in general and have some working knowledge of how Node.js works prior to digging into this tutorial.

Required npm packages

In this tutorial, we'll need to ensure the following packages have been install in your project directory: 
Note: This tutorial was written with Node.js (version  ~0.12.7).

Building the script

First off, let's create a new file called append_date.js. As with any typical Node.js script, we start off with a few variables requiring our packages:

var fs = require('fs');
var cheerio = require('cheerio');
var program = require('commander');
...


In this snippet, we use fs to read in the target file, cheerio to manipulate the content with jQuery-like features, and commander for the CLI features of our script. Commander is a wonderful library that I recently discovered and works very well with my documentation and web developing needs.

Setting up the CLI

Like any well written CLI, we should start off with defining how it works. In the following snippet, we define the version of the CLI, how to use it (.usage), and any options (.option) available to the user. Commander is kind enough to provide a basic help output if the user types in <command> -h/--help as well as any usage and options you provide. Commander will display the default help message plus any info defined in the snippet below:

...
program
  .version('0.0.3')
  .usage('[required options] -t <path/file.html> -d <date>')
  .option('-t, --target [target]', '*required* target of HTML file (including the path and file name)')
  .option('-d, --date [date]', '*required* date to append to the HTML file')
  .parse(process.argv);
...

Main processing

Next, we check if both the target file and date flags have been supplied. From there, the script will load the target file and convert it to a string. The script will next check to see if a container element (div#content) exist and decide to create or update it. If the div#content element exists, the script will then look to either create or update the span.date element. With the elements in place, the script will now write out the updated HTML file.

...
if (program.target && program.date) {
  console.log('Updating ' + program.target);
  $ = cheerio.load(fs.readFileSync(program.target).toString()); // load file and convert it to a string

  if ($('div#content').length <= 0) { // if div#content doesn't exist, create it
    console.log(program.target + ' does not have div#content element. Adding it and datestamp now.');
    $('body').append('\t<div id="content">\n\t\t\t<span class="date">Last updated: ' + program.date + '</span>\n\t\t</div>\n\t');
  } else { // if div#content exists, look for span.date
    console.log(program.target + ' has div#content element; looking for date stamp')
    if ($('span.date')) { // if the span.date exists, update it
      console.log(program.target + ' has span.date element; updating it now');
      $('span.date').text('Last updated: ' + program.date);
    } else { // span.date doesn't exist; create it
      console.log(program.target + ' dos not have span.date element; adding it now');
      $('div#content').append('\t<span class="date">Last updated: ' + program.date + '</span>\n\t\t'); // add date under the content div
    }
  }

  var updated = $.html(); // collect updated html content
  fs.writeFileSync(program.target,updated); // save out updated HTML file
...



Fail checks

Like any good CLI, it should fail gracefully by informing the user that something was missing or failed in some way. To accomplish that, our script should check if the target and date flags were not filled out as they will be required for the script to carry out it's purpose. In the following code snippet, the first else if statement confirms if the target file wasn't supplied by the user. The second else if statement confirms if the user supplied a date flag. The final else statement is a catch all if something goes skips a beat.

...
} else if (!program.target) {
  console.log('Target for the HTML file was not provided; quitting.');
  process.exit(1);
} else if (!program.date) {
  console.log('No date stamp was provided; quitting');
  process.exit(1);
} else {
  console.log('undefined error; quitting');
  process.exit(1);
}

This completes the script. At this point, we should save the file and call it appendDate.js.

Note: this script doesn't verify the date string you enter into the span.date element. You could add a check for the program.date content to conform to a specific format using Regex (like /(\d[0-9]){2}-(\d[0-9]{1})-(\d[0-9]{1})/g) but that is entirely up to you.

CLI in action

With script completed, lets test it out on a simple HTML file. Copy the following code and save the file name as test.html in the same directory as your node script:

<!DOCTYPE html>
<html>
<head>
  <title>Date Stamp Test</title>
</head>
<body>
</body>
</html>


Let's test our script by executing the following command: node appendDate.js -t test.html -d 2017-03-30 (or whatever date string you wish to use). Examine the test.html. You should see our date stamp appended to the document with our custom date stamp.


Review

In ~40 lines of code, we created a script that takes arguments from the command line for a target file and a date string, loaded the target file, modified the target file with the date string, and wrote out the target file with the update HTML.

I hope this tutorial was useful and thank you for reading it.

Friday, April 14, 2017

Automating Content Extraction From Confluence Using Exporter Plugins and Bash

Intro

As a technical writer, one of the many tasks I must work with on a regular basis is pulling content from multiple sources and compiling them into a source file for publication. Early on, the process I used was a very hands on process of manually generating a compressed file from Confluence and other sources, manipulate the contents of the extracted content, bundle everything up, and publish the refined package.

This tutorial's goal is to get novice users familiar with one method of automating content extraction by using a simple Bash script file that pulls content from Confluence using an exporter scheme URL.

The process contains two components: an export scheme URL and a Bash script. I have written two tutorials on how to extract content using two of Scroll's exporter plugins. Refer to Part 2 and/or Part 3 in the Export Content from Confluence series for details on how to generate the export URL using either Scroll's EclipseHelp or HTML Exporter plugins. You will need to complete at least one of these tutorials in order to complete this tutorial as you will need the export scheme URL. The Bash script we will create handles the manual part of extracting content from Confluence.

This tutorial is written for Mac users. The Bash features used in this document has not been tested on Linux but you know your way around wget, this tutorial should work just fine.

Prerequisites

Confluence Command Line Interface

Did you know that Confluence has a CLI? Check out and install Confluence Command Line Interface (CLI) as we may need it to gain access to Confluence via the command line to check against extracted content but I'll leave that up to you to decide if you want to use it or not.

For installation, please review Confluence CLI Installation and Use.

For getting started, reference, examples, and much more info, please review Confluence CLI User's Guide.

wget

Another component to automating the export process is using a command line network transfer tool such as wget. There are a few options out there for handling CLI transfers (such as curl) but I've found wget to be rather flexible, handles redirects well, and stable for my documentation needs. If you have arguments for or against, I'd love to hear them in the comments.

Setting up a "docbot" account for export

Prior to automating the export process from Confluence, you will want to create a non-human account that only has view and export permissions. If you don't plan on sharing or automating content extraction from Confluence, you can use your personal account but I've seen many tech writers get bitten by using their personal accounts.

In this case, I named this new account "docbot". If you don't have the proper credentials to create Atlassian accounts, please contact your Confluence administrator and request the account be created. Otherwise, create a new user account:
  1. Navigate to the Confluence Admin page.
  2. Click on Users under Users & Security section.
  3. Click Add Users.
  4. Enter docbot in Username field.
  5. Enter docbot in Full Name field.
  6. If you have a group email address that is shared with the tech writers, I recommend using that for the Email field. Otherwise, enter your email address.
With the docbot account created, we need to provide it with the proper permissions.
  1. Navigate to your target space's Space Admin page and click on Permissions.
  2. Under the Individual Users section, click the Edit Permission button.
  3. Locate the docbot account and enable only the following permissions:
    • All > View
    • Space > Export
  4. Once those two permissions have been set, click Save all.
Your docbot account should now have the proper permissions to export content from Confluence.

Bash script

Once you have ran through the process of creating and saving an export scheme from either one of the aforementioned plugins, you will need to apply the REST URL in our Bash script.

The Bash script will contain up to four lines: up to three variables and one command. You may wish to add a few additional lines before and after the export process to set up directories like adding a few commands for setting up an export archive and post processing the downloaded file (like renaming the .jar file to a .zip file, unzipping, it and so on).

...
USER='docbot'
PASS='<docbot's password>'
URL='<exporter scheme URL>'

wget --content-disposition "$URL&os_username=$USER&os_password=$PASS"
...


Breakdown of this script

The first three lines are just variables we will pass into the wget command. The fourth line is the backbone of the whole operation. Lets example each component of this command:
  • wget - network transfer command
  • --content-disposition - flag that will force the download to preserve the file name. Note: This flag is will experimental though I've never hand any problems with it.
  • "$URL&os_username=$USER&os_password=$PASS" - string that gets passed into the wget command. When pulling content from Confluence using the exporter scheme URL, you need to specify the exporter URL, provide the user requesting it (which it get checked for proper credentials, and the password). If everything checks out properly on Confluence's side, your command will pull down a compressed file based on the settings in your exporter scheme.
As mentioned earlier, one could use curl to pull content from Confluence but I found that command to be unreliable at times on my Mac. Maybe it was may flag options or some other setting but wget has worked very well for me for years using this configuration.

Note: if your Bash script will be shared with other users or in a hosted environment, you may want to localize the USER and PASS variables in your Bash profile.

From here, you can add any steps to the Bash script to include post processing, executing node scripts to modify extracted content, or whatever else you need to include in this automated process. Or, if you need to use this process on a regular basis, you can simply set a cron or Jenkins job.

Happy automating!