Automating Content Extraction From Confluence Using Exporter Plugins and Bash

Intro

As a technical writer, one of the many tasks I must work with on a regular basis is pulling content from multiple sources and compiling them into a source file for publication. Early on, the process I used was a very hands on process of manually generating a compressed file from Confluence and other sources, manipulate the contents of the extracted content, bundle everything up, and publish the refined package.

This tutorial's goal is to get novice users familiar with one method of automating content extraction by using a simple Bash script file that pulls content from Confluence using an exporter scheme URL.

The process contains two components: an export scheme URL and a Bash script. I have written two tutorials on how to extract content using two of Scroll's exporter plugins. Refer to Part 2 and/or Part 3 in the Export Content from Confluence series for details on how to generate the export URL using either Scroll's EclipseHelp or HTML Exporter plugins. You will need to complete at least one of these tutorials in order to complete this tutorial as you will need the export scheme URL. The Bash script we will create handles the manual part of extracting content from Confluence.

This tutorial is written for Mac users. The Bash features used in this document has not been tested on Linux but you know your way around wget, this tutorial should work just fine.

Prerequisites

Confluence Command Line Interface

Did you know that Confluence has a CLI? Check out and install Confluence Command Line Interface (CLI) as we may need it to gain access to Confluence via the command line to check against extracted content but I'll leave that up to you to decide if you want to use it or not.

For installation, please review Confluence CLI Installation and Use.

For getting started, reference, examples, and much more info, please review Confluence CLI User's Guide.

wget

Another component to automating the export process is using a command line network transfer tool such as wget. There are a few options out there for handling CLI transfers (such as curl) but I've found wget to be rather flexible, handles redirects well, and stable for my documentation needs. If you have arguments for or against, I'd love to hear them in the comments.

Setting up a "docbot" account for export

Prior to automating the export process from Confluence, you will want to create a non-human account that only has view and export permissions. If you don't plan on sharing or automating content extraction from Confluence, you can use your personal account but I've seen many tech writers get bitten by using their personal accounts.

In this case, I named this new account "docbot". If you don't have the proper credentials to create Atlassian accounts, please contact your Confluence administrator and request the account be created. Otherwise, create a new user account:

1. Navigate to the Confluence Admin page.
2. Click on Users under Users & Security section.
3. Click Add Users.
4. Enter docbot in Username field.
5. Enter docbot in Full Name field.
6. If you have a group email address that is shared with the tech writers, I recommend using that for the Email field. Otherwise, enter your email address.

With the docbot account created, we need to provide it with the proper permissions.

1. Navigate to your target space's Space Admin page and click on Permissions.
2. Under the Individual Users section, click the Edit Permission button.
3. Locate the docbot account and enable only the following permissions:
• All > View
• Space > Export
4. Once those two permissions have been set, click Save all.

Your docbot account should now have the proper permissions to export content from Confluence.

Bash script

Once you have ran through the process of creating and saving an export scheme from either one of the aforementioned plugins, you will need to apply the REST URL in our Bash script.

The Bash script will contain up to four lines: up to three variables and one command. You may wish to add a few additional lines before and after the export process to set up directories like adding a few commands for setting up an export archive and post processing the downloaded file (like renaming the .jar file to a .zip file, unzipping, it and so on).

...
USER='docbot'
PASS='<docbot's password>'
URL='<exporter scheme URL>'

wget --content-disposition "$URL&os_username=$USER&os_password=$PASS"
...


Breakdown of this script

The first three lines are just variables we will pass into the wget command. The fourth line is the backbone of the whole operation. Lets example each component of this command:

• wget - network transfer command
• --content-disposition - flag that will force the download to preserve the file name. Note: This flag is will experimental though I've never hand any problems with it.
• "$URL&os_username=$USER&os_password=$PASS" - string that gets passed into the wget command. When pulling content from Confluence using the exporter scheme URL, you need to specify the exporter URL, provide the user requesting it (which it get checked for proper credentials, and the password). If everything checks out properly on Confluence's side, your command will pull down a compressed file based on the settings in your exporter scheme.

As mentioned earlier, one could use curl to pull content from Confluence but I found that command to be unreliable at times on my Mac. Maybe it was may flag options or some other setting but wget has worked very well for me for years using this configuration.

Note: if your Bash script will be shared with other users or in a hosted environment, you may want to localize the USER and PASS variables in your Bash profile.

From here, you can add any steps to the Bash script to include post processing, executing node scripts to modify extracted content, or whatever else you need to include in this automated process. Or, if you need to use this process on a regular basis, you can simply set a cron or Jenkins job.

Happy automating!

Export Content from Confluence - Part 3

Intro

Welcome to the final tutorial of this three part series on how to extract HTML content from Confluence.

In the previous two tutorials, we explored how to export content using Confluence's native solution Export Content from Confluence - Part 1 and Export Content from Confluence - Part 2. In this tutorial, we will run through the steps to generate HTML content into a zip file from an user selected space in Confluence using Scroll's HTML Exporter plugin. The biggest difference in using this plugin versus EclipseHelp Exporter, is that plugin offers the option to generate a search index of your extract content and the wiki's space logo will be a part of the final presentation.

Setup

This document assumes that you have the Scroll HTML Exporter plugin installed and properly enabled and that your user account has the proper credentials (at least view and export permissions) for exporting and plugin usage. If it's not, contact your Confluence administrator and request it to be set up.

This process was tested on Confluence 5.6.4 and Scroll HTML Exporter version 3.5.0. Results may very with other versions.

Exporting using HTML Exporter

1. Navigate to the top most page in the space you wish to export from Confluence.
2. Go to Tools > Export HTML.
3. Click Customize Settings in lower left corner. For this tutorial, we are going to customize the content will wish to pull from Confluence.
4. In the General step, you will need to make some choices depending on the needs of how and what you want to export from Confluence:
1. In the Create drop down, you can choose between one HTML file for each Confluence page or a single large HTML file. For this tutorial, we'll be using the former option to export our content to the HTML format.
2. Since this may be our first time using this plugin, the Template drop down will just list one option, Scroll WebHelp Template. This tutorial will not cover how to create HTML templates.
3. With the Export option, we'll select This page and its children so we can collect every page starting with the current page and all of it's children.
5. In the Central Processing step, you will have several options to output a few macros with your content, exporting images with original resolution, converting labels to index terms, and merging single, first heading and page titles. For this step, only enable the Export images with original resolution. If your pages in Confluence used the other macros previously listed and you need your exported content to use them, go ahead and enable those as well. Otherwise, leave those options disabled. I have found that exporting the content in as raw form as possible works best when attempting some post processing in my documentation publication scripts.
6. In the File Naming step, we can leave the settings to their default values. These settings works very well with the vast majority of export settings and the content coming from Confluence.
7. In the Search Index step, depending on your export needs, you may want to leave this option disabled. For this tutorial, leave it disable. When enabled, the plugin will generate a full text search index and add it to the exported content. Note: if you decide to export your content into one large HTML file, this feature won't be supported.
8. Click Start Export and wait a moment or two while Confluence churns on the request. An Export in Progress window will appear informing you of the content export status (pages processed versus total number of pages in the request).
   
9. Once the request has finished, your browser will download a zip file with the name of the page you selected to start the process from followed by the version number of the page, a date stamp of when it was exported, and a time stamp (set by the server clock). For example, your zip file name may look like this: Home-v12-20170104_1435.zip. The Export Result window will present with the size and number of pages in the export and how long it took. This window will also allow you to save the export scheme you just created, collect the ReST URL, and manage export schemes. If you are going to use this export process repeatedly, you should follow these steps:
1. Click the Save Export Scheme ... button and select as new from the drop down. 
2. In the Save new Export Scheme window, decide if this scheme will be used only in the current space or in all spaces (globally). For this tutorial, select in this space.
3. Provide a good descriptive name and description in their respective fields.
4. Click the Save button.
10. Optional: back in the Export Result window, you can collect the ReST URL if you wish to automate the process in the future. Click the REST URL button and copy the URL listed in the middle of the REST URL window. Save this URL for your automation script.
11. Optional: back in the Export Result window, you can manage the export schemes in either the current space or in all spaces (globally). A new browser window will open up and present you with options to modify any export schemes you may already have. That is, if you have the proper user permissions to see or edit features in the Space Admin page of the current space.

With the completion of this tutorial, you should have an HTML copy of your selected content and an export scheme that can be used repeatedly to pull content from your desired space in Confluence into an HTML zip file without having to go through this entire process again.

For future exports, all you need to do now is go to Tools > Export to HTML and select the export scheme you created in this tutorial.

Export Content from Confluence - Part 2

This tutorial is part two in the three series of how to export content from Confluence. In the previous tutorial, we covered how to export content from Confluence using it's native solution Export Content from Confluence - Part 1.

In this post, we'll explore the process of how to export content from our favorite wiki using Scroll's EclipseHelp plugin into an HTML zip file.

Setup

This document assumes that you have the Scroll EclipseHelp plugin installed and properly enabled and that your user account has the proper credentials (at least view and export permissions) for exporting and plugin usage. If it's not, contact your Confluence administrator and request it to be set up.

This process was tested on Confluence 5.6.4 and Scroll EclipseHelp plugin version 3.5.0. Results may very with other versions.

Exporting using EclipseHelp

1. Navigate to the top most page in the space you wish to export from Confluence.
2. To start the process, go to menu for Tools > Export to EclipseHelp.
3. If this is your first time using this plugin, you may see a few global templates to choose from. This tutorial opts to create an export using custom settings so you can see what is available via this plugin. In the bottom left corner, click on the Customize Settings option.
4. From the General step, select the Default EclipseHelp Template from the Template option (selected by default).
5. From the Export option, you can select what you need to export (the current page and it's children, only the current page, or the current page and any children with an user specified label). For this tutorial, we'll use the option to export the current page and all it's children.
   
6. Click the Content Processing step next. From here, you have several options on what you want to export with your content like exporting the table of contents (toc), children, section, and numbered-headings macros, images with original resolution, and merging single, first heading and page title. To keep it simple, let's not include any of these options except for the images with the original resolution. By selecting this option, the process will download the original size of any and all images that are included on the selected wiki pages.
7. Skip down to EclipseHelp Features. In this step, you can add any additional features you would like to include in your export. The features you choose to include in your export will be up to you but if the content you are pulling from Confluence is meant to be a stand-alone doc set, I would at least include the following:
1. From the Index option, select the Create Index and Convert Labels to Index Terms. Including an index will generate additional documents called index.xml, toc.xml, and a few others associated with all your pages that you included in this export.
2. From the Compatibility option, Export to doc.zip. Once the export has finished, everything will be nicely wrapped up in a zip file called doc.zip.
8. In the File Naming step, you can define how the exported content should be handled. I normally leave this section alone as the default values work well for most common needs.
9. With you export options selected, click the Start Export button to start the export process. Depending on the size of the space or page selection you made, Confluence will churn for a moment or two. You should see an indicator window pop up telling you where you are in the export process. Once Confluence has finished with the export process, you will see your browser download a doc.zip file.
   
   
10. Upon the completion of the export, EclipseHelp will tell you the size of the content, how many pages you exported, how long the process took, and present you with a few options can you can use in the future to make this process go a little faster: Saving the export scheme, capture the ReST URL (for automation reasons), and managing the export schemes. If you know you will be exporting this space with these settings repeatedly, I recommend saving the export scheme you just created. Click the Save Export Scheme ... drop down and select as new.
11. Decide if this export scheme will be used globally throughout your Confluence instance or localized to just this particular space.
12. Provide a good name and description and click the Save button.
13. If you wish to automate the process by using the ReST URL, you will need to save the export scheme first (see steps 9 through 12 to save the scheme).
14. With the export scheme saved, you can now click the REST URL button to receive the URL you can use for automating this export process. 

With the completion of this tutorial, you should have an HTML copy of your selected content and an export scheme that can be used repeatedly to pull content from your desired space in Confluence into an HTML zip file without having to go through this entire process again.

For future exports, all you need to do now is go to Tools > Export to EclipseHelp and select the export scheme you created in this tutorial.

Export Content from Confluence - Part 1

Intro

This post is the first of a three part series (part 2) of tutorials in which we'll explore the basics of exporting content from Atlassian's Confluence. This series will cover three different methods for exporting content from Confluence to a HTML format. In the first tutorial, we will cover how to use Confluence's native solution for exporting content to HTML and the pros and cons of this method. In later tutorials, we will cover how to use Scroll's EclipseHelp and HTML Export plugins to accomplish the same result but without the pitfalls that the native solution brings to the table.

Confluence has the capability of exporting the contents of a space to many formats such as PDF, Word, XML, and HTML. In this series, we will only focus on exporting to the HTML format. The reason why we will be focusing on the HTML format is that is a flexiable format that allows technical writers to apply post operations to the content prior to posting the content to it's final presentation form.

By the end of this tutorial, you should have a zip file with all the content you wish to export from Confluence in the HTML format.

1. Navigate to the top most page in the space you wish to export from Confluence.
2. Go to Space Tools  > Content Tools. The location of this link will vary depending on the theme you are using.
3. Click on the Export tab.
4. If you wish to export in HTML, select HTML from the Export Formats option and click Next. You can also choose from XML and PDF but those options will not be covered in this post.
5. If you wish to export everything in your target space, select Normal Export. However, if you wish to export a particular set of pages, select Custom Export.
6. By selecting Custom Export, you will be presented with every page in the space that can be exported (by default). 
7. Click the Deselect All option and check off with pages you wish to export. If you select a page that has children pages, those pages will automatically be selected as well.
8. If you wish to include comments with your select pages, leave the Include comments option enabled.
9. With the pages selected, scroll to the bottom of the page and click Export. Confluence will churn for a moment and present you with a download link. Click the download link to receive your zipped file of HTML content.

This method is kind enough to generate an index.html file that lists, and links, the content you just generated along with space details at the top of the page, and document generation date in the footer of the page. This zip file will also include any and all images, and style sheets used in the pages you selected. If you were hoping that it would include any JavaScript code from features found on your Confluence pages, I'm afraid you're out of luck. While some JavaScript files may be exported, I didn't see any evidence of scripts that recreate the behaviors found in the macros typically used within Confluence.

Drawbacks

There are two drawbacks to this export method: 1: It isn't automatable. 2: No options to save export settings. I'm all about automating any documentation task. But, as far as I know, this method cannot be automated (please correct me if I'm wrong). Unlike the other export methods available to Confluence (see Scroll's plugin solution), this method doesn't offer any options to save the export settings or which pages to export.

Notes

If you applied any properties to the images found on your pages that you selected, they may not appear the same as before. At the time of writing this post, properties like floats and borders did not translate. The CSS rules and classes are in the code that is included in the export but the effects of the CSS didn't work out of the box.

This tutorial was tested on Confluence 5.6.4. Mileage may very on other versions.

Source

Exporting Confluence Pages and Spaces to HTML

Using Node.js for Text Processing

Intro

As a tech writer who is responsible for writing and publishing documentation in various formats, I've found a need to combine my hobby of toying around with JavaScript and document publication. In particular, I'm tasked with pulling information from an Atlassian's Confluence site down into a static HTML file set. However, the method I use (Export EclipseHelp with a custom template) doesn't reliably generate clean or consistent HTML documents. While the original intent of this tutorial was to update content extracted from Confluence, it can work on any HTML file.

I figure there are better ways of doing what I'm about to demonstrate, but my needs are rather particular (as in, this script needs to function as part of a bigger puzzle I employ for publication). If you have suggestions to improve it, I'd love to hear it.

This document doesn't cover how to export HTML from Confluence. What will be covered is a script I came up with that will complete a find and replace function on all HTML files in a particular directory.

Node.js requirements

This little script needs only three modules to read, write, gather a file list, and use jQuery-like features:

var fs = require('fs');
var cheerio = require('cheerio');
var shell = require('shelljs');


Documents, meet array

Using a shell module, I gather all the HTML files in a particular directory:

var fileNames = shell.ls('documents/*.html');


String it up

Read each document as a string if the document has the extension of .html:

for (i in fileNames) {
  if (fileNames[i].indexOf(".html") > -1) {
    $ = cheerio.load(fs.readFileSync(fileNames[i]).toString());
    ...
  }
}

While it may seem a bit redundant to look through the array matching the HTML file type, the array returned in fileNames can end with an empty element in the array and cause our script to throw an error at the end.

Here we use the cheerio module to add jQuery-like features to our script so we can do things like select elements and modify them in a number of ways.

Process the string

Check if an element with the class of footer. If it exists, remove it.

if ($('div.footer').length > 0) {
  console.log("Removing footer from ../" + fileNames[i]);
  $('div.footer').remove();
} else {
  console.log(fileNames[i] + " has no div.footer element.");
}

At this step in the script, we can have the actively selected HTML document be processed in a multitude of ways (e.g updating elements in the header, injecting Bootstrap grid system, swapping image locations, adding date stamps, and so on).

Update and save

Update the string (document) and save it out.

var removed = $.html();
fs.writeFileSync(fileNames[i],removed);


Full code:

var fs = require('fs');
var cheerio = require('cheerio');
var shell = require('shelljs');
var fileNames = shell.ls('documents/*.html');

for (i in fileNames) {
  if (fileNames[i].indexOf(".html") > -1) {
    $ = cheerio.load(fs.readFileSync(fileNames[i]).toString());
    if ($('div.footer').length > 0) {
      console.log("Removing footer from ../" + fileNames[i]);
      $('div.footer').remove();
    } else {
     console.log(fileNames[i] + " has no div.footer element.");
    }

    var removed = $.html();
    fs.writeFileSync(fileNames[i],removed); // save out HTML file
  }
}

Long Ticket Lists and Excel

Intro

When updating a release note that includes long lists of tickets (in my case, hundreds of bug fixes), going through the list can be problematic to keep the list up to date due to numerous factors that influences the ticket's timeline and release date. One trick I figured out was to use Excel to show me what has changed.

How this works

1. Copy and paste the list of tickets into Excel (column A for example) from your ongoing or pre-lease document. It might be best to remove any formatting features in the list to  ensure content consistency.
2. Go to your source of JIRA tickets and collect the list of tickets and copy it.
3. Back in Excel, paste the new list into column B in the same Excel document. Again, make sure any formatting has been stripped from this list as well.
4. Select both columns and set Conditional Formatting with Highlight Cell Rules set to Duplicate Values.
5. Since the tickets highlighted are good to go, I like to format with green file with dark green text.

Any ticket that isn't highlighted in column B after setting this should stand out pretty well. These unhighlighted tickets will be the new tickets you need to add to your pre-release document. Any tickets that aren't highlighted in column A could mean that the ticket was removed.

Regardless of which tickets aren't highlighted in either column, you should still review the JIRA ticket to confirm it was added or removed.

Tip: You can quickly tell if tickets have been added and/or removed by selecting all tickets in each individual column and looking at the cell count. If the count is off, then some tickets have been added and/or removed. This system isn't perfect because sometimes a ticket can be removed and another ticket added thus keeping the ticket count the same. However, the highlighting will pick up these changes and show you.

Is this a perfect system? No. Does it do 80% of the work for me between ticket updates? Yes.

Can this be done using Google Sheets? Yes, but it isn't as easy to do in Excel nor is it reliable.

I would appreciate any feedback and/or thoughts on how they manage ticket lists.