Developing Low-Cost Functional Class 3 IETM
Developing Low-Cost Functional Class 3 IETM
Betty Harvey
Copyright & 2012 Electronic Commerce Connection, Inc.
Table of Contents
Betty Harvey
As President of Electronic Commerce Connection, Inc. since 1995, Ms. Harvey
has led many federal government and commercial enterprises in planning and
executing their migration to the use of structured information for their
critical functions. Over the past 14 years she has helped develop strategic XML
solutions for her clients. Ms. Harvey has been instrumental in developing
industry XML standards. Ms. Harvey is a member of OASIS Open and is currently an
active participant in the Universal Business Language initiative. Previously she
was a member of the Core Components subcommittee of the ebXML initiative. She is
the co-author of "Professional ebXML Foundations" published by Wrox. Ms. Harvey
founded the Washington, DC Area SGML/XML Users Group in 1995. She still
coordinates the users group which is the longest standing XML users group. Ms.
Harvey is also a member of "The XML Guild" and recently coauthored the book
"Advanced XML Applications From the Experts at The XML Guild" published by
How to cite this paper
Developing Low-Cost Functional Class 3 IETM
Balisage: The Markup Conference 2012
Introduction
The MIL-STD-38784 data used as examples in this paper and demonstration is SGML
data that is provided
as part of MIL-STD-38784 (BV8). The graphics used in the
manual are generic bicycle graphics modified to support the functionality.
graphics and SGML/XML tagging have no correlation except they have been arbitrarily
linked for demonstration purposes..
IETM Background and History
U.S. Department of Defense (DoD) started down the path of Interactive Electronic
Technical Manuals (IETM) in the early 70&s.& The first initiative was the &Paperless
Ship& initiative.& The original idea behind the Paperless Ship initiative was to
digitize all paper manuals and provide them in electronic form.& The first initiative
took paper manuals, scanned and created microfiche images from the pages of the
manuals.& Every ship was provided microfiche readers.& Many of the microfiche images
were very poor quality. Scanning documents was new technology at the time and not
perfected.
In practice, most technicians printed the relevant pages from the manuals they were
interested in. After all the relevant pages were printed, the microfiche readers became
big paperweights and gaithered dust.& It wasn&t feasible for technicians to take the
microfiche readers to their maintenance areas or fit a microfiche reader to their bunk
to study. So much for microfiche!
The next phase of the Paperless Ship initiative was electronically indexed page images
which began in the early 80&s.& Maintenance manuals were digitized into TIFF graphics.&
Indices were created in the background which allowed readers to search and navigate to
the point in the graphical representation of the document.& This obviously was a vast
improvement over microfiche.& The original Navy Advanced Technical Information Support
(ATIS) program was a fairly successful initiative.
The Computer Aided Logistic Support (CALS)
initiative began in 1985.& In later years
the CALS acronym morphed into "Continuous Acquisition and Lifecycle Support" and finally
"Commerce at Light Speed". The CALS initiative resulted in numerous military
specifications for graphics, publishing, integrated database and IETMs.& Tri-Service
working groups were created which consisted of Army, Navy and Air Force representatives,
as well as government contractors and interested software vendors.& These working groups
cooperated to create joint specifications which were designed to be used by all U.S. DoD
organizations.& These specifications were also used as contractual requirements for DoD
logistics suppliers.&
DoD SGML Publishing and Specifications
The specification
(MARKUP REQUIREMENTS AND GENERIC STYLE SPECIFICATION FOR
EXCHANGE OF TEXT AND ITS PRESENTATION) was originally published in 1992.& The last
printing was MIL-PRF-280001C which was published in 1997.& MIL-PRF-28001 specified the
use of SGML for all new technical manuals within DoD.& Each branch of the service
developed DTD&s for use within their individual organizations based on their specific
requirements and adhering to the requirements in MIL-PRF-28001.&
In addition to specifying the SGML constructs for developing DTD&s, MIL-PRF-28001
provided a DTD and specification for applying styling to the SGML.& The styling
specification was called Formatting Output Specification Instance (FOSI).& Appendix B of
the specification contained the DTD that supported the use of FOSI for presentation.&
Several SGML vendors were part of the working group and were working toward developing a
FOSI-based publishing system.& Two vendors DataLogic and Arbortext successfully
developed FOSI-based formatting within their product.& However, the implementation was
slightly different in both systems based on different interpretations of the
specification and ambiguity of the DTD.& The result was that a FOSI developed for one
system could not be used in other systems.
About the time the FOSI was being finalized, the JCALS (Joint Computer Aided
Logistics) program began which implemented an integrated publishing system for the
components of CALS integrated into the hardware/software platform. JCALS included SGML
authoring, editing and publication components.&& Arbortext was the editor and DataLogic
performed the publishing in JCALS.& The system was available and started being used and
mandated for technical publications in the
timeframe.
IETM Specifications
The CALS initiative had three IETM specifications.
MIL-PRF-87268 - General Content, Style, Format and User Interface
MIL-PRF-87269 - Data Base, Revisable: Interactive Electronic Technical Manuals
(IETM) for the support of (Contains the IETM DTD
MIL-Q-87270 - Quality Assurance (rescinded in 1994 because the specification
was hard to interpret and there weren't any implementation of this
specification)
The IETM user interface (GCSFUI)
specification was developed in 1992.& The
presentation was very simplistic.& Window-based systems were in their infancy and the
user interface, even at the time of publication, was very linear.&
The SGML specification for the data was based on SGML and HyTime.& The IETM systems
were custom developed and were costly.& Getting data to valid MIL-PRF-87269 SGML was
very expensive.& In 1995 the estimate for authoring or converting data was $100 a page.&
Once the data was in an 87269 SGML format it was very difficult to produce written pages
because of the fault isolation and navigation linking that took place.
Small programs and component manufacturers could not economically afford such an
expensive endeavor.& If and organization could afford getting the data into a compliant
87269 format,
obtaining knowledgeable personnel was difficult to find.& Most
knowledgable individuals were employed by the large defense contractors and smaller
companies could not compete at that level.
DoD and the Navy in particular realized that a &one-size fits all& approach to IETM&s
wasn&t realistic.& The Navy adopted a class system for discussing
and identifying the types of IETM&s in 1994 .& The Navy defined 5 classes of IETMs.&
The class definitions only include SGML but
is a subset of SGML and there is a direct correlation.
&Class 1 - Page Turner (limited Navigation)
Raster Scanned, i.e., PDF
Class 2 - Electronically Scrolling Documents - Navigation
May be SGML but not necessarily
Class 3 - Linearly Structured IETM Documents - SGML
Class 4 - Hierarchical Structured IETM
Conforms to MIL-D-87269
Class 5 - Integrated Electronic Information System
Expert System, Artificial Intelligence, Integrated System
Diagnostics
About the time the U.S. DoD IETM specifications were published the
published the S1000D specification.& S1000D was in
direct competition with MIL-PRF-87269.& S1000D is being used on many European and now
U.S. DoD projects.& Today we see limited 87269 implementations.
Fast Forward to 2012
Today U.S. DoD is still requiring organizations to deliver technical data in SGML.&
Some contracts still stipulate that the documents have to be published using the FOSI&s
that were developed for the JCALS systems.& JCALS software and the hardware that
supports it is no longer available.& JCALS ran on Sun Microsystems platform.& The
software components were tightly integrated.&
Companies who have had long-standing contracts (5 years or more) with DoD to supply
manuals in whatever format that was specified, i.e., Word, WordPerfect, etc. are
receiving new contracts that have contractual language requiring SGML and published PDF
using JCALS FOSI's.
Most of these organizations have never heard of SGML or FOSI's.
The documentation part of the contract is a small component of their
deliverables.
This was case the for Cobham.& They
Air Force modified the documentation portion of
their contract to deliver technical manuals in SGML conforming to MIL-STD-38784C or
MIL-PRF-87269.& Initially there was a stipulation that a PDF for each manual be
delivered using the JCALS FOSI (DataLogic). &&These requirements are impossible for them
to achieve, especially with MIL-PRF-87269.& During the initial phase of the requirement
by the Air Force to move to SGML, Cobham management and staff received an education in
SGML and surrounding technologies. SGML, FOSI, and XML. This was brand new territory for
them.& &In order to understand the nuances of the requirements of the contract they
needed to be made aware of the realities of modern-day SGML publishing and it's
challenges and limitations, i.e., the lack of available software.& Through negotiation
the Air Force agreed to the following deliverables:
&SGML Manuals
TIFF graphics
Class 3 IETM
The first step was to educate the staff in SGML authoring and
MIL-STD-38784C in
particular.&Their manuals were in Word and conformed to the Air Force format as outlined
Technical Manual Contract Requirements (TMCR)
The staff was also provided Arbortext Editor training.&& The staff involved in the
editing of manuals included a team of 3 people.&
Jointly we &were able to create a Class 3 (with some Class 4 functionality) IETM at a
relatively low cost.& We developed a product that the Air Force was very happy with and
accepted in lieu of PDF.& The IETM met and exceeded the following capabilities:
Navigation bar for Table of Contents
Inline graphics with links to PDF file with full-size& graphics. Graphics and
links can be SVG if hot spots are provided.
Integrated Parts Breakdown (IPB) with linking to specific graphics.
Bi-directional linking is available with SVG graphics and hotspotting.
Full-text search.
Consolidated list of tables with links to appropriate locations
Consolidated list of figures with links to appropriate location.
Software Used
This section will describe the software that we used to create the Class 3 IETM.&
All the software is either open source or commercial-off-the-shelf (COTS) products.&
The cost of the software was relatively inexpensive.
Apache Subversion (SVN): SVN is an open-source
software versioning and revision control system.& SVN allowed Cobham personnel and
ECC to exchange files and configurations without the necessity of being co-located.&
SVN was critical during the training process for Cobham SME's.& It also provided
Cobham the ability to remotely test the development of the IETM and provide
real-time feedback.& An SVN repository was established on a Linux (CENTOS) box that
was accessible to both Cobham and ECC.
Tortoise SVN Client:& Tortoise SVN client was
used to communicate between the SVN repository and local Microsoft Windows
computers.& Tortoise allowed individuals to check-in and check-out data.
Editor:& Arbortext Editor & Arbortext Editor was
selected because it is one of the only editors that supports SGML.& Most XML editors
only support XML.& JustSystems XMetaL also supports SGML editing but we decided to
use Arbortext to take advantage of Arbortext included 38784 stylesheets.&
XSLT:& Custom XSLT scripts were created to create
the customized HTML and XSL-FO files.& Saxon is used as the XSLT transformation
XSL-FO Engine:& RenderX
Search Engine: &Zoom Search Engine by Wrensoft.&
Zoom is a low-cost search engine that provided quite a bit of functionality and
allowed the Javascript that controls the search to be bundled with each book.
SVG Editor:& Inkscape.
SVG was only used to test
the feasibility and currently isn't being used.
The SVN repository was created as a 'self-contained' environment.& This provided
Cobham with the ability to add manuals and convert them using a single batch file.&
The figure below shows the structure of the SVN repository.
The green check mark on the folders shows that the directory is up to date.& When
a red exclamation mark is displayed in the file navigator it is a clue to the editor
they need to update the SVN repository with their modifications.& I will describe
the contents of the directories later in this paper.
Conversion from Word to MIL-STD-38784C
manuals were initially written using Microsoft Word.& An XSLT script was
written to initially convert the Microsoft Word manuals to MIL-STD-38784C.& The
manuals that Cobham had written were very well formatted so the conversion was
better than most Word to XML conversions.& Cobham authors were consistent in the use
of styles within the Word file.& Consistency in use of styles within a desktop
editor file makes conversion to XML much easier and provides a better end
The initial conversion was to XML because XSLT only works on XML. Another option
would have been to use Omnimark and go directly to SGML but there are more XSLT
developers then Omnimark developers and if modifications were required any XSLT
developer could make the modifications.
Arbotext Editor supports both SGML and XML
so the obvious choice was XML.& The data was &saved as& SGML after the final edits
were made to the manual and it was ready to be delivered.& The SGML version of the
file is the official deliverable.& The XML version is used to create the IETM
There were some manual edits that were required after the Word to XML conversion.&
These edits required some reorganization of text. Most military technical manual
specifications are designed to be very hierarchical and are very stringent in the
organization of content.
Some of the areas that needed to be reorganized or rewritten by a subject-matter
expert (SME) was:
Multiple paragraphs in sections - MIL-STD-38784C only allows 1 paragraph
in a hierarchical structure, i.e., para0 (equivalent to a section).& The SME
was required to evaluate the text and determine the best approach for making
the SGML valid according to the DTD and not affect the functionality of the
manual negatively.
Alerts && &MIL-STD-38784C specifies that alerts (warnings, cautions and
notes) must appear before the text the alert refers too.& Alerts also must
appear in the &Warning, Caution, Notes& sequence.& The Word manuals followed
this construct for warnings and cautions, however, notes occasionally were
located after paragraphs.& The conversion tried to capture these occurrences
but wasn&t 100% successful because it required evaluating text to determine
if a paragraph inside the Word file was a note.
Tables & Tables converted fairly well.& A few of the complicated tables
required modification of spans and column and row borders.
Figures & Figures also converted fairly well.& Problems arose when
multiple graphics were included in a single figure.& Technical manuals tend
to have &sheet& information
Integrated Parts Breakdown (IPB)
The presentation of the IPB is in table form in the printed manual.& The only
differentiating characteristic to identify the hierarchy of assemblies is
leading dots (.) in front of the description.& Below is a fictional example of a
printed IPB. & The periods (.) in front of the part description defines the
hierarchy.& &Getting the hierarchy correct in the XML required analyzing the
text in the description before outputting the assembly level.
&DESCRIPTION
1& 2& 3& 4& 5& 6& 7
TOP LEVEL ASSEMBLY
................................
.& SUB 1, Assy of
...........................................
.& SUB 2, Assy of
.........................................
.& SUB 3, Desc.
............................................
.& .& SUBSUB 1
.............................................
.& .& SUBSUB 2
.............................................
.& .& .& SUBSUBSUB 1
..................................
A separate XSLT was written for the IPB after the entire manual had been
converted from Word to 38784C.& The IPB section was initially converted to a
CALS table model during the Word to XML conversion.& The IPB table was converted
in a second step to create a valid &ipbpl& structure from the converted
Microsoft Word IPB table.& Below is the above sample IPB tagged according to
MIL-STD-38784C:
&ipbplgrp&
&figure id="Examplefig6-2"&
&title&TOP LEVEL ASSEMBLY&/title&
&graphic boardno="Example-fig6-2" reprowid="5.0in" reprodep="8.0in"/&
&ipbpl0tbl&
&ipbfigindex& 6-2- &/ipbfigindex&
&partno&P-1234-1 &/partno&
&ipbcage&04577&/ipbcage&
&nomen&TOP LEVEL ASSEMBLY&/nomen&
&/ipbdesc&
&ipbunits& 1 &/ipbunits&
&ipbuseoncode/&
&ipbsmrcodes&
&ipbsmrcode&EX001&/ipbsmrcode&
&/ipbsmrcodes&
&ipbfigindex& -1 &/ipbfigindex&
&partno& P-&/partno&
&ipbcage&04577&/ipbcage&
&nomen&SUB 1&/nomen&
&/ipbdesc&
&ipbunits& 1 &/ipbunits&
&ipbuseoncode/&
&ipbsmrcodes&
&ipbsmrcode&EX002 &/ipbsmrcode&
&/ipbsmrcodes&
&ipbfigindex& -2 &/ipbfigindex&
&partno& P-&/partno&
&ipbcage&04577&/ipbcage&
&nomen&SUB 2&/nomen&
&/ipbdesc&
&ipbunits& 1 &/ipbunits&
&ipbuseoncode/&
&ipbsmrcodes&
&ipbsmrcode&EX003 &/ipbsmrcode&
&/ipbsmrcodes&
&ipbfigindex& -3 &/ipbfigindex&
&partno&P-&/partno&
&ipbcage&04577&/ipbcage&
&nomen&SUB 3, Desc &/nomen&
&/ipbdesc&
&ipbunits& 1 &/ipbunits&
&ipbuseoncode/&
&ipbsmrcodes&
&ipbsmrcode&EX004&/ipbsmrcode&
&/ipbsmrcodes&
&ipbfigindex& &/ipbfigindex&
&partno&P--1&/partno&
&ipbcage&04577&/ipbcage&
&nomen&SUBSUB 1&/nomen&
&/ipbdesc&
&ipbunits& 1 &/ipbunits&
&ipbuseoncode/&
&ipbsmrcodes&
&ipbsmrcode&EX005&/ipbsmrcode&
&/ipbsmrcodes&
&ipbfigindex& -4 &/ipbfigindex&
&partno&P--2 &/partno&
&ipbcage&04577&/ipbcage&
&nomen&SUBSUB 2 &/nomen&
&/ipbdesc&
&ipbunits& 1 &/ipbunits&
&ipbuseoncode/&
&ipbsmrcodes&
&ipbsmrcode&EX006 &/ipbsmrcode&
&/ipbsmrcodes&
&ipbfigindex& -5 &/ipbfigindex&
&partno&P--2-1&/partno&
&ipbcage&04577&/ipbcage&
&nomen&SUBSUBSUB 1
&/ipbdesc&
&ipbunits& 1 &/ipbunits&
&ipbuseoncode/&
&ipbsmrcodes&
&ipbsmrcode&EX007 &/ipbsmrcode&
&/ipbsmrcodes&
&/ipbpl0tbl&
&/ipbplgrp&
IETM Development
The development of the IETM code was developed using an agile, iterative process.& Cobham
staff was involved in each step of the development.& They provided feedback and guidance
on the look and feel of the IETM throughout each iteration.& Creating HTML from the XML
version of the manuals was relatively straight-forward, however, decisions had to be
made about the necessary functionality and look-and-feel.&
MIL-STD-38784C was developed
with paper or PDF output in mind.& It is not a true IETM specification.& The following
decisions were required:
1.&&&&&& Navigation from components
2.&&&&&& Level of granularity of text (chunking)
3.&&&&&& Specialty sections, i.e., List of Tables.
4.&&&&&& Handling graphics
Navigation From Components
We decided to use a framed-approach for the IETM.& A navigation panel is on the left-side
of the screen.& The user can navigate using a hierarchical table of contents.& We also
added widgets for expanding and collapsing navigation hierarchy.
We also added a button
for searching the entire contents.& Below is the start page of the sample IETM.
Search Engine
We selected a very inexpensive search engine called Zoom from Wrensoft.&Zoom will index
directories and subdirectories.& It is highly configurable and had all the functionality
required to be able to search and navigate to appropriate locations in the IETM. The
search engine can be used to create indexes of websites or documents for CD.& Currently
each manual is indexed separately however, if a family of manuals were included the
search engine index session the product could index more than a single manual.& The
process for creating the index is manual and is performed by Cobham SME's after the book
has completed the editorial process and the IETM has been generated and is ready to be
delivered.
Zoom creates 5 files that get included into the root directory where the indexed files.&
The conversion of
XSLT scripts create the interface that enables the search
functionality during conversion from XML to HTML.
Graphics we problematic.& Figures in a MIL-STD-38784C paper manual are full-page
graphics.& Large, highly detailed graphics do not scale well in a viewing pane.&During
initial design we provided the graphics in-line but they really weren't acceptable
because of the sheer size of the graphics. We solved this problem by providing links in
the IETM to the graphic.& A PDF file was created for every figure in the manual using
XSL-FO.& A
XSL-FO (.fo) file is created int he graphics directory for each graphic in
the manual.
The XSLT creates a separate Windows batch file that get run separately to
call RenderX and create the PDF file of the graphic.
The PDF file contained the figure
number, the title of the figure and the graphic itself.& All the graphics in the manual
are currently in TIFF formatted files.
SVG Graphics
We did a proof of concept for creating SVG. &&We tested putting warnings, cautions and
notes into a presentation defined in GCSFUI.& The challenge was ensuring that the &text not
overflow the boundary of the box.
SVG Generated Warning from XSLT (NOTE:& Text replaced - actual graphic will be used once
non-proprietary data is converted)
We also tested putting overlays on the on the bitmapped graphics.& We successfully created
hotspots on the graphics themselves and linked them
bi-directionally to and from the IPB to the PDF graphic.& This functionality is very
easy to obtain during the conversion by using id/idref functionality.& The time
consuming part of this approach was having manpower to manually create the hotspots.
This can be very time consuming when there are hundreds of call-outs in a graphic. There
were no requirements for graphic hotspots so this functionality was put on the back
None of the SVG graphics were used in the final product but we did prove that it could
be done and if the hotspots were available in the graphic during development of the
product it is relatively easy to accomplish.&
On-site (Cobham) Production of ETM
After the first few manuals Cobham was off and running on their own.& They were
authoring new manuals to MIL-STD-38784 and creating the ETM version based on the
environment we setup in SVN.& We have since broken the SVN connection because they
don't require support.
Conclusion
This project was very successful.& The Air Force was happy with the deliverables.&
Cobham was able to deliver a quality product to the Air Force.& The challenge is still
obtaining necessary technical editors who are knowledgeable about SGML (and XML).
Whenever there is a turnover in personnel new people need to either have an SGML/XML
background or be trained in the technology.&
This project could not have been as successful without the vision of Cobham and their
dedication to producing a quality product.& The military SGML specifications are
antiquated but with a little ingenuity and knowledge of the technology it is possible to
make them work with today's technology without costing the government and creating
organizations millions of dollars.
References
& MANUALS, INTERACTIVE ELECTRONIC TECHNICAL - GENERAL CONTENT, STYLE, FORMAT,
AND USER-INTERACTION REQUIREMENTS, October 1, 1995,
http://www.navsea.navy.mil/nswc/carderock/tecinfsys/etm/sta-spe/pdf/S87268A.PDF
MIL-PRF-87269A, DATA BASE, REVISABLE - INTERACTIVE ELECTRONIC TECHNICAL
MANUALS, FOR THE SUPPORT OF, October 1, 1995,
THE MOVE TO PAPERLESS TECHNICAL MANUALS IN THE US DOD by Eric L. Jorgensen,
Naval Surface Warfare Center, Carderock Division, U.S. Department of the Navy ()
MIL-PRF-28001C, MARKUP REQUIREMENTS AND GENERIC STYLE
SPECIFICATION FOR EXCHANGE OF TEXT AND ITS PRESENTATION,
DoD CLASSES OF ELECTRONIC TECHNICAL MANUALS, Eric L.
Jorgensen,
STANDARD PRACTICE FOR MANUALS, TECHNICAL: GENERAL STYLE
AND FORMAT REQUIREMENTS, July 2, 1995,
International Specification for Technical
Publications utilizing a Common Source Database ,}