Exploring the Domain

 

We employed a variety of research methods to understand the context of procedure execution.

 
WE ESTABLISHED A FOUNDATION OF RESEARCH TO INFORM OUR FUTURE DESIGN.

ON-SITE TOURS / INTERVIEWS
We interviewed 20 NASA employees at Kennedy Space Center and Ames Research Center to understand their pain points and work processes. 

ANALOGOUS DOMAIN RESEARCH
We performed competitive analysis research from other Maintenance, Repair and Overhaul (MRO) industries. By drawing from various industries, we gained access to work processes similar to our users (which we had limited access to) and allowed us to 
cross-pollinate ideas from one domain to another.

RESEARCH SYNTHESIS
From our body of research, we created visual diagrams to communicate our findings to each other. As a research method, it helped us focus and prioritize our most important data. 
20180212_112222

From this data, we distilled 5 key insights from our research.

noun_knowledge_1508652
Technician knowledge is valuable, but its capture and representation is not institutionalized
Technicians have detailed knowledge of work processes, but no formal process exists to ensure this knowledge loops back to WAD (work authorization document) authors. This leads to lost tribal knowledge.
noun_Space Shuttle_844911
Low-volume manufacturing demands that data be captured and surfaced in the most efficient manner
NASA does not have the luxury of learning from multiple trial-and-error sessions. Each launch is a rare opportunity to capture and analyze a rich context of data for future lessons learned. 
noun_process_194045
Process is a primary tool to mitigate risk. Process can be expedited or not followed
NASA faces a tension between “getting things done fast” and “getting things accounted for”. Process helps promote accountability and reduce risk, but certain situations require exceptions to that rule.
noun_Expansion_308097
Data bloat creates challenges in surfacing information effectively
NASA’s data is stored without an easy way to reference them. Older data is unstructured, named differently, and document revisions are created without notifying all relevant users.
noun_Paper_1231161
NASA is not quite ready for digital (paperless) systems
Digital tools are required in today’s industries, but NASA's implementation isn’t perfect. Common issues like lack of internet connectivity, no save states, and lack of flexibility leave employees frustrated.

 

These insights are supported by our research — key findings are shown below.

These insights are supported by our research —
key findings are shown below.

INTERVIEWS

Herschel_spacecraft_during_distance_measurements_between_the_primary_and_secondary_mirrors_node_full_image_2 (1)

TECHNICIANS

Authoring a WAD is just like writing a procedure for getting in your car. It would be written under the assumption of ideal conditions, but we'd like to say rarely does a plan sustain enemy contact. Any number of things can happen.
TAKEAWAY 
Some instruction deviations are avoidable, others aren't. When technicians deal with the unknown unknowns, the deviation becomes a learning opportunity.

ENGINEER [WAD AUTHOR]

There was a real issue in Shuttle where the head techs didn't look at the work orders because they knew the steps by heart. And it's led to an issue for us because the old work instructions are wrong since no one ever updated them.

You have some really great techs here during Shuttle that are excellent sources of info because they know how the procedures are supposed to be performed, and you do yourself a disservice if you don't at least get their opinion.
TAKEAWAY
Despite explicit policy, not everything gets written down. Personal relationships and communication are necessary to unearth tribal knowledge.  
ksc-314d-0132_0061b
wstf1209e13503_qa

QUALITY ASSURANCE ENGINEER

Sometimes we get a WAD draft that shouldn't be on our desks. For example, someone authored a WAD that I immediately rejected. They just copy/pasted from an old shuttle document. None of the terminologies were even changed. 
TAKEAWAY
Some documents are submitted to QA engineers without previous review, which can lead to unnecessary process delays.

TEST DIRECTOR

In the old days, there was carbon paper. You would go around, get signatures to build relationships, and talk to others. Now, it's all paperless. There are people in my office and I don't even know who they are.
TAKEAWAY
It is important to build relationships and trust to prevent fatal errors like Columbia. However, current digital processes remove the face-to-face interaction that paper processes required.
TAKEAWAY
It is important to build relationships and trust to prevent fatal errors like Columbia. However, current digital processes remove the face-to-face interaction that paper processes required.
nasa test director
NASA PM

PROJECT MANAGER

We don't typically do things repetitively. We're basically doing everything unique to launch this one vehicle, and then maybe four or five years later we'll do it again to the launch another vehicle.
TAKEAWAY
NASA has few opportunities to iteratively perfect a process. 

ANALOGOUS DOMAINS

 

WINDMILL TECHNICIAN

windtech
DESCRIPTION

In Wind MRO (maintenance, repair, operations) companies, technicians are the frontline data collectors for the company. Because technicians provide valuable feedback, these companies make it a priority to constantly improve their technicians’ workflow.

FINDINGS
  • Windmill technicians often work in inconvenient locations, which offers many constraints (carrying capacity, wireless connectivity, mobility) 

  • Portable devices (i.e. tablets) are often used to reduce the weight and provide multiple tools in one (video, photo, email)

  • Real-time feedback systems enable technicians to promptly make informed decisions 

  • Wireless data coverage may be unreliable, so digital tools require offline solutions

ARCJET TECHNICIAN

Arcjet test
DESCRIPTION

At NASA Ames, ArcJet technicians simulate atmospheric re-entry by accelerating gases to hypersonic velocities and blasting them at test samples. Like KSC technicians, ArcJet technicians deal with expensive, unique equipment.

FINDINGS
  • Documentation happens on a physical logbook, but are understood only by technicians who were present at that time

  • There was an attempt to digitally document "Lessons Learned", but it was never put into practice

  • There is a mistrust of digital tools due to networking security

  • Some processes are intentionally vague to expedite changes



COMPETITIVE ANALYSIS

SURFACING PAST INFORMATION

Engineers writing WADs go through a process of surfacing past information that can include looking at past WAD documents from shuttle, and using documentation delivered by the manufacturer.

We studied three research products by interviewing practitioners from different industries. We found that the timing of the information mattered: some need information ahead of time for thorough preparation, while others want it just in time, as another action is being performed.

  • Stack Overflow contains persistent and public solutions, which allow programmers to find answers without having to ask the question and wait for new answers.

  • Google Scholar uses a powerful search algorithm to search across knowledge domains. We found that most academic researches use citation frequency as a trust heuristic.

  • LexisNexis is used by paralegals, who have much in common with NASA Engineers when it comes to work process. They use a consistent set of rules when determining whether a document is admissible in court. They also rely on a paper trail and citations when defending the credibility of their cases.
Feature Matrix

RESEARCH SYNTHESIS

THE WAD LIFECYCLE

THE WAD LIFECYCLE

2018.0509_IntroWAD

We decided to narrow our scope to the Authoring, Execution, and Alteration phases.
Overall, we used our interviews to demarcate 7 phases in the "lifecycle" of a WAD.
 

We decided to narrow our scope to the Authoring, Execution, and Alteration phases.
Overall, we used our interviews to demarcate 7 phases in the "lifecycle" of a WAD.
 

STAKEHOLDERS IN THE WAD LIFECYCLE

STAKEHOLDERS IN THE WAD LIFECYCLE

2018.0426_Stakeholder

We focused on Engineers (WAD Authors) and Technicians as the key stakeholders.
We also visualized where other stakeholders were located, and who they interacted with. 

OFFICIAL PROCEDURE VS. INFORMAL WORKAROUNDS

OFFICIAL PROCEDURE VS. INFORMAL WORKAROUNDS

Screen Shot 2018-07-22 at 11.34.15 PM

We noted that some breakdowns led to workarounds.
These indicated to us that certain, official procedures were not always practical.

EMERGING TECHNOLOGY STUDY

EMERGING TECHNOLOGY STUDY

Emerging Tech

We explored possible design solutions with emerging technologies. 
We plotted them on a Gartner Hype Chart to visualize technological feasibility.

We explored possible design solutions with emerging technologies. 
We plotted them on a Gartner Hype Chart to visualize technological feasibility.