Movatterモバイル変換

Network Working Group                                         D. JohnsonRequest for Comments: 1297                           Merit Network, Inc.                                                            January 1992NOC Internal Integrated Trouble Ticket SystemFunctional Specification Wishlist("NOC TT REQUIREMENTS")Status of the Memo   This memo provides information for the Internet community.  It does   not specify an Internet standard.  Distribution of this memo is   unlimited.Abstract   Professional quality handling of network problems requires some kind   of problem tracking system, herein referred to as a "trouble ticket"   system.  A basic trouble ticket system acts like a hospital chart,   coordinating the work of multiple people who may need to work on the   problem.   Once the basic trouble ticket system is in place, however, there are   many extensions that can aid Network Operations efficiency.   Information in the tickets can be used to produce statistical   reports.  Operator efficiency and accuracy may be increased by   automating trouble ticket entry with information from the network   Alert system.  The Alert system may be used to monitor trouble ticket   progress.  Trouble tickets may be also used to communicate network   health information between NOCs, to telcom vendors, and to other   internal sales and engineering audiences.   This document explores competing uses, architectures, and desirable   features of integrated internal trouble ticket systems for Network   and other Operations Centers.Introduction   This RFC describes general functions of a Trouble Ticket system that   could be designed for Network Operations Centers.  The document is   being distributed to members of the Internet community in order to   stimulate discussions of new production-oriented operator-level   application tools for network operations.  Hopefully, this will   result both in more ideas for improving NOC performance, and in more   available tools that incorporate those ideas.Johnson                                                         [Page 1]

RFC 1297                  NOC TT REQUIREMENTS               January 1992PURPOSES OF A NOC TROUBLE TICKET SYSTEM   A good Network Operations Trouble Ticket System should serve many   purposes:      1) SHORT-TERM MEMORY AND COMMUNICATION ("Hospital Chart").  The      primary purpose of the trouble ticket system is to act as short-      term memory about specific problems for the NOC as a whole.  In a      multi-operator or multi-shift NOC, calls and problem updates come      in without regard to who worked last on a particular problem.      Problems extend over shifts, and problems may be addressed by      several different operators on the same shift.  The trouble ticket      (like a hospital chart) provides a complete history of the      problem, so that any operator can come up to speed on a problem      and take the next appropriate step without having to consult with      other operators who are working on something else, or have gone      home, or are on vacation.  In single-room NOCs, an operator may      ask out loud if someone else knows about or is working on a      problem, but the system should allow for more formal communication      as well.      2) SCHEDULING and WORK ASSIGNMENT.  NOCs typically work with many      simultaneous problems with different priorities.  An on-line      trouble ticket system can provide real time (or even constantly      displayed and updated) lists of open problems, sorted by priority.      This would allow operators to sort their work at the beginning of      a shift, and to pick their next task during the shift.  It also      would allow supervisors and operators to keep track of the current      NOC workload, and to call in and assign additional staff as      appropriate.      It may be useful to allow current priorities of tickets change      according to time of day, or in response to timer alerts.      3) REFERRALS AND DISPATCHING.  If the trouble ticket system is      thoroughly enough integrated with a mail system, or if the system      is used by Network Engineers as well as Network Operators, then      some problems can be dispatched simply by placing the appropriate      Engineer or Operator name in an "assigned to" field of the trouble      ticket.      4) ALARM CLOCK.  Typically, most of the time a trouble ticket is      open, it is waiting for something to happen.  There should almost      always be a timer associated with every wait.  If a ticket is      referred to a phone company, there will be an escalation time      before which the phone company is supposed to call back with an      update on the problem.  For tickets referred to remote site      personnel, there may be other more arbitrary timeouts such asJohnson                                                         [Page 2]

RFC 1297                  NOC TT REQUIREMENTS               January 1992      "Monday morning".  Tickets referred to local engineers or      programmers should also have timeouts ("Check in a couple of days      if you don't hear back from me").  A good trouble ticket system      will allow a timeout to be set for each ticket.  This alarm will      generate an alert for that ticket at the appropriate time.      Preferably, the system should allow text to be attached to that      timer with a shorthand message about what the alert involves      ("Remind Site: TT xxx") (The full story can always be found by      checking the trouble ticket).  These alerts should feed into the      NOC's standard alert system.      The Alarm Clock can also assist (or enforce!) administrative      escalation.  An escalation timer could automatically be set based      on the type of network, severity of the problem, and the time the      outage occurred.      5) OVERSIGHT BY ENGINEERS AND CUSTOMER/SITE REPRESENTATIVES.  NOCs      frequently operate more than one network, or at least have people      (engineers, customer representatives, etc) who are responsible for      subsets of the total network.  For these individual      representatives, summaries of trouble tickets can be filtered by      network or by node, and delivered electronically to the various      engineers or site representatives.  Each of these reports includes      a summary of the previous day's trouble tickets for those sites, a      listing of older trouble tickets still open, and a section listing      recurrent problems.  These reports allow the site reps to keep      aware the current outages and trends for their particular sites.      The trouble ticket system also allows network access to the the      details of individual trouble tickets, so those receiving the      general reports can get more detail on any of their problems by      referencing the trouble ticket number.      6) STATISTICAL ANALYSIS.  The fixed-form fields of trouble tickets      allow categorizations of tickets, which are useful for analyzing      equipment and NOC performance.  These include, Mean Time Between      Failure and Mean Time to Repair reports for specific equipment.      The fields may also be of use for generating statistical quality      control reports, which allow deteriorating equipment to be      detected and serviced before it fails completely.  Ticket      breakdowns by network a NOC costs to be apportioned appropriately,      and help in developing staffing and funding models.  A good      trouble ticket system should make this statistical information in      a format suitable for spreadsheets and graphics programs.      7) FILTERING CURRENT ALERTS.  It would be possible to use network      status information from the trouble ticket system to filter the      alerts that are displayed on the alert system.  For instance, if      node XXX is known to be down because the trouble ticket isJohnson                                                         [Page 3]

RFC 1297                  NOC TT REQUIREMENTS               January 1992      currently open on it, the alert display for that node could      automatically be acknowledged.  Trouble tickets could potentially      contain much further information useful for expert system analysis      of current network alert information.      8) ACCOUNTABILITY ("CYA"), FACILITATING CUSTOMER FOLLOW-THROUGH,      AND NOC IMAGE).  Keeping user-complaint tickets facilities the      kind of follow through with end-users that generates happy clients      (and good NOC image) for normal trouble-fixing situations.  But      also, by their nature, NOCs deal with crises; they occasionally      find themselves with major outages, and angry users or      administrators.  The trouble ticket system documents the NOC's      (and the rest of the organization's) efforts to solve problems in      case of complaints.FIXED FIELDS, FREE-FORM FIELDS, and TT CONFIGURATION   Information in trouble tickets can be placed in either fixed or   freeform fields.  Fixed fields have the advantage that they can be   used more easily for searches.  A series of fixed fields also acts as   a template, either encouraging or requiring the operators to fill in   certain standard data.  Fixed fields can facilitate data verification   (e.g., making sure an entered name is in an attached contacts   database, or verifying that a phone number consists of ten numeric   characters).  Fixed fields are also appropriate for data that is   automatically entered by the system, such as the operator's login id,   the name of the node that was clicked on if the trouble ticket is   opened via an alert tool, or names and phone numbers that are   automatically entered into the ticket based on other entries (e.g.,   filling in a contact name and phone based on a machine name).   Unfortunately, fixed fields work best where the problem-debugging   environment is uniform, well-understood, and stable; that is, trouble   tickets work best when their fields are well tailored to the specific   problem at hand.  It is easy to set up a large number of fields (or   even required fields) that are irrelevant to a given problem; this   slows down and confuses the operators.  Adding structure and validity   checking to a field tends to make the data more consistent and   reliable, but it also tends to force the operators into longer   procedures like menus to get the get the data accepted by the system.   It also forces there to be more maintenance on those verification   systems (adding new entries as they become new legal options), and in   some ways it reduces the accuracy of the system by forcing operators   to choose "canned" or authorized responses that may not always   represent the situation accurately.  Where statistical operational   reports are a primary purpose of the trouble ticket system, several   fixed fields may be appropriate.  If the primary intent of the system   is to keep notes for individual problems and to facilitateJohnson                                                         [Page 4]

RFC 1297                  NOC TT REQUIREMENTS               January 1992   communication between operators, then fixed fields may tend to be a   hindrance.  One reasonable guideline would be that fixed fields are   used ONLY where they are automatically filled in by the larger   system, or where the information in that field is explicitly used in   a report or standard search procedure.   Because of this close relationship between the structure of the   ticket and the problem to be solved, it is very very useful to be   able to define different ticket types for different classes of   problems.  This becomes even more true for those many NOCs whose   staff are responsible for other types of operations: mainframe   operations, workstation administration, help desk functions, or any   of the other real-time response functions.  Network operations to   justify the expense of an operations center.  This kind of operation   makes economic sense, and is becoming more prevalent.  In these kinds   of situations it is vital that the same tools that are used for   network operations also be available for the other operations.  This   means that the trouble ticket configurations need to be modifiable by   local staff.  Commercial RDBMS forms builder and report generator   packages and "fourth-generation languages" offer a good start at   this, although it is sometimes difficult to integrate full trouble   ticket functionality through these systems.TROUBLE TICKET STRUCTURE   1) HEADERS.  Inevitably, a trouble ticket begins with a number of   fixed fields.  These generally include:      Time and Date of problem start.      Initials or signon of the operator opening the ticket.      Severity of the problem  (possibly separating the "customer      severity" and the "NOC priority", since these could be different).      A one-line description of the problem for use in reports.   There can be many other fixed fields for specific purposes.  There   may also be different kinds of tickets for different problems, where   the ticket format differs mainly in fixed fields.  These include:      Who reported the problem?  (Name, organization, phone,                                                      email address)      Machine(s) involved.      Network involved (for multi-network NOCs).      User's machine address.      Destination machine address.      Next Action.      Time and date for alarm on this ticket.      Who should the ticket be dispatched to?      Ticket "owner" (one person designated to be responsible overall).Johnson                                                         [Page 5]

RFC 1297                  NOC TT REQUIREMENTS               January 1992   2) INCIDENT UPDATES.  The main body of trouble tickets is usually a   series of freeform text fields.  Optimally, each of these fields is   automatically marked with the time and date of the update, and with   the signon of the operator making the update.  Since updates are   frequently recorded sometime after the problem is fixed, however, it   is useful to allow the operators to override the current time stamp   with the time the update was actually made.  (In some   implementations, both times will be kept internally).   The first incident update usually is a description of the problem.   Since the exact nature of the problem is usually not known when the   ticket is first opened, this description may be complex and   imprecise.  For problems that are reported by electronic mail, it is   useful to be able to paste the original message in the ticket,   particularly if it contains cryptic or extensive information (such as   a user's traceroute output).  At least one such arbitrarily-long   freeform field seems necessary to contain this kind of output,   although it is better to allow arbitrarily long messages at any stage   (e.g., so future complex messages can also be archived in the   ticket).   Subsequent update fields may be as simple as "Called site;  no   answer".  Some systems allow these kinds of updates to be coded in   fixed fields; most use freeform text.   There should always be an indication of what the next action for this   ticket ought to be.  Again, this may be implemented as a special   fixed field, or by convention of using the last line of text.   Advanced systems may also need a facility to allocate the amount of   time a ticket is open between multiple sources.  A serious NOC will   want to use its trouble ticket system to statistically track its   performance on responding to problems. (e.g., Mean Time Between   Failure and Mean Time To Repair reports).  Frequently, though,   repairs are stopped at the customer's request.  ("It's not that   important a machine and I don't feel like coming in--can you defer it   until Monday Morning?").  In these cases the ticket needs to remain   open, but there needs to be a notation that the ticket is now in   "customer time" rather than "NOC time".  The durations of "customer   time" need to be excluded from MTBF and MTTR reports.  Complicated   repairs could move back and forth between "NOC time" "customer time"   repeatedly.  This probably implies that each Incident Update may have   a time and date of status change, and that these status changes can   be read and aggregated by by reporting programs.   3) RESOLUTION DATA.  Once a problem is resolved, it is useful to   summarize the problem for future statistical analysis.  The following   fields have been found to be useful:Johnson                                                         [Page 6]

RFC 1297                  NOC TT REQUIREMENTS               January 1992      - Time and Date of resulation (for outage duration).      - Durations (can be calculated from time of resolution and        incident report "customer/NOC time" stamps).      - Resolution (one line of description of what happened, for        reports).      - Key component affected (for MTBF and similar reports).      - Checked By -- a field for supervisors to sign off on ticket        review.      - Escalated to -- for reports on how many problems require        non-NOC help.      - Temp - a database field that can be used to store temporary        "check marks" while making statistical investigations.USER, TROUBLE, and ENGINEERING Ticket System(s)   The primary level of an Network Operations trouble ticket is the   "problem" or "trouble": a single malfunctioning piece of hardware or   software that breaks at some time, has various efforts to fix it, and   eventually is fixed at some given time.   The primary level of an Network Information Center ticket, however,   might well be the "user complaint".  A single network failure might   well produce a large number of individual user phone calls and hence   "user complaint" tickets.  A NIC may want to use tickets to track   each one of these calls, e.g., to make sure each user is informed and   satisfied about the eventual resolution of the single hardware   problem.   In addition, NOCs (or Engineering Staffs) may want to track   systematic problems.  The staff may know, for instance, that a   particular router is old and fragile, or that a particular section of   their network doesn't have enough redundancy.  It may be useful to   open an "Engineering Ticket" on these known problems, providing a   place to record history and notes about the problem, for use in   further engineering or funding discussions.   Even further "Meta" tickets could be described, having to do with   such issues as whether the current trouble ticket fields, reports,   and operation procedures were sufficient to handle current problems.   It would be very convenient to be able to build all of these systems   on the same platform, and to allow each type of ticket to easily   reference other types.  Multiple "user complaint" tickets, then,   might might explicitly point to a single "trouble" ticket.  Multiple   trouble tickets representing independent failures would then point to   a single "engineering" ticket, which described the systematic   problem.  Multiple engineering tickets could point to a single "meta"   ticket, if appropriate.Johnson                                                         [Page 7]

RFC 1297                  NOC TT REQUIREMENTS               January 1992ASSISTED ENTRY AND DATA VERIFICATION   Data (particularly in fixed fields) is only useful for searching if   it is entered in consistent formats.  A trouble ticket system needs   to help operators fill these fields with the correct format of   information.  This can be done using assisted entry (menus of   acceptable choices), verification routines which check against   internal lists or external databases (see next section), or other   computer checking.   Some database systems allow a customized "help" screen to be   associated with each field, helping new (and experienced) operators   by making context-sensitive trouble ticket system documentation   available at every field.   Very complicated help or operator-guidance systems can be built out   of Expert System technology.  This could be as simple as help   screens, or help screens with database information inserted (e.g.,   site contact names and phone numbers).  Or it could involve hints to   the operator, based on current network conditions.  Or it might even   ask the operator to run tests and to type in the results.  (See   EXPERT SYSTEMS, below).INTEGRATION   To be maximally efficient and useful, a Trouble Ticket system needs   to integrate well with most of the rest of the NOC tools.  These   include:      1) OPERATOR WINDOW ENVIRONMENT.  A NOC Operator needs access to      many pieces of information simultaneously, and therefore is well      served by a good windowing environment.  The Trouble Ticket system      needs to run within this larger windowing system, so that the      operator can debug, consult databases, use Email, field alerts,      and keep an eye out for other emergencies while working on a      trouble ticket.  It is also useful to be able to run two trouble      ticket sessions simultaneously, for example, to allow an operator      to search for related tickets while he is in the middle of      updating another ticket.  Cut and Paste between these various      screens is mandatory, to allow easy recording of technical details      in the trouble tickets.      2) ALERT MONITORING SYSTEM.  Trouble tickets are often opened in      response to machine alerts; it ought to be easy to open a trouble      ticket directly from the alert tool.  When a ticket is opened this      way, information about the alert and the machine involved ought to      be automatically filled into the ticket.  (There are various      opinions about whether trouble tickets ought to be openedJohnson                                                         [Page 8]

RFC 1297                  NOC TT REQUIREMENTS               January 1992      automatically without operator intervention.  This operator's      opinion is that an operator acknowledgement should be required,      but this point is debated enough that designers of a new system      probably should support either option).      The Alarm Clock feature of the trouble ticket system also      generates alerts.  These alerts ought to feed gracefully into the      Alert Monitor system, so that the operators will get all of their      alerts from one place.      3) DATABASE CONNECTIONS.  A good trouble ticketing system will      query NOC databases to automatically fill out trouble ticket      fields where possible.  This can be used for:      - Filling out Network Operator information (e.g., phone number)        based on the NetOp's signon id.      - Filling in contact information based on machine name.      - Filling in circuit numbers based on link description.      - Filling in alarm clock or escalation time fields based on the        machine or link name and on time of day.      - Filling in machine serial numbers based on configuration database.      4) MACHINE QUERYABLE INFORMATION.  It could also be possible for a      trouble ticket system to make standard queries of the network      itself when a trouble ticket is opened: e.g., the system could      request and store current machine configurations whenever a ticket      was opened for that machine.  On some systems, hardware serial      numbers are obtainable by software query directly to the machine.      5) ELECTRONIC MAIL.  Problem notification often comes via      electronic mail; it must be possible to easily open a ticket and      include the original mail message within the ticket as part of the      initial problem description.  When extremely technical messages      come in from network engineers, it is useful to allow those      messages to be included verbatim, rather than forcing less      technical network operators to rephrase the messages or to force      them into predefined formats.  Later update messages should also      be easily includable.  Possibly: tickets could be opened      automatically for mail messages to certain mailboxes.  A response      system saying "Your request has been received and assigned ticket      number ####" might be desirable.      Information within trouble tickets must also be easily available      (possibly just via the windowing system) for inclusion in Email      messages to engineers and others.      Scheduled (e.g., daily) reports must also be easily generated into      the Email system.Johnson                                                         [Page 9]

RFC 1297                  NOC TT REQUIREMENTS               January 1992      6) DISPATCHING AND NOTIFICATION SYSTEMS.  An important real-time      aspect of Network Operations is notifying users, technical      contacts, and administrators of various classes of problems.  The      rules for who gets notified of what can be very arbitrary and      complex, and can involve electronic mail, notices in computer      conferences, automatic beeper pages, and synthesized voice      announcements.  It would be good for a trouble ticket system to      provide for automatic (or operator initiated) notification of the      appropriate channels for the current ticket (based on network,      machine, severity of problem, duration of the problem, escalation      guidelines, etc).      Databases associated with the trouble ticket system may also have      lists of specific people to contact about outages for particular      machines.  These "who to inform" lists can facilitate customized      notification messages directly from the trouble ticket system.      It may also be possible to dispatch experts directly from the      trouble tickets system.  IBM's ECCO system allows allows customers      to directly dispatch Service Engineers from machine interactions.      Many NOCs also use computer hooked to modems to automatically page      engineers.  This kind of dispatching should be available from      within the trouble ticket system (along with an automatic note      into the trouble ticket that the engineer has been dispatched).      7) OTHER TROUBLE TICKET SYSTEMS.  When the NOC generates a trouble      ticket, it often immediately calls up a telco or another Internet      NOC, who proceed to open their own ticket.  The Internet      Engineering Task Force User Connectivity Working Group is also      proposing a national trouble ticket tracking system, which would      need updating from individual NOC trouble ticket systems.  A      state-of-the-art trouble ticket system could have provisions for      transferring tickets and ticket information in and out of other      such systems.      8) NETWORK ACCESS.  Some older trouble ticket systems assumed that      anyone with a need to access the information would obtain a signon      and learn to use that system.  The range of people with a need for      trouble ticket information is now too great to allow this      assumption.  A good system now needs to be able to support network      query for tickets and summary reports, as well as Email delivery      of scheduled reports.      9) SUBROUTINE INTERFACE.  To allow for ad-hoc and currently      unanticipated needs, the trouble ticket system needs to support a      full-function set of subroutine calls.  These subroutines will      allow construction of further trouble ticket functionality not yet      specified.Johnson                                                        [Page 10]

RFC 1297                  NOC TT REQUIREMENTS               January 1992      10) EXPERT SYSTEMS.  Network debugging is a very promising area      for expert system and artificial intelligence applications.  But      such an algorithm should require access to the alert monitoring      system, configuration and change control systems, to the network      itself, and also to the information in the trouble ticket system.      A good future system then needs to make this information available      (probably via the subroutine interface mentioned above), and to      also allow the Network Operators to invoke the artificially      intelligent debugging from within a trouble ticket (including its      output as part of the ticket dialogue).      11) GRAPHICS/REPORT Capability.  Statistical and graphical      displays about trouble ticket data need to be compatible with      tools used to generate reports, news letters, etc.OTHER CONSIDERATIONS:      1) INTERACTIVE SPEED.  The system must be fast enough to be used      interactively.  NetOps need to answer questions over the phone in      real time; good answers cannot be given if every query takes a      couple of minutes.  More importantly, the NetOps need the trouble      ticket system in order to get information necessary to fix the      network.  If looking for old or currently-open tickets takes more      than a few seconds, it won't be done.  If updates take very long      to make, then updates won't be recorded, or they will be recorded      long after the event (with corresponding loss of accuracy).  (Our      Operators have asked for a single-line "update this ticket with      this message" utility that would let them avoid even retrieving      the ticket for simple updates!)  Any time spent waiting reduces      NetOp productivity and Network reliability.      2) BACKUPS AND RELIABILITY.  The trouble ticket system is      absolutely crucial to both immediate and long-term operation of      the NOC.  Good systems could back up all data several times an      hour to an auxiliary processor.  That processor should be      accessible for immediate use in case of failure of the primary      system.      3) HISTORY AND ARCHIVING.  A trouble ticket system is a      constantly-growing database system.  Old tickets need to be      removed from the system at some interval (a year?  several years?)      and archived.  These archives should also be restorable for long-      term history processing.      4) PRIVACY AND SECURITY.  The ability to enter, append, and modify      tickets should be controlled by id and password.  Permissions      should be specifiable on a per-field basis.  General read access      to tickets (or portions of tickets) also needs to be restricted,Johnson                                                        [Page 11]

RFC 1297                  NOC TT REQUIREMENTS               January 1992      or else NetOps will be reluctant to be full and candid in their      reporting.UTILITY   There are quite a few ideas in this "Wishlist".  Ultimately, what an   Operations Center needs is a totally integrated set of tools which   completely model all of its activities, and which integrates cleanly   with all backup, peer, and vendor NOCs.  It is hard to imagine that   this whole system could come out of a shrinkwrapped box, even without   the local configuration.  But most of these facilities do exist, now,   in some system.  Hopefully, this document will foster an ongoing   discussion of ways in which NOC operator-level tools are used in real   operations, and will encourage systems implementors and vendors to   bring some of this functionality to the aid of real operations.  It   might even inspire current Operations Centers to add useful features   to their current operations.Security Considerations   This paper does not pose specific new security issues.  The systems   described herein would be host database applications, however, or   even distributed host database applications.  All of the normal   security considerations for that kind of system would apply.   Multiple classes of user access need to be specified for classes of   ticket data.  Possible security threats include disclosure of network   information, disclosure of confidential material (e.g., circuit   numbers or home phone numbers), and denial of service to the Network   Operations Center leading to degradation of network service.Author's Address   Dale S. Johnson   Merit NOC   1075 Beal Avenue   Ann Arbor, MI 48109-2112   Phone:  (313) 936-2270   Email:  dsj@merit.edu   Discussion/comments may be sent to noc-tt-req@merit.edu.  The list   is maintained by noc-tt-req-request@merit.edu.Johnson                                                        [Page 12]