US20060168134A1 - Method and System for Real-Time Observation Assessment - Google Patents

Abstract

Techniques for real-time observation assessment are provided. The techniques, which are designed for educators, take advantage of handheld computers, desktop/laptop computers and Internet access in order to reduce the paperwork associated with conventional educational assessments. An array of instructional assessment applications are designed to run on handheld computers. The instructional assessment applications may be based on existing and widely used paper methodologies. A common Web-based platform for assessment application distribution, selection, download, data management and reporting is also provided. Users can then periodically synchronize instructional data (assessments, diagnostic results, notes and/or schedules) to the Web site. At the Web site, browser-based reports and analysis can be viewed, administered and shared via electronic mail.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
• This application is a continuation of U.S. application Ser. No. 10/197,482, filed Jul. 18, 2002, and titled SYSTEM AND METHOD FOR REAL-TIME OBSERVATION ASSESSMENT, which claims priority from U.S. Provisional Application Ser. No. 60/305,887, filed Jul. 18, 2001, both of which are incorporated by reference in their entirety.
BACKGROUND OF THE INVENTION
• 1. Field of the Invention
• The present invention relates generally to computer information gathering and processing systems, and more particularly to a computer-based system and method for performing real-time educational assessments.
• 2. Related Art
• In today's technological environment, many advances have been made in the fields of software and computing. More specifically, the advent of smaller, faster and more powerful laptops, mobile computing devices, personal digital assistants (PDAs) and the like, along with the increasing accessibility of the global, public Internet, have transformed how people work, play and learn. Likewise, new software applications for these advanced computing devices have been developed which allow people to become more efficient and productive in almost every aspect—whether it be in business or personal—of daily life. These advances have combined to make present society less of a paper culture and more of an electronic (digital) culture.
• The above-mentioned advances, however, have not touched every aspect of present day life. That is, some tasks or processes are still done using “pen and paper.” One example of this is the elementary school assessment process which is still primarily a paper process.
• Classroom assessments are used by teachers as a powerful tool for improving student achievement. This is because classroom assessments can help teachers plan future lesson plans, adjust learning goals for particular students, and allows teachers to provide more frequent feedback to students about how to improve and actively engages students in their own learning. Using assessments allows teachers to adjust their lesson plans accordingly, thus resulting in higher student achievement. The administration and reporting of these assessments are often mandated by individual schools, school districts or state-wide departments of education.
• The method by which teachers observe and assess student's skills and progress in a particular subject has not changed much over the past few decades and thus, has not benefitted from the above-described advances in the fields of software and computing. For example, in the area of reading and literacy, teachers often administer (paper) assessments to observe and assess student's reading skills and measure progress. One such assessment is known as the “running record” as described in Marie M. Clay, “An Observation Survey: Of Early Literacy Achievement,” ISBN: 0435087630 (Heinemann 1993) [hereinafter “Clay”], which is hereby incorporated by reference in its entirety. The running record, as described in Clay, utilizes a system of coding, scoring, analysis and use of running record paper sheets and has been used in educational systems worldwide as a way of observing children's progress in the early years of learning about literacy and determining which children need supplementary teaching.
• For all its benefits, the paper running records add to the already significant amount of paperwork which teachers must perform. That is, teachers, for each student, must prepare the records before administering the assessments and record scores and comments during the assessment, and keep the records for later analysis and generating reports. This is in addition to other paper work—such as progress reports, report cards, lesson plans and the like—which teachers must already perform. Thus, there has been a general need for tools to help teachers manage, minimize of ease the burden of paperwork and more specifically, the burden of paperwork with respect to (oftentimes, mandatory) educational assessments.
• Therefore, given the above, what is needed is a system, method and computer program product for real-time observation assessment, designed for educators, that takes advantage of handheld computers, desktop/laptop computers and the increasing presence of Internet access in order to reduce the paperwork associated with conventional educational assessments.
SUMMARY OF THE INVENTION
• The present invention meets the above-identified needs by providing a system, method and computer program product for real-time observation assessment.
• The system of the present invention consists of an array of instructional applications designed to run on handheld computers, portable computing devices, personal digital assistants (PDAs) and the like. The instructional applications are all linked to a common Web-based infrastructure for application distribution, selection, download, data management and reporting.
• The method and computer program product of the present invention allows users (e.g., teachers, educators and administrators) to visit a central Web site, where they can register individually or as part of a school or school district. Once registered, they can choose an application appropriate for their subject and grade level from numerous available applications. They then download (and in some cases, customize) these applications and install them on their handheld device for daily use.
• Periodically (e.g., either daily or weekly), users synchronize instructional data (assessments, diagnostic results, notes and/or schedules) to the central Web site. At the Web site, browser-based reports and analysis can be viewed, student lists can be administered or further customizations can be made to previously-downloaded applications.
• When teachers are part of a school or school district license, data and reports may be shared with other teachers, tutors, administrators or parents from that particular school or school district, and other school-based features may be enjoyed.
• An advantage of the present invention is that it allows for more standardized data capture. That is, teachers often vary in the ways they capture and report assessment data. The present invention, however, helps to ensure that different teachers are generating comparable data for more meaningful observational assessment data comparison and analysis.
• Another advantage of the present invention is that it provides immediate analysis and calculation of key statistics and elimination of “after-the-fact” data entry. Because assessments are often administered under severe time constraints, teachers often postpone the important numerical analysis until much later. In some cases, teachers must enter their written data into a computer analysis system. Providing immediate analysis, however, can speed the teacher's ability to take follow-up actions with students.
• Another advantage of the present invention is that it provides streamlined reporting of assessment results and student needs. Teachers have to generate reading progress reports for administrators and parents. The conventional process currently involves a great deal of searching files for relevant assessments, and then “cutting and pasting” them into readable reports or entering assessment results into a computer spreadsheet. The automated data collection and report generation of the present invention reduces this process to a few minutes, thereby reducing teachers' paperwork burden. Furthermore, the reduced administrative burden and speed of the present invention encourages teachers to administer more frequent assessments, thereby benefitting students.
• Yet another advantage of the present invention is that it provides streamlined generation of class-wide progress reports. Teachers and administrators currently create class-wide progress reports to spot when a student or a class is falling behind. Automating these reports, using the present invention, now saves paperwork and can give teaching staff a continuous view of class progress.
• Another advantage of the present invention is that it allows for the aggregation, analysis and reporting of data at the school level, district level or even higher levels. By collecting assessment data from many classrooms into a central repository (unlike classroom desktop-based software applications), these reports and analyses are easily generated without time-consuming and costly collection and collation of data.
• Another advantage of the present invention is that it allows for discretion and non-intrusiveness. The handheld devices on which the assessments of the present invention execute can be used during the reading process without creating a physical barrier between teacher and student. The assessments are easy to use and allow the teacher to keep their attention on the student.
• Further features and advantages of the invention as well as the structure and operation of various embodiments of the present invention are described in detail below with reference to the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
• The features and advantages of the present invention will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference numbers indicate identical or functionally similar elements. Additionally, the left-most digit of a reference number identifies the drawing in which the reference number first appears.
• FIG. 1 is a block diagram illustrating the system architecture of an embodiment of the present invention, showing connectivity among the various components.
• FIG. 2 is a flowchart depicting an embodiment of the operational process of the real-time observation assessment system of the present invention.
• FIG. 3 is a block diagram illustrating the software architecture of an embodiment of the present invention, showing data flow among the various components.
• FIGS.4A-M are exemplary PDA-based windows or screen shots generated by the graphical user interface of the present invention.
• FIGS.5A-K are exemplary Web-based windows or screen shots generated by the graphical user interface of the present invention.
• FIG. 6A is a block diagram illustrating in more detail, according to an embodiment of the present invention, portions of the software architecture shown inFIG. 3.
• FIG. 6B is a sequence diagram illustrating the synchronization process in one embodiment of the present invention.
• FIG. 7 is a block diagram of an exemplary computer system useful for implementing the present invention.
DETAILED DESCRIPTION
• I. Overview
• The present invention provides a system, method and computer program product for real-time observation assessment.
• In an embodiment of the present invention, a service provider organization provides and allows access, perhaps on a subscriber fee or pay-per-use basis, to a Web-based tool that allows real-time observation assessment. The system, method and computer program product take advantage of handheld computers, desktop/laptop computers and the increasing presence of Internet access in order to reduce the paperwork associated with conventional educational assessments. That is, the service provider would provide the hardware (e.g., servers) and software (e.g., database) infrastructure (i.e., a platform), educational assessment application software, customer support, and billing mechanism to allow its customer-subscribers (e.g., individual teachers, reading specialist, educators, principals or school administrators, or entire schools, school districts or state-wide departments of education and the like) to download applications, upload assessment data, perform data analysis and generate and send reports via electronic mail. The tool would be used by subscribers to obtain both real-time and historical information, characteristics, and trend analysis to make educational planning decisions.
• In a preferred embodiment of the present invention, a service provider organization provides and allows access, on a subscriber fee basis, to a platform that allows real-time observation literacy assessment based on the running record methodology described in Clay. The platform makes use a personal digital assistant (PDA), a desktop with synchronization software and access to the global Internet, and the service provider organization's Web site to enable real-time literacy assessment at the point of instruction, with data capture and transfer to an Web-based data system.
• The present invention is now described in more detail herein in terms of the above example. This is for convenience only and is not intended to limit the application of the present invention. In fact, after reading the following description, it will be apparent to one skilled in the relevant art(s) how to implement the following invention in alternative embodiments (e.g., real-time observation assessment for subjects other than reading, applications that implement assessment methodologies other than that described in Clay, or utilizing other mobile computing devices other than a PDA).
• The terms “user,” “subscriber,” “customer,” “teacher,” “educator,” “school,” “school district,” “state-wide department of education,” and the plural form of these terms may be used interchangeably throughout herein to refer to those who would access, use, and/or benefit from the tool that the present invention provides for real-time, observation assessment.
• II. System Architecture
• Referring toFIG. 1, a block diagram illustrating the physical architecture of a real-time observation assessment (“RTOA”)system100, according to an embodiment of the present invention, is shown.FIG. 1 shows network connectivity among the various components ofsystem100.
• RTOA system100 includes a plurality of subscribers (e.g., individual teachers, educators or school administrators, or entire schools, school districts or state-wide departments of education and the like). Each subscriber is equipped with a mobile computing device (e.g., a Palm™ V Handheld PDA)102. Located within PDA102 (but shown externally for ease of reference herein) is alocal cache104 which stores recently entered or accessed memory (e.g., a recently-entered or accessed student running record).
• PDA102 is linked, either via a serial or universal serial bus (USB) connection, to a workstation106 (e.g., an IBMTM or compatible personal computer (PC) running the Microsoft® Windows 95/98™ or Windows NT™ operating system, Macintosh® computer running the Mac® OS operating system or the like), running a commercially-available Web browser (e.g., Microsoft® Internet Explorer). (For simplicity,FIG. 1 shows only onePDA102 andworkstation106.) Subscribers would connect to the infrastructure of RTOA system100 (i.e., a platform120) which is provided and maintained by the RTOA service provider. Access toplatform120 is granted to subscribers via theglobal Internet108.
• In alternative embodiments, subscribers may accessplatform120 ofRTOA system100 using any processing device including, but not limited to, a desktop computer, laptop, palmtop, workstation, set-top box, PDA with wireless Internet access and the like.
• Platform120 ofRTOA system100 includes aWeb server110, anapplication server112, a master repository database114 (located on adatabase server118, but shown externally for ease of reference herein), and a plurality of administrative workstations116 (shown as oneworkstation116 inFIG. 1 for simplicity).
• Web server110 provides the “front end” forRTOA system100. That is,server110 contains a Web server process running at a Web site which sends out Web pages in response to Hypertext Transfer Protocol (HTTP) or Hypertext Transmission Protocol, Secured (HTTPS) requests from remote browsers (i.e., subscribers of the RTOA service provider). More specifically, it provides a graphical user interface (GUI) “front end” screens to subscribers ofRTOA system100 in the form of Web pages. These Web pages, when sent to the subscriber's workstation106 (or the like), would result in GUI screens being displayed. In addition,server110 provides the central point of communication thatsubscriber workstation106 connects to for synchronization purposes.
• In an embodiment of the present invention,server110 is a Sun or NT workstation having access toapplication server112.Application server112 in turn has access tomaster repository database114, via adatabase server118, implemented with relational database management server software (e.g., Oracle 9 RDBMS).Database114 is the central store for information within RTOA system100 (e.g., book titles and text, subscriber information such as login names, passwords, etc., and student data such as running records, comments, historical data and demographics-related data for report generation).
• While onedatabase114 is shown inFIG. 1 for ease of explanation, it will be apparent to one skilled in the relevant art(s), thatRTOA system100 may utilize databases physically located on one or more computers which may be the same as any ofservers110,112 or118. Furthermore, in an embodiment of the present invention,database114 can be mirrored for fault tolerance.
• Application server112 ofplatform120 serves as the “back end” of the present invention. As such, it provides three main services (which are explained in more detail below with reference toFIG. 3). First,application server112 contains logic for producing dynamic content for the Web interface ofplatform120. Second,application server112 provides a system-wide logging service that collects data for support and monitoring functions. Third, it contains logic for the remote management of software onsubscriber desktops106 andPDAs102, as well as synchronization code logic that translates data first entered and stored inPDA102 and uploaded fromworkstation106 toWeb server110 for storage ontomaster database114. In an embodiment of the present invention,server112 is a Sun or NT workstation, although these services could be provided by (i.e., distributed among) a plurality of servers.
• Administrative workstation116 is used by the service provider organization as a management platform and contains a set of Web tools for remotely administering user, student, class and school data, as well as to update, maintain, monitor and log statistics related toserver110,server112 andRTOA system100 in general. Also,administrative workstation116 may be used “off-line” by the service provider organization's personnel in order to enter configuration, user, student, school and application specific data in order to customize the performance ofsystem100. Accordingly, FIGS.5D-F show exemplary windows or screen shots generated byWeb server110 that can be displayed onadministrative workstation116 in an embodiment of the present invention.
• As will be apparent to one skilled in the relevant art(s) after reading the description herein, all of the components insideplatform120 are connected and communicate via a wide or local area network (WAN or LAN) which is placed behind a firewall for security purposes.
• As will be appreciated by one skilled in the relevant art(s) after reading the description herein, a service provider organization can replicateplatform120 ofRTOA system100 in several market areas (i.e., schools or school districts) in which they offer services to support a plurality of subscribers (each having theirown PDAs102 and workstations106).
• More detailed descriptions ofRTOA system100 components, as well as their functionality, are provided below.
• III. System Operation Overview
• Referring toFIG. 2, a flowchart depicting an embodiment of anoperational process200 of the real-timeobservation assessment system100 of the present invention is shown. That is,process200 illustrates how, in an embodiment, an individual teacher (i.e., subscriber) may register with the service provider, download a running records literacy assessment application for their PDA (based on the methodology described in Clay), assess a student, upload the resulting data, and then generate a report.Flowchart200 begins atstep202 with control passing immediately to step204.
• Instep204, a teacher would access the service provider's Website using workstation106 in order to obtain an account (i.e., a login and password). In an embodiment, the service provider organization may have a contractual arrangement with a school, school district or state-wide department of education which would allow individual teachers to obtain accounts and access platform120 (and thus, the real-time observation assessment tool of the present invention).
• Instep206,Web server110 provides GUI screens toworkstation106 to enable the subscriber to set up a class roster (i.e., a list of student names, any other contact information and any demographics-related data kept by the subscriber for each student). FIGS.5G-I show exemplary windows or screen shots generated byWeb server110 that can be displayed onsubscriber workstation106 duringstep206 in an embodiment of the present invention.
• Instep208,Web server110 provides GUI screens to enable the subscriber to select and form a book list appropriate for the grade and reading levels for the subscriber's students.
• Instep210, all licensed applications and supporting files, as well as data including the student roster and the entire text (or portion) of each book in the book list, are downloaded fromplatform120 to the subscriber'sPDA102 via a synchronization process. (This would occur, in an embodiment, withPDA102 in its cradle which is connected either via a serial or USB connection toworkstation106 and the user pressing the HotSync® button.) In an embodiment of the present invention, the synchronization process is accomplished via a plug-in to the HotSync synchronization software (i.e., a conduit) that accompanies such (e.g., Palm V Handheld)PDAs102. In such an embodiment, the plug-in would be developed by the service provider organization using a software development kit (SDK) available from Palm, Inc. of Santa Clara, Calif.
• As will be apparent to one skilled in the relevant art(s) after reading the description herein, the service provider organization would cause the plug-in to be downloaded toworkstation106 during the registration process (step204). In an alternate embodiment, the plug-in could be loaded directly ontoworkstation106 from a CD-ROM or other media made available to subscribers by the service provider prior to the registration process. As will be also apparent to one skilled in the relevant art(s) after reading the description herein, the service provider organization would make several different plug-ins available, using several different vendors' SDKs, based on the specific make and model ofmobile computing device102 being used by the subscriber.
• Instep212, within the classroom environment and after the subscriber launches on theirPDA102 the previously-downloaded assessment application, an individual student's record is selected from the roster. This would be done as the subscriber is preparing to administer a real-time, observation assessment of the student's literacy.
• Instep214, a book title is selected from the previously downloaded book list. In an embodiment of the present invention, books are organized according to grade and reading level so that the subscriber can identify and select appropriate books quickly. This causes the text of the selected book to be loaded incache104 ofPDA102.
• Instep216, the previously-downloaded application is used to administer the real-time observation assessment of the student. (The real-time observation assessment process ofstep216, in one embodiment, is explained in more detail below.)
• Instep218, the student's scores and analysis from the assessment administered instep216 is displayed on the screen of thePDA102.
• Instep220, the subscriber would return toworkstation106,place PDA102 in its cradle, and the students running record, scores and any analysis would be uploaded to the Web site and stored ontodatabase114 via a synchronization process (similar to that described in step210) .
• In an embodiment, the synchronization process ofstep220 involves sending data stored inlocal cache104 toWeb server110, via theInternet108, in a compressed Extensible Markup Language (XML) format. Then, a synchronization service process executing onWeb server110 would translate the XML data into database records.Application server112 then stores such records ontodatabase114.
• Instep222, the subscriber would log onto the service provider's Web site (i.e., Web server110) usingworkstation106 and its connection to theInternet108. The log in process, in an embodiment, would use the login name and password assigned during the registration process (i.e., step204). FIGS.5J-K show exemplary windows or screen shots generated byWeb server110 that can be displayed onsubscriber workstation106 duringstep222 in an embodiment of the present invention.
• Instep224, the subscriber would be presented with GUI screens byWeb server110 using data stored indatabase114 to perform post-processing of the raw data uploaded during the synchronization process ofstep220. That is, in an embodiment, application-specific reports and data editors are provided to users to allow a gallery of pre-configured reports to be generated on demand.
• Instep226, the subscriber is allowed to view any of the generated reports on theirworkstation106. In an embodiment, these reports may be forwarded and shared with administrators, parents, other teachers and the like, via electronic mail and the like.
• The control flow ofprocess200 may then end as indicated bystep228.
• It should be understood thatprocess200, which highlights the functionality and other advantages ofRTOA system100, is presented for example purposes only. The architecture of the present invention is sufficiently flexible and configurable such that users may utilizesystem100 in ways other than that shown inFIG. 2. For example, as will be apparent to one skilled in the relevant art(s) after reading the description herein,workstation106 may actually be a plurality of workstations where a school subscriber may designate certain workstations as “synchronization workstations” (i.e., for use duringsteps210 and220), while designating certain other workstations as “Web workstations” (i.e., for use during steps204-208 and222-226).
• IV. Software Architecture
• Referring toFIG. 3, a block diagram illustrating asoftware architecture300 of an embodiment ofsystem100 is shown.FIG. 3 also shows communications (i.e., the data flow) among the various components. In an embodiment of the present invention,architecture300 is designed in the client-server paradigm where the client is the mobile computing device (i.e., handheld)102 andworkstation106, and the server is distributed among the components ofplatform120.
• Architecture300 includes a plurality of real-timeobservation assessment applications302. An additional application onhandheld device102 maintains key parameters for synchronization. These include the user's login information, server addresses and state information regarding the most recent synchronization. In an embodiment of the present invention, all of these applications are written in a high-level programming language such as C++.
• Applications302 execute onhandheld102 and read and write data to and from the on-board data cache104.Data cache104 stores assessment information specific to each application as well as shared data (i.e., student and class data) used by all real-timeobservation assessment applications302 offered by the provider organization.
• A desktop agent306 (residing on workstation106) is a modular, scriptable extension (i.e., plug-in) to the desktop-based synchronization software that comes with the handheld102.Desktop agent306 handles network synchronization, error and event logging, PDA-based assessment application updates, system configuration and updates toagent306 itself. In an embodiment of the present invention,desktop agent306 is written in a high-level programming language such as C++ and an interpreted, interactive, object-oriented programming language such as Python.
• Architecture300 includes anintegration service307 and aWeb interface service316, both residing onapplication server112. In an embodiment, these components are implemented usingJava™ 2 Platform Enterprise Edition.Integration service307 provides the scheduling of commands to be carried out bydesktop agent306 on a session-by-session basis. It routes communication from the client (i.e., subscriber workstation106) to necessary component ofplatform120.Web interface service316 sends out Web pages, viaWeb server110, in response to Hypertext Transfer Protocol (HTTP) or Hypertext Transmission Protocol, Secured (HTTPS) requests fromremote Web browsers306 executing onsubscriber workstations106. Instep226 ofprocess200, for example, the subscriber would be presented with GUI screens byWeb server110. This is accomplished byWeb interface service316 making Java Database Connectivity (JDBC)/SQL queries for master data314 (e.g., assessment content and results) stored indatabase114.
• Architecture300 includes a file anddata synchronization application310, residing onapplication server112.Synchronization application310 serves desktop agent306 (i.e., a Sync Package614 withinagent306 described in more detail below with reference toFIG. 6A). They communicate in SyncML—an XML-based open standard that specifies the protocol for synchronizing heterogeneous devices—in order to exchange and resolve file and data changes betweenmaster database114 and the client's data store (i.e., cache104). This interaction happens over a virtual communications channel carried over the HTTPS connection betweenagent306 and integration service307 (via a Sync Handler305 withinintegration service307 described in more detail below with reference toFIG. 6A).Synchronization application310 is also responsible for storing all master data314 (i.e., school, student and teacher information, assessment content and results) ontodatabase114 via SQL, handles permissions, and maintains data related to all synchronizations.
• Architecture300 includes a logging service318 (residing on application server112) and a log320 (residing, in an embodiment, on database server118) which are both explained in more detail below with reference toFIG. 6A.
• In an embodiment of the present invention,components307,310 and318, residing onapplication server112, communicate via the Java programming language library's Remote Method Invocation (RMI).
• V. Synchronization Process
• Architecture300 withinsystem100 allows real-time observation assessment application files, data and any updates to be installed transparently (i.e., without user intervention). The synchronization process (i.e., steps210 and220 of process200), in an embodiment of the present invention, is now explained in more detail.
• At first use, the user runs a one-time installation program, which registers the conduit (i.e., a shared library) with the synchronization manger software provided with their PDA102 (e.g., the Palm's HotSync Manager). In addition, an application that manages synchronization information is installed to thePDA102. The user then starts this application onPDA102 and enters their valid user name and password.
• When the user decides to synchronize, the user placesPDA102 in its cradle and presses the synchronization (e.g., HotSync) button. This action triggers the synchronization manger software of the PDA onworkstation106, which in turn calls thedesktop agent306.Agent306 connects with Web server110 (through a working connection to theInternet108 from workstation106) and automatically downloads and installs any new files required for any real-time observation assessment applications available from the service provider.Agent306 also recognizes any new, changed and deleted data onPDA102 and formats it for transmission toWeb server110.
• In one embodiment,agent306 communicates withWeb server110 using SyncML. SyncML messages are encrypted, compressed and exchanged using HTTP over secure sockets layer (SSL) (i.e., HTTPS), so firewalls and content-filtering agents will not block the communication channel.
• Web server110 forwards SyncML messages fromagent306 tosynchronization application310 running onapplication server112. There, code logic reconciles the information received fromagent306 with the contents of thecentral database114, resolves potential conflicts, and sends SyncML messages back toagent306 specifying modifications todata store104 onPDA102 to ensure the data indata store104 matches that in (i.e., is synchronized with)master data repository114.
• Application server112 maintains a historical record of successful synchronization sessions for all users. In the case of inadvertent data corruption or loss of data onPDA102,synchronization application310 is able to restore the state of all real-time observation assessment applications onPDA102 to an earlier, stable state.
• As will be apparent to one skilled in the relevant art(s) after reading the description herein, the use of a wireless, Web-enabledPDA102 would allow a user to bypass the use ofworkstation106 in accessingplatform120.
• The synchronization process withinsystem100, according to an embodiment of the present invention, is now described in more detail. This time, however, reference is made tosoftware architecture300 and the terminology defined in Table 1 below.

  TABLE 1
  Term	Definition
  Action	An operation that a properly equippedAgent 306
  	can execute.
  Agent	A set of programs (i.e., remote management utility)
  	residing ondesktop 106.
  Agent Core	The central part ofAgent 306 that coordinates
  	requests between Packages.
  Command	An order forAgent 306 to invoke a specific Action,
  	accompanied by arguments.
  Context	A description of the circumstances under which
  	Agent 306 is operating; key elements include Station
  	identification and information pertaining to the
  	particular Trigger that invoked the Agent. In the
  	case of a device synchronization, this would
  	include device identification as well as
  	user authentication information.
  Handler	An extension to the Integration Server that connects
  	Desktop Agent 306 with another component
  	ofplatform 120.
  Package	An extension (i.e., shared library or script) that enables
  	Agent 306 to perform one or more Actions.
  Trigger	A program or other means that invokesAgent 306.
  Schedule	A sequence of Commands to be sent toAgent 306.
  Scheduling Rule	A mapping between a set of Contexts and a list
  	of Commands.
  Session	A full operation ofAgent 306, beginning with a
  	Trigger and ending with either a hang-up Command
  	fromserver 110 or an error.
  Station	A desktop computer (i.e., workstation 106) that has
  	Agent 306 installed.

• In general, the communication betweendesktop agent306 andintegration service307 manifests itself as a series of message exchanges. Each exchange includesagent306 assembling a message, making a connection tointegration service307, sending the message, receiving a response, and then breaking the connection. This “conversation” begins whenagent306 firstcontacts integration service307.Desktop agent306 continues to initiate exchanges withservice307 untilservice307 explicitly indicates that the session has been completed with a hang-up command.
• Shown inFIG. 6A is a block diagram illustrating in more detail, according to an embodiment of the present invention, portions of the software architecture originally shown inFIG. 3 is shown. The synchronization process (i.e., step220 of process200) is now described with reference toFIG. 6A in order to detail the interaction betweendesktop agent306 andintegration service307.
• When a user starts the HotSync process with handheld102 (i.e., placing it in the cradle and pressing the HotSync button), a HotSync Manager620 (residing on station106) responds by starting aHotSync Trigger618 fordesktop agent306, which in turn invokes an agent core610.
• Agent Core610 connects to an integration servlet601 (via a connector package612 that initiates and manages communications betweendesktop agent306 and Web server110), and delivers a message identifying the station and an indication that the user has requested synchronization ofhandheld102.
• Integration servlet601 contacts asession manager602 which is responsible for maintaining session data across exchanges withagent307.Session manager602 then creates a session identifier, which is used to recall session state for later exchanges.Session manager602 then contacts a scheduler604, which returns a list of commands (i.e., a schedule) foragent306 to execute. The schedule includes a command to begin a data synchronization, as well as commands to download any additional application files fromserver110 that might be needed bystation106.Session manager602 returns this session identifier and the schedule to integration servlet601, which in turn packages them into a message which is sent toagent306.
• Agent core610 then invokes a synchronization package614, which initiates a synchronization process with synchronization application310 (residing on application server112). Synchronization package614 initiates this process by assembling an initial message, including device identification as well as user name and password information from the handheld102, which it obtains via a HotSync database package616. HotSync database package616 is a package that can read and/or write data to a data store on handheld102 (e.g., cache104).
• This initial message is returned to agent core610, which then sends the message to integration servlet601. Servlet601 recognizes that the message is to be handled by asynchronization handler605, which performs any necessary translation before passing the message on tosynchronization application310.Application310 responds with a series of messages for synchronization package614, which are returned viasynchronization handler605 to servlet601. This cycle continues untilsynchronization application310 indicates that it requires no further information. At this point, servlet601 requests thatagent306 send any logging information it has gathered up to this point. This logging information (e.g., failure to contact theintegration service307 on a prior attempt, events leading up to a prior connection breakage, or failure to write a record to the PDA'sdata store104 and the like) is sent to a log handler606, which in turn interprets the information and passes it to logging service318 (residing on application server112) for storage into log320 (residing, in an embodiment, on database server118). Finally, servlet601 instructsagent306 to hang up, which ends the conversation.
• A sequence diagram illustrating the above-described synchronization process (i.e., step220 of process200), in one embodiment of the present invention, is shown inFIG. 6B.
• VI. Real-Time Observation Assessment
• As mentioned above, the present invention takes advantage of handheld computers, desktop/laptop computers and the increasing presence of Internet access in order to reduce the paperwork associated with conventional educational assessments. Thus, an embodiment of the present invention allows a service provider organization to provide a platform that allows real-time observation literacy assessment based on the running record methodology described in Clay. Such an embodiment is now described in more detail.
• Referring to FIGS.4A-H, exemplary windows or screen shots generated by a real-time observation literacy assessment application executing onPDA102 during administration of an assessment are shown.
• FIG. 4A is an exemplary “splash” window or screen shot410, displayed onPDA102 during the execution of the real-time observation literacy assessment application. That is, screen410 is displayed onPDA102 immediately after the assessment application is launched.
• FIG. 4B is an exemplary window or screen shot420 which would be displayed onPDA102 during the student and book selection steps (i.e., steps212-214 of process200) of the real-time observation literacy assessment application. In an embodiment of the present invention, when the teacher selects a student from the class roster (top portion of screen420), books that match that student's reading level automatically appear in the book selection list (lower portion of screen420).
• FIG. 4C is an exemplary window or screen shot430 displayed onPDA102 during the administration of the literacy assessment (i.e., step216 of process200) of the real-time observation literacy assessment application.Screen430 includes aticker area432, a palette of code buttons434a-f, a display (“lens”)area436, afreehand area438 and a set of control buttons439a-e.
• The assessment would begin when the teacher taps on start/pause control button439c.This causes atimer display437 to begin timing the duration of the assessment.Ticker area432 sequentially displays a pre-determined number of words from the text of the book selected in screen420 (i.e., step214 of process200). As a student physically reads the book aloud, the teacher can follow along onPDA screen430. As a student moves through the text of the selected book, (and thus, the pre-determined number of words displayed in ticker area432), the teacher may use the PDA's stylus to tap an uparrow431aor adown arrow431baccordingly. At any time, the teacher may tap cancelcontrol button439dwhich would cancel the current assessment and not save any of its data onto the handheld102.
• In an embodiment of the present invention, as the student reads the book and the teacher follows along usingticker area432, every time the student says a word correctly, the teacher needs to do nothing. Thus, in an embodiment, the real-time observation literacy assessment application is non-linear. That is, it saves effort and time for the teacher by emphasizing the capture of error-related (or behavior of interest-related) data during the assessment.
• In such an embodiment, when a student says a word incorrectly (or exhibits some other behavior of interest), however, the teacher taps the word inarea432 which causes the word to be placed in a highlightedarea435 withinticker area432. This also causes the selected word to be displayed inlens area436.Freehand area438 may then be used by the teacher to make a note about the error (e.g., writing the word the student mistakenly substituted for the highlighted word actually in the book's text). In an embodiment, these notes are taken in freehand thereby eliminating the need to use Graffiti. At any time during this process,clear control button439bmay be tapped to clear any freehand text placed inarea438 by the teacher in error.
• Next, the teacher taps one or more of the code buttons434 in the palette to identify the error(s) or behavior(s) of interest. In an embodiment of the present invention that implements the Clay methodology, as will be appreciated by one skilled in the relevant art(s), buttons434 provided in the palette by the literacy assessment application are described in Table 2 below.

  TABLE 2
  Button Label/
  Code	Behavior of Interest
  S	Substitution:button 434a is tapped when student
  	substitutes another word, which may be written inarea
  	438 by the teacher, for the tapped word inarea 432.
  O	Omission:button 434b is tapped when student omits
  	the tapped word inarea 432.
  {circumflex over ( )}	Insertion:button 434c is tapped when student inserts a
  	word not in the text before/after the word tapped inarea
  	432.
  T	Told:button 434d is tapped when student could not
  	read the word tapped inarea 432 and had to be told by
  	teacher.
  SC	Self-Correction:button 434e is tapped when the
  	student said the word tapped inarea 432 incorrectly, but
  	then corrected themselves.

• In an embodiment of the present invention, as shown inscreen430, after the teacher taps on one of the buttons in palette434, this would cause the button label to appear inlens area436 and a unique annotation (e.g., an annotation equal to the button label) to appear above the relevant word inticker area432 to correspond to the each button in palette434. A series of such annotations is shown inarea433a.(An “s” annotation also appears above the word “said” in highlightedarea435 indicating the student substituted another word—i.e., the word “says” which the teacher noted inarea438—for the word “said.”) Further, in such an embodiment, other annotations can appear inticker area432. For example, anote icon annotation433bappears above the word “get” which indicates that the teacher created a note concerning what they observed during the child's reading of that word. In another example, a familiar “accurate” checkmark would appear above each word the student read correctly.
• In an embodiment, deletecontrol button434facts as an “undo” button that negates the effect of the last button in palette434 tapped by the teacher. This would also cause any resulting annotation inarea432 to be erased or replaced by anote icon annotation433bdepending on the state ofarea438. That is, in an embodiment, ifarea438 does not contain a note,button434fwill cause the previously-created annotation to be erased. If, however,area438 does contain a note,button434fwill cause the previously-created annotation to be replaced by anote icon annotation433b,thereby indicating the completion of the note creation for the previously-tapped word.
• In an embodiment of the present invention that implements the Clay methodology, the reading running records include the M/S/V coding system that indicates what cues sent the student in the wrong direction when they made an error and what cues were used by the student when they corrected themselves. That is, as will be appreciated by those skilled in the relevant art(s): an “M” code indicates that the student took a cue from the meaning or context of the text (i.e., passage) they were reading in making the mistake or correcting themselves; an “S” code indicates that the student took a cue from the syntax of the text they were reading in making the mistake or correcting themselves; and a “V” code indicates that the student took a visual cue from the text they were reading in making the mistake or correcting themselves.
• Accordingly, the teacher may tapmenu control button439aduring the assessment in order to display a second palette of buttons corresponding to the M/S/V coding system.FIG. 4D is an exemplary window or screen shot440 displayed onPDA102 during the administration of the literacy assessment (i.e., step216 of process200) of the real-time observation literacy assessment application.Screen440 shows the expansion ofmenu control button439a.This causes a palette of control buttons442a-ccorresponding to the M/S/V codes, respectively. The teacher can then indicate the use of meaning, syntax, or visual cues by pressing the appropriate control button442. These indications are captured in the running record, along with the error, for later review, display or modification.
• In an embodiment, palette442 would include a Try That Again (“TTA”)button442dthat allows a teacher, when a student make too many errors (e.g., after being distracted), to instruct the student to start over. TappingTTA button442dplaces a TTA expandicon444 inticker area432 before the word that was tapped by the teacher as the beginning word for the TTA text portion. The application then calculates the last word of the TTA passage to be the last word inticker area432 that has been previously-coded with a behavior of interest button434a-eby the teacher. Then, all the error-related annotations and notes are cleared fromarea432 for the TTA text portion. By tapping on TTA expandicon444, it turns into aTTA contract icon452 as shown inscreen450 ofFIG. 4E. Then, the text that was selected to be tried again is grayed out and the previously-recorded errors are displayed above the text. Tapping onTTA contract icon452 will cause the handheld to return toscreen440.
• In an alternate embodiment, the set of buttons434 would include a “Repetition” button that allows the teacher to cause an arrow to appear inticker area432 indicating the beginning and ending point where the student repeated themselves while reading the (or those) word(s) aloud.FIG. 4F is an exemplary window or screen shot460 which would be displayed onPDA102 during such a repetition operation. That is,screen460 illustrates the teacher tapping on the center of a word (i.e., “the”) and dragging to center of any previous word (i.e., “fishing”) to select a block of text the student repeated. In an embodiment, when the stylus moves off the original word, a left facingelbow bracket462 displays on upper right hand corner of word to denote the end of the selected area. As the stylus moves along the text, apath464 is drawn up to a right facing elbow bracket466 that snaps to the upper left hand coner of the word that denotes the beginning of the selected block. In an alternate embodiment, this action may causeTTA button442dto automatically appear for selection. In yet another embodiment, a note can be created about the repetition (e.g., number of repetitions, etc.) infreehand area438.
• FIGS. 4G-4iare exemplary window orscreen shots470,480 and490, respectively, displayed onPDA102 during the display scores and analysis step (i.e., step218 of process200) of the real-time observation literacy assessment application. That is, after the teacher taps the “Done”control button439e,the assessment application is stopped and the assessment is complete.
• Summary screen470 may then be displayed, using the “Summary” tab shown, which gives the teacher instant information to support their teaching. As shown inscreen470, the real-time, observation assessment application immediately reports and charts the student's results (e.g., accuracy rate, total words, total errors, etc.), M/S/V analysis and other important statistics that were not previously available to teachers in prior paper-based assessments.
• M/S/V analysis screen480 may also be displayed, using the “MSV” tab shown, which gives the teacher additional instant information to support their teaching and provides an opportunity to record or modify M/S/V analyses. As shown inscreen480, the real-time, observation assessment application immediately reports more detailed M/S/V analysis than that previously-shown inscreen470. The teacher can select an error by clicking on the appropriate word, which causeslens area436 to display the word, and any associated note and M/S/V analysis already recorded for that error to be displayed infreehand area438. The teacher can then record or modify the M/S/V analysis for that error by pressing a set of M/S/V toggle buttons displayed inarea438.
• Results edit screen490 may also be displayed, accessible using the “Edit” tab shown, which gives the teacher the ability to edit and correct portions of the reading running record, including any notes infreehand area438, before synchronization withWeb server110. In an embodiment of the present invention, such notes are stored in the handheld'slocal cache104 using a platform-independent vector encoding. Upon synchronization, this encoded information is transmitted toWeb server110 and eventually stored indatabase114. During report generation and viewing (i.e., steps224-226 of process200), the notes may be recreated byWeb server110 in Flash format for browser-based viewing onworkstation106.
• FIGS. 4J-4K are additional exemplary window orscreen shots492 and494, respectively, displayed onPDA102 during the display scores and analysis step (i.e., step218 of process200) of the real-time observation literacy assessment application. That is, after the teacher taps the “Done”control button439e,the assessment application is stopped and the assessment is complete.Screen492 may then be displayed which gives the teacher instant information to support their teaching. The application (i.e., screen492) immediately reports the student's accuracy rate and other important statistics that were not previously available to teachers in prior, paper-based assessments.
• In an embodiment, detailed results may be displayed graphically and are available to the teacher onscreen492. In the example ofscreen492, a graph that displays the tested student's progress over time, including any change in reading level, is shown. Other view results screens, such as screen494, take data that teachers previously had to interpret in purely numerical form, and make it easy to grasp visually. That is, the teacher can exactly determine how the tested student is progressing by using, for example, the ratio of accurately-read words to errors, self-corrections to errors and the like can be shown in pie chart form, and a graphical comparison of relative frequencies of meaning, syntax and visual miscues for errors and self-corrections can be displayed.
• FIGS. 4L-4M are two additional exemplary window orscreen shots496 and498, respectively, displayed onPDA102 during the display scores and analysis step (i.e., step218 of process200) of the real-time observation literacy assessment application. The present invention also provides more “classroom management” tools, such asscreen496, that help teachers see the particulars of each student's reading progress, that help manage changes in reading level and the administration of reading groups. Many of these reports, such as screen498, are useful for conferences among reading teachers and for conferences with parents in which a broad review of reading history is needed.
• As mentioned above, the teacher can place theirPDA102 in its cradle, synchronize and then log onto the service provider's Web site via theInternet108. Once there, several report window or screen shot can be displayed onworkstation106 during the view report step (i.e., step126) of process200). In general, these report screens allow the teacher to view a running record in the same manner as they would be on a paper record. However, more detailed analysis can be done, teachers can more easily type observations, and records can be shared in a discussion with colleagues or experts. Progress reports also can be generated by grade level, by school, and even by district.
• FIGS.5A-C are three exemplary report window orscreen shots510,520 and530, respectively, displayed onworkstation106 during the view report step (i.e., step226 of process200).
• Inreport screen510, the teacher can see a graphical report of a student's assessment. Each error shows up in adiagnostic window502 which contains the kind of error and any M/S/V analysis that was conducted during the assessment. Astatistics area504 contains important statistics about the student's error types. At the bottom of the screen, teachers can record any notes they wish to associate with the assessment. In an embodiment, ane-mail button506 is provided which allows the teacher to communicate with students and parents directly.
• Screen510 also includes aprogress link508. Clickinglink508causes report screen520 to be displayed.Report screen520 illustrates a student's progress over time as they have moved through various reading levels. A chart512 is displayed with each point514 being a link to a full report for the previous assessment it represents.
• Screen520 also includes aclass summary link516. Clicking onlink516 causes screen530 to be displayed.Report screen530 is a view of the entire class' progress over time. The number of days each student has been on his or her current reading level is displayed in an area522. In an embodiment,screen530 will highlight all students who have been on a specific reading level too long (based on the teacher's pre-set requirements).Screen530 also contains a plurality of graphs (shown as graphs524a-b) which can display various comparisons of the class' achievement to that of other classes or categories of students.
• VII. Example Implementations
• The present invention (i.e.,system100,process200,architecture300 and/or any part(s) or function(s) thereof) may be implemented using hardware, software or a combination thereof and may be implemented in one or more computer systems or other processing systems. In fact, in one embodiment, the invention is directed toward one or more computer systems capable of carrying out the functionality described herein. An example of acomputer system700 is shown inFIG. 7.
• Computer system700 includes one or more processors, such as processor704. The processor704 is connected to a communication infrastructure706 (e.g., a communications bus, cross-over bar, or network). Various software embodiments are described in terms of this exemplary computer system. After reading this description, it will become apparent to a person skilled in the relevant art(s) how to implement the invention using other computer systems and/or architectures.
• Computer system700 can include adisplay interface702 that forwards graphics, text, and other data from the communication infrastructure706 (or from a frame buffer not shown) for display on thedisplay unit730.
• Computer system700 also includes amain memory708, preferably random access memory (RAM), and may also include asecondary memory710. Thesecondary memory710 may include, for example, ahard disk drive712 and/or aremovable storage drive714, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, etc. Theremovable storage drive714 reads from and/or writes to a removable storage unit718 in a well known manner. Removable storage unit718, represents a floppy disk, magnetic tape, optical disk, etc. which is read by and written to byremovable storage drive714. As will be appreciated, the removable storage unit718 includes a computer usable storage medium having stored therein computer software and/or data.
• In alternative embodiments,secondary memory710 may include other similar devices for allowing computer programs or other instructions to be loaded intocomputer system700. Such devices may include, for example, a removable storage unit722 and aninterface720. Examples of such may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an erasable programmable read only memory (EPROM), or programmable read only memory (PROM)) and associated socket, and other removable storage units722 andinterfaces720, which allow software and data to be transferred from the removable storage unit722 tocomputer system700.
• Computer system700 may also include acommunications interface724. Communications interface724 allows software and data to be transferred betweencomputer system700 and external devices. Examples ofcommunications interface724 may include a modem, a network interface (such as an Ethernet card), a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, etc. Software and data transferred viacommunications interface724 are in the form ofsignals728 which may be electronic, electromagnetic, optical or other signals capable of being received bycommunications interface724. Thesesignals728 are provided tocommunications interface724 via a communications path (e.g., channel)726. Thischannel726 carriessignals728 and may be implemented using wire or cable, fiber optics, a telephone line, a cellular link, an radio frequency (RF) link and other communications channels.
• In this document, the terms “computer program medium” and “computer usable medium” are used to generally refer to media such asremovable storage drive714, a hard disk installed inhard disk drive712, and signals728. These computer program products provide software tocomputer system700. The invention is directed to such computer program products.
• Computer programs (also referred to as computer control logic) are stored inmain memory708 and/orsecondary memory710. Computer programs may also be received viacommunications interface724. Such computer programs, when executed, enable thecomputer system700 to perform the features of the present invention, as discussed herein. In particular, the computer programs, when executed, enable the processor704 to perform the features of the present invention. Accordingly, such computer programs represent controllers of thecomputer system700.
• In an embodiment where the invention is implemented using software, the software may be stored in a computer program product and loaded intocomputer system700 usingremovable storage drive714,hard drive712 orcommunications interface724. The control logic (software), when executed by the processor704, causes the processor704 to perform the functions of the invention as described herein.
• In another embodiment, the invention is implemented primarily in hardware using, for example, hardware components such as application specific integrated circuits (ASICs). Implementation of the hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art(s).
• In yet another embodiment, the invention is implemented using a combination of both hardware and software.
• VIII. Conclusion
• It should be understood thatFIGS. 4 and 5, which highlight the functionality and other advantages ofRTOA system100, are presented for example purposes only. The architecture of the present invention is sufficiently flexible and configurable such that users may utilizesystem100 in ways other than that shown inFIGS. 4 and 5 (e.g., the use of different button labels, screen annotation characters, button codes for different behaviors of interests, report formats and the like).
• While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example and not limitation. It will be apparent to persons skilled in the relevant art(s) that various changes in form and detail can be made therein without departing from the spirit and scope of the invention. Thus, the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims (11)

1. A method for providing access to a real-time, observation assessment system, the method comprising:

enabling a desktop computer to access a Web server, via a communications network, in order to enable a subscriber to access the real-time, observation assessment system;

accessing a master database, coupled to the Web server, including a class roster for the subscriber;

receiving, at the Web server, a transmission via the communications network of assessment data previously stored on a mobile computing device used by the subscriber when performing observation assessments, wherein the mobile computing device is linked to the desktop computer during the transmission;

matching the received assessment data with the class roster for the subscriber; and

storing, by an application server coupled to the Web server and the master database, the assessment data to the master database in association with the class roster for the subscriber,

wherein browser-based reports and analysis using the class roster and the stored assessment data can be created by said Web server and displayed to the subscriber on the desktop computer.

2. The method ofclaim 1 wherein the communications network is at least a portion of the Internet.

3. The method ofclaim 2 further comprising sending, by the Web server, the browser-based reports and analysis to a recipient specified by the subscriber via electronic mail.

4. The method ofclaim 2 wherein:

the transmission utilizes Hypertext Transmission Protocol, Secured (HTTPS), and

the assessment data is in compressed Extensible Markup Language (XML) format.

5. The method ofclaim 1 wherein matching the received assessment data with the class roster for the subscriber comprises matching the received assessment data with a student record that is associated with the class roster for the subscriber.

6. A real-time, observation assessment system comprising:

a master database containing login information and a class roster for a subscriber;

a Web server, coupled to the master database, that accepts assessment data previously-stored on a mobile computing device used by the subscriber when performing observation assessments and transmitted via a communications network; and

an application server, coupled to the master database and the Web server, that matches the assessment data received from the mobile computing device with the class roster for the subscriber and stores the assessment data received from the mobile computing device to the master database in association with the class roster for the subscriber,

wherein the Web server can generate browser-based reports and analysis using the class roster and the translated and stored assessment data.

7. The system ofclaim 6 wherein said communications network is at least a portion of the Internet.

8. The system ofclaim 6 wherein the Web server allows the browser-based reports and analysis to be sent a recipient specified by the subscriber via electronic mail.

9. The system ofclaim 6 wherein the assessment data is transmitted from the mobile computing device to the Web server in compressed Extensible Markup Language (XML) format over the Hypertext Transmission Protocol, Secured (HTTPS).

10. The system ofclaim 6 further comprising a desktop agent, connected to the communications network, that facilitates the transmitting of the assessment data from the mobile computing device to the Web server.

11. The system ofclaim 6 wherein matching the received assessment data with the class roster for the subscriber comprises matching the received assessment data with a student record that is associated with the class roster for the subscriber.

Images