US7162643B1 - Method and system for providing transfer of analytic application data over a network - Google Patents

Abstract

A method and system providing a high speed and secure data link for moving large amounts of data across a network, such as the data used in an analytic application. Featured are simultaneous compression and encryption of the data, as well as means for recovery in the event the network connection is lost.

Description

FIELD OF THE INVENTION

The invention relates generally to computer system networks, and more particularly, to securing rapid, reliable, and private communication between networked computers in a multiple data warehouse/analytic application environment.

BACKGROUND OF THE INVENTION

Computers are used to perform a wide variety of applications in such diverse fields as finance, traditional and electronic commercial transactions, manufacturing, health care, telecommunications, etc. Most of these applications typically involve inputting or electronically receiving data, processing the data according to a computer program, then storing the results in a database, and perhaps transmitting the data to another application, messaging system, or client in a computer network. As computers become more powerful, faster, and more versatile, the amount of data that can be processed also increases.

Unfortunately, the raw data found in operational databases often exist as rows and columns of numbers and codes which, when viewed by individuals, appears bewildering and incomprehensible. Furthermore, the scope and vastness of the raw data stored in modern databases is overwhelming to a casual observer. Hence, applications were developed in an effort to help interpret, analyze, and compile the data so that it may be readily and easily understood by a human. This is accomplished by sifting, sorting, and summarizing the raw data before it is presented for display, storage, or transmission. Thereby, individuals can now interpret the data and make key decisions based thereon.

Extracting raw data from one or more operational databases and transforming it into useful information (e.g., data “warehouses” and data “marts”) is the function of analytic applications. In data warehouses and data marts, the data are structured to satisfy decision support roles rather than operational needs. A data warehouse utilizes a business model to combine and process operational data and make it available in a consistent way. Before the data are loaded into the data warehouse, the corresponding source data from an operational database are filtered to remove extraneous and erroneous records; cryptic and conflicting codes are resolved; raw data are translated into something more meaningful; and summary data that are useful for decision support, trend analysis and modeling or other end-user needs are pre-calculated. A data mart is similar to a data warehouse, except that it contains a subset of corporate data for a single aspect of business, such as finance, sales, inventory, or human resources.

In the end, the data warehouse or data mart is comprised of an “analytical” database containing extremely large amounts of data useful for direct decision support or for use in analytic applications capable of sophisticated statistical and logical analysis of the transformed operational raw data. With data warehouses and data marts, useful information is retained at the disposal of the decision makers and users of analytic applications and may be distributed to data warehouse servers in a networked system. Additionally, decision maker clients can retrieve analytical data resident on a remote data warehouse servers over a computer system network.

An example of the type of company that would use data warehousing is an online Internet bookseller having millions of customers located worldwide whose book preferences and purchases are tracked. By processing and warehousing these data, top executives of the bookseller can access the processed data from the data warehouse, which can be used for sophisticated analysis and to make key decisions on how to better serve the preferences of their customers throughout the world.

The rapid increase in the use of networking systems, including Wide Area Networks (WAN), the Worldwide Web and the Internet, provides the capability to transmit operational data into database applications and to share data contained in databases resident in disparate networked servers. For example, vast amounts of current transactional data are continuously generated by business-to-consumer and business-to-business electronic commerce conducted over the Internet. These transactional data are routinely captured and collected in an operational database for storage, processing, and distribution to databases in networked servers.

The expanding use of “messaging systems” and the like enhances the capacity of networks to transmit data and to provide interoperability between disparate database systems. Messaging systems are computer systems that allow logical elements of diverse applications to seamlessly link with one another. Messaging systems also provide for the delivery of data across a broad range of hardware and software platforms, and allow applications to interoperate across network links despite differences in underlying communications protocols, system architectures, operating systems, and database services. Messaging systems and the recent development of Internet access through wireless devices such as enabled cellular phones, two-way pagers, and hand-held personal computers, serve to augment the transmission and storage of data and the interoperability of disparate database systems.

In the current data warehouse/data mart networking environment, one general concern involves the sheer volume of data that must be dealt with. Often massive, multi-terabyte data files are stored in various server sites of data warehouses or in operational databases. Transmitting these massive amounts of data over WANs or the Internet is a troublesome task. The time needed to move the data is significant, and the probability that the data may contain an error introduced during transmission is increased. Also, the data are also vulnerable to interception by an unauthorized party. Furthermore, when the connection is lost in the process of transmitting the data over a network, there often is a need to retransmit large amounts of data already transmitted prior to the loss of connection, further increasing the time needed to move the data.

Accordingly, there is a need for a reliable, secure, authenticated, verifiable, and rapid system and/or method for the transmission of huge amounts of data, such as data in a data warehouse/mart, over networks such as WANs and the Internet. The present invention provides a novel solution to this need.

SUMMARY OF THE INVENTION

The present invention satisfies a currently unmet need in a networked data warehouse/analytic application environment to provide a method and system that provide reliable, secure, authenticated, verifiable, and rapid system and method for the transmission of huge amounts of data over a network (e.g., operational data, and transformed data in a data warehouse/data mart). The data can be moved from a source to a target (e.g., from a server to a client, or from a client to a server) in the computer system network. The source represents any centralized source on the network, while the target can represent a remotely located device (e.g., at a customer site) or a local device (e.g., a device in communication with the source via a local area network).

In one embodiment, in a source (e.g., server) computer system, an incoming request is received from a target (e.g., client) for a large amount of data (e.g., a data file) resident in a mass storage unit on the server, for example. The incoming request is authenticated and is then used to spawn a session thread between the server and the client. The incoming request includes a command that, in one embodiment, uses Extensible Markup Language (XML). The command is parsed and translated into a set of tasks which can be executed by the server as part of the session thread.

In one embodiment, the data are separated into blocks, and each block is sequentially compressed and encrypted, then sent to the client. In one embodiment, the blocks are processed in parallel, saving time. The transfer of the data to the client is checked to make sure it is complete and accurate.

On the client side, the session thread between the server and client is spawned in response to a message from the server. The compressed and encrypted data blocks are received from the server, then decompressed, decrypted, and assembled into the requested data.

The present invention provides a recovery mechanism for automatically or manually restoring a connection between the server and client when the connection is lost. As part of the recovery mechanism, data transfer is resumed from the point where it was terminated when the connection was lost, so that previously transmitted data do not have to be retransmitted.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1A illustrates a schematic block diagram of an exemplary client/server computer system network upon which embodiments of the present invention may be implemented.

FIG. 1B illustrates an exemplary computer system upon which embodiments of the present invention may be practiced.

FIG. 2A illustrates a general functional block diagram of a computer system network in accordance with one embodiment of the present invention.

FIG. 2B illustrates a more detailed functional block diagram of the computer system network generally illustrated inFIG. 2A.

FIG. 3A illustrates data flow through a first embodiment of an output channel of the present invention.

FIG. 3B illustrates data flow through a first embodiment of an input channel of the present invention.

FIG. 4A illustrates data flow through a second embodiment of an output channel of the present invention.

FIG. 4B illustrates data flow through a second embodiment of an input channel of the present invention.

FIG. 5 illustrates data flow through one embodiment of a session thread in accordance with the present invention.

FIGS. 6A,6B, and6C illustrate data transfer recovery after a failure of a network connection in accordance with one embodiment of the present invention.

FIG. 7A is flowchart of the steps in a server-side process for transferring data over a network in accordance with one embodiment of the present invention.

FIG. 7B is flowchart of the steps in a client-side process for transferring data over a network in accordance with one embodiment of the present invention.

DETAILED DESCRIPTION

Reference will now be made in detail to the preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. While the invention will be described in conjunction with the preferred embodiments, it will be understood that they are not intended to limit the invention to these embodiments. On the contrary, the invention is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be obvious to one of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present invention.

Some portions of the detailed descriptions that follow are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In the present application, a procedure, logic block, process, or the like, is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as sessions, objects, blocks, parts, threads, or the like.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present invention, discussions utilizing terms such as “establishing,” “issuing,” “authenticating,” “spawning,” “transmitting,” “accumulating,” “restoring,” “resuming,” “translating,” “storing,” “executing,” “receiving,” “writing,” “compressing,” “decompressing,” “encrypting,” “decrypting,” “sending,” “verifying,” or the like, refer to actions and processes (e.g., processes700 and750 ofFIGS. 7A and 7B, respectively) of a computer system or similar electronic computing device. The computer system or similar electronic computing device manipulates and transforms data represented as physical (electronic) quantities within the computer system memories, registers or other such information storage, transmission or display devices. The present invention is well suited to the use of other computer systems.

FIG. 1A illustrates a block diagram of client/servercomputer system network100 upon which embodiments of the present invention may be practiced. This server/client system100 is made up of server computer system110 (e.g., Unix or NT server computer),client computer system102, andremote computer systems103–105, (e.g., personal computers, laptop computers, workstations, terminals, etc.) which may be used to access the information accessible toserver computer system110.Server110 can represent any centralized source on thenetwork100, whileclient102 andremote computer systems103–105 can represent a remotely located device (e.g., at a customer site) or a local device (e.g., a device in communication withserver110 via a local area network).

Eachremote computer system103–105 has its own physical memory system (e.g., hard drive, random access memory, read only memory, etc.) for storing and manipulating data.Client computer system102,server computer system110, andremote computer systems103–105 are connected for intercommunication and transfer of data bynetwork bus107. However, it is appreciated that these devices may instead by coupled in a wireless network.

Server computer system110 is coupled to servermass storage device112 that is or is not accessible byclient computer system102 andcomputer terminals103–105 throughnetwork bus107 directly.Client system102 also has its own clientmass storage device170. The present invention includes threads and objects within the application software that are executed byserver computer system110 and/orclient system102 to transfer data therebetween (refer toFIG. 2B, below).

Located withinmass storage device112 isoperational database116a, which receives and stores the current raw data for a data mart or data warehouse. Raw data received and stored withinoperational database116aare transformed by an analytic application into information that is more meaningful for decision support. Data marts/warehouses113a, located withinmass storage device112, include transformed data processed by the analytic application. It is important to point out that data marts/warehouses113aandoperational database116acould each reside within a separate mass storage devices and each mass storage device could be connected bynetwork bus107 to a separate server.

A data file120 is a file stored within eitheroperational database116a, within the database of data warehouse/data mart113b, or elsewhere in servermass storage device112. In accordance with the present invention, data file120 is securely, quickly and reliably transmitted overnetwork bus107 toclient computer system102 or toremote computer systems103–105 for display or storage on these systems or for use in analytic applications resident on these systems. Data file120 is a large file containing, for example, operational data such as customer data or third party data. Data file120 may instead contain data transformed according to an analytic application. It is appreciated that the present invention also can be used to transmit a data stream fromserver computer system110 to a target device (e.g.,client computer system102 or toremote computer systems103–105).

Operational database116band data warehouse113bare also shown residing within clientmass storage device170. A data file120 is shown also residing in clientmass storage device170 to represent the transmission and receipt of the data file as mentioned in the preceding paragraph. It is appreciated that the present invention can likewise be used to transmit a data is file (or a data stream) fromclient102 toserver110, or from these devices to any other device onnetwork100. Generally speaking, the present invention can be used to transmit a data file or a data stream from a source device to a target device. For simplicity of discussion, the present invention is described in the context of a transfer of a data file from a server to a client.

Refer now toFIG. 1B, which illustrates anexemplary computer system1090 upon which embodiments of the present invention may be practiced.Computer system1090 exemplifiesserver110,client102, andremote computer systems103–105 ofFIG. 1A.

In general,computer system1090 ofFIG. 1B comprisesbus1000 for communicating information, one ormore processors1001 coupled withbus1000 for processing information and instructions, random access (volatile) memory (RAM)1002 coupled withbus1000 for storing information and instructions forprocessor1001, read-only (non-volatile) memory (ROM)1003 coupled withbus1000 for storing static information and instructions forprocessor1001,data storage device1004 such as a magnetic or optical disk and disk drive coupled withbus1000 for storing information and instructions, an optional user output device such asdisplay device1005 coupled tobus1000 for displaying information to the computer user, an optional user input device such asalphanumeric input device1006 including alphanumeric and function keys coupled tobus1000 for communicating information and command selections toprocessor1001, and an optional user input device such ascursor control device1007 coupled tobus1000 for communicating user input information and command selections toprocessor1001. Furthermore, an optional input/output (I/O)device1008 is used to couplecomputer system1090 onto, for example, a network.

Display device1005 utilized withcomputer system1090 may be a liquid crystal device, cathode ray tube, or other display device suitable for creating graphic images and alphanumeric characters recognizable to the user.Cursor control device1007 allows the computer user to dynamically signal the two-dimensional movement of a visible symbol (pointer) on a display screen ofdisplay device1005. Many implementations of the cursor control device are known in the art including a trackball, mouse, joystick or special keys onalphanumeric input device1006 capable of signaling movement of a given direction or manner of displacement. It is to be appreciated that thecursor control1007 also may be directed and/or activated via input from the keyboard using special keys and key sequence commands. Alternatively, the cursor may be directed and/or activated via input from a number of specially adapted cursor directing devices.

FIG. 2A illustrates a functional block diagram of an embodiment of the client/servercomputer system network100 ofFIG. 1A in accordance with one embodiment of the present invention. In this embodiment, with reference also toFIG. 1A, the present invention provides asecure data stream118 that transfers data file120 fromserver computer system110 toclient computer system102. Data file120 is originally resident indata warehouse113aoroperational database116ain servermass storage device112, and is transmitted to clientmass storage device170.

For simplicity of discussion, communication is shown as occurring fromserver110 toclient102; however, it is appreciated that communication can similarly occur fromclient102 toserver110. In this latter case, the arrows indicating the direction of data flow would be in the opposite direction, the “output channel” would refer to the channel onclient102, and the “input channel” would refer to the channel onserver110.

FIG. 2B is a functional block diagram providing additional details of the client/servercomputer system network100 in accordance with one embodiment of the present invention. Listener object130ais designed to receive an incoming connection request and to spawn asession thread140ain response (specifically, listener object130acalls session manager object138ato createsession thread140a). Server listener object130areceives, at a well known port132, anincoming request134 fromclient computer system102. In the present embodiment, therequest134 is generated in the session thread140bexecuting onclient102. Therequest134 is a request to establish aclient socket connection136 and to transmit data from server mass storage device112 (e.g., data file120) toclient computer system102.

Arequest134 may be received from a remote device on thenetwork100, or from a local device. That is, for example, arequest134 may be received over the Internet from a device that is outside of a firewall or a company's intranet (e.g., a local area network), or therequest134 may be from a local device within the company's intranet. However, in the present embodiment, instead of trying to determine the source ofrequest134, all requests are delivered/received in encrypted form.

In the present embodiment, all computers in thenetwork100 are considered as non-secure. All keys are stored in encrypted form, with a password-based encryption algorithm used to encrypt and decrypt keys.

All registered users seeking to implement the data retrieval process of the present invention have their own account. Keys for each account are stored in separate files. Each file has the name of the account and is encrypted using the account password. All registered account keys are stored in a central repository (not shown) and are encrypted using a repository password. In the present embodiment, the repository is initialized with a single password hidden and secured by an administrator password. The repository password is used for all secured (encrypted) objects.

The central repository resides in main memory of the server110 (and likewise, a central repository resides in main memory of client102). The central repository has the capability to encrypt any stored object using pass word-based encryption. In one embodiment, the central repository asks the object if it needs encryption.

In the present embodiment, there are three repository object types: an object to represent an account, an object that represents a repository password for the administrative account, and an object to represent each session instance for recovery purposes. The object representing an account contains a key (or password) that is used to establish a secure connection betweenclient102 andserver110 ofFIGS. 2A and 2B. The object representing the repository password for the administrative account is used to gain access to all repository objects, and is secured with an administrator account password. The object representing each session instance contains all needed information for session recovery (refer toFIGS. 6A,6B and6C, below).

With reference again toFIG. 2B, in the present embodiment, listener object130ais part of a multi-thread software application that is capable of parallel execution in coordination with other threads of the present invention. A “thread” is a part of the application that can execute independently of the other parts that may be executing simultaneously in this parallel mode. Parallel execution of threads in this embodiment of the present invention is accomplished by sharing as much as possible of the application execution between the different threads. Such a multi-threading embodiment increases the speed of the transfer of data in the present invention.

Listener object130aestablishesclient socket connection136 at well known port132 requested byclient computer system102 onserver computer system110. When auser request134 is received, listener thread130acalls the session manager object138ato create a new session thread and to start to handle the user request. After completing these tasks, listener object130areturns to the task of listening for another incoming user request (e.g., another request fromclient computer system102 or from another computer system innetwork100 ofFIG. 1A).

In the present embodiment, session manager object138aofFIG. 2B spawns thesession thread140athat reads a command from the connection that is established by listener object130a, parses the command, and executes it (refer also toFIG. 5, below). Session manager object138aincludes (tracks) information regarding all of the ongoing sessions.

Asession140aincludes a channel object with multiple threads (e.g., channel139a), over which the data are actually transferred.Channel object139ais designed to transfer data in a secure manner. Additional information regardingchannel object139ais provided below in conjunction withFIGS. 3A,3B,4A and4B.

With reference toFIG. 2B, for security purposes, only clients authorized to accessdata warehouse113aandoperational database116aare allowed to receivedata file120. A protocol incorporated within the present invention is designed to authenticate thatincoming request134 originated at aclient computer system102 that is authorized to receivedata file120. Authentication protocols could include passwords, intelligent tokens, or other well-known techniques.

In the present embodiment, session manager object138aprovides the application program interfaces (APIs) for monitoring and managing sessions. An object is an application data item that includes instructions for the operations to be performed on it. After listener object130areceives auser request134, session manager object138areceives a call from listener object130ato create and start asession thread140afor processing user commands. In this embodiment, session manager object138aspawns session thread140a.

Session manager object138aprovides the functionality to generate a unique identifier (ID) for a session, and to maintain logs of sessions for recovery purposes. The APIs provided by session manager object138ainclude an API to create a session, an API to run the session, an API to stop the session by passing the session ID, an API to update the session's status by passing the session ID, an API to query session information, and an API to recover a failed session.

In the present embodiment, after session manager object138acreates a new session, it assigns a unique ID to that session. Session manager object138asaves (e.g., into an internal hash table) a session information object. Session manager object138a, when it creates a new session, passes its own reference as an initialization parameter to the session. The session then uses this reference, in combination with its unique session ID, to invoke a session manager callback function to provide an update of its status information to session manager object138a.

Thus, in the present embodiment of the present invention, listener object130aofserver110 receives anincoming connection request134 fromclient102 and passes the incoming connection request to the session manager object138a. Session manager object138aofserver110 spawnssession thread140athat reads the command from the connection request134 (e.g., a command requesting transfer of data file120 ofFIG. 2A). The command contains all of the information needed to open data file120 and to send data file120 through a connection betweenclient102 andserver110. Thesession thread140aparses the command and creates and initializes achannel object139a(with its threads), and runs the channel.Channel object139awill be initialized with the currently established network connection betweenserver110 andclient102.

Channel object139arepresents the set of objects needed for sending a file (e.g., data file120). In the present embodiment, the set of objects includes the reader, compressor, encryptor, and writer objects described in conjunction withFIGS. 3A and 4A.

On the client side, listener object130breceives the incoming connection fromserver110 and passes this connection to session manager138b. Session manager138bspawns a session thread140b. Session thread140breads the command from the connection. This command contains all of the information needed to connect toserver110, read the data file120, and to have the data file120 sent toclient102. Session thread140bparses the command and executes the following: establishes a connection withserver110, sends the command to start the transfer of data file120, and initializes and runs the channel.

FIG. 3A illustrates data flow through an embodiment of anoutput channel object139ain accordance with the present invention. In this embodiment,output channel139acomprises four data transformation threads or objects:reader channel object142a,compressor channel object146,encryptor channel object156, andwriter channel object152a. The data transformers (reader channel object142a,compressor channel object146,encryptor channel object156, andwriter channel object152a) can work in parallel (e.g., they each can have their own threads).Output channel139aalso comprises block manager object154a.

Block manager object154acontains multiple data blocks (e.g., vectors of data blocks). A data block is designed to store byte arrays between transformations. In the present embodiment, each data block is associated with one data transformation object, and only that object can write to the data block; however, a data transformation object may be associated with multiple data blocks. A size of a data block can vary, so that if a data transformation object finds that it cannot fit output data into the data block buffer, the data transformation object can create a new (larger) buffer that replaces the old (smaller) buffer. Also, for example, a data block containing compressed data can be smaller than one containing uncompressed data.

Block manager object154acontrols the total count of data blocks for each data transformation object; that is, the block manager object154ahas a parameter limiting the number of data blocks per data transformation object. Generally, about four data blocks are specified per data transformation object. If a data transformation object requests a data block, but the amount of available data blocks is equal to the maximum allowed for the transformation object (that is, there are no free data blocks), then block manager object154aand the data transformation object will wait for a data block to be freed.

Referring still toFIG. 3A, in the present embodiment,reader channel object142areads a part of data file120 and writes that part into a first data block buffer in the main memory ofserver computer system110. Data file120 is typically a large file. By reading a only a part of the file, downstream transformations relating to compression and encryption may commence in parallel, while subsequent parts of data file120 are read and written to first data block buffer.

Compressor channel object146 reads the data in the first data block buffer, transforms it (compresses it), and writes the compressed data to a second data block buffer.Compressor channel object146 encodes the data contained in data file120 in a way that makes it more compact.

Encryptor channel object156 reads the compressed data from the second data block buffer, encrypts it, and writes it a third data block buffer.

Writer channel object152areads the encrypted data block and writes it to the network socket stream (to the input channel139b;FIG. 3B).

In the present embodiment,output channel139afunctions as follows.Output channel139areceivesdata file120. Data file120 may be received byoutput channel139ain its entirety and then broken into smaller blocks of data, or data file120 may be read from mass storage device112 (FIG. 1A) a portion at a time.

Reader channel object142arequest a free data block. Block manager object154asearches for the free data block, and if there is no such block, then block manager object154acreates a new block, marks it as free, and assigns it toreader channel object142a.Reader channel object142awrites data from data file120 to the data block and marks the data as ready for compression.Compressor channel object146 receives this data block from block manager object154aand requests a free block (for its output). Block manager object154acreates a new data block, marks it as free, and assigns it tocompressor channel object146.

In parallel,reader channel object142arequests another data block, and as described above, block manager154acreates a new block, marks it as free, and assigns the new data block toreader channel object142a.Reader channel object142acan then write another portion of data file120 to this block and mark it as ready for compression.

In the meantime,compressor channel object146 compresses (encodes) the data contained in its respective data block, marks it as ready for encryption, and frees its respective data block.Encryptor channel object156 receives this (compressed) data block from block manager object154aand requests a free block for its output. Block manager object154acreates a new data block, marks it as free, and assigns it toencryptor channel object156.

Encryptor channel object156 encrypts the data contained in its respective data block, marks it as ready for ready for writing, and frees its respective data block.Writer channel object152areceives the encoded (compressed) and encrypted data block from block manager object154aand writes to the networksocket output stream307.

The process described above is repeated until thereader channel object142areads the last block of data in data file120. Each block of data is stepped throughoutput channel139a, with data blocks being created and freed as needed by the respective transformation objects. In this manner, the number of data blocks can be reduced so that memory resources are not unduly consumed.

In this embodiment of the present invention, means well known to those of ordinary skill in the art are utilized to verify that data file120 was completely and accurately transmitted toclient102.

Thus, the goals of the present invention are achieved. A data file120 is sent on request fromserver computer system110 to at least one computer system remote from server computer system110 (e.g., client102). The data transfer is accomplish securely, rapidly, and reliably.

FIG. 3B illustrates data flow through an embodiment of an input channel139bof the present invention. As shown inFIGS. 2B and 3B, in another aspect of the present invention, the operations ofserver computer system110 may be mirrored on theclient computer system102. A network stream of compressed/encrypted data blocks307 are received fromserver computer system110 byclient computer system102 and are decrypted, decompressed, and ultimately assembled intodata file120. Alternatively, the data blocks may received byclient102 en masse fromserver110.

On the client side, reader channel object142breads formatted (encrypted and compressed) data fromnetwork input stream307. A decryptor channel object164 reads data from a data block inblock manager object154b, decrypts the data, and writes the data to a data block inblock manager object154b.

Adecompressor channel object158 reads data from a data block inblock manager object154b, decompresses (decodes) the data, and writes the data to a data block inblock manager object154b. Writer channel object152 writes the data to the data file120 output stream to mass storage device170 (FIG. 2A), for example.

The data transformers (reader channel object142b, decryptor channel object164, decompressor channel object168, and writer channel object152) function similar to that described above in conjunction withFIG. 3A. Means well known to those skilled in the art can be utilized to verify that data file120 was completely and accurately transmitted.

Thus, the goals of the present invention are again achieved in the client-side embodiment of the present invention. A data file120 is sent on request ofclient computer system102 fromserver computer system110 toclient computer system102. The data transfer is accomplished securely, rapidly and reliably.

The forgoing discussion illustrates a “pull” of data from theserver computer system110 byclient server system102. As can be appreciated, in another aspect of the present invention, data transfer could be accomplished by “push” of data from theserver computer system110 wherein,server computer system110 would command a “listening”client computer system102 to receive data. Appropriate socket connections, threads, and objects would be created in accordance with the present invention, and data would be transfer over the computer network from theserver computer system110 to theclient computer system102.

FIGS. 4A and 4B illustrate alternative embodiments ofoutput channel139aand input channel139bofFIGS. 3A and 3B (the alternative embodiments are designatedoutput channel122 and input channel124). The operation and function of numbered elements inFIGS. 4A and 4B is the same as like numbered element inFIGS. 3A and 3B, respectively.

In the embodiment ofFIG. 4A, a streaming mechanism is used for compressingdata file120, encryptingdata file120, and then sending data file120 over the network to theclient102 or another remote device. Thus, instead of dividing the file into blocks as described above in conjunction withFIG. 3A, and treating each block as a small file, the file can instead be treated as a contiguous file.

The compressor channel object164 and theencryptor channel object156 may produce an output that is different in size from their input. Thestage output streams186 and188 write data to an output data block buffer until that block is full, or until there is no more data to write. If the current output data block is full, then output streams186 and188 indicate that the current block is ready to be read by the next transformation object, and asks block manager object154afor the next available output data block. By accumulating streaming parts of data file120 fromcompressor channel object146 andencryptor channel object156, better management of data block buffers by block manager object154 may be realized.

For example,compressor channel object146 may be capable of a 10:1 compression ratio. Instead of specifying an output data block buffer size equal to one-tenth the size of the input buffer, better use of the data block buffers may be achieved by accumulating ten parts of compressed data blocks in thestage output stream186 before writing to the output data block buffer, so that the output buffer is sized the same as the input buffer size.

InFIG. 4B, similar to the discussion above,stage output stream190 andstage output stream192 accumulate data for decryptor channel object164 anddecompressor channel object158, respectively. Additionally, stage input stream194 is provided ininput channel124 to receive the entire stream of compressed data blocks until the end of data file120 is reached, because decompressor channel object168 may require a complete compressed data file120 in order to operate.

FIG. 5 illustrates data flow through one embodiment of asession thread140ain a server computer system110 (FIG. 2B) in accordance with the present invention. It is appreciated that a similar data flow occurs through a session thread140bin a client computer system102 (FIG. 2B).Session threads140aand140bare executed under direction of the session manager objects138aand138b, respectively (refer toFIG. 2B).

Referring toFIG. 5, asession thread140ais created for eachincoming request134. Requests can include commands such as status commands, commands to send one or more files (e.g., data file120) to one or more remote devices (e.g., client102), and commands to receive one or more files from a remote device. In one embodiment, the commands inrequest134 use the Extensible Markup Language (XML).

In this embodiment,protocol174 validatesincoming request134 and directs it toXML command translator173.Protocol174 is used to send and receive commands in an encrypted format, and is used for authentication.Protocol174 is used bychannel factory182 to open a communication socket for a channel (e.g.,input channels139bor124, oroutput channels139aor122 ofFIGS. 3B,4B,3A and4A, respectively).

Continuing with reference toFIG. 5, XMLcommand translator object173 parses theincoming request134, generates optimized low-level internal tasks, and inserts the tasks with associated parameters into task table176. Each parameter is passed to all of the tasks that require it using the “declare” task (mentioned below), so that when a parameter is declared, it is shared at multiple locations and in multiple tasks where it is needed.

Incoming request134 is translated into low-level incoming tasks because the number of these tasks can be limited, while the number of XML commands inincoming request134 could be large. However, the XML commands can be translated into low-level internal tasks to facilitate implementation and to make the implementation more extensible. Also, use of low-level internal tasks instead of XML commands facilitates parallel processing and avoids the need for redundant processing steps.

The low-level internal tasks include but are not limited to the following:

Connect: to establish a connection between two devices (e.g.,server110 and client102), to create and initializeprotocol174, and to pass an initialization command to session manager object138a;

Create Channel: to create and initialize a channel;

Run Channel: to perform the file transfer;

External Execute: to execute an external command on the local device;

Get Session Status: to get a status report on one or more sessions;

Stop Session: to stop a session;

Declare: to insert a row in a separate variable vector maintained by task executor178 (the variable vector provides a mechanism to share objects across multiple tasks);

Wait: to wait until a channel has finished transferring a file or files;

Terminate: to terminate a session;

Create Account: to create a new account;

Edit Account: to edit an existing account; and

Remove Account: to remove an existing account.

XML command translator173 places these tasks into task table176 for execution bytask executor178. Task table176 is a memory object and comprises a hash table of the tasks.

Task executor178 executes the tasks from task table176 in a given order.Task executor178 uses an API of session manager object138a(Figure2B) to execute the tasks.Task executor178 executes in a loop through task table176 until terminate command is found. OnceXML command translator173 creates a task table176 for a command141,task executor178 takes control of the session thread and starts executing the tasks in the session thread.Task executor178 also updates session statistics for the recovery option described below in conjunction withFIGS. 6A,6B, and6C.

With reference toFIG. 5, a channel object (e.g.,channel object139a, and alsochannel object122 ofFIG. 4A) is generated and initialized by achannel factory182. In the present embodiment,channel factory182 initializeschannel object139awith input and/or output streams.

Continuing with reference toFIG. 5,channel object139arepresents the set of data transformation objects needed for sending and receiving a file (e.g., data file120). In the present embodiment, the set of data transformation objects includes the reader, compressor, encryptor, decompressor, decryptor, and writer objects described in conjunction withFIGS. 3A–3B and4A–4B.Protocol174 direct executed tasks to aremote session184.

FIGS. 6A,6B and6C illustrate another aspect of the present invention relating to data transfer recovery after a temporary loss and/or failure of a network connection. Because large amounts of data are being transferred in accordance with the present invention, recovery is an important consideration. In the present embodiment, two recovery modes are considered: automatic network connection recovery, and manual session recovery.

Automatic network connection recovery means that both the input channel139band theoutput channel139a(FIG. 2B) are running but the network connection is lost or has failed. In this case, the network connection can be recovered automatically.

When a network connection fails, both channels get notification via an “exception.” When a channel receives an exception, it calls the API for its respective session manager (e.g., session manager object138aor138bofFIG. 2B) to restore the connection. The API returns a new session thread for the connection if the connection can be restored, or a NULL if the connection cannot be restored.

Referring toFIGS. 6A,6B and6C, there are two session threads involved with the recovery process. InFIG. 6A, session manager object138a(FIG. 2B) creates session1 (e.g.,session thread140a) onserver computer system110 and initiates the connection withclient computer system102. Session manager object138b(FIG. 2B) receives the request to start a session and creates session1 (e.g., session thread140b) onclient computer system102.

InFIG. 6B, the connection is lost. InFIG. 6C, once the connection is lost and the initiating session manager (session manager object138aon server110) gets called to recover the session, session manager object138asends a request to client102 (session manager object138b) to restore the network connection forsession1. When client102 (specifically, session manager object138b) receives the request fromserver110, it spawns a second session thread (session2, e.g., session thread140c). Session2 passes the connection to thewaiting session1 onclient102. Once the connection is restored, both channels (output channel139aand input channel139b) continue to transfer data from the point at which they stopped.

Information about a session is stored in session manager objects138aonserver110. Session manager138astores all session parameters and the count of bytes of data written insession1 before the failure. Accordingly, reading can start reading from the correct byte offset, and writing can proceed by appending those data blocks created but not yet sent.

In the recovery mode, task table176 (FIG. 5) contains the same tasks but, when executed, some of the tasks may be skipped depending on the results of previous task execution and settings.

Manual session recovery is used when either of the channels fails. Recovery typically does not occur automatically because it is desirable to first determine and resolve the cause of the failure.

FIG. 7A is a flowchart of the server side steps in aprocess700 for transferring data over a network100 (FIG. 1A) in accordance with one embodiment of the present invention. In this embodiment,process700 is implemented by a server computer system110 (FIG. 1A), exemplified by computer system1090 (FIG. 1B), as computer-readable instructions stored in a memory unit (e.g.,ROM1003,RAM1002 ordata storage device1004 ofFIG. 1B) and executed by a processor (e.g.,processor1001 ofFIG. 1B). However, it is appreciated that some aspects ofprocess700 may be implemented onserver computer system110 with other aspects of theprocess700 performed on client computer system102 (FIG. 1A).

Instep702 ofFIG. 7A,server110 receives a request134 (FIG. 2B) for a data file residing in a mass storage unit on the server (e.g., data file120 residing inmass storage device112 ofFIG. 1A). In one embodiment, a listener object130a(FIG. 2B) is listening for such a request. In the present embodiment, therequest134 includes commands; refer toFIG. 5. In one embodiment, request134 uses XML.

Instep704 ofFIG. 7A, therequest134 is authenticated to make sure that the request is from an authorized user. In one embodiment, protocol174 (FIG. 5) validatesincoming request134.

Instep706 ofFIG. 7A, a session thread (e.g.,session thread140a) is spawned in response to the user request134 (FIG. 2B). In one embodiment, listener object130a(FIG. 2B) calls session manager object138a(FIG. 2B), and session manager object138acreatessession thread140aand assigns a unique ID to it.Session thread140areads the command(s) contained in theuser request134, and translates therequest134 into a set of low-level incoming tasks that are to be executed forsession thread140a(refer toFIG. 5). Session manager138aalso sends a message toclient102 directing it to spawn a session thread (e.g., session thread140b). Achannel object139ais also generated, providing the set of data transformation objects needed for sending and receivingdata file120; refer toFIGS. 3A and 4A.

Instep708 ofFIG. 7A, the data in data file120 are read from server mass storage device112 (FIG. 1A) and compressed as described in conjunction with eitherFIG. 3A orFIG. 4A.

Instep710 ofFIG. 7A, the compressed data are encrypted as described in conjunction withFIG. 3A orFIG. 4A.

Instep712 ofFIG. 7A, the data are sent to the requesting device (e.g., toclient102 over network bus101 innetwork100 ofFIG. 1B), and complete and accurate data transfer are verified.

Steps708,710 and712 are performed in parallel for different parts of the data file120.

FIG. 7B is a flowchart of the client side steps in aprocess750 for transferring analytical data over a network in accordance with one embodiment of the present invention. In this embodiment,process750 is implemented by client computer system102 (FIG. 1A), exemplified by computer system1090 (FIG. 1B), as computer-readable instructions stored in a memory unit (e.g.,ROM1003,RAM1002 ordata storage device1004 ofFIG. 1B) and executed by a processor (e.g.,processor1001 ofFIG. 1B). However, it is appreciated that some aspects ofprocess750 may be implemented onclient computer system102 with other aspects of theprocess750 performed onserver computer system110.

Instep752 ofFIG. 7B, a requesting device (e.g.,client102 ofFIG. 1A) issues a request134 (FIG. 2B) to a server (e.g.,server110 ofFIG. 1A) for a data file120 (FIG. 1A).

Instep754 ofFIG. 7B,client102 receives from server110 (specifically, from session manager object138aofFIG. 2B) amessage directing client102 to spawn a session thread (e.g., session thread140bofFIG. 2B).

Instep756 ofFIG. 7B,client102 receives fromserver110 encrypted and compressed data blocks that represent data file120; refer toFIG. 3B orFIG. 4B.

Instep758 ofFIG. 7B, the data are decrypted as described byFIG. 3B orFIG. 4B.

Instep760 ofFIG. 7B, the data are decompressed as described byFIG. 3B orFIG. 4B.

Steps756,758 and760 are performed in parallel for different parts of the data file120.

In summary, the present invention provides a reliable, secure, authenticated, verifiable, and rapid system and method for the transmission of huge amounts of data over a network, such as the data used in an analytic application (e.g., operational data, and transformed data in a data warehouse/data mart).

The foregoing descriptions of specific embodiments of the present invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the Claims appended hereto and their equivalents.

Claims (32)

What is claimed is:

1. In a multithreaded analytic application executed by a source computer system and capable of concurrent execution of multiple session threads, a method for transferring data, the method comprising:

receiving an incoming request for analytic data resident in a mass storage unit on the source computer system;

authenticating the incoming request;

spawning a session thread that reads and parses a command received via the incoming request, the command for sending the data to a second computer system; and

concurrently executing a plurality of data transformation threads within the session thread, comprising

a reader thread that reads data and writes at least a part of the data to a first data block buffer;

a compressor thread that compresses the part of the data in the first data block buffer into a compressed data block and writes the compressed data block to a second data block buffer;

an encryptor thread that encrypts the compressed data block in the second data block buffer into an encrypted and compressed data block and writes the encrypted and compressed data block to a third data block buffer; and

a writer thread that reads the encrypted and compressed data block in the third data block buffer and sends the encrypted and compressed data block to the second computer;

restoring a connection with the second computer system when an ongoing connection is lost; and

resuming transfer of data to the second computer system at the point in the data where the ongoing connection was lost.

2. The method ofclaim 1 further comprising:

verifying that data transfer to the second computer system is complete.

3. The method ofclaim 1 further comprising:

verifying that data transfer to the second computer system is without error.

4. The method ofclaim 1 wherein the source computer system and the second computer system are networked via the Internet.

5. The method ofclaim 1 wherein the data comprises data processed by an analytic application.

6. The method ofclaim 1 wherein the incoming request uses Extensible Markup Language (XML).

7. The method ofclaim 1 wherein spawning a session thread further comprises:

translating the command into a plurality of tasks;

storing the tasks in a task table in a given order; and

executing the tasks in order until a task ending the session thread is found.

8. The method ofclaim 1 wherein the first data block buffer and the second data block buffer are substantially equal in size and wherein enough compressed data blocks are accumulated to fill the second data block buffer before the compressor thread writes to a second data block buffer.

9. The method ofclaim 1 wherein the second data block buffer and the third data block buffer are substantially equal in size and wherein enough encrypted and compressed data blocks are accumulated to fill the third data block buffer before the encryptor thread writes to the third data block buffer.

10. In a first multithreaded analytic application executed by a target computer system and capable of concurrent execution of multiple session threads, a method for receiving data transferred from a source computer, the method comprising:

issuing a request for data to the source computer system on which the data resides, the source computer system executing a second multithreaded analytic application;

spawning a session thread in response to a message from the source computer system;

receiving from the source computer system at least one encrypted and compressed data block of the data; and

concurrently executing a plurality of data transformation threads within the session thread, comprising

a reader thread for writing the encrypted and compressed data block to a first data block buffer;

a decryptor thread for decrypting the encrypted and compressed data block into a compressed data block and writing the compressed data block to a second data block buffer; and

a decompressor thread for decompressing the compressed data block in the second data block buffer and writing a resultant data block to a third data block buffer;

restoring a connection with the source computer system when an ongoing connection is lost; and

resuming transfer of data from the source computer system at the point in the data where the ongoing connection was lost.

11. The method ofclaim 10 further comprising:

verifying that data transfer from the source computer system was complete.

12. The method ofclaim 10 further comprising:

verifying that data transfer from the source computer system was without error.

13. The method ofclaim 10 wherein the target computer system and the source computer system are networked via the Internet.

14. The method ofclaim 10 wherein the data comprises data processed by an analytic application.

15. The method ofclaim 10 wherein a plurality of encrypted and compressed data blocks accumulate before the decryptor thread executes.

16. The method ofclaim 10 wherein a plurality of compressed data blocks accumulate before the decompressor thread executes.

17. A source computer system comprising:

a bus;

a memory unit coupled to the bus; a multithreaded analytic application stored in the memory unit, and comprising:

a listener object for receiving an incoming request for data resident in a mass storage unit on the source computer system;

protocol for authenticating the incoming request;

a session manager object for spawning a session thread that reads and parses a command received via the incoming request, the command for sending the data to a second computer system;

a reader channel object for reading data and writing at least a part of the data to a first data block buffer;

a compressor channel object for compressing the part of the data in the first data block buffer into a compressed data block and writing the compressed data block to a second data block buffer;

an encryptor channel object for encrypting the compressed data block in the second data block buffer into an encrypted and compressed data block and writing the encrypted and compressed data block to a third data block buffer; and

a writer channel object for reading the encrypted and compressed data block in the third data block buffer and sending the encrypted and compressed data block to the second computer, wherein the application executes the reader channel object, the compressor channel object, the encryptor channel object, and the writer channel object concurrently; and

a processor coupled to the bus, the processor configured for executing the multithreaded analytic application, wherein the processor is further configured for:

restoring a connection with the second computer system when an ongoing connection is lost; and

resuming transfer of data to the second computer system at the point in the data where the ongoing connection was lost.

18. The source computer system ofclaim 17 wherein the processor is further configured for verifying that data transfer to the second computer system is complete.

19. The source computer system ofclaim 17 wherein the processor is further configured for verifying that data transfer to the second computer system is without error.

20. The source computer system ofclaim 17 wherein the source computer system and the second computer system are networked via the Internet.

21. The source computer system ofclaim 17 wherein the data comprises data processed by an analytic application.

22. The source computer system ofclaim 17 wherein the incoming request uses Extensible Markup Language (XML).

23. The source computer system ofclaim 17 wherein the session manager object is further configured for:

translating the command into a plurality of tasks;

storing the tasks in a task table in a given order; and

executing the tasks in order until a task ending the session thread is found.

24. The source computer system ofclaim 17 wherein the first data block buffer and the second data block buffer are substantially equal in size and wherein the compressor channel object is further configured for:

accumulating compressed data blocks before data are written to the second data block buffer, wherein enough compressed data blocks are accumulated to fill the second data block buffer.

25. The source computer system ofclaim 17 wherein the second data block buffer and the third data block buffer are substantially equal in size and wherein the encryptor channel object is further configured for:

accumulating encrypted and compressed data blocks before data are written to the third data block buffer, wherein enough encrypted and compressed data blocks are accumulated to fill the third data block buffer.

26. A target computer system comprising:

a bus;

a memory unit coupled to the bus;

a multithreaded analytic application stored in the memory unit, and comprising:

a first session thread for issuing a request for data to a source computer system on which the data resides;

a session manager object for spawning a session thread in response to a message from the source computer system;

a listener object for receiving from the source computer system at least one encrypted and compressed data;

a reader channel object for reading data and writing at least part of the encrypted and compressed data to a first data block buffer;

a decryptor channel object for decrypting the encrypted and compressed data block into a compressed data block and writing the compressed data block to a second data block buffer; and

a decompressor channel object for decompressing the compressed data block in said the second data block buffer and writing a resultant data block to a third data block buffer, wherein the application executes the reader channel object, the decryptor channel object, and the decompressor channel object concurrently; and

a processor coupled to the bus, the processor configured for executing the multithreaded analytic application, wherein the processor is further configured for:

restoring a connection with the source computer system when an ongoing connection is lost; and

resuming transfer of data from the source computer system at the point in the data where the ongoing connection was lost.

27. The target computer system ofclaim 26 wherein the processor is further configured for verifying that data transfer from the source computer system was complete.

28. The target computer system ofclaim 26 wherein the processor is further configured for verifying that data transfer from the source computer system was without error.

29. The target computer system ofclaim 26 wherein the target computer system and the source computer system are networked via the Internet.

30. The target computer system ofclaim 26 wherein the data comprises data processed by an analytic application.

31. The target computer system ofclaim 26 wherein the decryptor channel object is further configured for accumulating encrypted and compressed data blocks before decrypting the encrypted and compressed data blocks.

32. The target computer system ofclaim 26 wherein the decompressor channel object is further configured for accumulating compressed data blocks before decompressing the compressed data blocks.

Images