US7155380B2 - System and method for designing a logical model of a distributed computer system and deploying physical resources according to the logical model - Google Patents

Abstract

A system facilitates the design and implementation of large-scale applications, such as Internet Services and Websites, for distributed computer systems, such as server data centers, Internet data centers (IDCs), Web farms, and the like. The system has a modeling system and a deployment system. The modeling system permits developers to architect the hardware and software used to implement the applications in an abstract manner. The modeling system defines a set of components used to describe the functionality of an application. The model components are arranged and interconnected to form a scale-independent logical model of the application. Once a logical model is created, the deployment system uses the logical model to automatically deploy various computer/software resources to implement the application. The deployment system converts each of the model components into one or more instances that correspond to physical resources.

Description

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 09/696,707, filed on Oct. 24, 2000 now U.S. Pat. No. 6,907,395, entitled “System and Method for Designing a Logical Model of a Distributed Computer System and Deploying Physical Resources According to the Logical Model” and naming Robert Welland, Galen Hunt, Aamer Hydrie, Steven Levi, Jakob Rehof and Bassam Tabbara as inventors, the disclosure of which is hereby incorporated herein by reference.

TECHNICAL FIELD

This invention relates to distributed computer systems, such as Internet-based Services or Websites. More particularly, this invention pertains to a way to convert a logical, scale-independent model of an application for a distributed computer system to an actual physical configuration.

BACKGROUND

It is no secret that Internet usage has exploded over the past few years and continues to grow rapidly. People have become very comfortable with many services offered on the World Wide Web (or simply “Web”), such as electronic mail, online shopping, gathering news and information, listening to music, viewing video clips, looking for jobs, and so forth. To keep pace with the growing demand for Internet-based services, there has been tremendous growth in the computer systems dedicated to hosting Websites, providing backend services for those sites, and storing data associated with the sites.

One type of distributed computer system is an Internet data center (IDC), which is a specifically designed complex that houses many computers for hosting Internet-based services. IDCs, which also go by the names “Webfarms” and “server farms”, typically house hundreds to thousands of computers in climate-controlled, physically secure buildings. These computers are interconnected to run one or more programs supporting one or more Internet services or Websites. IDCs provide reliable Internet access, reliable power supplies, and a secure operating environment.

FIG. 1 shows anInternet data center100. It hasmany server computers102 arranged in a specially constructed room. The computers are general-purpose computers, typically configured as servers. An Internet data center may be constructed to house a single site for a single entity (e.g., a data center for Yahoo! or MSN), or to accommodate multiple sites for multiple entities (e.g., an Exodus center that host sites for multiple companies).

The IDC100 is illustrated with three entities that share the computer resources: entity A, entity B, and entity C. These entities represent various companies that want a presence on the Web. The IDC100 has a pool ofadditional computers104 that may be used by the entities at times of heavy traffic. For example, an entity engaged in online retailing may experience significantly more demand during the Christmas season. The additional computers give the IDC flexibility to meet this demand.

While there are often many computers, an Internet service or Website may only run a few programs. For instance, one Website may have 2000–3000 computers that run only 10–20 distinct software components. Computers may be added daily to provide scalability as the Website receives increasingly more visitors, but the underlying programs change less frequently. Rather, there are simply more computers running the same software in parallel to accommodate the increased volume of visitors.

Today, there is no conventional way to architect Internet Services in a way that abstracts the functionality of the Service from the underlying physical deployment. Little thought has gone into how to describe a complete Internet Service in any manner, let alone a scale-invariant manner. At best, Internet Service operators might draft a document that essentially shows each and every computer, software program, storage device, communication link, and operational relationship in the Website as of a specific time and date. The downside with such physical schematics is, of course, that the document is always out of date, must be updated as the Service grows in physical resources and hence, it is of limited usefulness as a management tool. Furthermore, while a human may understand such a document, it holds no meaning to a computer.

Moreover, managing the physical resources of the distributed computer system for an Internet Service is difficult today. Decisions such as when to add (or remove) computers to carry out portions of the Internet Service are made by human operators. Often times, these decisions are made based on the operators' experience in running the Internet Service. Unfortunately, with the rapid growth of services, there is a shortage of qualified operators who can make real-time decisions affecting the operation of a Website. Accordingly, it would be beneficial if some of the managerial aspects of running a Internet service could be automated.

SUMMARY

A system facilitates the design and implementation of large-scale distributed computer applications, such as Internet Services and Websites. The applications are implemented as software distributed over many interconnected computer nodes, such as server data centers, Internet data centers (IDCs), Web farms, and the like.

The system has a modeling system and a deployment system. The modeling system permits developers to architect the hardware and software used to implement the applications in an abstract manner. The modeling system defines a set of components used to describe the functionality of an application in a logical, scale-independent manner. In the described implementation, the modeling system defines several model components: a module, a port, and a wire. The model also admits an unlimited set of model extensions including, but not limited to stores, event sources, event sinks, and event wires.

The module is the basic functional unit and represents a container of behavior that may be implemented by one or more computers running one or more software programs. For instance, in the context of a Website, one module might represent a front end that renders HTML pages, another module might represent a login database, and another module might represent a mailbox program. A port is a service access point for the module. All communications into and out of the module go through a port. A wire is the logical binding that defines an allowed communication route between two ports.

While the model consists of the three basic components described above (namely modules, ports, and wires), the model can be augmented with numerous extensions, specializations of the basic components. For example, a store is a basic unit of storage and a specialization of the module. A store represents a logical amount of storage, which may be implemented by any number of physical disks or other storage media. Like the module, the store represents behavior, in this case, the ability to save and retrieve data. Also like the module, the store can communicate with other modules and stores through ports and wires. A store differs from a module in that it is labeled with additional attributes such as the amount of storage required, required access speed, or a minimum number of outstanding storage requests. The store extends the model by adding a specialized type of module with additional semantic information.

The model can be further augmented with ports extensions. For example, an event source and an event sink are used for discrete semantic messaging between modules and module extensions, such as stores. Event sinks are specialized ports in that they are communication access points between model components, but with additional semantics, namely the specific events.

The model can also be augmented with wires extensions. For example, an event wire is a logical connection between event sources and event sinks, and carries event messages used to inform modules and implement policy. While most wire extensions allow communication at run time, it is possible for some wire extensions to transfer data only at compile or initialization time.

The model components are arranged and interconnected to form a scale-independent model of the application. Each component specifies some functionality of the application.

Once a logical model is created, the deployment system uses the logical model to automatically deploy various computer/software resources to implement the application. The deployment system converts each of the model components into one or more instances that correspond to physical resources. As one example, the resources correspond to computer nodes of a distributed computer system that are loaded with specific types of software to implement the function represented by the model components. The deployment system initially installs the application and then dynamically and automatically modifies the resources used to implement the application in an ongoing basis as the operating parameters of the application change.

In one implementation, the deployment system includes a service running state to store the logical model and track instances of the model components as they are created (or destroyed). A resource manager tracks the computer nodes available for allocation and tracks the nodes as they are allocated to correlate the nodes with the instances. The deployment system further includes a loader to load software onto newly allocated computer nodes to implement the logical functions represented by the model components.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a conventional Internet data center (IDC).

FIG. 2 illustrates a set of model components that form the building blocks for modeling an Internet Service, along with the associated schema.

FIG. 3 illustrates a database application for an Internet Service that is modeled in terms of the components.

FIG. 4 illustrates an Internet-based email Internet Service.

FIG. 5 is a block diagram of a computer that may be used to implement the modeling software for modeling an Internet Service.

FIG. 6 is a flow diagram of a process for modeling an Internet Service.

FIG. 7 is a block diagram of a deployment system that converts a logical model to a fully functioning physical implementation

FIG. 8 illustrates a translation of the logical model into real-world instances.

FIG. 9 illustrates exemplary data records of an instance database used to store the real-world instances.

FIG. 10 is a flow diagram of a process for deploying resources for the application based on the logical model.

DETAILED DESCRIPTION

A design system for designing applications of distributed computer systems includes a modeling system and a deployment system. The modeling system permits developers of applications for distributed computer systems (e.g., server data centers, Internet data centers (IDCs), Web farms, and the like) to architect the hardware and software in an abstract manner. The modeling system defines a set of components used to describe the functionality of an application in a logical, scale-independent manner. An “application” within this context refers to an entire service hosted on the distributed computers. For instance, an Internet data center may host a Website for an online retailer, where the application entails the entire software and hardware configuration that implements the online retailer's Internet presence. The application might include, for example, a front end to handle client requests, an order processing system, a billing system, an inventory system, and a database system.

The model components are arranged and interconnected to form a scale-independent model of the application. Each component specifies some functionality of the application. The model can then be used to construct a scalable physical blueprint in terms of which machines run which pieces of software to form the application.

The deployment system uses the logical model to deploy various computer/software resources in real-time as the applications need them. The deployment system converts each of the model components into one or more instances that correspond to physical resources. The deployment system tracks the instances and all available resources. The deployment system decides when resources should be added (or removed) and monitors the current state of implementation. The deployment system installs the application and then dynamically and automatically modifies the resources used to implement the application in an ongoing basis as the operating parameters of the application change.

The design system is described in the context of Internet Services and Websites, such as might be deployed in Internet data centers, because modeling Internet Services represents one suitable use of the system. However, the design system may be implemented to model other large size and scalable applications for computer systems. Accordingly, the design system can be implemented in a wide variety of ways, including Internet-based implementations and non-Internet-based implementations.

Model Components and Schema

The modeling system defines several model components that form the building blocks of a logical, scale-independent application: a module, a port, and a wire. It also defines a set of model extensions including, but not limited to: a store, an event source, an event sink, and an event wire. In a design tool, the components are represented pictorially as graphical elements or symbols that may be arranged and interconnected to create scale-independent models of Website applications. The graphical elements have an associated schema that dictates how the functional operations being represented by the graphical elements are to be specified.

FIG. 2 illustrates a set ofmodel components200 that form the building blocks of logical, scale-independent Internet Services. The components include a module, as represented by modules202(A)–202(C),ports206,wires208, and extensions such as astore204,event sources210, event sinks212, andevent wires214. Thecomponents200 are arranged in a no particular manner other than to foster discussion of their individual traits.

Amodule202 represents a basic unit of functionality for the Internet Service. It is a logical entity that represents some portion of the application as might be deployed at the IDC, but it does not necessarily have a physical manifestation. The module often corresponds to a software program that handles a logical set of tasks for the Service. For instance, one module might represent a front end for a Website, another module might represent a login database, and another module might represent an electronic mail program.

Eachmodule202 is a container of behavior. A simple module is indivisible and has associated a unique identifier. Modules can be nested into a hierarchy of modules to form more complex behaviors. In a module hierarchy, the leaf modules are simple modules, and the non-leaf modules are compound modules.

Eachmodule202 defines a unit of scaling. While one module logically represents a functional operation of the Service, the module may be deployed to any number of computers when actually implemented. In this way, the module is scale-independent, allowing the number of underlying computers used to implement the module to change at over time. When converted to a physical implementation, “module instances” are created from the modules. The module instances are assigned a unique identifier and maintain ancestral data regarding which module created them. The module instances of simple modules are often called “engines”, which are software programs that run on an individual computer.

Extensions to the model are additional components that specialize the role, behavior, and possibly graphical representation of the base components. Exemplary extensions include, but are not limited to,store204,event source210,event sink212, andevent wire214.

Astore204 is the most basic unit of storage. It represents a logical storage partition, which may be implemented by any number of physical disks or other storage media.

Aport206 is a service access point (SAP) for amodule202 orstore204. All service-related communications into and out of a module go through aport206. Eachport206 has a “type”, which is a set of attributes describing format, semantics, protocol, and so forth. At runtime, the port represents a set of physical ports associated with the instantiated engines of the modules. Note that a given module might have any number of ports representing different services or functionality provided by the module.

Awire208 is the logical binding that defines an allowable communication route between twoports206. Eachwire208 can be type-checked (i.e., with respect to protocols, roles) and defines protocol configuration constraints (e.g., HTTP requires TCP, TCP requires IP, etc.).

Event sources210 and event sinks212 are used for discrete semantic messaging between modules and module extensions, such as stores. Anevent wire214 is a logical connection between sources and sinks, and carries event messages used to inform modules or module extensions and implement policy (e.g., scaling, fail-over, monitoring, application processes, etc.).

The event sources210 and event sinks212, together with theports206, collectively form interfaces for communications to and from themodules202 and module extensions, such asstores204. The event sources and sinks may be implemented as ports that are configured for message handling.

Themodel components200 are depicted as graphical icons or symbols that may be selected and interconnected using a modeling system (described below in more detail). In the illustrated example, themodules202 are depicted as blocks, thestore204 is depicted as a disk storage icon, and theports206 are depicted as spherical knobs projecting from the modules or module extensions, such as stores. Additionally, thewires208 are depicted as bold lines, theevent sources210 are depicted as triangles pointing away from the module or module extension, the event sinks212 are depicted as triangles pointing toward the module or module extension, and theevent wire214 is depicted as a dashed line.

The graphical icons have an associated schema that dictates how the functional operations being represented by the icons are to be specified. For instance, a module icon may have a predefined schema that specifies the hardware and software resources used to implement the functionality represented by the module. Thus, a module for a database function might have characteristics pertaining to the kind of database (e.g., relational), the data structure (e.g., tables, relationships), software (e.g., SQL), software version, and so forth.

FIG. 2 also illustrates the schema underlying the graphical elements as exemplary data structures associated with the model components. Module202(A) has an associatedstructure220 that contains various characteristics for the module, such as functionality, processing requirements, software, and so forth. Modules202(B) and202(C) have similar structures (not shown). Model extensions also have associated structures. Thestore204 has acorresponding structure222 that defines the requirements for storage. Thestore schema structure222 might include, for example, the kind of storage (e.g., disk), the storage format, and so on.

Eachport206 has a schema structure, as represented bystructure224, which dictates the port's type. Eachwire208 is also associated with a schema structure, such asstructure226, which outlines the protocols implemented by the connection. Similar schema structures may also be provide for event sources event sinks, and event wires.

Using the model components, a developer can logically describe and configure scale-independent Internet Service prior to physically laying them out in Internet data centers. The developer drafts a model using a user interface to select and interconnect the model components. Once constructed, the modeling software generates the Internet Service based on the depicted model and the underlying schema. The Service may subsequently be converted into a physical blueprint that details the computers and software needed to implement the Service for a specified number of clients.

The scale-invariant nature of the modeling system allows Internet Service developers to focus only on designing software for a specific functional task (e.g., front end, login database, email program, etc.). All external communications can then be expressed in terms of transmitting to and receiving from one or more associated ports. In this manner, the Service developers need not worry about how many machines will be used to run the module, or how other modules of the scale-independent Internet Service are being configured.

Exemplary Module and Application

FIG. 3 shows a fault-tolerant SQL (structure query language)database module300 to demonstrate how the model components may be organized and connected to represent a portion of an application. In this example, thedatabase module300 represents a SQL database that may be used independently or as a component in a larger application. TheSQL database module300 has a module interface composed of asingle port302 that implements the TDS (Tabular Data Stream) protocol.

TheSQL database module300 is a compound module made up of three simple modules: a fail-overpolicy module310, aprimary SQL module312, and asecondary SQL module314. The primary and secondary SQL modules represent dual programs that operate in parallel so that, in the event that theprimary module312 crashes, thesecondary module314 can assume the role without loss of service. Thedatabase module300 also has adata store316 that represents the memory storage for the SQL database module.

Theprimary SQL module312 has a module interface that includes afirst port320 for communicating with thecompound module port302 and asecond port322 for communicating with thestore316. Theprimary SQL module312 also has anevent source324 and anevent sink326 for handling event messages from the fail-overpolicy module310. Similarly, thesecondary SQL module314 has a module interface with afirst port330 for communicating with thecompound module port302, asecond port332 for communicating with thestore316, and anevent sink334 for receiving events from the fail-overpolicy module310. Awire336 interconnects the externalcompound module port302 with theports320 and330 of the primary and secondary SQL modules, respectively.

Thestore316 has aport340 to communicate with the primary312 and secondary314 SQL modules and an event sink342 to receive event messages from the fail-overpolicy module310. Awire344 interconnects thestore port340 with theports322 and332 of the primary and secondary SQL modules, respectively.

The fail-overpolicy module310 has a module interface that includes three event sources and one event sink. Anevent sink350 receives a “fail” event from theevent source324 of theprimary SQL module312 via anevent wire352 when the primary SQL module experiences some failure. In response to receiving a “fail” event, the fail-overpolicy module310 concurrently issues a first event to stop the failedprimary module312, another event to assign thesecondary module314 as the new owner of thestore316, and a third event to start thesecondary module314. The “stop” event is issued via anevent source354 over anevent wire356 to theevent sink326 of theprimary SQL module312. The “stop” event directs theprimary SQL module312 to halt operation.

The fail-overpolicy module310 issues an “assign owner” (AO) event from anevent source358, over the event wire360 to the event sink342 of thestore316. The assign owner event directs the storage mechanisms to switch to allowing access by thesecondary SQL module314, rather than theprimary SQL module312. The fail-overpolicy module310 also issues a “start” event fromevent source362 overevent wire364 to theevent sink334 of thesecondary module314. The start event directs the secondary SQL module to start operation in place of the primary SQL module.

TheSQL database module300 illustrates how the base model components and exemplary model extensions—modules, ports, wires, stores, event sources, event sinks, and event wires—may be arranged and interconnected to form a complex module. The developer specifies the characteristics associated with each component according to the prescribed schema. The complex module may in turn be added to other simple or complex modules to form other complex modules. Eventually, the largest complex module becomes the Internet Service, which may then be used to form a blueprint for deploying to the data center.

FIG. 4 shows asimplified application400 for an online retailer. Theapplication400 includes afront end module402, acatalog module404, anorder processing module406, and afulfillment module408. Theapplication400 also includes acustomer database410 and the fault-tolerantSQL database module300. Notice that theSQL database module300 is the same as that shown inFIG. 3 to illustrate how complex modules can be nested into even greater complex modules to form an application.

Thefront end module402 handles requests from clients who wish to shop with the online retailer. Thefront end module402 has aport420 that accommodates communications with external clients using the TCP/IP protocol over the Internet. Thefront end module402 also has anorder port422 to define a communication exchange with theorder processing module406 and acatalog port424 for communication flow to thecatalog module404. Theports422 and424 may be configured according to any of a variety of types, which support any of a number of protocols including SOAP, TCP, or UDP. Anevent sink426 is also provided to receive a “new product” message from thecatalog module404 when a new product has been added to the catalog.

Thecatalog module404 provides catalog information that may be served by the front end to the requesting clients. Thecatalog module404 has afront end port430 connected via awire432 to thecatalog port424 of thefront end module402. Thefront end port430 has a type that matches thecatalog port424. Thecatalog module404 also has anevent source434 for communicating the “new product” messages overwire436 to theevent sink426 of thefront end module402.

ASQL port438 interfaces thecatalog module404 with theSQL database module300. TheSQL port438 has a type that utilizes the TDS protocol for the communication exchange with theexternal port302 of theSQL database300.

Theorder processing module406 has afront end port440 to define a communication interface with thefront end module402 via awire442. Theorder processing module406 also has afulfillment port444 to facilitate communication with thefulfillment module408 overwire446 and adatabase port448 to facilitate communication with thecustomer database410 viawire450.

Anevent source452 is provided at theorder processing module406 to pass “order complete” events to thefulfillment module408 viawire454. These events inform thefulfillment module408 that an order is complete and ready to be filled. Asecond event source456 passes “new account” events to thecustomer database410 viawire458 whenever a new customer orders a product.

Thefulfillment module408 has anorder port460 to provide access to thewire446 to theorder processing module406 and adatabase port462 to interface with thecustomer database410. Thefulfillment module408 also has anevent sink464 to receive the “order complete” events from theorder processing module406.

Thecustomer database410 has anorder port470 to provide access towire450 and afulfillment port472 to facilitate communication with thefulfillment module408 viawire474. Thecustomer database410 further has anevent sink476 to receive the “new account” events from theorder processing module406.

The modeling approach illustrated inFIGS. 3 and 4 is tremendously beneficial because it allows developers and IDC operators to view the entire Internet Service in terms of functional pieces independent of deployment scale. The onlineretailer Internet Service400, for example, requires a front end unit, a catalog unit, an order processing unit, and a fulfillment unit regardless of whether the retailer is handling 100 hits a day or 10 million hits per day.

The scale-independent nature frees the developer to focus on his/her little piece of the Service. For instance, a developer assigned the task of building thefront end module402 need only be concerned with writing software code to facilitate response/reply exchanges. Any communication to and from the module is defined in terms of order-related data being passed to theorder processing module406 via theorder port422 and product data being received from thecatalog module404 via thecatalog port424. The developer defines the data flow to and from theorder port422 and thecatalog port424 according to their respective associated protocol types.

TheInternet Service400 can then be used to construct a computer system that hosts the online retailer. Initially, the online retailer may not receive very much traffic, especially if launched away from the Christmas season. So, perhaps thefront end module402 deploys initially to only a few computers to handle the light traffic from the Internet. But, suppose that over time the site becomes more popular and the Christmas season is fast approaching. In this situation, the online retailer may authorize the IDC operator to add many more computers for the front end tasks. These computers are equipped with software and configured to accept HTTP requests for product information and to serve Web pages containing the product information. The computers are added (or removed) as needed, without altering the basic description of theInternet Service400.

Computer-Based Modeling System and Method

FIG. 5 shows anexemplary computer system500 that implements modeling software used to design Internet Services. The modeling computer may be implemented as one of the nodes in a Internet Service, or as a separate computer not included as one of the nodes. The modeling computer has aprocessor502, volatile memory504 (e.g., RAM), and non-volatile memory506 (e.g., ROM, Flash, hard disk, optical, RAID memory, etc.). Themodeling computer500 runs anoperating system510 andmodeling system512.

For purposes of illustration,operating system510 andmodeling system512 are illustrated as discrete blocks stored in thenon-volatile memory506, although it is recognized that such programs and components reside at various times in different storage components of thecomputer500 and are executed by theprocessor502. Generally, these software components are stored innon-volatile memory506 and from there, are loaded at least partially into the volatilemain memory504 for execution on theprocessor502.

Themodeling system512 includes a user interface514 (e.g., a graphical UI) that presents the pictorial icons of the model components516 (e.g., modules, ports, sources, sinks, etc.), acomponent schema database518, a logical-to-physical converter520, and an instance-trackingdatabase522. Themodeling system512 allows a developer to design an Internet Service by defining modules, ports, wires, and event message schemes. Theuser interface514 presents symbols of thecomponents516, such as the symbols shown inFIGS. 2–4, and permits the developer to arrange and interconnect them. TheUI514 may even support conventional UI techniques as drag-and-drop operations.

The symbols depicted on the screen represent anunderlying schema518 that is used to define the model. For instance, a block-like module symbol is associated with the characteristics of the functionality that the module is to represent in the Internet Service. Thus, the developer may define a database module that has characteristics pertaining to the kind of database (e.g., relational), the data structure (e.g., tables, relationships), software (e.g., SQL), software version, and so forth. Accordingly, by drafting the model on the UI, the developer is architecting the entire schema that will be used to design the scale-independent Internet Service.

Once the Internet Service is created, the logical-to-physical converter520 converts the Service to a physical blueprint that details the number of computers, software components, physical ports, and so forth. The converter takes various parameters—such as how many site visitors are expected, memory requirements, bandwidth requirements, processing capabilities, and the like—and scales the Internet Service according to theschema518 created by the developer. Theconverter520 specifies the number of computers needed to implement each module, the number of disks to accommodate the stores, and the types of communications protocols among the modules and stores. The identity of every component instance is recorded in an instance-trackingdatabase522. Instances in the instance-trackingdatabase522 include those for modules, port, wires, and instances of model extensions such as stores, event ports, and event wires.

In one embodiment, the developer writes management policy, which issues commands on theschema518 to create new instances of modules, port, and wires to deploy the Internet Service. Developers may choose to write management policy instead of using fully automatic logical-to-physical converter520 when they want finer control over the growth and management of the Internet Service. The management code issues commands using the namespace defined by theschema518, but the commands operate on individual module, port, and wire instances. The commands are still dispatched through theconverter520, which allocates nodes to the management policy. Whether operating automatically or driven by management policy code, theconverter520 records in the instance-trackingdatabase522 the individual instances of modules, port, and wires.

In this manner, the modeling system changes the development effort from a node-centric approach for architecting Internet Services to an application-centric approach. Within conventional node-centric methodology, the focus was on the computers and how they were laid out. The Internet Service was then loaded onto the nodes in an ad hoc manner. With the new application-centric approach, the focus is initially on the Internet Service itself. The physical nodes used to implement the Internet Service are derived in terms of the Service schema once it is specified. The instance-trackingdatabase522 gives both developers and operators information about how many instances of each module are running at any time and how the modules are connected using port instances and wires instances within the Service schema.

FIG. 6 shows a method for modeling a scale-independent Internet Service. Themethod600 may be implemented, for example, by themodeling system512 executing on themodeling computer500. In such an implementation, the method is implemented in software that, when executed oncomputer500, performs the operations illustrated as blocks inFIG. 6.

Atblock602, themodeling system512 allows the developer to define the modules and extensions, such as stores, that form the functional elements of the Internet Service. TheUI514 enables the developer to create modules and extensions, such as stores, and to define their characteristics as prescribed by a predetermined schema. This entry process begins to construct the logical building blocks of the Service.

Atblock604, themodeling system512 enables the developer to define the ports for the modules and module extensions, such as stores. The developer selects the type of ports. The modeling system ensures compatibility of ports that are connected to one another. Atblock606, the developer also defines other extensions, such as events that may be passed among modules and module extensions, such as stores. For example, the developer creates event sources and event sinks to accommodate the various events. Atblock608, the developer uses themodeling system512 to interconnect the ports with wires and the port extensions, such as event sources/sinks with wire extensions, such as event wires. By joining the various modules and module extensions, such as stores, the developer effectively forms a logical representation of the Internet Service.

At block610, themodeling system512 generates an Internet Service using the graphical representation constructed by the developer through its associated schema. Themodeling system512 generates the logical specifications associated with the graphical model, including the characteristics of the modules and module extensions, such as stores, as well as the types of the ports and port extensions, such as event sources/sinks. The Internet Service provides a complete logical representation of the Service that will eventually be implemented at the Internet data center. The Internet Service description may be stored on disk or some other form of computer-readable medium (block612).

Atblock614, themodeling system512 converts the Internet Service description to a physical blueprint that specifies the computers, the software run by each of the computers, and the interconnections among the computers. This physical blueprint may be used by the operator to install and manage the Internet Service.

Computer-Based Deployment System and Method

Once a logical model is created, an automatic computer-based deployment system uses the logical model to deploy various computer/software resources to implement the Internet Service. The deployment system converts each of the model components into one or more instances that correspond to physical resources, such as nodes of a distributed computer system that are loaded with specific types of software to implement the function represented by the model components. The deployment system initially installs an Internet Service in the physical resources according to the logical model. It then dynamically and automatically modifies the resources used to implement the Internet Service in an ongoing basis as the operating parameters of the application change.

In one embodiment, the deployment system installs an Internet Service under the direction of management policy code authored by the Service developers. When operated under the direction of management code, the deployment system creates instances as instructed by the management code. The management code assumes partial or complete responsibility for monitoring the Service and determining when instances should be created and destroyed.

FIG. 7 shows adeployment system700 that converts the logical model to a fully functioning physical implementation. Thedeployment system700 includesmanagement policy702, a core logical-to-physical converter704, and hardware/software resources706 that are all interconnected via a wireless and/or wire-based communication network708 (e.g., a LAN, a WAN, intranet, Internet, combinations thereof, etc.). In this example, the hardware/software resources are illustrated as computer nodes of a distributed computer system, as represented by computers706(1),706(2), . . . ,706(N). Themanagement policy702 andcore converter704 may be implemented on one or more computers, which may or may not be part of the nodes in the distributed computer system.

For purposes of discussion, thedeployment system700 is described in the context of an Internet Service that is executed at an Internet data center having an abundance of generic computer nodes. The nodes can be allocated to one or more Internet Services from a reserve pool of nodes, as illustrated inFIG. 1.

Themanagement policy702 implements one or more policies devised by the developer or operator of the Internet Service. The policies specify when instances derived from the logical model should be created, manipulated, and destroyed. The management policy monitors various events generated by the nodes and implements policy decisions regarding how to handle the events. By specifying when and which instances should be created (or destroyed), themanagement policy702 effectively dictates when hardware/software resources706 should be added (or removed) to support the changing demands of the Internet Service.

Thecore converter704 implements the policy decisions made by themanagement policy702. Theruntime converter704 has aservice running state710 that tracks all instances of the model components currently in existence. That is, theservice running state710 tracks the elements in the physical world with respect to the logical model. Theservice running state710 maintains a copy of the logical model, such asonline retailing model400 ofFIG. 4. The logical model is created by the modeling system described above with respect toFIGS. 5 and 6. The current instances are maintained in the instance-trackingdatabase522. The records in instance-trackingdatabase522 include such information as identity of the instance, the name of the logical component from which it is derived, the node on which it is running, the network addresses representing the ports of the modules and module extensions, such as stores, type of software loaded on the node, various protocols supported by the instance, and so forth. The instance-trackingdatabase522 tracks not only module instances, but also port instances, wire instances, and can also track instances of extensions such as stores, event ports, and event wires.

The instances are derived from the logical model. Themanagement policy702 articulates the number of instances of each model component used to implement the Internet Service at any given time. For example, suppose the Internet Service requires one hundred computers to effectively implement a front end that handles site traffic at 99.9% efficiency with each computer running at 70% utilization. Themanagement policy702 might further specify that more computers should be added if some policy threshold is met (e.g., efficiency rating drops below some threshold or computer utilization rises above some threshold) or removed if another threshold is met.

Aresource manager716 tracks all of the physical resources available to the Internet Service. These resources include computer nodes, storage, software, and so forth. Records identifying all of the resources are kept in theresource database718. For instance, there might be one record for each computer node, storage device, and software module in the Internet data center. The records contain such information as the identity of the allocated nodes, computing characteristics and capabilities, the application(s) to which they are allocated, the date and time of allocation, and so forth.

Theresource manager716 allocates the resources as needed or requested by the Internet Service according to the policy implemented by thepolicy manager702. The allocation depends upon the availability of resources at the time of request. Theresource manager716 may also recover resources that are no longer needed by the Internet Service and return the resources to the pool of available resources.

Upon allocation (or recovery) of a resource, theresource manager716 posts a record to theresource database718 reflecting which resource is allocated to (or recovered from) which Internet Service. As an example, when an Internet Service desires more nodes for the front end tasks, theresource manager716 allocates one or more free nodes from the pool of resources to the Internet service.

The core logical-to-physical converter704 manages the creation of new instances in the physical world from the model components specified in thelogical model400 through aloader722. Aloader722 carries out the configuration of newly allocated resources to the functions dictated by the new instances. In this manner, when another instance of a component is desired, the management policy720 communicates with theresource manager716 to allocate a node from the resource pool and with theloader722 to load the appropriate software programs onto the node.

Thenode loader732 performs the actual loading tasks specified by theloader722 in thecore converter704. It is the local code on an otherwise generic node to which theruntime loader722 may communicate when configuring the node.

FIG. 8 shows an example of a portion of thelogical model400 ofFIG. 4 being converted into actual instances. To illustrate the conversion, thefront end module402 and theorder processing module406 are extracted from the online retailer service application400 (FIG. 4). Awire442 interconnects the two modules by logically coupling theports422 and440.

Thefront end module402 in the logical model translates to one or more computer nodes in the physical world that runs software for handling client queries. These physical instances are represented diagrammatically by the rectangles with ovals. Here, based on a policy, thefront end module402 is converted into multiple front end instances800(1),800(2),800(3), . . . ,800(J), which each corresponds in one-to-one fashion with a computer node loaded with software used to implement the front end of the service. Theorder processing module406 translates to one or more computer nodes that run a program for processing client orders. InFIG. 8, theorder processing module406 is converted into plural order processing instances802(1),802(2), . . . ,802(K).

Theports422 and440 represent network ports with protocols and roles within protocols in the logical world. They translate to the physical world as a set of network addresses used to communicate with the software at the respective modules. For instance, the physical translation of alogical port422 might be IP Port804(n), using HTTP (hypertext transport protocol). InFIG. 8, one logical port in the logical model is converted to a port address for each instance of the module.Logical port422 converts into physical address ports804(1)–804(J) andlogical port440 converts into physical ports806(1)–806(K).

Thewire442 represents a set of ports with allowable communication connections. It translates to a physical mesh of all possible communication wires between the instances of each port. The maximum number of physical lines created from onewire442 is determined as the cross product of the number of instances of each port. InFIG. 8, thewire442 can convert to a physical connection between every port instance804(1)–804(J) of thefront end module402 and every port instance806(1)–806(K) of theorder processing module406. The number of actual physical connections created from thewire442 is determined by themanagement policy702.

FIG. 9 illustratesexemplary records900 in the instance-trackingdatabase522 that tracks the instances derived from thelogical model400. In this example, the database is a relational database that stores records in tables, and the records may be linked to one another via relationships. Here, there are three tables: a module table902, a port table904, and a wire table906. Each table holds one record for a corresponding instance of the logical model. Each record has a number of fields relating to the type of information that is being tracked for each instance.

The module table902 tracks instances of modules in the logical model. There is one record for each module instance. Thus, with respect to the front end module ofFIG. 8, there are “J” records in the module table902 corresponding to the “J” front end instances800(1)–800(J). Each record in the module table902 contains an instance ID to identify the individual instance, an identity of the module component from which the instance is derived, a node ID to identify the computer node to which the instance is associated, the type of software loaded on the node to implement the module functionality, a software ID, an identity of the various ports to the modules, and various protocols supported by the module instance. It is noted that other implementations may include additional fields or fewer fields than those illustrated.

The port table904 tracks instances of the port in the logical model, such asports422 and440 inFIG. 8. A record from this table includes such information as the port ID, the model component identity, a node ID, the network address represented by the port, the instance ID of the corresponding instance, the protocol used by the port, and an ID of the wire to which the port is connected. Again, more or fewer fields may be used in other implementations.

The wire table906 tracks instances of the wires in the logical model, such aswire442 inFIG. 8. A record in the wire table includes a wire ID, a model component identity, the protocol supported by the wire, and information to identify each of the ports on the wire, such as a node ID, a port ID, and an instance ID.

Notice that the three tables can be correlated with one another via various relationships. For example, the module table902 and the port table904 are related by various data fields, such as the port ID field and the instance ID field. The wire table906 correlates with the port table904 via wire ID and port ID and with the module table902 via instance ID. Notice also that the tables have a node ID field that provides a reference into the resource database by identifying which node the instance is associated with.

It is noted that the illustrated arrangement of the database is merely for discussion purposes. Many other table arrangements with more or fewer tables than illustrated may be used in other implementations.

FIG. 10 shows amethod1000 for deploying resources for an Internet Service based on the logical model. Themethod1000 is implemented by thedeployment system700 ofFIG. 7 and hence can be embodied as software that, when executed on one or more computers, performs the operations illustrated as blocks inFIG. 10.

Atblock1002, themanagement policy702 monitors various operating parameters and listens for events from theindividual nodes706 or the core logical-to-physical converter704. Themanagement policy702 evaluates the parameters and events against the policy backdrop to determine whether the current physical implementation is satisfactorily supporting the Internet Service, or whether a new instance of some component should be added (block1004). It is noted that the continuing process is described in the context of adding a new instance. However, the policy may alternatively dictate that one or more instances should be removed. Removal of instances is somewhat easier in that instances are deleted from the instance-trackingdatabase522 and the computer nodes are returned to the additional pool for reallocation.

Assuming that a new instance is desired (i.e., the “yes” branch from block1004), themanagement policy702 consults theservice running state710 to understand the current number and arrangement of instances (block1006). Themanagement policy702 can request various types of information of theservice running state710, such as:

How many instances of a given module?

What nodes are associated with a given model component?

Which ports are attached to a given wire?

What is the network address of a node?

What wire is attached to a port on a given component?

What components does a given component own?

Depending upon the event or operating condition, themanagement policy702 can request information on a particular module in thelogical model400. For example, assume that an event has been received from a front end node indicating that the utilization has risen to above 90%. In response, a policy specifying the addition of another instance at such utilization levels is triggered. Themanagement policy702 asks theservice running state710 how many instances of the front end module currently exist, and information regarding how to specify an instance for the front end module.

Atblock1008, themanagement policy702 calls theresource manager716 to request allocation of a new node (assuming one is available). Theresource manager716 examines the resources using data from theresource database718 and allocates a new node that is currently available from a pool of free nodes. Theresource manager716 records the allocation in theresource database718, identifying which node is allocated, what application it is being allocated to, the date and time that it is allocated, and so forth.

Atblock1010, themanagement policy702 calls theloader722 to install software onto the allocated node and configure the node to perform the functions represented by the logical module from which the instance is created. In response, theloader722 initializes the node by communicating with thenode loader732 of the new node via thenetwork708 to install the appropriate software; such an operation might install an image of an operating system (e.g., Windows NT server operating system from Microsoft Corporation). Theloader722 then loads the appropriate software and configures the node to perform the functions represented by the logical model component from which the instance is derived. For example, for an instance of an SQL module in thelogical model400, thenode loader732 loads SQL server software. Theloader722 registers the physical port addresses with theservice running state710, which records the new instance in the instance-trackingdatabase522.

Atblock1012, theservice running state710 is notified when the newly allocated and configured node is up and running. Theservice running state710 records a new instance of the logical model in the instance-trackingdatabase522. The record reflects an ID of the instance, the name of the logical component from which it is derived, the allocated node, the network addresses of the node, software type and IDs, various protocols supported by the instance, and so on.

Atblock1014, themanagement policy702 is notified by the core logical-to-physical converter704 that a new instance is up and running. The new instance should relieve the event or operating condition to bring operation back into compliance with the policy.

CONCLUSION

Although the description above uses language that is specific to structural features and/or methodological acts, it is to be understood that the invention defined in the appended claims is not limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the invention.

Claims (26)

1. An apparatus comprising:

service running state means configured to maintain a logical model of a service application to be implemented by software as instances derived from the logical model and distributed across a plurality of computer nodes, the logical model having multiple components representing logical functions of the application;

resource manager means configured to allocate computer nodes for the instances; and

loader means configured to load various software onto the computer nodes allocated by the resource manager, the software being executable on the computer nodes to implement the logical functions represented by the model components from which the instances are derived.

2. The apparatus ofclaim 1, wherein the service running state means is configured to track the instances.

3. The apparatus ofclaim 1, wherein the resource manager means is configured to track whether the computer nodes are allocated or unallocated.

4. The apparatus ofclaim 1, further comprising management policy means configured to monitor operation of the application and to specify when new instances of the model components are to be created or removed based on the operation of the application.

5. The apparatus ofclaim 4, wherein the management policy means is configured to listen to events generated by the computer nodes as a way to monitor operation of the application.

6. The apparatus ofclaim 1, further comprising node loader means resident at each of the computer nodes to install the software onto the computer nodes.

7. A process comprising:

maintaining a scale-independent logical model of a service application to be implemented by software distributed across a plurality of computer nodes, the logical model having multiple components representing logical functions of the application;

creating one or more instances of the model components according to a desired scale of the service application; and

allocating the computer nodes to associated instances of the model components, the computer nodes being configured to perform the logical functions represented by the components from which the instances are created.

8. The process ofclaim 7, further comprising tracking the instances.

9. The process ofclaim 7, further comprising tracking the allocated computer nodes.

10. The process ofclaim 7, further comprising facilitating policy that specifies when and what instance of a particular model component is created.

11. The process ofclaim 7, further comprising loading software onto the computer nodes following allocation, the software being executed on the computer nodes to perform the logical functions represented by the components from which the instances are created.

12. A process comprising:

maintaining a logical model of an application to be implemented by software distributed across a plurality of computer nodes, the logical model having multiple components representing logical functions of the application; and

converting the model components into one or more instances representative of physical resources used to implement the logical functions.

13. The process ofclaim 12, further comprising directing one or more computing devices to allocate the physical resources for each of the instances.

14. The process ofclaim 12, further comprising directing one or more computing devices to track the instances in a database.

15. An apparatus comprising:

means for logical modeling of an application for a distributed computer system, the logical modeling means having at least one module that represents a functional behavior of the application, at least one port that represents a communication access point for the module, and at least one wire that represents a logical connection between the port of the module and a port of another module;

a first structure to store module information pertaining to one or more module instances of the module that correspond to physical resources used to implement the functional behavior represented by the module;

a second structure to store port information pertaining to one or more port instances of the port; and

a third structure to store wire information pertaining to one or more wire instances of the wire.

16. The apparatus ofclaim 15, wherein the module information in the first structure is correlated with the port information in the second structure to associate certain ports with certain modules.

17. The apparatus ofclaim 15, wherein the port information in the second structure is correlated with the wire information in the third structure to associate certain wires with certain ports.

18. The apparatus ofclaim 15, wherein the module information includes data fields selected from a group of fields including an identity of the module instance, an identity of the module in the logical model from which the module instance is created, an identity of a physical computer node on which the module instance is instantiated, an identity of a port for the module, and a protocol supported by the module.

19. The apparatus ofclaim 15, wherein the port information includes data fields selected from a group of fields including an identity of the port instance, an identity of the port in the logical model from which the port instance is created, a network address of a physical computer node on which the port instance is instantiated, an identity of a module with which the port is the communication access point, and a protocol supported by the port.

20. The apparatus ofclaim 15, wherein the wire information includes data fields selected from a group of fields including an identity of the wire instance, an identity of the wire in the logical model from which the wire instance is created, an identity of a port with which the wire is coupled, and a protocol supported by the wire.

21. An apparatus having a plurality of computer nodes and configured to deploy an application adapted to a scale-independent model for a distributed computer system, the distributed computer system including:

logical modeling means configured to provide a logical model of the application, the logical model having multiple components representing logical functions of the application; and

core conversion means configured to create one or more instances of the model components and to allocate computer nodes of the distributed computer system for the instances to implement the logical functions represented by the model components from which the instances are created.

22. The apparatus ofclaim 21, wherein the core conversion means comprises:

service running determination means configured to state to track the instances created for the model components; and

resource manager means configured to track the computer nodes available to be allocated.

23. The apparatus ofclaim 21, wherein the core conversion means comprises a loader to load software on the computers nodes to implement the logical functions.

24. The apparatus ofclaim 21, wherein the core conversion means comprises:

service running state means configured to track the instances created for the model components;

resource manager means configured to track the computer nodes available to be allocated; and

loader means configured to load software onto the new computer node, the software being executable on the computer node to implement the logical functions represented by the particular model component.

25. The apparatus ofclaim 21, further comprising a management policy to monitor operation of the application and to specify when new instances of the model components are to be created or removed based on the operation of the application.

26. The apparatus ofclaim 21, further comprising a management policy to monitor operation of the application and to specify when new instances of the model components are to be created or removed based on the operation of the application, wherein the management policy listens to events generated by the computer nodes as a way to monitor operation of the application.

Images