BACKGROUNDMany computer applications load data from databases before performing operations to process the data. In some scenarios, for example where an application accesses a relatively small amount of data, all of the required data may be loaded into memory by the application from the database when the application is started. However, in other scenarios, for example where an application accesses a relatively large amount of data, the application may retrieve portions of the data from the database on-demand as the data is required by the application's operations. Such on-demand retrieval can be referred to as “lazy loading” since the data is loaded from the database as it is needed (or “lazily”).
Some applications organize data in memory according to one or more logical domains of the applications. For example, an application may define data structures (such as object graphs) that arrange data into objects according to logical relations. However, the organization of the data in such object graphs may diverge (in some cases significantly) from the organization of the data in the database. Some previous techniques for loading data into such an object graph require each object to separately retrieve the data it needs from the database.
Therefore, there is room for improvement in technologies related to the loading of structured data.
SUMMARYThis Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
In an example embodiment, a method is performed by one or more computing devices, wherein the method comprises receiving a request identifying a root data object of a hierarchical data structure; identifying a plurality of child data objects of the root data object in the hierarchical data structure, wherein the plurality of child data objects are associated with records stored in separate tables of a database; using a loaded child record table to identify a subset of the records associated with the plurality of child data objects that are not stored on one of the one or more computing devices; and transmitting a request to the database to retrieve the subset of the records that are not cached on the one of the one or more computing devices.
In another example embodiment, a system comprises a database and a server, wherein the server comprises a loaded child record table, one or more processors, and a memory storing instructions that, when executed by the one or more processors, cause the server to perform operations, the operations comprising: receiving a request identifying a plurality of parent data records, wherein the plurality of parent data records are roots of structured data objects comprising a plurality of child data records; determining, using the loaded child record table, which of the plurality of child data records is stored on the server for the plurality of parent data records; transmitting a request to the database, comprising a plurality of queries for a subset of the plurality of child data records that are not stored on the server; receiving the subset of the plurality of child data records that are not stored on the server from the database; and populating the structured data objects using the plurality of parent data records and the subset of the plurality of child data records received from the database.
Another example embodiment comprises one or more computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising: analyzing a hierarchical data structure to identify a plurality of child data objects of a root data object of the hierarchical data structure; determining, using a loaded child record table, that a plurality of database records associated with the plurality of child data objects have not been received from a database; generating a database request comprising a plurality of database queries for the plurality of database records associated with the plurality of child data objects; transmitting the generated request to a database; receiving the plurality of database records from the database; initializing the plurality of child data objects of the hierarchical data structure using the received plurality of database records; and adding entries, associating the received plurality of database records with the plurality of child data objects, to the loaded child record table.
As described herein, a variety of other features and advantages can be incorporated into the technologies as desired.
BRIEF DESCRIPTION OF THE DRAWINGSFIG.1 is a system diagram depicting an example system for bulk lazy loading structured data.
FIG.2 is a flowchart of an example method for the bulk lazy loading of a hierarchical data structure.
FIG.3 is a system diagram depicting an example system comprising a loaded child record table for use in bulk loading hierarchical data structures.
FIGS.4A-C are system diagrams depicting an example system for processing a data load request using a loaded child record table to initialize hierarchical data structures.
FIG.5 is a block diagram of an example computing system in which some described embodiments can be implemented.
FIG.6 is an example cloud computing environment that can be used in conjunction with the technologies described herein.
DETAILED DESCRIPTIONExample 1OverviewThe description provided herein is directed to various technologies for the loading of data from databases into data structures.
Some techniques for loading data into structured data objects (such as hierarchical data structures) require each object to retrieve the data it requires from the database separately. In at least some cases, this can lead to inefficient database accesses. For example, if several objects need to be initialized, and each requires a record from the database, each object would have to establish a connection with the database and send a separate data access request to the database. In applications where there is a limited number of database connections available, the problem may be exacerbated, as several of the objects may have to wait for an available connection before sending their request(s) to the database. In another example, a hierarchical data structure may have a root object that has several child objects (such as a collection of child objects of a same data type or multiple child objects of different data types). Loading the hierarchical data structure may require each child object to independently retrieve its data from the database. In addition to potentially requiring the transmission of several separate requests for the multiple child objects to the database, this technique can lead to data access inefficiencies. For example, in a scenario where the root object has a collection of child objects of a same data type. The child objects may make several requests to retrieve individual records from a same table in the database. In another example, in a scenario where the child objects have different data types, two of the child objects may require different fields from a same database record. This may result in two separate requests to retrieve data from the same database record in the database.
Various technologies described herein can be used to address these problems by bulk lazy loading data for structured data objects. For example, rather than having multiple objects load their data independently, the structures of the objects can be analyzed to identify data records that are required by child objects of the structured data objects. Additionally or alternatively, a process can identify one or more tables in a database storing the data records required to populate the child objects of the structured data objects. A single request can then be transmitted to the database comprising one or more queries to retrieve the required data records. The child objects of the structured data objects can then be loaded with the requested data records. Once the database records have been received, they can be stored locally (for example in a cache). A loaded child record table can be defined that identifies which database records have been stored and/are associated with which structured data objects. When one or more structured data objects need to subsequently be loaded, the loaded child record table can be inspected to determine which required database records have already been retrieved from the database and stored in the cache. A single request can be transmitted to the database that comprises one or more queries for only the required records which are not already in the cache. The structured data objects can then be initialized using the database records stored in the cache and any additional database records retrieved from the database.
In a different or further example, a request can be received to load data for a hierarchical data structure. The hierarchical data structure can be analyzed to identify one or more child data objects of the hierarchical data structure. Database records required by each of the child data objects can be identified, for example, by a process which identifies one or more tables required by the process to populate the hierarchical data structure. A single request can be generated and sent to the database to retrieve all the required database records. The child data objects of the hierarchical data structure can then be populated using the requested database records. The retrieved database records can be stored in a cache. A loaded child record table can be used to identify which database records are stored in the cache for the various hierarchical data structures. When a subsequent request to load data for a hierarchical data structure is received, the loaded child record table can be used to determine which required database records required by child data objects of the hierarchical data structure are already stored in the cache. A request can be sent to the database, requesting only the required database records that have not been cached. The hierarchical data structure can then be populated using the cached database records and any additional database records requested from the database.
Example 2Example Systems and Methods for Bulk Lazy Loading of Structured DataIn any of the examples described herein, systems and methods can be provided for processing requests for the bulk lazy loading of structured data.
FIG.1 is a system diagram depicting anexample system100 for the bulk lazy loading of structured data. Theexample system100 comprises aserver110 comprising a loaded child record table113. Theserver110 can be configured to use the loaded child record table113 to process requests for data to initialize one or more structured data objects. For example, theserver110 can be configured to receive arequest120 identifying a plurality of parent data records121-125. Each of the identified parent data records121-125 can be a root of a structured data object comprising a plurality of child data records.
As used herein, the term “structured data object” refers to a data structure comprising a plurality of child data structures. Such child data structures can be defined separately as complex data types (e.g., data types comprising more than one value). For example, a first data structure can be defined as comprising a plurality of data values and a second data structure can be defined as comprising a different plurality of data values. Then, a third data structure can be defined with a first value that is an instance of the first data structure and a second value that is an instance of the second data structure. In such a configuration, the third data structure can be referred to as a structured data object and the instances of the first and second data structures can be referred to as child data objects (or child data records). In at least some embodiments, a structured data object can be a hierarchical data structure comprising a root (or parent) record and a plurality of child records that have a parent-child relationship to the root (or parent) record.
Theexample system100 comprises adatabase130 storing a plurality of child data records133. In at least some embodiments, thedatabase130 comprises a plurality of tables in which the plurality ofchild data records133 are stored. Theserver110 can be configured to cache at least some of thechild data records163 in achild record cache160 on theserver110. Thechild record cache160 can be stored in whole, or in part, in one or more memories and/or one or more computer readable storage media of theserver110. Additionally or alternatively, all or part of thechild record cache160 can be stored on one or more other computing devices.
The loaded child record table113 can identify which of thechild data records133 is/are available on theserver110. For example, the loaded child record table113 can identify which of thechild data records133 is/are cached in thechild record cache160 on theserver110. Additionally or alternatively, the loaded child record table113 can identify which of thechild data records133 is/are already loaded into the structured data objects115. Entries in the loaded child record table113 can be organized according to the relationships between the plurality of parent data records and the plurality of child data records. For example, an entry can be added to the loaded child record table113 for a given parent data record. The entry can comprise one or more fields associated with the one or more child data records that are defined for the structured data object for which the given parent data object is a root.
Each of the one or more fields can contain a value which indicates whether or not a child data record associated with the given field is stored in theserver110. For example, the value can indicate that a child data record associated with the given field is stored in thechild record cache160. Alternatively, the value can indicate that a child data record associated with the given field has already been used to populate one or more structured data objects (e.g.,115). In at least some embodiments, each field can have a value set equal to one or more indicator values: a first value indicating that a child data record associated with the given field is not available on theserver110, a second value indicating that the child data record is not yet available on theserver110 but has been requested from the database, a third value indicating that the child data record has been loaded onto theserver110 but has not yet been used to populate a structured data object, and/or a fourth value indicating that the child data record has been used to populate a structured data object.
Subsequent to receiving therequest120, theserver110 can use the loaded child record table113 to determine which of the child data records associated with the plurality of parent data records121-125 is/are cached on theserver110. For example, for a given parent data record (e.g.121) theserver110 can locate an entry for the given parent data record in the loaded child record table113. Theserver110 can then use the fields of the located entry to determine which of the child data records associated with the given parent data record have already been cached on theserver110. Theserver110 can repeat this process for each of the plurality of parent data records121-125.
Theserver110 can then transmit arequest140 to thedatabase130, comprising a plurality of queries141-145 for a subset of the plurality of child data records that are not cached on theserver110. Theserver110 can generate the queries141-145 based on the contents of the loaded child record table113. For example, for a given parent data record, as a child data record is identified which the loaded child record table113 indicates is not yet cached on theserver110, theserver110 can generate a query for the identified child data record. In at least some embodiments, where multiple uncached child data records are identified that are stored in a same table in thedatabase130, theserver110 can generate a single query to retrieve the multiple uncached child data records from the table. Theserver110 can then transmit the plurality of the generated queries141-145 to thedatabase130 in thesingle database request140. In a particular example, theserver110 can use the loaded child record table113 to determine that a first child data record of a given parent data record is stored in theserver110, but that a second child data record of the given parent data record is not stored in theserver110. Theserver110 can include a query for the second child data record in thedatabase request140 and can omit a query for the first child data record from therequest140.
Subsequent to transmitting therequest140, theserver110 can receive aresponse150 from thedatabase130 comprising the requested child data records151-155. In at least some embodiments, theserver110 can be configured to transmit aresponse119 comprising the subset of the plurality of child data records received from the database and the remaining subset of the plurality of child data records that were stored on theserver110. Additionally or alternatively, theserver110 can be configured to populate structured data objects115 using the plurality of parent data records121-125 and the subset of the plurality of child data records151-155 received from thedatabase130. Populating the structured data objects115 can further comprise using a plurality of child data records stored on theserver110. For example, for each of the parent data records121-125, theserver110 can create a new structured data object using the parent data record as the root of the structured data object. Creating a new structured data object can comprise, for example, creating a new instance of a data type (such as a structure, class, etc.). In at least some embodiments, this can comprise instantiating a structured data object of a data type associated with the given parent data record. Additionally or alternatively, one or more members of the newly created structured data object can be populated using fields of the parent data record. The child data records retrieved from thedatabase130 and/or thechild record cache160 which are associated with the given parent data record can then be used to initialize child data structures of the structured data object. In such an embodiment, the populated structured data objects can be transmitted in aresponse117.
Thus, in at least some embodiments, the loaded child record table113 can be used to identify the set of child data records that are required to instantiate a plurality of structured data objects. The loaded child record table113 can further identify which of this set of child data records have not yet been loaded on theserver110. A single request can be generated to retrieve the non-cached child data records from thedatabase130. The retrieved child data records can then be used to instantiate one or more child data structures of the plurality of structured data objects. In at least some embodiments, a previously initiated structured data object can be identified using the loaded child record table and the retrieved child data records can be used to add child data structures to the previously initiated structured data object. Additionally or alternatively, a combination of the retrieved and previously stored child data records can then be used to instantiate the plurality of structured data objects. Thus, in at least some scenarios, the overall amount of interaction with thedatabase130 required to instantiate the plurality of structured data objects can be reduced.
In at least some embodiments, theserver110 can be configured to use the loaded child record table113 to determine that a request for one of the plurality of child data records was previously transmitted to thedatabase130 and to omit a query for the one of the plurality of child data records from therequest140. For example, when a query for a given child data record is transmitted to thedatabase130, theserver110 can be configured to update an entry in the loaded child record table113 for the child data record to indicate that the child data record has been requested (but has not yet been received from) thedatabase130. If a subsequent request is received for a parent data record that is associated with the child data record, the updated entry in the loaded child record table113 can be used to avoid sending a redundant request for the child data record to thedatabase130.
Subsequent to receiving the child data records151-155 from thedatabase130, theserver110 can be configured to store the received child data records151-155 on the server110 (for example, by storing them in thechild record cache160 and/or loading them into one or more of the structured data objects115). Theserver110 can then add entries to the loaded child record table113 for the received child data records151-155. For example, for each of the received child data records151-155, theserver110 can create or update an entry in the loaded child record table113 to indicate that the child data record is now stored on theserver110.
In at least some embodiments, thechild data records133 can be managed in a non-relational structure, such as a hierarchical structure. For example, thechild data records133 can be managed in a hierarchical structure following the structure of the structured data objects115. However, in some cases the hierarchical structure in which thechild data records133 are managed, and the structure of the structured data objects115 may differ.
FIG.2 is a flowchart of anexample method200 for the bulk lazy loading of a hierarchical data structure. Any of the example systems described herein can be used to perform theexample method200. For example, theexample system100 can be used to perform theexample method200.
At210, a request is received identifying a root data object of a hierarchical data structure. The hierarchical data structure comprises the root data object and a plurality of child data objects that are in a parent-child relationship with the root data object. Example hierarchical data structures include tree data structures, graph data structures, collection data structures (such as arrays, vectors, lists, or the like), object graphs, etc. A particular example of a hierarchical data structure is an instance of an object graph in which the root data object is an instance of a data type comprising multiple members (the child data objects) that are instances of other data types.
At220, a plurality of child data objects of the root data object in the hierarchical data structure are identified. The identified plurality of child objects can be associated with records stored in a database. In at least some embodiments, at least some of the records can be stored in separate tables of the database. For example, in an embodiment where the child data objects are associated with different data types, the different data types can be associated with separate tables in the database. In such an embodiment, for a given child data object, a corresponding record can be stored in the database in a table associated with the child data object's data type. In another example, the data types can be associated with records in a database that are organized in hierarchical data structures. In a different or further embodiment, the root data object can be associated with a record stored in a database. For example, a data type of the root data object can be associated with a table in the database in which a record corresponding to the root data object is stored. In such an embodiment, the request received at210 can comprise a unique identifier associated with the database record for the root data object.
In at least some embodiments, a data type associated with the root data object can be used to identify the child data objects of the hierarchical data structure. Example data types include data structure definitions, object class definitions, etc. Additionally or alternatively, the request received at210 can comprise one or more child data object identifiers that are associated with the identifier for the root data object. Additionally or alternatively, the request can comprise an identifier for an operation or process to be performed using the hierarchical data structure. The operation or process can be used to identify the plurality of child data objects. For example, the operation or process may require less than all of the child data objects of the root data object. This subset of the root object's child data objects can be identified and used for the subsequent steps of theexample method200.
At230, a loaded child record table is used to identify a subset of the database records associated with the child data objects that are not cached. The loaded child record table is a data structure that can be used to identify database records which have been stored in one or more memories and/or computer readable storage media of a given computing device. In at least some embodiments the loaded child record table can be stored in the same computing device in which the database records are cached. The loaded child record table can comprise a plurality of entries which correspond to hierarchical data structures. For example, entries can be added to the loaded child record table for one or more root data objects of one or more hierarchical data structures. An entry for a given root data object can comprise one or more fields corresponding to one or more child data objects of the root data object. A field for a given child data object can comprise a value, indicating whether or not a database record for the child data object is cached on the computing device.
The loaded child record table can be used to identify one or more of the database records associated with the plurality of child objects for the root data object identified by the request that are not cached on the computing device. For example, the request can comprise an identifier for the root data object. This identifier can be used to locate an entry in the loaded child record table associated with the root data object. The entry can comprise fields corresponding to the plurality of child data objects of the root data object. The values of the fields can be used to identify a subset of the plurality of child data objects whose associated database records are not cached on the computing device. In an embodiment where the root data object is associated with a database record, the loaded child record table can also be used to determine whether the database record associated with the root data object is cached on the computing device.
Data records can be cached on the computing device in a record cache. Additionally or alternatively, the data records can be used to populate one or more hierarchical data structures that are stored on the computing device. In such an embodiment, the loaded child record table can identify data records that have already been retrieved and used to populate the one or more hierarchical data structures.
At240, a request is transmitted to the database to retrieve the subset of records that are not stored locally. The request can comprise one or more queries to retrieve the subset of unstored records from one or more tables in the database. In at least some embodiments, when the loaded child record table indicates that a database record for a given child data object is not stored on the computing device, a data type associated with the given child data object can be used to identify a table in the database that is associated with the data type. An identifier for the database record associated with the given child data object can be used to generate a query to retrieve the database record from the identified database table. For example, an identifier associated with the root data object can be used as a key to retrieve the database record for the child data object. Additionally or alternatively, the root data object can comprise an identifier for the child data object that can be used as a key in the query to retrieve the database record for the child data object. Other identifier schemes are also possible.
In an embodiment where the identified subset of records comprises records for child data objects with the data types that are associated with separate database tables, the request can comprise a plurality of queries to retrieve the subset of records from the separate tables of the database. Additionally or alternatively, if it is determined that multiple uncached records are stored in a same table in a database, a single query can be generated to retrieve the multiple records from the database table.
In a scenario where the root data object is associated with the database record, and the loaded child record table indicates that the database record is not cached on the computing device, the request can include a query for the record associated with the root data object.
In at least some embodiments, the request received at210 can identify a plurality of root data objects of a plurality of hierarchical data structures. In such an embodiment, a plurality of child data objects can be identified for each of the plurality of root data objects. The loaded child record table can be used to identify a subset of uncached records associated with the plurality of child data objects for the plurality of root data objects, and the request transmitted at240 can be a request to retrieve the uncached records associated with the plurality of child data objects for the plurality of root data objects.
In a different or further embodiment, the database is stored in a separate database server. In such an embodiment, transmitting the request can comprise transmitting the request to the database server via a computer network.
Optionally, at250, the plurality of child data objects of the hierarchical data structure are populated using the subset of records retrieved from the database. The records which were cached can also be used to populate child data objects of the hierarchical data structure.
In at least some scenarios, the subset of records retrieved from the database can be stored in a cache. Entries can be added to the loaded child record table to indicate that the records have been stored in a cache. Subsequent requests to populate instances of the hierarchical data structure can be processed using the cached records instead of re-retrieving the records from the database. For example, if the loaded child record table indicates that a record was previously stored on the computing device, then it can be retrieved from its storage location and used to populate one or more of the child data objects. Additionally or alternatively, the child record table can indicate that the record was previously retrieved and used to populate one or more of the child data objects. In such a scenario, the previously populated one or more child data objects can be used without re-populating them.
In at least some embodiments, the loaded child record table can be used to determine that a record associated with one of the plurality of child data objects is not cached, but that another request for the record has been transmitted to the database. For example, an entry for the child data object in the loaded child record table can comprise a value which indicates that the query for the database record associated with the child data object has been transmitted to the database, but the record has not yet been received from the database. In such scenarios, a query for the database record can be omitted from the request that is transmitted to the database to retrieve the subset of records that are not cached on the computing device.
Example 3Example Cached Child Record TableIn any of the examples described herein, systems and methods can be provided for using a loaded child record table to process requests for bulk loading of hierarchical data structures.
FIG.3 is a system diagram depicting anexample system300 comprising a loaded child record table350 for use in bulk loading of hierarchical data structures (e.g.,303). Theexample system300 further comprises adatabase330. The hierarchical data structure and303, the loaded child record table350, and thedatabase330 can be stored by one or more computing devices (not shown) such as one or more server computers.
Thehierarchical data structure303 can be analyzed to identify a plurality of child data objects311-315 of a root data object310 of thehierarchical data structure303. The loaded child record table350 can be used to determine that a plurality of database records associated with the plurality of child data objects311-315 are not cached. A database request comprising a plurality of database queries for the plurality of database records associated with the plurality of child data objects311-315 can be generated. The generated request can be transmitted to thedatabase330. Responsive to the transmitted request, the plurality of database records can be received from thedatabase330. In at least some embodiments, the received plurality of database records can be stored in a cache. Entries, associating the received plurality of database records with the plurality of child data objects311-315, can be added to the loaded child record table350.
In at least some embodiments, it can be determined that the plurality of database records are stored in separate tables (e.g.331-335) in thedatabase330. In such an embodiment, generating the database request can comprise generating separate queries to retrieve the plurality of database records from the separate database tables.
Optionally, the plurality of child data objects311-315 can be initialized using the received plurality of database records. For example, each of the plurality of child data objects311-315 can represent a member of the root data object310 that is associated with a particular data type. For a given child data object, an instance of the associated data type can be created and populated with values contained in the database record associated with the child data object.
In a particular example, thehierarchical data structure303 can be analyzed to identify the root data object310 and its child data objects311-315. Data types associated with the child data objects311-315 can be used to identify associated tables in thedatabase330 in which records for the respective child data objects are stored. For example, a data type of the child data object311 can be used to determine that the table331 in thedatabase330 stores a database record (e.g.,341) associated with thechild data object311. A data type of the child data object313 can be used to determine that the table333 stores a database record (e.g.,343) associated with thechild data object313. A data type of the child data object315 can be used to determine that the table335 stores a database record (e.g.,345) associated with thechild data object315. It is possible to use an alternative or additional data source (such as metadata associated with a child data object or a data type of the child data object, etc.) to identify a database table that corresponds to a given child data object.
The loaded child record table350 can comprise one or more rows associated with the various hierarchical data structures and/or root data objects of various hierarchical data structures. For example, inFIG.3 the loaded child record table350 is depicted as comprising a plurality of rows wherein each row is associated with a different root data object. A row in the loaded child record table350 can be identified that is associated with theroot data object310. The row for the root data object310 comprises a plurality of fields associated with database tables331,333, and335, respectively. The field for table331 contains a value that indicates that the database record for the child data object311 (which is associated with the table331) is already stored in a cache. The fields for tables333 and335 contain values that indicate that the database records for the child data objects313 and315 (which are associated with the tables333 and335, respectively) are not stored in the cache.
Additionally or alternatively, the fields in the loaded child record table350 can indicate whether or not associated child data objects have already been populated with records from thedatabase330. For example, the value of the field for table331 can indicate that the database record for thechild data object311 was previously retrieved and used to populate thechild data object311. The values of the fields for the tables333 and335 can indicate that the child data objects313 and315, respectively, have not yet been populated with records from thedatabase330.
Based on this analysis of the loaded child record table350, a database request can be generated which comprises a query to retrieve a database record (e.g.,343) for child data object313 from the database table333 and a query to retrieve a database record (e.g.,345) for the child data object315 from the database table335. However, a query to retrieve a database record for thechild data object311 is not included in the request because that database record is already stored in the cache. The generated request can be transmitted to thedatabase330, responsive to which the queried database records can be received from thedatabase330.
The received database records for the child data objects313 and315, and the cached database record for thechild data object311, can be used to instantiate the child data objects311-315 of thehierarchical data structure303. In at least some embodiments, the received database records (e.g.,343 and345) can be stored in the cache. The row for the root data object310 in the loaded child record table350 can be updated to indicate that the database records for thechild data object313 and315 (which are associated with the database tables333 and335, respectively) are now stored in the cache.
Thus, database accesses can be reduced in at least some cases. Rather than initializing the child data objects311-315 in isolation, which would require separate database requests to retrieve associated database records for the respective child data objects, a single request is generated that includes queries for all data records necessary to initialize thehierarchical data structure303. Furthermore, this single database request includes queries only for database records that are not already cached. Thus, in at least some cases, using at least some of the techniques described here, the overall volume of database accesses can be reduced.
In at least some embodiments, thehierarchical data structure303 can include multiple levels of child data objects. For example, the root data object310 can comprise agrandchild data object317, which is a child data object of thechild data object311. In such an embodiment, the process of data loading described above with respect to the root data object310 can be repeated, but in this case treating the child data object311 as if it were a root data object. For example, another loaded child record table360 can be used to track whether database records for child data objects (e.g.,317) of the data object311 are stored in the cache. The grandchild data object317 can be associated with a record (e.g.,347) stored in a table337 of thedatabase330. The loaded child record table360 can be used to determine that the database record stored in the table337 is not cached. The request transmitted to thedatabase330 can include a query to retrieve the database record (e.g.,347) from the database table337. The retrieved record can then be used to initialize thegrandchild object317 as part of the initialization of thehierarchical data structure303. Although only a single grandchild data object is depicted inFIG.3, other numbers of grandchild data objects are possible (as are additional levels of ancestry, such as great-grandchild data objects, etc.). In at least one embodiment, the loaded child record tables350-360 can be organized in a hierarchical structure which follows an object structure of thehierarchical data structure303. For example, the loaded child record table360 can be stored as a value of a field ofrow310 in the loaded child record table350.
Example 4Example Bulk Lazy Loading of Hierarchical Data StructuresIn any of the examples described herein, a loaded child record table can be provided for determining the storage status of database records associated with child data objects.
FIGS.4A-C are system diagrams depicting anexample system400 for processing adata load request410 using a loaded child record table420 to initialize one or more hierarchical data structures (e.g.,450-480). The loaded child record table420 and the one or more hierarchical data structures (e.g.,450-480) can be stored on one or more computing devices, such as one or more server computers.
FIG.4A depicts the loaded child record table420 and uninitialized hierarchical data structures450-480. Thehierarchical data structure450 comprises aroot data object451 which comprises a key411 for theroot data object451. Thehierarchical data structure460 comprises aroot data object461 which comprises a key413 for theroot data object461. Thehierarchical data structure470 comprises aroot data object471 which comprises a key417 for theroot data object471. Thehierarchical data structure480 comprises aroot data object481 which comprises a key417 for theroot data object481. Each of the hierarchical data structures450-480 is depicted inFIG.4A as having two child data objects (453 and455,463 and465,473 and475,483 and485), however different numbers of child data objects are possible.
Theexample system400 comprises adata load request410. Thedata load request410 identifies the root data object keys411-417 and a plurality of database records which should be loaded into child data objects of the respective root data objects. For example, thedata load request410 indicates that a child data object of the root data object451 should be initialized with a record from Table 2 (“T2”) of a database and that another child data object of the root data object451 should be initialized with a record from Table 3 (“T3”) of the database. Thedata load request410 further indicates that a child data object of the root data object461 should be initialized with a record from Table 1 (“T1”) of the database and that another child data object of the root data object461 should be initialized with a record from Table 2. Thedata load request410 further indicates that a child data object of the root data object471 should be initialized with a record from Table 1 and that another child data object of the root data object471 should be initialized with a record from Table 2. Thedata load request410 further indicates that a child data object of the root data object481 should be initialized with a record from Table 1 and that another child the data object of the root data object481 should be initialized with a record from Table 3.
The loaded child record table420 can be used to determine which database records for child data objects of the various root data objects451-481 have already been retrieved from the database. For example, as depicted inFIG.4A, the loaded child record table420 indicates that a database record for a child data object of the root data object with key411 (451) that is stored in Table 1 of the database has already been retrieved from the database, but database records for child data objects of the root data object451 that are stored in Tables 2 and 3 have not been retrieved from the database. The loaded child record table420 further indicates that no database records for child data objects of the root data object with key413 (461) have been retrieved from the database. The loaded child record table420 further indicates that a database record for a child data object of the root data object with key415 (471) stored in Table 2 has been retrieved from the database, but the database records for child data objects of the root data object471 that are stored in Tables 1 and 3 have not been retrieved from the database. The loaded child record table420 further indicates that database records for child data objects of the root data object with key417 (481) that are stored in Tables 1 and 3 have been retrieved from the database, but that a database record for a child data object of the root data object481 that is stored in Table 2 has not been retrieved from the database.
FIG.4B depicts theexample system400 comprising thedata load request410, the loaded child record table420, the hierarchical data structures450-480, and a generateddatabase request430. The loaded child record table420 is analyzed to determine which of the database records identified by thedata load request410 have not yet been retrieved from the database. Based on this analysis, thedatabase request430 is generated, comprising queries431-439 for the data records that have not yet been retrieved.
For example, it can be determined using the loaded child record table420 that records stored in Tables 2 and 3 of the database that are associated with child data objects of the root data object with key411 (451) are not stored locally (for example, in a cache and/or as part of one or more of the hierarchical data structures450-480).Queries431 and433 are thus included in thedatabase request430 to retrieve the records stored in Tables 2 and 3 that are associated with child data objects of the root data object with key411 (451). It can be further determined using the loaded child record table420 that records stored in Tables 1 and 2 that are associated with child data objects of the root data object with key413 (461) are not stored locally.Queries435 and437 are thus included in thedatabase request430 to retrieve the records stored in Tables 1 and 2 that are associated with child data objects of the root data object with key413 (461). It can be further determined using the loaded child record table420 that a record stored in Table 1 of the database that is associated with a child data object of the root data object with key415 (471) is not stored locally and that a record stored in Table 2 that is associated with a child data object of the root data object with key415 (471) is stored locally (for example, a copy of the record may be stored in a cache and/or a child data object of one of the hierarchical data structures may have previously been initialized using the record). Thequery439 is thus included in thedatabase request430 to retrieve the record stored in Table 1 that is associated with a child data object of the root data object with key415 (471). However, a query is not included to retrieve the record from Table 2 for the root data object withkey415 because it was previously retrieved from the database. It can be further determined using the loaded child record table420 that records stored in database Tables 1 and 3 that are associated with child data objects of the root data object with key417 (481) are stored locally. Thus, queries to retrieve these records are not included in thedatabase request430.
Although a separate query is listed for each individual record in thedatabase request430, this is not intended to be limiting. For example, it is possible in at least some embodiments to a create single query to retrieve multiple records that are stored in a same table in the database. In such an embodiment, thedatabase request430 can comprise three queries: a first query to retrieve the records for413 and415 from Table 1, a second query to retrieve records for411 and413 from Table 2, and a third query to retrieve the record for411 from Table 3.
Entries in the loaded child record table420 for queried records can be updated to indicate that a request for the records has been transmitted to the database. For example, the entries in the loaded child record table420 that are included in the queries431-439 have been updated inFIG.4B to indicate that the records have been requested (“R”). This can be useful, for example, in scenarios where various operations for retrieving the database records and/or initializing the hierarchical data structures or performed asynchronously. If, after thedatabase request430 is transmitted to the database, another data load request (not shown) identifying one or more of the same database records is received, the updated entries in the cache child record table420 can be used to determine that the one or more database records has been requested from the database but is not yet stored locally. Based on such a determining, queries for the database records that have already been requested can be omitted from any subsequently generated database request.
In at least some embodiments, different values can be used in the entries of the loaded child record table420 to indicate different statuses of the hierarchical data structures with respect to the database records. For example, a first value can be used to indicate that a record for a child data object has not been retrieved from the database (e.g., a null value), a second value can be used to indicate that the record has not been retrieved but a request for the record has been sent to the database but the record has not yet been received (e.g., a value of “R”), a third value can be used to indicate that the record has been retrieved from the database but has not yet been used to populate the corresponding child data object (e.g., a value of “L”), and/or a fourth value can be used to indicate that the record has been retrieved and has been used to populate the corresponding child data object (e.g., a value of “X”).
FIG.4C depicts theexample system400, comprising the child record table420, adatabase response440, and the hierarchical data structures450-480. Subsequent to transmitting thedatabase request430, depicted inFIG.4B, thedatabase response440 is received from the database. Thedatabase response440 includes the requested database records441-449 that were identified by the database queries431-439. The received database records441-449 can be used to initialize the hierarchical data structures450-480. Initializing a hierarchical data structure can comprise populating one or more fields of one or more data objects of the hierarchical data structure with data in the one or more received database records441-449.
For example, it can be determined that the child data object453 of the root data object451 is associated with Table 2. Therecord441, which is a Table 2 record for a child data object of theroot data object451, can be used to initialize thechild data object453. It can be determined that the child data object455 of the root data object451 is associated with Table 3. Therecord443, which is a Table 3 record for a child data object of theroot data object451, can be used to initialize thechild data object455. It can be determined that the child data object463 of the root data object461 is associated with Table 1. Therecord445, which is a Table 1 record for a child data object of theroot data object461, can be used to initialize thechild data object463. It can be determined that the child data object465 of the root data object461 is associated with Table 2. Therecord447, which is a Table 2 record for a child data object of theroot data object461, can be used to initialize thechild data object465. It can be determined that the child data object473 of the root data object471 is associated with Table 1. Therecord449, which is a Table 1 record for a child data object of theroot data object471, can be used to initialize thechild data object473.
It can be determined that the child data object475 of the root data object471 is associated with Table 2. However, thedatabase response440 does not include a record from Table 2 for a child data object of theroot data object471. This is because the loaded child record table420 indicated that a Table 2 record associated with the child data object of the root data object471 was stored locally and, thus, a query for this database record was not included in thedatabase request430. In an embodiment where this database record (491) is stored locally in a cache therecord491 can be retrieved from the cache and used to initialize thechild database object475. Alternatively, therecord491 may have been previously retrieved from the database and previously used to initialize thechild database object475.
It can be determined that the child data objects483 and485 are associated with Tables 1 and 3, respectively. Thedatabase response440 does not include database records for these child data objects of theroot data object481. This is because the loaded child record table420 indicated that database records stored in Tables 1 and 3 that are associated with the root data object481 were stored locally and, thus, queries for these database records were not included in thedatabase request430. In an embodiment where these database records (493 and495) are stored in a cache, the database records493 and495 can be retrieved from the cache and used to initialize the child database objects483 and485, respectively. Alternatively, therecords493 and495 may have been previously retrieved from the database and previously used to initialize the child data objects483 and485, respectively.
The entries for these records in the loaded child record table420 can be updated to indicate that the data from the database records441-449 are now stored locally in the hierarchical data structures450-480. Optionally, the database records441-449 can be stored in the cache. In an embodiment where thedatabase request430 is transmitted to the database asynchronously, after thedatabase request430 has been transmitted to the database, theresponse440 can be received asynchronously from the database. Once the requested database records441-449 have been received, their entries in the loaded child record table420 can be asynchronously updated to indicate that the records are now stored locally (for example, in the cache). In at least some such embodiments, the loaded child record table420 can be asynchronously inspected to determine when entries for the data records have been added or updated. When it is detected, based on the loaded child record table420, that the data records have been stored locally, the plurality of child data objects for the root data objects451-481 can be initialized using the stored plurality of records.
In at least some embodiments, the loaded child record table420 can be used to track whether data records for root data objects (e.g.,451,461,471, and481) have been retrieved from the database. For example, the loaded child record table420 can include a column for another table (not shown) in the database that stores data records for one or more of the root data objects. In such an embodiment, the loaded child record table420 can be used to selectively retrieve data records for the root data objects and initialize the root data objects as described above with respect to the child data objects.
The hierarchical data structures450-480 are depicted inFIGS.4A and4B as comprising child data objects that are instantiated, but that do not yet contain data. However, this is not intended to be limiting. One or more of the child data objects can be instantiated on-demand. For example, a child data object can be instantiated when a corresponding record is received from the database. The instantiated child data object can then be initialized using data in the corresponding database record.
Example 5Example Computing SystemsFIG.5 depicts a generalized example of asuitable computing system500 in which the described innovations may be implemented. For example, thecomputing system500 can be used as a client computing device and/or server computer as described herein. Thecomputing system500 is not intended to suggest any limitation as to scope of use or functionality, as the innovations may be implemented in diverse general-purpose or special-purpose computing systems.
With reference toFIG.5, thecomputing system500 includes one ormore processing units510,515 andmemory520,525. InFIG.5, thisbasic configuration530 is included within a dashed line. Theprocessing units510,515 execute computer-executable instructions. A processing unit can be a general-purpose central processing unit (CPU), processor in an application-specific integrated circuit (ASIC) or any other type of processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. For example,FIG.5 shows acentral processing unit510 as well as a graphics processing unit orco-processing unit515. Thetangible memory520,525 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, solid state drives, etc.), or some combination of the two, accessible by the processing unit(s). Thememory520,525 can storesoftware580 implementing one or more innovations described herein, in the form of computer-executable instructions suitable for execution by the processing unit(s).
A computing system may have additional features. For example, thecomputing system500 includesstorage540, one ormore input devices550, one ormore output devices560, and one ormore communication connections570. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of thecomputing system500. Typically, operating system software (not shown) provides an operating environment for other software executing in thecomputing system500, and coordinates activities of the components of thecomputing system500.
Thetangible storage540 may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, solid state drives, CD-ROMs, DVDs, or any other medium which can be used to store information in a non-transitory way and which can be accessed within thecomputing system500. Thestorage540 can store instructions for thesoftware580 implementing one or more innovations described herein.
The input device(s)550 may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to thecomputing system500. For video encoding, the input device(s)550 may be a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video samples into thecomputing system500. The output device(s)560 may be a display, printer, speaker, CD-writer, or another device that provides output from thecomputing system500.
The communication connection(s)570 enable communication over a communication medium to another computing entity. The communication medium conveys information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can use an electrical, optical, RF, or other carrier.
The innovations can be described in the general context of computer-executable instructions, such as those included in program modules, being executed in a computing system on a target real or virtual processor. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Computer-executable instructions for program modules may be executed within a local or distributed computing system.
For the sake of presentation, the detailed description uses terms like “determine” and “use” to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer, and should not be confused with acts performed by a human being. The actual computer operations corresponding to these terms vary depending on implementation.
Example 6Example Cloud Computing EnvironmentFIG.6 depicts an examplecloud computing environment600 in which the described technologies can be implemented. Thecloud computing environment600 comprises cloud computing services610. Thecloud computing services610 can comprise various types of cloud computing resources, such as computer servers, data storage repositories, networking resources, etc. Thecloud computing services610 can be centrally located (e.g., provided by a data center of a business or organization) or distributed (e.g., provided by various computing resources located at different locations, such as different data centers and/or located in different cities or countries).
Thecloud computing services610 are utilized by various types of computing devices (e.g., client computing devices), such ascomputing devices620,622, and624. For example, the computing devices (e.g.,620,622, and624) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smart phones), or other types of computing devices. For example, the computing devices (e.g.,620,622, and624) can utilize thecloud computing services610 to perform computing operators (e.g., data processing, data storage, and the like).
Example 7Example ImplementationsAlthough the operations of some of the disclosed methods are described in a particular, sequential order for convenient presentation, it should be understood that this manner of description encompasses rearrangement, unless a particular ordering is required by specific language set forth below. For example, operations described sequentially may in some cases be rearranged or performed concurrently. Moreover, for the sake of simplicity, the attached figures may not show the various ways in which the disclosed methods can be used in conjunction with other methods.
Any of the disclosed methods can be implemented as computer-executable instructions or a computer program product stored on one or more computer-readable storage media and executed on a computing device (e.g., any available computing device, including smart phones or other mobile devices that include computing hardware). Computer-readable storage media can include any tangible media that can be accessed within a computing environment (e.g., one or more optical media discs such as DVD or CD, volatile memory components (such as DRAM or SRAM), or nonvolatile memory components (such as flash memory, solid state drives, or magnetic media such as hard drives)). By way of example and with reference toFIG.5, computer-readable storage media includememory520 and525, andstorage540. The term computer-readable storage media does not include signals and carrier waves. In addition, the term computer-readable storage media does not include communication connections (e.g.,570).
Any of the computer-executable instructions for implementing the disclosed techniques as well as any data created and used during implementation of the disclosed embodiments can be stored on one or more computer-readable storage media. The computer-executable instructions can be part of, for example, a dedicated software application or a software application that is accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software can be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the Internet, a wide-area network, a local-area network, a client-server network (such as a cloud computing network), or other such network) using one or more network computers.
The disclosed technology is not limited to any specific computer language or program. For instance, the disclosed technology can be implemented by software written in C++, Java, Perl, JavaScript, assembly language, or any other suitable programming language. Likewise, the disclosed technology is not limited to any particular computer or type of hardware.
Furthermore, any of the software-based embodiments (comprising, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed through a suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, software applications, cable (including fiber optic cable), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.
The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Instead, the present disclosure is directed toward all novel and nonobvious features and aspects of the various disclosed embodiments, alone and in various combinations and sub combinations with one another. The disclosed methods, apparatus, and systems are not limited to any specific aspect or feature or combination thereof, nor do the disclosed embodiments require that any one or more specific advantages be present or problems be solved.
The technologies from any example can be combined with the technologies described in any one or more of the other examples. In view of the many possible embodiments to which the principles of the disclosed technologies may be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technologies and should not be taken as a limitation on the scope of the disclosed technologies. Rather, the scope of the disclosed technologies includes what is covered by the scope and spirit of the following claims.