US20110202854A1 - Metadata Capture for Screen Sharing - Google Patents

Abstract

A method for metadata capture for screen sharing of a graphical user interface (GUI) screen includes determining by a screen share sending module a region of the GUI screen to be shared; analyzing the GUI screen by a metadata capture module to determine metadata relevant to the region to be shared; and sending a screen share of the region comprising the relevant metadata to a receiver by the screen share sending module.

Description

BACKGROUND
• This disclosure relates generally to the field of screen sharing during online communication.
• Screen sharing may be used to communicate information from a user's (or sender) graphical user interface (GUI) screen in, for example, an online chat or web conference. The screen share may comprise an image of the user's GUI screen that is sent to a receiver. The GUI screen may comprise metadata, including but not limited to confidentiality information, audit trail information, text, table cell color, a uniform resource locator (URL) link, a search query, or a persistent session ID. The metadata is not captured in the screen share image; for example, metadata such as selectable text or clickable URL links may be lost. The metadata must be captured and shared separately.
• Different types of metadata require different capturing methods. For example, text may be highlighted and copied, table cell color may be extracted from HTML code, a search query may be copied from a search textbox, a URL link may be right clicked, and “Copy Link Location” selected, a screen capture of a bitmap may be extracted by pushing the printscreen key, or a session ID may be invisible, with no available extraction method. A sender of a screen share may not be able to extract more than one of type of metadata at a time. The receiver may determine some metadata via post processing and analysis of received screen share image, using, for example, optical character recognition (OCR) or lexicographic analysis. However, such analysis does not capture all types of metadata that are present in the sender's GUI screen at the time of the screen share, and may perform inefficiently or inaccurately. Therefore, in an instant messaging scenario, the sender may need to copy and paste the metadata information separately and manually to the receiving party, which limits the usefulness of the screen share. In a web conference scenario, the meeting host may not be able to copy and paste metadata without interrupting the realtime nature of the web conference.
SUMMARY
• An exemplary embodiment of a method for metadata capture for screen sharing of a graphical user interface (GUI) screen includes determining by a screen share sending module a region of the GUI screen to be shared; analyzing the GUI screen by a metadata capture module to determine metadata relevant to the region to be shared; and sending a screen share of the region comprising the relevant metadata to a receiver by the screen share sending module.
• An exemplary embodiment of a system for metadata capture for screen sharing of a graphical user interface (GUI) screen includes a screen share sending module configured to determine a region of the GUI screen to be shared; a metadata capture module configured to analyze the GUI screen to determine metadata relevant to the region to be shared; and wherein the screen share sending module is further configured to send a screen share of the region comprising the relevant metadata to a receiver.
• An exemplary embodiment of a computer program product comprising a computer readable storage medium containing computer code that, when executed by a computer, implements a method for metadata capture for screen sharing of a graphical user interface (GUI) screen, wherein the method includes determining by a screen share sending module a region of the GUI screen to be shared; analyzing the GUI screen by a metadata capture module to determine metadata relevant to the region to be shared; and sending a screen share of the region comprising the relevant metadata to a receiver by the screen share sending module.
• Additional features are realized through the techniques of the present exemplary embodiment. Other embodiments are described in detail herein and are considered a part of what is claimed. For a better understanding of the features of the exemplary embodiment, refer to the description and to the drawings.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
• Referring now to the drawings wherein like elements are numbered alike in the several FIGURES:
• FIG. 1 illustrates an embodiment of a system for metadata capture for screen sharing.
• FIG. 2 illustrates an embodiment of a method for metadata capture for screen sharing.
• FIG. 3 illustrates an embodiment of a GUI screen with a region selected for screen sharing.
• FIG. 4 illustrates an embodiment of a sender interface for metadata capture for screen sharing.
• FIG. 5 illustrates an embodiment of a receiver interface for metadata capture for screen sharing.
• FIG. 6 illustrates an embodiment of a computer that may be used in conjunction with systems and methods for metadata capture for screen sharing.
DETAILED DESCRIPTION
• Embodiments of systems and methods for metadata capture for screen sharing are provided, with exemplary embodiments being discussed below in detail.
• Capture of metadata may be performed automatically in a screen share of a GUI screen, rather than through manual copying and pasting by the sender. The sender may select a region within the GUI screen for which metadata is to be extracted, or the sender may select the entire GUI screen. Once a region of the screen is selected, object recognition technology is applied to the selected region, and to objects outside of the selected region that are relevant to the selected region, to recognize and capture the metadata. Some or all of the captured metadata may be sent to a receiver along with the screen share, according to the sender's preferences. The metadata may be rendered in the receiver's system as HTML, an image map, native object, or any other desired format. The captured metadata is ready for use by the receiver, without the need for additional processing. Metadata may be captured across traditional boundaries, such as windows, frames, images or tables, and from outside of the designated metadata capture region. The metadata may be captured and shared in a real-time, collaborative environment, to enhance and simplify collaboration and information sharing.
• Metadata may include but is not limited to authorship information, confidentiality information, audit trail information, table cell color, a persistent session ID, text (including characteristics such as content, font size, or line spacing), window information such as title or whether the window is minimized/maximized, or any other application specific information, such as a search string URL for a browser application. Shared metadata may be re-rendered in the most appropriate format with respect to the application from which the metadata is extracted. For example, captured metadata may be presented to appear as if it is on its native application. Application specific support for application windows in the screen share may be provided through application specific plugins. For example, when sharing a Power Point® presentation or Microsoft Office® document, metadata such as author or last modified date may be shared. When sharing a browser window, a mechanism may be used that that introspects the HTML document described by a standard format such as document object model (DOM). When sharing a webpage with persistence and shareable sessions, the URL to access the session may also be sent as metadata.
• FIG. 1 illustrates an embodiment of asystem100 for capturing metadata for screen sharing. Sendingcomputer system101 comprises aGUI screen102. A user may select some or all ofGUI screen102 for a screen share. Screenshare sending module103 invokesmetadata capture module104 to identify the metadata present in the selected region ofGUI screen102. Screenshare sending module103 then applies the user's preferences to the identified metadata, and shares the screen share with the desired metadata with receivingcomputer system106 vianetwork105. The screen share and metadata are displayed to the receiver via screenshare receiving module107, which may comprise a chat or web conferencing application in some embodiments.
• Metadata capture module104 may implement any appropriate GUI object navigation technology to determine properties such as location, size, content, and visibility of objects in theGUI screen102. By determining each object's coordinates in the GUI screen,metadata capture module104 may determine if an object is within the selected region, partially within the region, or relevant to the selected region. Themetadata capture module104 may then capture metadata from objects determined to be within or relevant to the selected region.Metadata capture module104 may implement technology including but not limited to object map from the .NET framework, Rational Function Tester, Spy++, Public Application Programming Interface (API), Microsoft Foundation Class (MFC), or SpyLib in some embodiments.
• FIG. 2 illustrates an embodiment of amethod200 for capturing metadata for screen sharing.FIG. 2 is discussed with reference toFIG. 1. Inblock201, a sender selects a region of aGUI screen102 to be shared as a screen share. The selected region may comprise a portion ofGUI screen102, or all ofGUI screen102. Inblock202, the selected region is analyzed bymetadata capture module104 to identify any metadata present in the selected region, and in any region of theGUI screen102 relevant to the selected region if the selected region is less than theentire GUI screen102. Data not currently visible in theGUI screen102, but that may be made visible by scrolling, may also be captured. Inblock203, the sender's preferences regarding which metadata to share in the screen share are determined and applied to the identified metadata to determine the metadata to be shared by screenshare sending module103. The sender preferences may be entered by the user at the time of the screen share, by, for example, displaying a list of available metadata to the user, or the preferences may comprise a previously stored set of preferences. Inblock204, the screen share is transmitted by screenshare sending module103 to screenshare receiving module107 on receivingcomputer system106 along with the metadata to be shared that was determined inblock203.
• Metadata may be captured from any portion of theGUI screen102 that is determined to be relevant to the region of the GUI screen that is selected inblock201.FIG. 3 shows an embodiment of aGUI screen300 with aregion305 of theGUI screen300 selected for metadata capturing.GUI screen300 comprises various objects, including icons301a-dand windows302-304. The selectedregion305 may be selected in any appropriate manner, for example, clicking and dragging a mouse to form a box on a computer screen, or by making a circular region with finger on a touch-screen device in some embodiments. A selectedregion305 may comprise any portion or all ofGUI screen300. InFIG. 3, selectedregion305 includes portions ofwindows302 and304. Therefore, in a metadata capture for a screen share ofregion305, anymetadata regarding windows302 and304 may be captured, whether the metadata is actually located inside ofregion305 or not, as anymetadata regarding windows302 and304 is relevant to objects located insideregion305. Information regarding icons301a-dandwindow303, however, may not be captured.
• The sender may designate preferences regarding the type and amount of metadata from selectedregion305 to share. This may be accomplished via a sender interface such as is shown inFIG. 4.FIG. 4 illustrates an embodiment of asender interface400 for capturing metadata.Interface400 may be launched when a sender selects a region of a GUI screen for metadata capture; however,interface400 is shown for illustrative purposes only, metadata capture may be performed via any appropriate interface.Interface400 comprisesMyCapture401, which corresponds to the selected region of the sender's GUI screen, such as selectedregion305 ofFIG. 3.MyCapture401 comprises a web browser application having aURL402, asearch string403, and various clickable links such as clickable link404.MyCapture401 also comprises applications MyDisclosures405 andInfoprint Footprint Manager406.Available metadata window407 lists the metadata available inMyCapture401, and allows the sender to use checkboxes to designate which of the listed metadata to send in a screen share ofMyCapture401. Once the user has selected the desired metadata for sending with the screen share, the user may select “Share With Metadata”button408.
• FIG. 5 illustrates an embodiment of areceiver interface500 for receiving ascreen share501 comprising captured metadata.Receiver interface500 comprises a chat window; however,receiver interface500 is shown for illustrative purposes only, ascreen share501 comprising metadata may be received in type of communication.Screen share501 corresponds to MyCapture401 as is shown inFIG. 4, and comprisesselectable URL link502,copyable search string503,clickable link502, andinformation regarding applications505 and506.
• Referring again toFIG. 1, the following code segment shows an embodiment of computer code that may be embodied inmetadata capture module104 to capture all metadata in a window:

• Windows using the ManagedSpyLib (available for free from MSDN):

  using System;

  using System.Text;

  using System.Diagnostics;

  using System.Windows.Forms;

  using System.Collections.Generic;

  using Microsoft.ManagedSpy;

  class Program

  {

  static void Main(string[ ] args)

  {

  Dictionary<int, int> topWindowCounts = new Dictionary<int, int>( );

  foreach (ControlProxy proxy in ControlProxy.TopLevelWindows)

  {

  if (!topWindowCounts.ContainsKey(proxy.OwningProcess.Id))

  {

  topWindowCounts.Add(proxy.OwningProcess.Id, 0);

  }

  topWindowCounts[proxy.OwningProcess.Id]++;

  }

  foreach (int pid in top WindowCounts.Keys)

  {

  Process p = Process.GetProcessById(pid);

  Console.WriteLine(“Process:”+p.ProcessName+“has”+

  topWindowCount[pid].TString( ) + “top level windows”);

  }

  }

  }

• The following code segment shows an embodiment an embodiment of computer code that may be embodied inmetadata capture module104 to capture metadata for a button in a selected region:

• 	foreach (Process p in
  	Process.GetProcessesByName(“WindowsApplication1”))
  	{
  	if (p.Id != Process.GetCurrentProcess( ).Id)
  	{
  	ControlProxy proxy =
  	ControlProxy.FromHandle(p.MainWindowHandle);
  	string val = (string)proxy.GetValue(“MyStringValue”);
  	.....
  	}
  	}

• The above code segments are shown for illustrative purposes only;metadata capture module104 may implement any appropriate metadata recognition technology.
• FIG. 6 illustrates an example of acomputer600 which may be utilized by exemplary embodiments of systems and methods for metadata capture for screen sharing as embodied in software. Various operations discussed above may utilize the capabilities of thecomputer600. One or more of the capabilities of thecomputer600 may be incorporated in any element, module, application, and/or component discussed herein.
• Thecomputer600 includes, but is not limited to, PCs, workstations, laptops, PDAs, palm devices, servers, storages, and the like. Generally, in terms of hardware architecture, thecomputer600 may include one ormore processors610,memory620, and one or more input and/or output (I/O)devices670 that are communicatively coupled via a local interface (not shown). The local interface can be, for example but not limited to, one or more buses or other wired or wireless connections, as is known in the art. The local interface may have additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communications. Further, the local interface may include address, control, and/or data connections to enable appropriate communications among the aforementioned components.
• Theprocessor610 is a hardware device for executing software that can be stored in thememory620. Theprocessor610 can be virtually any custom made or commercially available processor, a central processing unit (CPU), a digital signal processor (DSP), or an auxiliary processor among several processors associated with thecomputer600, and theprocessor610 may be a semiconductor based microprocessor (in the form of a microchip) or a macroprocessor.
• Thememory620 can include any one or combination of volatile memory elements (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and nonvolatile memory elements (e.g., ROM, erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), tape, compact disc read only memory (CD-ROM), disk, diskette, cartridge, cassette or the like, etc.). Moreover, thememory620 may incorporate electronic, magnetic, optical, and/or other types of storage media. Note that thememory620 can have a distributed architecture, where various components are situated remote from one another, but can be accessed by theprocessor610.
• The software in thememory620 may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. The software in thememory620 includes a suitable operating system (O/S)650,compiler640,source code630, and one ormore applications660 in accordance with exemplary embodiments. As illustrated, theapplication660 comprises numerous functional components for implementing the features and operations of the exemplary embodiments. Theapplication660 of thecomputer600 may represent various applications, computational units, logic, functional units, processes, operations, virtual entities, and/or modules in accordance with exemplary embodiments, but theapplication660 is not meant to be a limitation.
• Theoperating system650 controls the execution of other computer programs, and provides scheduling, input-output control, file and data management, memory management, and communication control and related services. It is contemplated by the inventors that theapplication660 for implementing exemplary embodiments may be applicable on all commercially available operating systems.
• Application660 may be a source program, executable program (object code), script, or any other entity comprising a set of instructions to be performed. When a source program, then the program is usually translated via a compiler (such as the compiler640), assembler, interpreter, or the like, which may or may not be included within thememory620, so as to operate properly in connection with the O/S650. Furthermore, theapplication660 can be written as an object oriented programming language, which has classes of data and methods, or a procedure programming language, which has routines, subroutines, and/or functions, for example but not limited to, C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP scripts, FORTRAN, COBOL, Perl, Java, ADA, .NET, and the like.
• The I/O devices670 may include input devices such as, for example but not limited to, a mouse, keyboard, scanner, microphone, camera, etc. Furthermore, the I/O devices670 may also include output devices, for example but not limited to a printer, display, etc. Finally, the I/O devices670 may further include devices that communicate both inputs and outputs, for instance but not limited to, a NIC or modulator/demodulator (for accessing remote devices, other files, devices, systems, or a network), a radio frequency (RF) or other transceiver, a telephonic interface, a bridge, a router, etc. The I/O devices670 also include components for communicating over various networks, such as the Internet or intranet.
• If thecomputer600 is a PC, workstation, intelligent device or the like, the software in thememory620 may further include a basic input output system (BIOS) (omitted for simplicity). The BIOS is a set of essential software routines that initialize and test hardware at startup, start the O/S650, and support the transfer of data among the hardware devices. The BIOS is stored in some type of read-only-memory, such as ROM, PROM, EPROM, EEPROM or the like, so that the BIOS can be executed when thecomputer600 is activated.
• When thecomputer600 is in operation, theprocessor610 is configured to execute software stored within thememory620, to communicate data to and from thememory620, and to generally control operations of thecomputer600 pursuant to the software. Theapplication660 and the O/S650 are read, in whole or in part, by theprocessor610, perhaps buffered within theprocessor610, and then executed.
• When theapplication660 is implemented in software it should be noted that theapplication660 can be stored on virtually any computer readable medium for use by or in connection with any computer related system or method. In the context of this document, a computer readable medium may be an electronic, magnetic, optical, or other physical device or means that can contain or store a computer program for use by or in connection with a computer related system or method.
• Theapplication660 can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a “computer-readable medium” can be any means that can store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.
• More specific examples (a nonexhaustive list) of the computer-readable medium may include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic or optical), a random access memory (RAM) (electronic), a read-only memory (ROM) (electronic), an erasable programmable read-only memory (EPROM, EEPROM, or Flash memory) (electronic), an optical fiber (optical), and a portable compact disc memory (CDROM, CD R/W) (optical). Note that the computer-readable medium could even be paper or another suitable medium, upon which the program is printed or punched, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
• In exemplary embodiments, where theapplication660 is implemented in hardware, theapplication660 can be implemented with any one or a combination of the following technologies, which are well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon data signals, an application specific integrated circuit (ASIC) having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.
• The technical effects and benefits of exemplary embodiments include automatic sending of metadata information in a screen sharing application without interrupting real-time communication.
• The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
• The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims (20)

1. A method for metadata capture for screen sharing of a graphical user interface (GUI) screen, the method comprising:

determining by a screen share sending module a region of the GUI screen to be shared;

analyzing the GUI screen by a metadata capture module to determine metadata relevant to the region to be shared; and

sending a screen share of the region comprising the relevant metadata to a receiver by the screen share sending module.

2. The method ofclaim 1, wherein the metadata relevant to the region is located outside of the region.

3. The method ofclaim 2, wherein the metadata relevant to the region is located outside of a visible region of the GUI screen.

4. The method ofclaim 1, further comprising receiving the screen share of the region comprising the relevant metadata by a screen share receiving module, wherein the received screen share comprises the relevant metadata in the same format in which it is presented in the GUI screen.

5. The method ofclaim 1, further comprising displaying a list of the relevant metadata to a sender of the screen share, receiving a selection of a piece of the relevant metadata from the sender, and sending the screen share of the region with selected metadata.

6. The method ofclaim 1, further comprising applying a set of stored preferences to the relevant metadata, and sending the screen share of the region with metadata designated by the set of stored preferences.

7. The method ofclaim 1, wherein the relevant metadata comprises authorship of a document, confidentiality information, audit trail information, table cell color, a persistent session ID, text content, text font size, text line spacing, window title, whether a window is minimized or maximized, or a search string.

8. The method ofclaim 1, wherein the screen share is sent as part of a web conference or chat session.

9. A system for metadata capture for screen sharing of a graphical user interface (GUI) screen, the system comprising:

a screen share sending module configured to determine a region of the GUI screen to be shared;

a metadata capture module configured to analyze the GUI screen to determine metadata relevant to the region to be shared; and

wherein the screen share sending module is further configured to send a screen share of the region comprising the relevant metadata to a receiver.

10. The system ofclaim 9, wherein the metadata relevant to the region is located outside of the region.

11. The system ofclaim 10, wherein the metadata relevant to the region is located outside of a visible region of the GUI screen.

12. The system ofclaim 9, further comprising a screen share receiving module configured to receive the screen share of the region comprising the relevant metadata, wherein the received screen share comprises the relevant metadata in the same format in which it is presented in the GUI screen.

13. The system ofclaim 9, wherein the screen share sending module is further configured to display a list of the relevant metadata to a sender of the screen share, receive a selection of a piece of the relevant metadata from the sender, and send the screen share of the region with selected metadata.

14. The system ofclaim 9, wherein the screen share sending module is further configured to apply a set of stored preferences to the relevant metadata, and send the screen share of the region with metadata designated by the set of stored preferences.

15. The system ofclaim 9, wherein the relevant metadata comprises authorship of a document, confidentiality information, audit trail information, table cell color, a persistent session ID, text content, text font size, text line spacing, window title, whether a window is minimized or maximized, or a search string.

16. The system ofclaim 8, wherein the screen share is sent as part of a web conference or chat session.

17. A computer program product comprising a computer readable storage medium containing computer code that, when executed by a computer, implements a method for metadata capture for screen sharing of a graphical user interface (GUI) screen, wherein the method comprises:

determining by a screen share sending module a region of the GUI screen to be shared;

analyzing the GUI screen by a metadata capture module to determine metadata relevant to the region to be shared; and

sending a screen share of the region comprising the relevant metadata to a receiver by the screen share sending module.

18. The computer program product according toclaim 15, wherein the metadata relevant to the region is located outside of the region.

19. The computer program product according toclaim 15, further comprising receiving the screen share of the region comprising the relevant metadata by a screen share receiving module, wherein the received screen share comprises the relevant metadata in the same format in which it is presented in the GUI screen.

20. The computer program product according toclaim 15, further comprising displaying a list of the relevant metadata to a sender of the screen share, receiving a selection of a piece of the relevant metadata from the sender, and sending the screen share of the region with selected metadata.

Images