US20050273761A1 - Freehand system and method for creating, editing, and manipulating block diagrams - Google Patents

Abstract

A method of integrating freehand user input into a block diagram environment is disclosed. The freehand user input is a user's approximation of a diagram component or feature of a component which is received by the block diagram environment and compared to multiple patterns stored in a storage location. The storage location holds patterns of block diagram components and block diagram component features. The freehand user input may be displayed, superimposed on a block diagram being shown to the user. Upon the freehand user input being matched to one of the patterns representing a block diagram component or feature of a component, the freehand user input is replaced on the displayed block diagram with an electronic device drawn rendering of the matched diagram feature component or feature of a component. Partial matches of the user drawn input may result in a menu of choices being presented to the user for selection.

Description

FIELD OF THE INVENTION
• The illustrative embodiment of the present invention relates generally to block diagrams and more particularly to the use of freehand input data in the creation, editing and manipulation of block diagrams.
BACKGROUND
• Freehand user input such as input from a mouse or optical pen operated by a user of a computer system or other electronic device is accepted by many different types of applications. For example, freehand input from a mouse is accepted by Microsoft's Paint program and enables the user to construct freehand drawings on a display surface connected to a computer system or other electronic device. One type of freehand user input is Graffiti®, a text recognition system which is used in conjunction with the Palm OS® from PalmSource of Sunnyvale, Calif. in handheld PDAs (Personal Digital Assistants). The acceptance of the freehand input enables an application to interact with the user in an easy and well understood manner and enables the user to customize the displayed output of the particular application.
• Block diagram environments such as Simulink(g from The MathWorks of Natick, Mass., enable a user to construct models of dynamic systems. These models include graphical references to system components which may be used to simulate the system operations. Current block diagram environment tools enable a user to drag items from pre-defined templates into a block diagram. The components being dragged into the block diagram include attributes enabling the component to be modeled in the system during the simulation of the diagram. Unfortunately, conventional block diagram environments do not allow a user to sketch in system components using freehand input. The result is a lengthy series of drag and drop operations that the user is required to perform when constructing the block diagram. Additionally, the user must also first search to find the desired component in the proper template.
SUMMARY OF THE INVENTION
• The illustrative embodiment of the present invention provides a method of integrating freehand user input into a block diagram environment. The freehand user input is a user's approximation of a diagram component or feature of a component which is received by the block diagram environment and compared to multiple patterns stored in a storage location. The storage location holds patterns of block diagram components and block diagram component features. The freehand user input may be displayed, superimposed on a block diagram being shown to the user. Upon the freehand user input being matched to one of the stored patterns representing a block diagram component or feature of a component, the freehand user input is replaced on the displayed block diagram with an electronic device drawn rendering of the matched diagram feature component or feature of a component. The component is also added to the block diagram model data. In one implementation, the user input displayed on the block diagram and the electronic device drawn rendering of the feature or component may be simultaneously displayed pending user confirmation of the selection. Partial matches of the user drawn input may result in a menu of choices being presented to the user for selection.
• In one embodiment in an electronic device holding a block diagram environment with at least one block diagram, the electronic device also interfaced with a display surface displaying a block diagram, a method receives freehand user input entered with a pointing device into the block diagram. The method also compares the user input to at least one of multiple stored patterns of block diagram components and features of block diagram components. In the event of a match, the method adds the block diagram component or block diagram component feature to the block diagram model data and updates the displayed block diagram with a program-drawn block diagram component/feature representing the input data.
• In another embodiment, in an electronic device interfaced with a display surface, a system includes a block diagram environment with at least one block diagram displayed on the display surface. The system also includes a pointing device interfaced with the electronic device, the pointing device being used to transmit freehand user input to the block diagram displayed on the display surface. The system additionally includes a storage location holding multiple block diagram component patterns and block diagram component feature patterns to which the freehand user input is compared. Matching block diagram components and component features from the storage location are rendered on the electronic device drawn diagram in the event of a match.
• In a different embodiment in an electronic device holding a block diagram environment with at least one block diagram, the electronic device also interfaced with a display surface displaying a block diagram, a method receives freehand user input entered with a pointing device into the block diagram. The method also analyzes the user input to identify the user input as a block diagram component or block diagram component feature. In the event of an identification, the method adds the block diagram component or block diagram component feature to the block diagram model data and updates the displayed block diagram with a program-drawn block diagram component/feature representing the input data.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1 is a block diagram of an environment suitable for practicing the illustrative embodiment of the present invention;
• FIG. 2A is a flow chart of the sequence of steps followed by the illustrative embodiment of the present invention to compare freehand user input to stored diagram components and features;
• FIG. 2B is a flowchart examining the comparison and normalization process in greater detail;
• FIG. 3A is a view of the illustrative embodiment of the present invention showing received freehand user input displayed in a block diagram environment;
• FIG. 3B is a view of the illustrative embodiment of the present invention showing a electronic device drawn diagram component based on the freehand user input;
• FIG. 4A is a view of the illustrative embodiment of the present invention showing freehand user input being appended to a previously drawn electronic device generated diagram component;
• FIG. 4B is a view of the illustrative embodiment of the present invention showing a electronic device drawn replacement of the freehand user inputFIG. 4A;
• FIG. 5A is a view of the illustrative embodiment of the present invention showing an initial display of the received freehand user input appended to a feedback loop in a block diagram;
• FIG. 5B shows an electronic device rendering of the partial component depicted in the freehand user input ofFIG. 5A;
• FIG. 5C is a view of the illustrative embodiment of the present invention showing a displayed menu of choices of finished components possibly completing the previously drawn component ofFIG. 5A andFIG. 5B;
• FIG. 5D is a view of the illustrative embodiment of the present invention showing the view ofFIG. 5C wherein the user has selected one of the possibilities displayed in the menu by entering additional freehand input corresponding to a menu choice;
• FIG. 5E shows the final electronic device drawn rendering of the additional component ofFIG. 5D.
• FIG. 6A shows a user drawn output being added to the diagram ofFIG. 5E;
• FIG. 6B shows a electronic device drawn replacement for the user input displayed inFIG. 6A;
• FIG. 7A shows freehand user input indicating a request for an additional view of the diagram being displayed; and
• FIG. 7B shows a zoomed-out viewer superimposed on a portion of the block diagram in response to the received freehand user input displayed inFIG. 7A.
• DETAILED DESCRIPTION
• The illustrative embodiment of the present invention allows a user to construct, modify, and manipulate block diagram elements using a pointing device. A block diagram environment displaying a block diagram receives freehand user input from a pointing device such as an optical pen or a mouse. The freehand user input is entered by patterns and gestures and is not limited to displayed elements on a template as is found in conventional drag and drop systems. The entered patterns and symbols are compared to stored patterns of block diagram components and features of components and used as the basis for electronic device drawn diagram components or component features which are added to the block diagram. By allowing freehand entry of input data in such a manner, the illustrative embodiment of the present invention greatly expands and simplifies the creation and modification of block diagram features and components.
• FIG. 1 depicts an environment suitable for practicing the illustrative embodiment of the present invention. Anelectronic device2 holds ablock diagram environment4. Theelectronic device2 may be a personal computer, workstation, laptop, server, mainframe, PDA or some other type of electronic device equipped with a processor and able to support theblock diagram environment4. Theblock diagram environment4 may be a block diagram environment such as that found in Simulink® from The MathWorks, Inc. of Natick, Mass., and Labview® from National Instruments of Austin, Tex. Theblock diagram environment4 includes a least one set of block diagram model data6 for a block diagram. The block diagram model data6 may be a block diagram model of a system. Also included on theelectronic device2 is astorage location8. Thestorage location8 includes a stored collection of block diagram components and block diagramcomponent feature patterns10. Those skilled in the art will recognize thatstorage location8 may also be located at other locations accessible to theblock diagram environment4. For example, thestorage location8 may be located on a remote server in a distributed environment. A user12 manipulates apointing device18 which is interfaced with theelectronic device2. The pointing device12 may be a mouse, trackball, joystick, mousepad, and light pen. Adisplay14 holds a rendering of block diagram16. Input received by theblock diagram environment4 from the user manipulating thepointing device18 is compared to the block diagram component/feature patterns10 which are used to render a electronic device drawn version of the user input on thedisplay14.
• FIG. 2A depicts the sequence of steps followed by the illustrative embodiment of the present invention to match freehand user input to stored block diagram components/features of components. The sequence begins when the user manipulates thepointing device18 to send input to the displayed block diagram16 via the block diagram environment4 (step30). The technique used for the user input varies with the type of input device being utilized. For example, if a mouse is being used, the user may drag a cursor around the displayed block diagram16 in the outline of the shape of the component/feature of a component that the user is attempting to add to the block diagram16. Similarly, if an optical pointing device is being used, the user may gesture at the diagram in the shape of the component/feature of a component. The input data is received at the electronic device2(step32) and a graphical representation of the input is displayed in the rendering of the block diagram16 (step34). For example, a user may draw a shape in the displayed block diagram using a mouse connected to theelectronic device2. The user12 may position the mouse cursor at the appropriate spot in thedisplay14. By depressing a mouse button and moving the mouse, a line is rendered in the displayed block diagram by theblock diagram environment4. The rendering is the result of theblock diagram environment4 intercepting and handling the mouse messages generated by the depression of the button on the mouse. The handling of mouse messages is well-known to those with experience in the Windows® programming environment (e.g.: WM_RBUTTONDOWN, WM_LBUTTONDOWN). The message handler cross references the coordinates associated with the intercepted message with the coordinates of the model data being displayed on the display in order to orient the input data with the block diagram data.
• The shape of the user input as captured is then compared to stored patterns of block diagram components and features of block diagram components (step36). The comparison may be made when the input data ceases, for example, through a WM_LBUTTONUP message or the failure to extend a line being rendered in response to the movements of an optical pen. Alternatively, the illustrative embodiment of the present invention may attempt comparisons based on a time parameter. A determination is made for each comparison as to whether a definitive match has been determined (step37). The determination as to what percentage of the user input must correspond to the stored pattern to constitute a definitive match in the illustrative embodiment is an implementation choice. The illustrative embodiment of the present invention checks for points of correspondence between the user input and the stored diagram component/feature patterns10. For some shapes, the borders of enclosed input shapes may be compared to the stored diagram component/feature patterns10. For other shapes, both the borders, interior and/or exterior shapes may be checked for correspondence. The size of the input shapes may be normalized against the stored features/patterns with a percentage of deviation being allowed from the stored component/feature size. The stored diagram components/feature patterns10 may be stored in a database including a size attribute associated with the component/feature. The comparison may only compare the entered pattern to stored components/features within the deviation range. Distances and angles of the input data are compared to the stored/components features accounting for pre-defined amount of size deviation.
• The process of determining a definitive match while normalizing for the size differential between the input data and the stored patterns is illustrated in greater detail in the flowchart ofFIG. 2B. After receiving the freehand user input (step39), the illustrative embodiment of the present invention analyzes the received input to identify attributes of the input data (e.g.:length of line, direction of line, angle between lines, etc.) (step41). The stored components/features of components are retrieved and similarly analyzed (step43). Alternatively, the stored patterns may have already been analyzed in which case the attributes for the stored patterns are retrieved. The input attributes are then compared to the stored attribute patterns while normalizing for size differential (step45). The normalization process is necessary since in most cases the user's freehand input will not be to exactly the same scale as the stored patterns. Any matches from the comparison are identified (step47).
• In the event of a definitive match, an electronic device drawn component or feature of a component corresponding to the user input data is rendered in the block diagram16 (step38). Alternatively, if there is not a definitive match, but rather several possibilities, the user may be presented with a menu of possible block diagram components or features of components which may correspond to component/feature the user was attempting to add to the block diagram (step40). Following a user selection of the appropriate component or feature (step42) the electronic device adds the input data to the block diagram model data and draws the component or feature of the component in the rendering of the block diagram16(step44).
• The visual effect of the matching of freehand user input to the stored patterns is depicted inFIGS. 3A and 3B.FIG. 3A depicts aview50 of a block diagram environment in the illustrative embodiment of the present invention in which a user has enteredfreehand input52 representing a triangular shape. The illustrative embodiment of the present invention matches the freehand user input to a stored pattern in the collection of patterns of block diagram components and component features. Theblock diagram environment4 generates theview50 ofFIG. 3B showing an electronic device renderedgain block54.
• The illustrative embodiment of the present invention may also be used to add additional components to an already existing block diagram as depicted inFIGS. 4A andFIG. 4B.FIG. 4A depicts theview50 and computer generatedgain block54 ofFIGS. 3A and 3B to which freehand user input of a rough approximation of afeedback signal56 has been added to the electronic device drawn components previously appearing on the display. As before, the freehand user input for thefeedback signal approximation56 is compared to the diagram component andfeature patterns10 that are stored at alocation8 accessible to theblock diagram environment4. InFIG. 4B a computer renderedfeedback signal58 is then generated to replace the user drawninput56. Depending upon the implementation, the electronic device drawn feedback signal may be superimposed over the userfreehand input signal56 until such time as the user indicates a confirmation of the substituted element. Those skilled in the art will recognize that such confirmation may be given in a number of different manners such as by clicking on the electronic device drawn component or component feature or clicking a control button in a popup dialog box which requests verification.
• The sequence of views generated by the illustrative embodiment of the present invention to choose a component from among many components is illustrated inFIGS. 5A through 5E.FIG. 5A shows aview50 with the electronic device generatedgain block54 andfeedback signal58 as well as additional userfreehand input60 of a circular shape near the end of the feedback signal.FIG. 5B shows theview50 including the components previously discussed inFIG. 5A with theuser input60 replaced by an electronic device generatedcircle62. The electronic device generatedcircle62 represents a partially completed block diagram component.FIG. 5C depicts theview50 with thegain block54,feedback signal58 andcircle62 as well as additional choices displayed on amenu64. Themenu64 displays a list of potential matches for the partially completedelement62.FIG. 5D depicts theprevious view50 as well as an additionalfreehand feature66 added by the user to the previously displayed computer renderedcircle62. The additionalfreehand feature66 matches one of thecomponents67 displayed in themenu64.FIG. 5E shows the electronic device generated replacement of the user drawnfeature66. The result is asum block68 added on to the end of thefeedback signal58.
• A user may also add additional components to previously stored diagrams using the illustrative embodiment of the present invention. For example,FIG. 6A depictsview50 including thegain block54,feedback signal58, andsum block68 to which the user indicates a desire to add an output port through the creation of a freehand user input. Those skilled in the art will recognize that the previously stored block diagram may be retrieved from any location accessible to theblock diagram environment4.FIG. 6B shows the result after matching the freehand user input to a stored pattern, the generation of a computer renderedoutput port72.
• Those skilled in the art will recognize that alternate forms of analysis rather than an explicit comparison may also be used by the illustrative embodiment of the present invention to identify the freehand user input. For example, a neural net may be used to determine the type of block diagram component/feature the user is attempting to add via freehand user input. The neural net may attempt to analyze small pieces of user input in parallel to identify the desired component/feature the user is attempting to add to the block diagram.
• The illustrative embodiment of the present invention may also be used to call up block diagram tools through the use of freehand gestures.FIG. 7A depicts aview80 of a portion of a block diagram82. A user sketch enclosing the lower left hand corner of the displayedview80 creates arough box84. The gesture is evaluated by comparing it to stored patterns and results in the calling of a zoomed-out viewer showing the entire block diagram or subsystem86 (of which themain view80 is only displaying a portion) superimposed in the lower left corner of the view.
• Similar to the procedure used to call up block diagram tools, a pre-defined input may also be used to cause the deletion of a component. For example, an “x” written over a diagram component may be interpreted as a delete signal. Similarly, pre-defined input such as a symbol may result in the insertion of a complete block or subsystem into the block diagram.
• In one aspect of the illustrative embodiment of the present invention, a user may switch between an editing mode and an interpreted mode in theblock diagram environment4. The editing mode allows the user to make changes to the block diagram without the pattern comparison taking place. In the interpreted mode, once the user activates the mode, pattern comparison is performed in response to the user input. This ability to toggle the interpreted mode on and off prevents unwanted components and component features from being inadvertently added to a block diagram.
• In another aspect of the illustrative embodiment of the present invention, the user may teach the block diagram environment to interpret specific freehand user input gestures. User input may be entered and the environment subsequently instructed to interpret the entered input in a certain manner and perform an associated action. For example, the user may enter a learning mode and enter freehand input. After entering the input in the learning mode, the user indicates to the block diagram environment a specific action (e.g.: perform function, add component) that is to be associated with the input. The entered pattern and its association is saved , and subsequent user entries of that pattern in interpreted mode result in the specified action being performed.
• Although the examples contained herein have been discussed with reference to a block diagram environment, it should be appreciated that the present invention may also be implemented in other design environments. For example, the method of accepting and handling freehand user input that have been discussed above may be implemented in a user interface design system where various controls are added to a user interface in response to the received freehand user input. The illustrative embodiment of the present invention may also be practiced in a software diagram environment such as that found in Stateflow® from The MathWorks, Inc. of Natick, Mass. or Unified Modeling Language (UML) environments.
• Since certain changes may be made without departing from the scope of the present invention, it is intended that all matter contained in the above description or shown in the accompanying drawings be interpreted as illustrative and not in a literal sense. Practitioners of the art will realize that the system configurations depicted and described herein are examples of multiple possible system configurations that fall within the scope of the current invention. Likewise, the sequence of steps utilized in the illustrative flowcharts are examples and not the exclusive sequence of steps possible within the scope of the present invention.

Claims (52)

1. In an electronic device holding a block diagram environment with at least one block diagram, said block diagram being a model of a system, said electronic device interfaced with a display surface displaying said block diagram, a method, comprising:

receiving freehand user input entered with a pointing device into said block diagram;

comparing said user input to at least one of a stored plurality of patterns of at least one of block diagram components and block diagram component features;

adding said at least one of said block diagram components and block diagram component features to said model, said adding occurring as a result of a matching comparison between said freehand user input and at least one of said plurality of patterns; and

rendering programmatically with the block diagram environment at least one of said block diagram components and said block diagram component features in said diagram.

2. The method ofclaim 1, further comprising:

displaying said freehand user input in said diagram prior to said rendering.

3. The method ofclaim 2, further comprising:

replacing said displayed freehand user input with a programmatic rendering of said matching pattern.

4. The method ofclaim 1 wherein said freehand user input is added to an existing programmatically rendered diagram component in said diagram.

5. The method ofclaim 4, further comprising:

modifying said existing programmatically rendered pattern by programmatically rendering at least one additional feature based on matching said user input to a stored pattern.

6. The method ofclaim 1 wherein said freehand user input is one of a complete diagram component and complete diagram component feature.

7. The method ofclaim 1 wherein said freehand user input is one of a partial diagram component and partial diagram component feature.

8. The method ofclaim 1, further comprising:

providing a choice of at least two of diagram components and diagram component features to said user prior to said rendering, said choice prompting a user for a selection of one of said at least two diagram components and diagram component features.

9. The method ofclaim 8, further comprising:

rendering programmatically at least one of said diagram components and diagram component features based on said selection.

10. The method ofclaim 8 wherein the identity of the at least two diagram components and diagram component features appearing in said choice is based on the context of said diagram.

11. The method ofclaim 1 wherein said freehand user input is a symbol.

12. The method ofclaim 1 wherein said diagram is a block diagram.

13. The method ofclaim 1, further comprising:

manipulating at least one of a diagram component and diagram component feature displayed in said diagram as a result of said user input.

14. The method ofclaim 1, further comprising:

deleting at least one of a diagram component and diagram component feature displayed in said diagram as a result of said user input.

15. The method ofclaim 1, further comprising:

providing diagramming tools to a user in said block diagram environment in response to said freehand user input.

16. The method ofclaim 15 wherein said tool is a zoomed-out viewer superimposed over at least a portion of said displayed diagram.

17. The method ofclaim 1, further comprising:

providing at least one visually displayed symbol associated with an attribute type;

selecting said symbol associated with an attribute type with said pointing device;

subsequently selecting with said pointing device at least one of a diagram component and diagram component feature displayed in said diagram; and

assigning programmatically said attribute type associated with said symbol to the selected at least one of a diagram component and diagram component feature.

18. The method ofclaim 1, further comprising:

displaying said freehand user input on said display surface;

superimposing the programmatically rendered pattern over the displayed freehand user input; and

removing said freehand user input from said display surface following a user indication of the correctness of the programmatically rendered at least one of said diagram component and diagram component feature.

19. The method ofclaim 1 wherein said comparing occurs only after said user selects an interpreted mode option for said block diagram environment.

20. In an electronic device interfaced with a display surface, a system comprising:

a block diagram environment with at least one block diagram displayed on said display surface;

a pointing device interfaced with said electronic device, said pointing device used to transmit freehand input from a user to said block diagram displayed on said display surface; and

a storage location holding a plurality of block diagram component patterns and diagram component feature patterns, said user input being compared to said plurality of block diagram component patterns and diagram component feature patterns, said block diagram environment programmatically rendering one of a diagram component and diagram component feature in said block diagram based on a matching comparison between said freehand user input and at least one of said plurality of patterns.

21. The system ofclaim 20 wherein said rendering superimposes said programmatically rendered one of a diagram component and diagram component feature over said user input in said diagram displayed on said display surface.

22. The system ofclaim 20 wherein said user input represents one of a partial diagram component and partial diagram component feature.

23. The system ofclaim 20 wherein said user input represents one of a complete diagram component and complete diagram component feature.

24. The system ofclaim 20, further comprising:

a menu of choices displayed to said user prior to the rendering of said one of a diagram component and diagram component feature, said choices representing possible diagram components and diagram component features matching said user input.

25. The system ofclaim 24 wherein a user selects one of the choices, the selected choice then being rendered as a matching comparison.

26. The system ofclaim 20 wherein said user input is of an additional feature to a previously rendered diagram component.

27. The system ofclaim 20 wherein said pointing device is one of a mouse, trackball, joystick, mousepad, and light pen.

28. The system ofclaim 20 wherein said block diagram environment is a block diagram environment and said block diagram is a block diagram.

29. In an electronic device holding a block diagram environment with at least one block diagram, said electronic device interfaced with a display surface displaying said diagram, a medium holding executable steps for a method, said method comprising:

comparing freehand user input entered with a pointing device into said diagram to at least one of a stored plurality of patterns; and

rendering programmatically with the block diagram environment one of a diagram component and diagram component feature in said diagram, said diagram component based on a matching comparison between said freehand user input and at least one of said plurality of patterns.

30. The medium ofclaim 29 wherein said method further comprises:

displaying said freehand user input in said diagram prior to said rendering.

31. The medium ofclaim 30 wherein said method further comprises:

replacing said displayed freehand user input with said programmatically rendered one of a diagram component and diagram component feature.

32. The medium ofclaim 29 wherein said freehand user input is added to an existing electronic device drawn diagram component in said diagram.

33. The medium ofclaim 32 wherein said method further comprises:

modifying said existing programmatically rendered component by programmatically rendering at least one additional feature based on matching said user input to a stored pattern.

34. The medium ofclaim 29 wherein said freehand user input is one of a complete diagram component and complete diagram component feature.

35. The medium ofclaim 29 wherein said freehand user input is one of a partial diagram component and partial diagram component feature.

36. The medium ofclaim 29 wherein said method further comprises:

providing a choice of at least two diagram components and diagram component features to said user prior to said rendering, said choice prompting a user for a selection of one of said at least two diagram components and diagram component features.

37. The medium ofclaim 36, wherein said method further comprises:

rendering one of said diagram components and diagram component features based on said selection.

38. The medium ofclaim 36 wherein the identity of the at least two of said diagram components and diagram component features appearing in said choice is based on the context of said diagram.

39. The medium ofclaim 29 wherein said freehand user input is a symbol.

40. The medium ofclaim 29 wherein said diagram is a block diagram.

41. The medium ofclaim 29 wherein said method further comprises:

manipulating at least one of a diagram component and diagram component feature displayed in said diagram as a result of said user input.

42. The medium ofclaim 29 wherein said method further comprises:

deleting at least one of a diagram component and diagram component feature displayed in said diagram as a result of said user input.

43. The medium ofclaim 29 wherein said method further comprises:

providing diagramming tools to a user in said block diagram environment in response to said freehand user input.

44. The medium ofclaim 43 wherein said tool is a zoomed-out viewer superimposed over at least a portion of said displayed diagram.

45. The medium ofclaim 29 wherein said method further comprises:

providing at least one visually displayed symbol associated with an attribute type;

selecting said symbol associated with an attribute type with said pointing device;

subsequently selecting with said pointing device at least one of a diagram component and diagram component feature displayed in said diagram; and

assigning programmatically said attribute type associated with said symbol to the selected at least one of a diagram component and diagram component feature.

46. In an electronic device holding a block diagram environment with at least one block diagram, said block diagram being a model of a system, said electronic device interfaced with a display surface displaying said block diagram, a method, comprising:

receiving freehand user input entered with a pointing device into said block diagram;

analyzing said user input to identify said user input as at least one of a block diagram component and block diagram component feature;

adding said at least one of said block diagram components and block diagram component features to said model, said adding occurring as a result of said analysis; and

rendering programmatically at least one of said block diagram components and said block diagram component features in said diagram.

47. The method ofclaim 46 wherein said block diagram component is a block diagram subsystem.

48. The method ofclaim 46 wherein said analysis is performed using a neural net.

49. In an electronic device holding a block diagram environment with at least one block diagram, said block diagram being a model of a system, said electronic device interfaced with a display surface displaying said block diagram, a method, comprising:

receiving freehand user input entered with a pointing device into said block diagram environment executing in a learning mode;

receiving a user indication associating said freehand user input with an action;

storing said user input and its associated action as one of a plurality of patterns;

receiving freehand user input entered with a pointing device into said block diagram environment executing in an interpretive mode;

comparing said user input to at least one of a stored plurality of patterns of at least one of block diagram components and block diagram component features; and

performing the associated action, said performing occurring as a result of a matching comparison between said freehand user input and at least one of said plurality of patterns.

50. In an electronic device holding a block diagram environment with at least one block diagram, said block diagram being a model of a system, said electronic device interfaced with a display surface displaying said block diagram, a medium holding executable steps for a method, said method comprising:

receiving freehand user input entered with a pointing device into said block diagram environment executing in a learning mode;

receiving a user indication associating said freehand user input with an action;

storing said user input and its associated action as one of a plurality of patterns;

receiving freehand user input entered with a pointing device into said block diagram environment executing in an interpretive mode;

comparing said user input to at least one of a stored plurality of patterns of at least one of block diagram components and block diagram component features; and

performing the associated action, said performing occurring as a result of a matching comparison between said freehand user input and at least one of said plurality of patterns.

51. In an electronic device interfaced with a display surface, a system comprising:

a software diagram environment with at least one software diagram displayed on said display surface;

a pointing device interfaced with said electronic device, said pointing device used to transmit freehand input from a user to said software diagram displayed on said display surface; and

a storage location holding a plurality of software diagram component patterns and diagram component feature patterns, said user input being compared to said plurality of software diagram component patterns and diagram component feature patterns, said software diagram environment programmatically rendering one of a diagram component and diagram component feature in said software diagram based on a matching comparison between said freehand user input and at least one of said plurality of patterns.

52. The system ofclaim 51 wherein said software diagram is one of a Stateflow® diagram and Unified Modeling Language (UML) diagrams.

Images