Repository files navigation

The Snapshot Debugger lets you inspect the state of a running cloudapplication, at any code location, without stopping or slowing it down. It’s notyour traditional process debugger but rather an always on, whole app debuggertaking snapshots from any instance of the app.

You can use the Snapshot Debugger with any deployment of your application,including test, development, and production. The debugger typically adds lessthan 10ms to the request latency only when the application state is captured. Inmost cases, this isn’t noticeable by users.

This project has been archived and is no longer supported. There will be nofurther bug fixes or security patches. The repository can be forked by usersif they want to maintain it going forward.

This OSS solution provides an alternative to Cloud Debugger. Cloud Debugger wasdeprecated on May 16, 2022 and the service was shut down on May 31, 2023

• Python 2.7 is not supported

SeeCOMMAND_REFERENCE.md.

Install the debugger CLI in your local environment, or in your Cloud Shell$HOME directory. SeeUsing CloudShell for information onusing Cloud Shell.

python3 -m pip install snapshot-dbg-cli

Note: When using the Snapshot Debugger in Cloud Shell you will be asked toAuthorize using your account credentials.

There are two options to run the CLI once the pip install has been completed.

As part of the pip install process, a script,snapshot-dbg-cli will beinstalled which can be used to run the CLI.

Example running thelist_debuggees command:

snapshot-dbg-cli list_debuggees

NOTE: To run the script this way from any directory, you must ensure thescript's install directory is in your PATH. Pip should emit a warning if theinstall location is not in the PATH, and also provide the install location inthis case, so that you can add it to your PATH.

Example running thelist_debuggees command:

python3 -m snapshot_dbg_cli list_debuggees

To complete the setup you’ll need to have the following permissions in yourGoogle Cloud project. If you are anOwner orEditor of the Google Cloudproject, then you have these permissions.

• firebase.projects.create
• firebase.projects.update,
• firebasedatabase.instances.create
• firebasedatabase.instances.get
• resourcemanager.projects.get
• serviceusage.services.enable
• serviceusage.services.get

For more information on permissions and roles in Google projects, readUnderstandingroles

Snapshot Debugger requires Python 3.6 or above and thegcloud CLI. If you areworking in Cloud Shell, you already have Python andgcloud installed.

In addition, the environment should already configured correctly by default. Youcan verify this by running the following commands:

1. gcloud config get-value project
2. gcloud config get-value account

NOTE: When running the cli, you may encounter a popup warning you thatgcloud is requesting your credentials to make a GCP API call. You'll need toclickAUTHORIZE to proceed.

The Snapshot Debugger CLI requiresPython3.6 or newer.

The Snapshot Debugger CLI depends on thegcloud CLI. To install thegcloudCLI, follow theseinstructions. Ifyou already have thegcloud CLI installed, rungcloud components update toupdate all of your installed components to the latest version.

1. Rungcloud auth login, be sure to use the account that has permissions onthe Google Cloud project you are working on.
2. Rungcloud config set project PROJECT_ID. Where PROJECT_ID is the projectyou want to use. The Snapshot Debugger CLI always acts on the currentgcloud configured project.

The Snapshot Debugger CLI and agents use a Firebase Realtime Database (RTDB) tocommunicate.

If you already use Firebase in your project, skip to theSet up the FirebaseRTDB section.

1. Add Firebase to your project:

   https://console.firebase.google.com/?dlAction=MigrateCloudProject&cloudProjectNumber=PROJECT_ID

   Where PROJECT_ID is your project ID

2. Select yourProject IDand clickContinue. If you have billing enabled in your project, thepay-as-you-go Blaze plan is selected, otherwise the free Spark plan isselected. If you have billing enabled and want to use the free Spark plan,set up a new project without billing enabled.

   Note: The Snapshot Debugger uses the Firebase RTDB service. Most users' usagewill be low enough to remain under thefree usagelimits.

3. If you are using the pay-as-you-go Blaze plan, clickConfirm plan. Youare not prompted to confirm if you are on the free Spark plan.

4. Read the information underA Few things to remember when adding Firebase toa Google Cloud project then clickContinue.

5. Toggle theenable analytics option to enable or disable Google Analyticsfor Firebase. Google Analytics isn't required for Debugger use.

6. ClickContinue.

7. ClickGet Started.

8. Click on your project.

The instructions are slightly different depending on whether you are on theSpark or Blaze billing plan. Follow the steps in the following section for theplan you have.

You can check what billing plan is in effect for your project on the Firebase Usage & Billing page:

https://console.firebase.google.com/project/PROJECT_ID/usage/details

Where PROJECT_ID is your project ID

This will instruct the debugger CLI to create and use a database with the namePROJECT_ID-cdbg

1. Runsnapshot-dbg-cli init.
2. The output resembles the following:

Project 'test-proj' is successfully configured with the Firebase RealtimeDatabase for use by Snapshot Debugger.The full database information is below. If you have specified a custom databaseID the url below is the one you'll need to specify when using the other clicommands.  name:         projects/23498723497/locations/us-central1/instances/test-proj-cdbg  project:      projects/23498723497  database url: https://test-proj-cdbg.firebaseio.com  type:         USER_DATABASE  state:        ACTIVE

Note: The information printed by theinit command can be accessed from withinyour Firebase project. It’s safe to run thesnapshot-dbg-cli init commandmultiple times to view this information.

This will instruct the CLI to create and use a database with the namePROJECT_ID-default-rtdb. It will only be created if it does not currentlyexist.

1. Runsnapshot-dbg-cli init --use-default-rtdb
2. The output resembles the following:

Project 'test-proj' is successfully configured with the Firebase RealtimeDatabase for use by Snapshot Debugger.The full database information is below. If you have specified a custom databaseID the url below is the one you'll need to specify when using the other clicommands.  name:         projects/23498723497/locations/us-central1/instances/default  project:      projects/23498723497  database url: https://test-proj-default-rtdb.firebaseio.com  type:         USER_DATABASE  state:        ACTIVE

Note: The information printed by theinit command can be accessed from withinyour Firebase project. It’s safe to run thesnapshot-dbg-cli init --use-default-rtdb command multiple times to view this information.

By default,snapshot-dbg-cli init will create a Firebase Realtime Database inus-central1. It is possible to create and use a database in any regionsupported by Firebase Realtime Database. Seesupported RTDB locations.

Setting up your database in a non-default region comes with some trade-offs:

• As a positive, you get to control where your snapshot data will be stored.This may be important for compliance reasons.
• As a negative, the vsCode extension and agents will be unable toautomatically find the database. The database URL will need to be providedexplicitly via configuration, see the following for details:
  • Configuring the Java Agent
  • Configuring the Python Agent
  • Configuring the Node.js Agent
  • Configuring the VsCode Extension

You can set up your database in a non-default location as follows:

snapshot-dbg-cli init --location={YOUR_LOCATION}

For example, you may want to set up your database in Belgium, and so would runsnapshot-dbg-cli init --location=europe-west1

Make note of the database URL provided in the command output; you will need toprovide this to your debug agent(s) and the vsCode plugin.

Working examples of using the Snapshot Debugger with Java, Python and Node.jsapplications across different Google Cloud environments can be found in:

• snapshot-debugger/samples

See the following for agent specific documentation:

• Java
• Node.js
• Python

You create a breakpoint (snapshot or logpoint) on debuggees. Debuggees representinstances of the running application. In general all instances of the sameversion of the application will have the same debuggee ID, and breakpoints seton a debuggee will be installed on all running instances of it.

Run the following command

snapshot-dbg-cli list_debuggees --include-inactive

The output resembles the following:

Name           ID          Description                              Last Active           Status-------------  ----------  ---------------------------------------- --------------------  --------test-app - v2  d-8dd7f149  node index.js module:test-app version:v2 2022-12-16T21:45:07Z  ACTIVEtest-app - v1  d-24abc4f1  node index.js module:test-app version:v1 2022-10-16T21:45:07Z  INACTIVE

Snapshots capture local variables and the call stack at a specific line locationin your app's source code. You can specify certain conditions and locations toreturn a snapshot of your app's data, and view it in detail to debug your app.

Set snapshots with the following command:

snapshot-dbg-cli set_snapshot index.js:21 --debuggee-id d-8dd7f149

Where:

• index.js:21 is thefile:line for the snapshot
• d-8dd7f149 is the debuggee ID

A snapshot condition is a simple expression in the app's language that mustevaluate to true for the snapshot to be taken. Snapshot conditions are evaluatedeach time the line is executed, by any instance, until the condition evaluatesto true or the snapshot times out.

Use of snapshot conditions is optional.

The condition is a full boolean expression that can include logical operators.Conditions are specified using the--condition flag of theset_snapshotscommand.

Example:

snapshot-dbg-cli set_snapshot index.js:26 --debuggee-id d-8dd7f149 --condition="ultimateAnswer <= 42 && foo==bar"

You can use the following language features to express conditions:

Most Java expressions are supported, including:

• Local variables:a == 8.
• Numerical and boolean operations:x + y < 20.
• Instance and static fields:this.counter == 20,this.myObj.isShutdown,myStatic, orcom.mycompany.MyClass.staticMember.
• String comparisons with the equality operator:myString == "abc".
• Function calls. Only read-only functions can be used. For example,StringBuilder.indexOf() is supported, butStringBuilder.append() is not.
• Type casting, with fully qualified types:((com.myprod.ClassImpl) myInterface).internalField

The following language features arenot supported:

• Unboxing of numeric types, such asInteger; usemyInteger.value instead.

Most Python expressions are supported, including:

• Reading local and global variables.
• Reading from arrays, lists, slices, dictionaries and objects.
• Calling simple methods.

The following language features are not supported:

• Calling functions that allocate new objects or use complex constructs.
• Creating new objects inside the expression.

Most Javascript expressions are supported, with the following caveat:

Expressions that may have static side effects are disallowed. The debug agentensures all conditions and watchpoints you add are read-only and have no sideeffects, however, it doesn’t catch expressions that have dynamic side-effects.

For example,o.f looks like a property access, but dynamically, it may end upcalling a getter function. The debugger presently doesn't detect suchdynamic-side effects.

Snapshot Debugger's Expressions feature allows you to evaluate complexexpressions or traverse object hierarchies when a snapshot is taken. Expressionssupport the same language features assnapshot conditions, described above.

Use of expressions is optional.

Typical uses for expressions are:

• To view static or global variables that are not part of the local variableset.
• To easily view deeply nested member variables.
• To avoid repetitive mathematical calculations. For example, calculating aduration in seconds with(endTimeMillis - startTimeMillis) / 1000.0.

Expressions are specified using the --expression flag of the set_snapshotscommand.

Example:

snapshot-dbg-cli set_snapshot index.js:26 --debuggee-id d-8dd7f149 --expression="histogram.length"

List snapshots with the following command:

snapshot-dbg-cli list_snapshots --debuggee-id d-8dd7f149 --include-inactive

Where:

• d-8dd7f149 is the debuggee ID

The output resembles the following:

Status     Location     Condition    CompletedTime         ID---------  -----------  -----------  --------------------  ------------ACTIVE     index.js:21                                     b-1648008775ACTIVE     index.js:21                                     b-1648044994ACTIVE     index.js:21                                     b-1648045010COMPLETED  index.js:21               2022-03-23T02:52:23Z  b-1648003845

Get a snapshot with the following command:

snapshot-dbg-cli get_snapshot b-1649947203 --debuggee-id d-8dd7f149

Where:

• b-1649947203 is the snapshot ID
• d-8dd7f149 is the debuggee ID

The output resembles the following:

--------------------------------------------------------------------------------| Summary--------------------------------------------------------------------------------Location:    index.js:30Condition:   No condition set.Expressions: No expressions set.Status:      CompleteCreate Time: 2022-05-13T14:14:01ZFinal Time:  2022-05-13T14:14:02Z-------------------------------------------------------------------------------| Evaluated Expressions--------------------------------------------------------------------------------There were no expressions specified.--------------------------------------------------------------------------------| Local Variables For Stack Frame Index 0:--------------------------------------------------------------------------------[  {    "req (IncomingMessage) ": {      "_readableState (ReadableState) ": {        "objectMode": "false",        "highWaterMark": "16384",        "buffer (BufferList) ": {          "head": null,          "tail": null,          "length": "0"        },[... snip]--------------------------------------------------------------------------------| CallStack:--------------------------------------------------------------------------------Function              Location--------------------  -----------(anonymous function)  index.js:30

Adds a debug logpoint to a debug target (debuggee). Logpoints inject logginginto running services without changing your code or restarting your application.Every time any instance executes code at the logpoint location, SnapshotDebugger logs a message. Output is sent to the standard log for the programminglanguage of the target (java.logging for Java, logging for Python, etc.)

Logpoints remain active for 24 hours after creation, or until they are deletedor the service is redeployed. If you place a logpoint on a line that receiveslots of traffic, Debugger throttles the logpoint to reduce its impact on yourapplication.

Set logpoints with the following command:

snapshot-dbg-cli set_logpoint index.js:21 "a={a} b={b}" --debuggee-id d-8dd7f149

Where:

• index.js:21 is thefile:line for the logpoint
• a={a} b={b} is the logpoint message format
• d-8dd7f149 is the debuggee ID

Note: A common issue that users have run into is that logging at INFOlevel is often suppressed by the default logger and so logpoints will appear tobe broken. See the--log-level option for setting a higher priority log level.

The format string is the message which will be logged every time the logpointlocation is executed. If the string contains curly braces ('{' and '}'), anytext within the curly braces will be interpreted as a run-time expression in thedebug target's language, which will be evaluated when the logpoint is hit. Somevalid examples are {a}, {myObj.myFunc()} or {a + b}. The value of theexpression will then replace the {} expression in the resulting log output. Forexample, if you specify the format string "a={a}, (b+1)={b+1}", and the logpointis hit when local variable a is 1 and b is 3, the resulting log output would be"a=1, (b+1)=3".

For more detailed information on valid expressions seeSnapshotexpressions as the rules are the same forlogpoint expressions.

A logpoint condition is a simple expression in the application language thatmust evaluate to true for the logpoint to be logged. Logpoint conditions areevaluated each time the line is executed, by any instance, until the logpointexpires or is deleted.

Use of logpoint conditions is optional.

For more detailed information on valid conditions seeSnapshotconditions as the rules are the same forlogpoint conditions.

List logpoints with the following command:

snapshot-dbg-cli list_logpoints --debuggee-id d-8dd7f149 --include-inactive

Where:

• d-8dd7f149 is the debuggee ID

The output resembles the following:

User Email    Location        Condition  Log Level  Log Message Format   ID            Status------------  --------------  ---------  ---------  -------------------  ------------  -------------------------------------------foo1@bar.com  Main.java:23               INFO      a={a} b={b}           b-1660681047  EXPIREDfoo2@bar.com  Main.java:25    a == 3     WARNING   Line hit              b-1660932877  EXPIREDfoo2@bar.com  Main.java:9999             INFO      Log msg               b-1661203071  SOURCE_LOCATION: No code found at line 9999

Get a logpoint with the following command:

snapshot-dbg-cli get_logpoint b-1660681047 --debuggee-id d-8dd7f149

Where:

• b-1660681047 is the logpoint ID
• d-8dd7f149 is the debuggee ID

The output resembles the following:

Logpoint ID:        b-1660681047Log Message Format: a == 3Location:           Main.java:23Condition:          No condition setStatus:             EXPIREDCreate Time:        2022-08-19T18:14:38ZFinal Time:         2022-08-20T18:14:39ZUser Email:         foo1@bar.com

The following commands can be used to delete debuggees and breakpoints(snapshots and logpoints).

Run the following command

snapshot-dbg-cli delete_debuggees

The output resembles the following:

This command will delete the following debuggees:Name                       ID          Last Active           Status-------------------------  ----------  --------------------  ------default - 20221125t224954  d-39f7082e  2022-12-05T03:13:42Z  STALEdefault - 20221125t154414  d-dba89292  2022-12-04T03:02:48Z  STALEDo you want to continue (Y/n)?Deleted 2 debuggees.

When deleting a debuggee, all breakpoints that belong to it are also deleted.

Delete snapshots with the following command:

snapshot-dbg-cli delete_snapshots --debuggee-id d-8dd7f149 --include-inactive

Where:

• d-8dd7f149 is the debuggee ID

The output resembles the following:

This command will delete the following snapshots:Status     Location     Condition    ID---------  -----------  -----------  ------------ACTIVE     index.js:28               b-1649959801ACTIVE     index.js:27               b-1649959807COMPLETED  index.js:19               b-1649702213COMPLETED  index.js:22               b-1649702753Do you want to continue (Y/n)? YDeleted 4 snapshots.

Delete logpoints with the following command:

snapshot-dbg-cli delete_snapshots --debuggee-id d-8dd7f149 --include-inactive

Where:

• d-8dd7f149 is the debuggee ID

The output resembles the following:

This command will delete the following logpoints:Location         Condition  Log Level  Log Message Format  ID---------------  ---------  ---------  ------------------  ------------Main.java:25     a == 3     WARNING    Line hit            b-1660927187Main.java:9999              INFO       Log msg             b-1660927272Do you want to continue (Y/n)?Deleted 4 snapshots.

There is a VSCode extension for the Snapshot Debugger. You can use thisextension to set logpoints, set breakpoints and view snapshots in the comfortof your IDE. See theextension's README for more details.

You can install the extension by downloading it from themost recent release, and then runningcode --install-extension snapshotdbg-*.vsix.

Note that you will still need to use the Snapshot Debugger CLI to set up yourenvironment and to purge old data. Gcloud needs to be installed for credentialmanagement.

SeeSnapshot Debugger Troubleshooting

SeeSnapshot Debugger Firebase DB Schema