- Notifications
You must be signed in to change notification settings - Fork104
Data Collection
Before you can begin collecting data from an AWS account, you'll need toconfigure an account to use.
Once you're set up, you can run the ingestor, usingdefault options, with:awspx ingest
[root@localhost ~]#awspx ingest ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Create profile │ ├────────────────────────────────────────────────────────────────────────────────────────────────┤ │ AWS Access Key ID [None]: ***************E9UX5 │ │ AWS Secret Access Key [None]: ***********************************Vukdh │ │ Default region name [None]: │ │ Default output format [None]: │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ Savedprofile 'default' Identity: arn:aws:iam::000000000000:user/awspx ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━Services: IAM, EC2, S3, LambdaDatabase: default.db Account: 000000000000Region: eu-west-1...
Next,explore this environment.
You'll need to setup an IAM identity for each AWS account you plan on ingesting:
Attach the following built-in managed policies:
- SecurityAudit (required)
- ReadOnlyAccess (only if you want to ingest S3 Objects)
Touse a role instead, see the--env
ingestion option.
Theawspx
command-line utility is a bash script (/usr/local/bin/awspx
) that wrapscli.py
: all commands are forwarded and executed within the context of theawspx
docker container hosting Neo4j.
You can use the CLI to manage what is available from the web interface using ingestion, database, and supporting functions.
Argument | Description |
---|---|
ingest | Ingest data from an AWS account. |
db | Manage databases used for visualization, ingestion, and attack computation. |
profile | Manage AWS credential profiles used for ingestion. |
attacks | Compute attacks using the active database. |
update | Update awspx to the latest version. |
For a full list of command-line options run:awspx --help
The ingestor collects and processes AWS account data, expressing and storing it as a graph. This data includes resources and their relationships. Action information — expressed as Policies, or variants thereof — is automatically resolved; as are theattacks that incorporate them. For additional information on how it all works, and the thinking behind it, check outthis blog post.
Argument | Description |
---|---|
--assume-role | ARN of a role to assume for ingestion (useful for cross-account ingestion). |
--assume-role-duration | Maximum session duration in seconds (for--assume-role ). |
--database | Database to store results (defaults to<profile>.db ). |
--env | Use AWS credential environment variables. |
--include-conditional-attacks | Include conditional actions when computing attacks (default False). |
--max-attack-depth | Maximum search depth for attacks (default None). |
--max-attack-iterations | Maximum number of iterations to run each attack (default False). |
--only-arns | Resources to include by ARN, all other resources will be excluded. |
--only-attacks | Attacks to include by name, all other attacks will be excluded. |
--only-types | Resource to include by type, all other resource types will be excluded. |
--profile | Profile to use for ingestion (corresponds to a[section] in~/.aws/credentials). . |
--region | Region to ingest (defaults to profile region, oreu-west-1 if not set). |
--services | One or more services to ingest (eg: IAM EC2 S3 Lambda). |
--skip-arns | Resources to exclude by ARN. |
--skip-attacks | Attacks to exclude by name. |
--skip-types | Resources to exclude by type. |
--skip-actions-all | Skip policy resolution (actions will not be processed). |
--skip-attacks-all | Skip attack path computation (it can be run later withawspx attacks ). |
--quick | Skips supplementary ingestion functions (i.e. speed at the cost of infromation). |
--verbose | Enable verbose output. |
Runawspx ingest --help
for a full list of ingestion options.
Running the ingestor without any arguments will default to the following behavior:
- Theprofile will be set todefault (
--profile default
). - Thedatabase will be set todefault.db (
--db default.db
) - Allservices will be ingested
- Actions will automatically be resolved from Policy and ACL information (
--skip-actions-all False
) - Attack paths are computed automatically (
--skip-actions-all False --skip-attacks-all False
).
Assuming the roleAudit, using the profileprd
awspx ingest --profile prd --assume-role arn:aws:iam::000000000000:role/Audit --assume-role-duration 3600
Usingenvironment variables
export AWS_ACCESS_KEY_ID="WY4XH9ZUFXVNIIG6C7JN"export AWS_SECRET_ACCESS_KEY="EDzukCRFBVDgq0//5yr01usTpHhniJHJ"export AWS_SESSION_TOKEN="keRJJBEWc3usCFS4I7wxq9MT6AYd7A53zN1bhvZ2ojHgfNZbmf5/////NWkyKutOZGw=="awspx ingest --env
SkippingALL S3 objects
awspx ingest --skip-types AWS::S3::Object
Skipping the S3 Bucketslogs andpii (and all of their Objects)
awspx ingest --skip-arns arn:aws:s3:::logs arn:aws:s3:::pii
Ingesting onlyIAM andEC2 resources, from the regioneu-west-2
awspx ingest --services IAM EC2 --region eu-west-2
Skipping Policy/ACL resolution (attacks will be skipped implicitly)
awspx ingest --skip-actions-all
SeeContributing for more information on how to extend this list.
Some services (e.g. IAM) are region independent, while others (e.g. EC2) are not. You can specify the target region using the--region
option, or omit it to fallback to the value associated with theprofile. If neither value is set, awspx will default toeu-west-1.
For more information on supported regions, see the relevantAWS docs.
Credential profiles essentially wrapaws configure
, allowing you to manage andstore credentials for those accounts you frequently interact with.
Argument | Description |
---|---|
--create | Create a new profile usingaws configure . |
--delete | Delete a saved profile. |
--list | List saved profiles. |
Runawspx profile --help
for a full list of profile options:
Creating a profile calledprd (once completed, it will be available for ingestion, e.g.
awspx ingest --profile prd
)awspx profile --create prd
Deleting a profile calleddev
awspx profile --delete dev
Listing all profiles
awspx profile --list
Each database corresponds to a single.db
file used by Neo4j and, by extension, awpsx's web interface. These files are created by the ingestor and named according to theprofile that was used at runtime.
Runawspx db --help
for a full list of available options.
Argument | Description |
---|---|
--use | Switch to the specified database. |
--list | List available databases. |
--load-zip | Create/overwrite database with ZIP file content. |
--verbose | Enable verbose output. |
Listing all available databases
awspx db --list
Switching the active database todev.db (you will need to refresh your browser)
awspx db --use dev.db
Loading the sample databasesample.zip intosample.db
awspx db --load-zip sample.zip
Before ingested data is written to a database, it is saved as a.zip
file. This means historical data can be readily loaded without rerunning the ingestor.
To load a ZIP, saysample.zip
included with awspx, you can executeawspx db --load-zip sample.zip
, which will:
- Search
/opt/awspx/data
— or~/bin/awspx/data
on macOS — forsample.zip
- Set the database to
sample.db
(based on the last field delimited by_
preceding.zip
) - Load
sample.zip
intosample.db
, overwriting any preexisting data.
ZIPs do no not include attack information.awspx attacks
` must be run separately.
Attacks are the result of post-processing graph data to contextualize those actions that together constitute compromise. In graph terms, each new Attack that is discovered will translate to one or more new edges. Since this may give rise to additional attacks, this process is repeated until it converges, or the maximum number of iterations is reached. Attacks are automatically computed as part of the ingestion process unless explicitly excluded with--skip-attacks-all
, or implicitly with--skip-actions-all
.
Argument | Description |
---|---|
--skip-attacks | Attacks to exclude by name. |
--only-attacks | Attacks to include by name, all other attacks will be excluded. |
--max-attack-iterations | Maximum number of iterations to run each attack (default 5). |
--max-attack-depth | Maximum search depth for attacks (default None). |
--include-conditional-attacks | Include conditional actions when computing attacks (default False). |
--verbose | Enable verbose output. |
Runawxpx attacks --help
for a full list of available options.
Running through attacks at most once
awspx attacks --max-attack-iterations 1
Setting the maximum search depth to 0 hops (i.e. directly available actions only)
awspx attacks --max-attack-depth 0
Utilizing conditional actions as well (i.e. policies includingConditions orResource fields incorporating variables)
awspx attacks --include-conditional-attacks
Skipping attacks involvingiam:CreatePolicyVersion
awspx attacks --skip-attack CreatePolicyVersion
Searching for attacks that would directly enable adding a user to a group
awspx attacks --only-attacks AddUserToGroup