Repository files navigation

Introspector is a tool and schema for importing cloud infrastructure configuration.The goal is to unlock the expressive power of SQL and relational databases to ask questions about your infrastructure's security, compliance, and governance posture.

We (@Gold Fig Labs) were inspired byosquery to bring the same level of structure and consistency to the data backing our cloud deployments. All of this information is available from the underlying platform but is in disparate places and relationships can be difficult to find. Additionally, the expressivity of SQL far outstrips the querying functionality built into the existing CLI tools (aws,gcloud, etc.). At the cost of needing to import the data, Introspector allows you to issue more specific or complex queries, or even join against internal data sources (like an org chart) to produce customized reports. Introspector is not intended to replace provider tools, but instead standardize the process of analyzing your infrastructure.

1. Import - Run an import job against a cloud platform (currently AWS is supported) to retrieve your deployment details. This takes a snapshot of your current deployment's configuration, settings, and policies. Your database is updated to match the status of your infrastructure, and observed deltas from the previous snapshot are logged.

2. Analyze - Introspector comes with sometools out of the box to start analyzing your cloud infrastructure. But, these tools are mostly just wrappers around SQL queries. You can extend these tools or implement your own by writing SQL. SeeExample Queries below.

• Docker

  $ docker --versionDocker version 19.03.8, build afacb8b$ docker-compose --versiondocker-compose version 1.25.5, build 8a1c60f6

• AWS command line interface

  aws configure list

1. Download the latest Introspectorrelease:

Linux:

 curl -LO https://github.com/goldfiglabs/introspector/releases/latest/download/introspector_linux.zip unzip introspector_linux.zip

OSX:

 curl -LO https://github.com/goldfiglabs/introspector/releases/latest/download/introspector_osx.zip unzip introspector_osx.zip

2. Start Introspector containers:

   docker compose up

   Alternatively, in older versions of Docker:

    docker-compose up -d

Initialize Introspector system and schemas:

./introspector init

Import data from provider:

./introspector account aws import

Note that this may take a couple of minutes.

At this stage the underlying data is ready for querying, analysis, or alerting. You can get a summary of the import using:

./introspector status

Find all untagged resources:

./introspector tags find-untagged

Get a report on all tags used across every resource:

./introspector tags report

Run several queries demonstrating a sample of theCIS

AWS Foundation Benchmark:

./introspector cis foundation

Run an arbitrary SQL query against your data:

./introspector run "SELECT COUNT(*) FROM aws_ec2_instance"

Get every S3 bucket:

cat sample_queries/aws_storage_buckets.sqlSELECT  name,  uri,  creationdateFROM  aws_s3_bucket./introspector run sample_queries/all_storage_buckets.sql

Get all public IP addresses across all AWS instances:

cat sample_queries/aws_ec2_instance_ips.sqlSELECT  uri,  instanceid,  publicipaddressFROM  aws_ec2_instance./introspector run sample_queries/aws_ec2_instance_ips.sql

Get every AWS S3 bucket where payer is the bucket owner:

cat sample_queries/aws_owner_pays_buckets.sqlSELECT  name,  uri,  requestpayment->>'Payer' AS PayerFROM  aws_s3_bucketWHERE  requestpayment->>'Payer' = 'BucketOwner'./introspector run sample_queries/aws_owner_pays_buckets.sql

Get total size for all disks:

cat sample_queries/aws_total_disk_size.sqlSELECT  SUM(size)FROM  aws_ec2_volume./introspector run sample_queries/aws_total_disk_size.sql

After running an import job multiple times, you can also query for resource that have been flagged as 'update' or 'delete':

./introspector run "SELECT * FROM resource_delta WHERE change_type = 'delete'"

See more in thesample_queries/ folder.

1. What's currently supported?

   Introspector is being released with support for most common AWS services, including IAM, EC2, and S3, as well as higher level services such as Lambda, ECS, and plumbing such as SNS and SQS. Please check out theschema docs to see all of the currently supported resources.

2. What's the set of permissions needed to run an import?

   Introspector uses read-only API calls, will not make any changes to your infrastructure, and does not require any write permissions for any API.

   • AWS: the available credentials when running the import must have at least permissions in the following policies:

     • arn:aws:iam::aws:policy/SecurityAudit
     • arn:aws:iam::aws:policy/job-function/ViewOnlyAccess

     The following commands can create the read-only account credentials which should be saved to ~/.aws/credentials:

   export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account'  | awk -F '"' '{print $2}')aws iam create-group --group-name Introspectoraws iam create-policy --policy-name Introspector-Ro-Additions --policy-document file://$(pwd)/permission-policies/aws-introspector-ro.jsonaws iam attach-group-policy --group-name Introspector --policy-arn arn:aws:iam::aws:policy/SecurityAuditaws iam attach-group-policy --group-name Introspector --policy-arn arn:aws:iam::aws:policy/job-function/ViewOnlyAccessaws iam attach-group-policy --group-name Introspector --policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/Introspector-Ro-Additionsaws iam create-user --user-name introspectoraws iam add-user-to-group --user-name introspector --group-name Introspectoraws iam create-access-key --user-name introspector

3. How does Introspector compare Terraform, Deployment Manager, Cloudformation, etc?

   Infrastructure-as-code tools (which are great!) impose structure and assert how portions of your infrastructureshould be. Introspector is focused on surveying what your infrastructureactually is and makes no changes to your deployment. This is a complementary tool to IAC, and indeed one use case could be aiding in migrating to and enforcing the usage of IAC.

4. What's next on the Roadmap?

   Increasing the breadth of services supported and normalization of data that appears in different forms throughout a provider's data. See something missing? File an issue—we'd love your contributions!

Schema documentation can be found online:

• https://www.goldfiglabs.com/introspector-schema-docs/

Alternatively, your running Docker instance will have the docs for your build:

• http://localhost:5000/

Copyright (c) 2019-2021Gold Fig Labs Inc.

This Source Code Form is subject to the terms of the Mozilla Public License, v.2.0. If a copy of the MPL was not distributed with this file, You can obtain one athttp://mozilla.org/MPL/2.0/.

Mozilla Public License v2.0