- Notifications
You must be signed in to change notification settings - Fork67
Add QL4QL to analyze the queries#511
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Draft
rvermeulen wants to merge26 commits intogithub:mainChoose a base branch
base:main
Could not load branches
Branch not found:{{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline, and old review comments may become outdated.
Uh oh!
There was an error while loading.Please reload this page.
Draft
Changes fromall commits
Commits
Show all changes
26 commits Select commitHold shift + click to select a range
e68fb0e Add minimal working Nix env
rvermeulene0841eb Remove dependency on CodeQL CLI
rvermeulen3efa7b5 Add derivative CodeQL QL for QL queries and libraries
rvermeulenc5be6b1 Add support for additional packs and extractors
rvermeulen3fbd6a6 Rename derivatives and include QL for QL pack.
rvermeulen39126fb Move CodeQL tooling and shell derivative into tooling directory
rvermeulencda3971 Remove output of environment when building CodeQL derivative
rvermeulenab731b2 Copy the contents of the pack derivative instead of the derivative dir
rvermeulen749ea80 Add workflow to run CodeQL on QL queries
rvermeulen93b27e8 Run analyzes on PRs or pushes to main
rvermeulene60bb08 Allow usage of non-free derivatives
rvermeulenb628c0c Pin Nix packages
rvermeulen67baed8 Select the hash based on the platform
rvermeulenf80e31e Address issue with JDK on Linux
rvermeulen6fe5e35 Prepare all tools for other derivatives
rvermeulen131f6ab Add shell derivate to test CodeQL CLI
rvermeulen4e5d514 Cache the Nix store for improved runtime
rvermeulen4121f79 Fix incorrect option specification
rvermeulencdf9295 Close Nix expression
rvermeulen250fe90 Preserve the GitHub token environment variable to upload results
rvermeulen6f4124e Pass GitHub token through env
rvermeulen750e8be Limit GH token to security event writes
rvermeulen1dda311 Key the Nix store on the workflow file
rvermeulen9537f6a Attempt bind mount to address cache write issues
rvermeulenb824f61 Attempt use of Nix copy command for caching store
rvermeulene7ea5b7 Unify cache key
rvermeulenFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
There are no files selected for viewing
66 changes: 66 additions & 0 deletions.github/workflows/codeql.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| name: CodeQL workflow for analyzing QL queries | ||
| on: | ||
| workflow_dispatch: | ||
| push: | ||
| branches: [main] | ||
| pull_request: | ||
| branches: [main] | ||
| permissions: | ||
| security-events: write | ||
| env: | ||
| NIXPKGS_ALLOW_UNFREE: 1 | ||
| jobs: | ||
| analyze-ql-files: | ||
| name: Analyze QL files | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Install Nix | ||
| uses: cachix/install-nix-action@6004951b182f8860210c8d6f0d808ec5b1a33d28 | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
| - name: Cache Nix store | ||
| uses: actions/cache@v4 | ||
| id: cache | ||
| with: | ||
| path: ${{ runner.temp }}/nix/store | ||
| key: ${{ runner.os }}-${{ hashFiles('tooling/**', '.github/workflows/codeql.yml') }} | ||
| - name: Copy Nix store from cache | ||
| if: steps.cache.outputs.cache-hit == 'true' | ||
| run: | | ||
| nix copy --all --from "file://$RUNNER_TEMP/nix/store" | ||
| - name: Prepare Nix store | ||
| if: steps.cache.outputs.cache-hit != 'true' | ||
| run: | | ||
| nix-shell --pure --command "codeql version" tooling/shell.nix | ||
| - name: Copy Nix store to cache location | ||
| if: steps.cache.outputs.cache-hit != 'true' | ||
| run: | | ||
| nix copy --all --to "file://$RUNNER_TEMP/nix/store" | ||
| - name: Create database | ||
| run: | | ||
| nix-shell --pure --command "codeql database create --language=ql --source-root=. $RUNNER_TEMP/ql-db" tooling/shell.nix | ||
| - name: Analyze database | ||
| run: | | ||
| nix-shell --pure --command "codeql database analyze --output=$RUNNER_TEMP/ql.sarif --format=sarif-latest --sarif-category=ql4ql -- $RUNNER_TEMP/ql-db codeql/ql" tooling/shell.nix | ||
| - name: Upload results | ||
| env: | ||
| GITHUB_TOKEN: ${{ github.token }} | ||
| run: | | ||
| nix-shell --pure --keep GITHUB_TOKEN --command "codeql github upload-results --sarif=$RUNNER_TEMP/ql.sarif --repository=$GITHUB_REPOSITORY --ref=$GITHUB_REF --commit=$GITHUB_SHA" tooling/shell.nix | ||
4 changes: 4 additions & 0 deletionstooling/all-tools.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| let | ||
| codeql = import codeql/default.nix; | ||
| in | ||
| codeql |
61 changes: 61 additions & 0 deletionstooling/codeql/codeql-cli/2.16.0.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| { lib, stdenv, fetchzip, jdk17, withExtractors ? [], withPacks ? [] }: | ||
| stdenv.mkDerivation rec { | ||
| pname = "codeql-cli"; | ||
| version = "2.16.0"; | ||
| platform = if stdenv.isDarwin then "osx64" else "linux64"; | ||
| dontConfigure = true; | ||
| dontBuild = true; | ||
| dontStrip = true; | ||
| src = fetchzip { | ||
| url = "https://github.com/github/codeql-cli-binaries/releases/download/v${version}/codeql-${platform}.zip"; | ||
| hash = if platform == "osx64" then "sha256-trWUSMOT7h7J5ejjp9PzhGgBS3DYsJxzcv6aYKuk8TI=" else "sha256-ztvKlNbqWcH93AB/Mum9jM81ssxiGcbkBROEANFGXis="; | ||
| }; | ||
| buildInputs = if (lib.length withExtractors) == 0 then [ ] else withExtractors; | ||
| inherit withExtractors withPacks; | ||
| nativeBuildInputs = [ jdk17 ]; | ||
| installPhase = '' | ||
| # codeql directory should not be top-level, otherwise, | ||
| # it'll include /nix/store to resolve extractors. | ||
| mkdir -p $out/{codeql/qlpacks,bin} | ||
| cp -R * $out/codeql/ | ||
| if [ "$platform" == "linux64" ]; then | ||
| ln -sf $out/codeql/tools/linux64/lib64trace.so $out/codeql/tools/linux64/libtrace.so | ||
| fi | ||
| # many of the codeql extractors use CODEQL_DIST + CODEQL_PLATFORM to | ||
| # resolve java home, so to be able to create databases, we want to make | ||
| # sure that they point somewhere sane/usable since we can not autopatch | ||
| # the codeql packaged java dist, but we DO want to patch the extractors | ||
| # as well as the builders which are ELF binaries for the most part | ||
| rm -rf $out/codeql/tools/$platform/java | ||
| ln -s ${jdk17} $out/codeql/tools/$platform/java | ||
| ln -s $out/codeql/codeql $out/bin/ | ||
| for extractor in $withExtractors; do | ||
| # Copy the extractor, because CodeQL doesn't follow symlinks. | ||
| cp -R $extractor $out/codeql/ql | ||
| done | ||
| for pack in $withPacks ; do | ||
| # Copy the pack, because CodeQL doesn't follow symlinks. | ||
| cp -R $pack/* $out/codeql/qlpacks/ | ||
| done | ||
| ''; | ||
| meta = with lib; { | ||
| description = "Semantic code analysis engine"; | ||
| homepage = "https://codeql.github.com"; | ||
| platforms = lib.platforms.linux ++ lib.platforms.darwin; | ||
| license = licenses.unfree; | ||
| }; | ||
| } |
7 changes: 7 additions & 0 deletionstooling/codeql/codeql-cli/default.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| let | ||
| nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0e148322b344eab7c8d52f6e59b0d95ba73fb62e.tar.gz"; | ||
| pkgs = (import nixpkgs { config = {}; overlays = []; }); | ||
| in | ||
| { | ||
| codeql-cli_2_16_0 = pkgs.callPackage ./2.16.0.nix {}; | ||
| } |
11 changes: 11 additions & 0 deletionstooling/codeql/codeql-cli/shell.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| let | ||
| nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0e148322b344eab7c8d52f6e59b0d95ba73fb62e.tar.gz"; | ||
| pkgs = (import nixpkgs { config = {}; overlays = []; }); | ||
| codeql = import ./default.nix; | ||
| in | ||
| pkgs.mkShell { | ||
| packages = with codeql; [ | ||
| codeql-cli_2_16_0 | ||
| ]; | ||
| } |
29 changes: 29 additions & 0 deletionstooling/codeql/codeql-ql/0.0.1-dev.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| { stdenv, lib, fetchFromGitHub, rustPlatform, gh, libiconv, which, jq, withCodeQlCli }: | ||
| stdenv.mkDerivation rec { | ||
| pname = "codeql-ql"; | ||
| version = "0.1.0-dev"; | ||
| dontConfigure = true; | ||
| dontStrip = true; | ||
| dontInstall = true; | ||
| dontFixup = true; | ||
| src = fetchFromGitHub { | ||
| owner = "github"; | ||
| repo = "codeql"; | ||
| rev = "codeql-cli/v2.16.0"; | ||
| sha256 = "x2EFoOt1MZRXxIZt6hF86Z1Qu/hVUoOVla562TApVwo="; | ||
| }; | ||
| nativeBuildInputs = [ gh libiconv which withCodeQlCli jq]; | ||
| platform = if stdenv.isLinux then "linux64" else "osx64"; | ||
| buildPhase = '' | ||
| runHook preBuild | ||
| mkdir -p $out | ||
| codeql pack create --threads=0 --output=$out --no-default-compilation-cache --compilation-cache=$TMP_DIR ql/ql/src | ||
| runHook postBuild | ||
| ''; | ||
| } |
7 changes: 7 additions & 0 deletionstooling/codeql/codeql-ql/default.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| let | ||
| nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0e148322b344eab7c8d52f6e59b0d95ba73fb62e.tar.gz"; | ||
| pkgs = (import nixpkgs { config = {}; overlays = []; }); | ||
| in | ||
| { | ||
| codeql-ql-0_1_0-dev = pkgs.callPackage ./0.0.1-dev.nix {}; | ||
| } |
10 changes: 10 additions & 0 deletionstooling/codeql/default.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| let | ||
| nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0e148322b344eab7c8d52f6e59b0d95ba73fb62e.tar.gz"; | ||
| pkgs = (import nixpkgs { config = {}; overlays = []; }); | ||
| in | ||
| rec { | ||
| ql-extractor-0_0_1 = pkgs.callPackage ./ql-extractor/0.0.1.nix {}; | ||
| codeql-cli-2_16_0 = pkgs.callPackage ./codeql-cli/2.16.0.nix {}; | ||
| codeql-ql-0_0_1-dev = pkgs.callPackage ./codeql-ql/0.0.1-dev.nix { withCodeQlCli = codeql-cli-2_16_0; }; | ||
| codeql-cli-2_16_0-with-ql = pkgs.callPackage ./codeql-cli/2.16.0.nix { withExtractors = [ql-extractor-0_0_1]; withPacks = [codeql-ql-0_0_1-dev];}; | ||
| } |
42 changes: 42 additions & 0 deletionstooling/codeql/ql-extractor/0.0.1.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| { stdenv, lib, fetchFromGitHub, rustPlatform, gh, libiconv, which, jq}: | ||
| rustPlatform.buildRustPackage rec { | ||
| pname = "codeql-ql-extractor"; | ||
| version = "0.0.1"; | ||
| dontConfigure = true; | ||
| dontStrip = true; | ||
| src = fetchFromGitHub { | ||
| owner = "github"; | ||
| repo = "codeql"; | ||
| rev = "codeql-cli/v2.16.0"; | ||
| sha256 = "x2EFoOt1MZRXxIZt6hF86Z1Qu/hVUoOVla562TApVwo="; | ||
| }; | ||
| sourceRoot = "${src.name}/ql"; | ||
| cargoLock = { | ||
| lockFile = "${src.outPath}/ql/Cargo.lock"; | ||
| outputHashes = { | ||
| "tree-sitter-json-0.20.0" = "sha256-fIh/bKxHMnok8D+xQlyyp5GaO2Ra/U2Y/5IjQ+t4+xY="; | ||
| "tree-sitter-ql-0.19.0" = "sha256-2QOtNguYAIhIhGuVqyx/33gFu3OqcxAPBZOk85Q226M="; | ||
| "tree-sitter-ql-dbscheme-0.0.1" = "sha256-wp0LtcbkP2lxbmE9rppO9cK+RATTjZxOb0EWfdKT884="; | ||
| }; | ||
| }; | ||
| nativeBuildInputs = [ gh libiconv which jq]; | ||
| platform = if stdenv.isLinux then "linux64" else "osx64"; | ||
| installPhase = '' | ||
| runHook preInstall | ||
| mkdir -p $out/tools/$platform | ||
| cargo run --profile release --bin codeql-extractor-ql -- generate --dbscheme ql/src/ql.dbscheme --library ql/src/codeql_ql/ast/internal/TreeSitter.qll | ||
| # For some reason the fixupPhase isn't working, so we do it manually | ||
| patchShebangs tools/ | ||
| cp -r codeql-extractor.yml tools ql/src/ql.dbscheme ql/src/ql.dbscheme.stats $out/ | ||
| cp $(cargo metadata --format-version 1 | jq -r '.target_directory')/release/codeql-extractor-ql $out/tools/$platform/extractor | ||
| runHook postInstall | ||
| ''; | ||
| } |
7 changes: 7 additions & 0 deletionstooling/codeql/ql-extractor/default.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| let | ||
| nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0e148322b344eab7c8d52f6e59b0d95ba73fb62e.tar.gz"; | ||
| pkgs = (import nixpkgs { config = {}; overlays = []; }); | ||
| in | ||
| { | ||
| ql-extractor-0_0_1 = pkgs.callPackage ./0.0.1.nix {}; | ||
| } |
15 changes: 15 additions & 0 deletionstooling/shell.nix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| let | ||
| nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0e148322b344eab7c8d52f6e59b0d95ba73fb62e.tar.gz"; | ||
| pkgs = (import nixpkgs { config = {}; overlays = []; }) // (import ./all-tools.nix); | ||
| in | ||
| pkgs.mkShell { | ||
| packages = with pkgs; [ | ||
| clang-tools_14 | ||
| python39 | ||
| git | ||
| gh | ||
| jq | ||
| codeql-cli-2_16_0-with-ql | ||
| ]; | ||
| } |
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.