Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Clean up old dashboard files on upgrade of local builds and static builds.#21370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Draft
Ferroin wants to merge7 commits intonetdata:master
base:master
Choose a base branch
Loading
fromFerroin:dashboard-manifest

Conversation

@Ferroin
Copy link
Member

@FerroinFerroin commentedDec 1, 2025
edited
Loading

Summary

This adds a manifest file to the root of the web directory to keep track of what files were installed by the current version of the Netdata Agent. On upgrade, static builds and local builds will remove any files in that directory that are not in the manifest file.

Test Plan

Requires manual testing to ensure that only files that are supposed to be removed are removed.

Additional Information

Fixes:#21327

This will be used to clean up old files from the web/v3 directory afteran update/reinstall for a static build or local build.
@github-actionsgithub-actionsbot added the area/packagingPackaging and operating systems support labelDec 1, 2025
@FerroinFerroin marked this pull request as ready for reviewDecember 2, 2025 12:15
@FerroinFerroin requested review froma team andvkalintiris ascode ownersDecember 2, 2025 12:15
@FerroinFerroin requested a review froma teamDecember 2, 2025 12:15
Copy link
Contributor

@cubic-dev-aicubic-dev-aibot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

No issues found across 3 files

Copy link
Contributor

CopilotAI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Pull request overview

This PR adds a manifest file (.MANIFEST) to track installed dashboard files and automatically removes obsolete files during upgrades for both static builds and local builds.

Key changes:

  • CMake generates a.MANIFEST file listing all installed dashboard files
  • Install scripts use the manifest to identify and remove old dashboard files that are no longer part of the current version
  • Cleanup occurs after new files are installed to ensure only truly obsolete files are removed

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 10 comments.

FileDescription
packaging/cmake/Modules/NetdataDashboard.cmakeGenerates.MANIFEST file containing list of dashboard files to be installed; adds manifest to CMake install rules
packaging/makeself/install-or-update.shImplements cleanup logic for static builds using the manifest to remove old dashboard files
netdata-installer.shImplements cleanup logic for local builds using the manifest to remove old dashboard files

💡Add Copilot custom instructions for smarter, more guided reviews.Learn how to get started.

Comment on lines +91 to +94
file(GLOBfiles
LIST_DIRECTORIESFALSE
RELATIVE"${dashboard_src_prefix}"
"${dashboard_src_prefix}/${dir}/*")
Copy link

CopilotAIDec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Both the install rules and manifest generation only handle one level of files within subdirectories (using${dashboard_src_prefix}/${dir}/*). If the dashboard contains nested subdirectories (e.g.,static/css/fonts/), those nested files won't be installed or tracked in the manifest.

This appears to be a limitation of the current implementation. If the dashboard structure is guaranteed to be flat (no nested subdirectories), this is fine. Otherwise, both the install rules and manifest generation should useGLOB_RECURSE or handle nested directories explicitly.

Verify that the dashboard structure doesn't contain nested subdirectories, or update the code:

file(GLOB_RECURSEfiles     LIST_DIRECTORIESFALSE     RELATIVE"${dashboard_src_prefix}""${dashboard_src_prefix}/${dir}/**")

Copilot uses AI. Check for mistakes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@Ferroin, we still have this suggestion.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is a known tradeoff taken based on discussion with the frontend team. Actually recursively listing all the files while still making CMake’s install directives work correctly is absurdly complicated to do right, and we have a general guarantee from the frontend team that the dashboard file layout is not going to change significantly from what is currently.

thiagoftsm reacted with thumbs up emoji
Copy link
Member

@ilyam8ilyam8Dec 2, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

nested
e.g. static/css/fonts/

This actually happened with GIF files (added and removed). I don't remember the exact directory, but there was some nesting.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Correction to my earlier statement, Copilot is wrong here, and I forgot about the specifics of how thesubdirlist() function that’s being used to generate the list of directories to process works (it has been more than a year since I wrote this code...).

That function generates arecursive list of all subdirectories of the specified base directory, so it does in fact pick up everything under${dashboard_src_prefix}.

Seehttps://github.com/netdata/netdata/blob/master/packaging/cmake/Modules/NetdataUtil.cmake#L202 for the actual code.

The strange looking split of processing all the directories one by one instead of doing a recursive glob is because CMake‘sinstall() command does not preserve the relative paths of the files listed for it (soinstall(foo/bar DESTINATION baz) will install the filebar directly into the directorybaz without creating a directorybaz/foo to install it into) and does not take directory trees (eachfile has to be listed for installation).


message(STATUS" Generating manifest for dashboard code")
list(APPEND manifest
.MANIFEST
Copy link

CopilotAIDec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The manifest includes.MANIFEST itself on line 85, but this creates a potential issue: the manifest file won't exist in the source directory when it's being generated, so it shouldn't be listed as a file to preserve.

While self-referencing prevents the manifest from being deleted during cleanup, it's semantically incorrect since the manifest describes the installed dashboard files, not itself. The cleanup logic already reads the manifest file, so it's implicitly protected.

Consider removing.MANIFEST from the manifest list.

Suggested change
.MANIFEST

Copilot uses AI. Check for mistakes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@Ferroin, we also have this,

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The cleanup logic already reads the manifest file, so it's implicitly protected.

This is blatantly wrong, there’s no special protections given to the manifest file by virtue of it being read by the cleanup logic.

thiagoftsm
thiagoftsm previously approved these changesDec 3, 2025
Copy link
Contributor

@thiagoftsmthiagoftsm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

PR ran as expected after local building and generation of static installer to install on CentOS 7. LGTM!

@ilyam8
Copy link
Member

ilyam8 commentedDec 3, 2025
edited
Loading

This is a known tradeoff taken based on discussion with the frontend team. Actually recursively listing all the files while still making CMake’s install directives work correctly is absurdly complicated to do right

If this version has known "tradeoffs"... why don't we make the logic easier and just remove the/usr/share/netdata/web/v3/ directory?

If the dashboard contains nested subdirectories (e.g., static/css/fonts/), those nested files won't be installed or tracked in the manifest.

It does contain. We added and removed 500 MB of gifs. This PR will not remove them, right?

@ilyam8
Copy link
Member

@thiagoftsm

ran as expected

What do you mean by that? What did you test?

@Ferroin
Copy link
MemberAuthor

This is a known tradeoff taken based on discussion with the frontend team. Actually recursively listing all the files while still making CMake’s install directives work correctly is absurdly complicated to do right

If this version has known "tradeoffs"... why don't we make the logic easier and just remove the/usr/share/netdata/web/v3/ directory?

Because the approach I’m taking here works pretty much independently of how the user chooses to update things (the only case it doesn’t work in is building by hand without netdata-installer.sh).

The static build can’t run things before the files are extracted and put in place, so weneed to track what files are installed and remove only those that aren’t part of the current version.

If the dashboard contains nested subdirectories (e.g., static/css/fonts/), those nested files won't be installed or tracked in the manifest.

It does contain. We added and removed 500 MB of gifs. This PR will not remove them, right?

The logic is that anythingnot in the manifest for the newly installed version gets removed, so yes, it will remove them.

That said,if they got installed in the first place, they would be in the manifest for the versions that they were installed in. The logic being used to generate the list of files for the manifest is literally identical to that used to generate the installation directives for the dashboard files.

@ilyam8ilyam8 marked this pull request as draftDecember 3, 2025 17:44
@ilyam8
Copy link
Member

Converting to draft to make it non-measurable - I found some issues while testing, give me some time.

@ilyam8
Copy link
Member

@Ferroin, the original cleanup logic was functionally broken. I don't know what@thiagoftsm was testing 🤷‍♂️.@thiagoftsm, if you don't understand the change, please ask or don't review/approve.

Two issues causedgrep to match all files:

  1. .MANIFEST contains relative paths, whilefind -print0 produced
    absolute paths, leading to unintended substring matches.
  2. A blank line in.MANIFEST created an empty pattern, causing
    grep -Ff to match every file.

As a result,grep -vzFf filtered out the entire file list andxargs
invokedrm with no operands.

This change replaces the pipeline with a correct, POSIX-safe loop that:

  • converts NUL-separated paths to newline-separated ones,
  • strips the NETDATA_WEB_DIR prefix to match manifest entries,
  • performs exact whole-line comparisons withgrep -Fxq,
  • removes only files not present in.MANIFEST.

The cleanup now works as intended and handles all filenames safely.

Copy link
Contributor

CopilotAI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 8 comments.


💡Add Copilot custom instructions for smarter, more guided reviews.Learn how to get started.

@Ferroin
Copy link
MemberAuthor

The cleanup now works as intended and handles all filenames safely.

Handles all current filenames.

There’s areason this was using null terminated records instead of newline terminated records, that’s theonly character that is guaranteed to not be in a filename on all systems.


Separately, there are a shit ton of files being processed here, which is exactlywhy I structured this as one pipeline instead of a long form loop that invokes multiple commands for every single file.

@Ferroin
Copy link
MemberAuthor

Oh the irony, Copilot pointing out exactly what I was as I was writing my comment...

@Ferroin
Copy link
MemberAuthor

Force pushed to fix the codeproperly without wasting resources and correctly accounting for the fact that a newline is actually a valid character in filenames on most UNIX-like systems.

@ilyam8
Copy link
Member

ilyam8 commentedDec 3, 2025
edited
Loading

Separately, there are a shit ton of files being processed here, which is exactly why I structured this as one pipeline instead of a long form loop that invokes multiple commands for every single file.

Consider testing your changes before marking them ready for review.


if [-r"${NETDATA_WEB_DIR}/.MANIFEST" ];then
old_pwd="$(pwd)"
cd"${NETDATA_WEB_DIR}"&& remove_files="$(find"." -type f -print0| grep -vzF""| grep -vzFf"${NETDATA_WEB_DIR}/.MANIFEST")"
Copy link
Member

@ilyam8ilyam8Dec 8, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

grep -vzF ""

This filters everything because every string contains the empty string. What are you trying to do here? Can you please test before pushing commits?

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@ilyam8ilyam8ilyam8 left review comments

Copilot code reviewCopilotCopilot left review comments

@cubic-dev-aicubic-dev-ai[bot]cubic-dev-ai[bot] left review comments

@thiagoftsmthiagoftsmthiagoftsm left review comments

@vkalintirisvkalintirisAwaiting requested review from vkalintirisvkalintiris is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

area/packagingPackaging and operating systems support

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Growing /opt/netdata/usr/share/netdata/web/v3

3 participants

@Ferroin@ilyam8@thiagoftsm

[8]ページ先頭

©2009-2025 Movatter.jp