Does this PR introduce a breaking change, and is
titled accordingly?
Nope, hopefully.

Doesn't this PR make significant breaking changes to the API of the Go packages? Are those considered part of Arduino CLI's public API?

It breaks Arduino Lint. I'm OK with breaking changes if they are necessary. But I think it's important to communicate about it to the users.

Good point@per1234, I didn't take that into consideration at all.

Nonetheless am not sure how we could handle breaking changes of the public API, we don't have versioning for it and neither a clearly defined interface for those that want to use it as a library.

I always thought that the breaking changes policy applied only to the CLI and gRPC consumers, changes in the public API is certainly something that we must take into account for the future if we want to keep using the CLI as a library but we first need to define a public/private boundary.

am not sure how we could handle breaking changes of the public API

It could be handled the same way as any other breaking change:
https://arduino.github.io/arduino-cli/dev/CONTRIBUTING/#pull-request-checklist

neither a clearly defined interface for those that want to use it as a library.

I've always thought that all exported components of non-internal packages were the interface. This is why I have taken the measure of making the packages of Arduino Lint andarduino/libraries-repository-engine internal, since those projects are not intended to be used as libraries at this time.

I always thought that the breaking changes policy applied only to the CLI and gRPC consumers

The Go library is advertised as one of the "three pillars of Arduino CLI":
https://arduino.github.io/arduino-cli/dev/integration-options/#the-third-pillar-embedding

Embedding the Arduino CLI is limited to Golang applications and requires a deep knowledge of its internals. For the average use case, the gRPC interface might be a better alternative. Nevertheless, this remains a valid option that we use and provide support for.

This is where I got the idea that it was intended to be part of the public API of the project. However, I notice now that almost every interface used by the example program in that "Integration Options" article has been broken in the year since it was written.

It could be handled the same way as any other breaking change:
https://arduino.github.io/arduino-cli/dev/CONTRIBUTING/#pull-request-checklist

Am doing this right now.

I've always thought that all exported components of non-internal packages were the interface. This is why I have taken the measure of making the packages of Arduino Lint andarduino/libraries-repository-engine internal, since those projects are not intended to be used as libraries at this time.

Rightly so I'd say, fact is that the CLI is kinda burdened with old decisions that make it hard to separate the public from the private APIs, it would be cool to do it. With the Arduino Lint it has been easier since we started the project from the ground up with a clear idea in mind of what we wanted it to be, not so much for the CLI I'd say.

The Go library is advertised as one of the "three pillars of Arduino CLI":
https://arduino.github.io/arduino-cli/dev/integration-options/#the-third-pillar-embedding

Yeah, I know that, my main pain point about this is the one I stated above though.

This is where I got the idea that it was intended to be part of the public API of the project. However, I notice now that almost every interface used by the example program in that "Integration Options" article has been broken in the year since it was written.

That part of the documentation needs some love probably, the example are really outdated. I think we must avoid using images for those kind of examples, it makes it harder to update.

One last suggestion following on the discussion of breaking changes: there are two things which are supposed to be done to communicate breaking changes:
https://arduino.github.io/arduino-cli/latest/CONTRIBUTING/#pull-request-checklist

The UPGRADING.md part is done now, but not yet the use of the "[breaking]" prefix on the PR title. I don't think it's so important to add it to the commit messages in this case because when a PR contains multiple commits as this one does, the squashed commit is titled with the PR title. The primary reason for the instructions to use the prefix on commit messages is because if a PR contains only a single commit then the PR title is not used by the merged commit.

I think it's a really nice improvement Silvano. Thanks for putting up with my nitpickery.

Nitpickery is always welcome. 😁

The UPGRADING.md part is done now, but not yet the use of the "[breaking]" prefix on the PR title. I don't think it's so important to add it to the commit messages in this case because when a PR contains multiple commits as this one does, the squashed commit is titled with the PR title. The primary reason for the instructions to use the prefix on commit messages is because if a PR contains only a single commit then the PR title is not used by the merged commit.

I'll update the PR title, it uses that when squashing and merging.

While revisiting#1201, I came upon this PR and looked at it in a bit more detail. It seems this makes three changes that I'm not entirely sure are intentional:

1. Insketch.New, open failures are now ignored and the offending file is silently skipped. In general, I would argue that ignoring errors is a bad idea, and if this causes those files to be silently omitted from further processing (i.e. omitted from compilation), that could lead to very puzzling (linker) errors that would be hard to diagnose. I would suggest to not ignore these errors, and instead fail to load the sketch (at least for files that will actually be processed, i.e. files after file extension filtering). Note that some of these errors (that cause a Stat() failure, such as a broken symlink) are currently reported by go-paths-helper already, but I think that is the wrong place (better have arduino-cli have control over this error reporting, so it can apply the reportingafter the file extension filtering). I am preparing a PR that implements this.
2. When collecting the list of files in a sketch, theCVS andRCSwere previously omitted, but now only hidden files are omitted. Even though CVS and RCS are old and might not be worth supporting, it looks like this change might have been overlooked. Note that these directories (including some others) are still excluded in the actual build process, but AFAICS not excluding them here means unecessarily enumerating them, uneccessarily keeping them in memory, and probably also unecessarily copying them into the build directory.
3. Previously, hidden directories (as well as CVS/RCS) where ignoredduring the tree walk. Now, they are fully enumerated, and only filtered out later. This means that the tree enumeration takes needlessly long, especially when a big hidden directory is present (such as a.git directory).
4. Previously, hidden files and directories were filtered out anywhere in the tree. Now, it seems that only top-level hidden files or directories are filtered out (I suspect because the filtering happens after enumeration and only seems to check for a. prefix against the enumerated path relative to the sketch directory). But this would also be fixed if the previous point is fixed.

As I said, for the file error handling of point 1, I'm preparing a draft PR. For 2-4, it would probably be good to modify go-paths-helper's ReadDirRecursive to accept a callback that tests file/dir names (and probably an isDir boolean) for inclusion in the result?