elastic/integrationsPublic

NotificationsYou must be signed in to change notification settings
Fork545
Star318

Migrate obs-infraobs-integrations to package-spec v3 #3#8203

Merged

tommyers-elastic merged 50 commits intoelastic:mainfrom

shmsr:migrate-infraobs-ps-v3-batch-3-serverless

Oct 25, 2023

Merged

Migrate obs-infraobs-integrations to package-spec v3 #3#8203
tommyers-elastic merged 50 commits intoelastic:mainfrom
shmsr:migrate-infraobs-ps-v3-batch-3-serverless

Conversation

Copy link

Member

shmsr commentedOct 15, 2023•
edited
Loading

Migrate a batch of packages from v1/v2 to v3 package-spec. Used the following scripts and some manual changes to generate the changes in this PR:

yq_hack.zsh

#!/bin/bashset -eset -uset -o pipefailset -xpushd $1find data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].release" {} \;find data_stream -type f -path "*fields*" -name "ecs.yml" -exec yq e -i "unique_by(.name)" {} \;find data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].fields.[].required" {} \;yq e -i 'del .release' manifest.ymlyq e -i "with(select(.license != null); .conditions.elastic.subscription = .license) | del .license" manifest.ymlpopdgit add -u $1/git commit -m "[$1]: migration with yq"

ecs-update.zsh

#!/bin/bashset -uset -o pipefailset -xecho "-- executing ecs-update for $1 --"ecs-update -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-infraobs-integrations -v -skip-format=true $1if (( $? )) then    echo "-- write validation for $1 --"    pushd $1    go run ../scripting/write_validation.go    popd    # Try again?    git checkout .    ecs-update -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-infraobs-integrations -v -skip-format=true $1fi

And another Go program to write the validation.yml by parsing the elastic-package lint errors.

Then I hooked this scripts to run with:

$ pkgs=($(cat .github/CODEOWNERS| grep'@elastic/obs-infraobs-integrations'| cut -d'' -f 1| cut -d'/' -f 3| sort| uniq))$for((idx=16; idx<=30; idx++))doscripting/yq_hack.zsh$pkgs[idx]done

$for((idx=16; idx<=30; idx++))doscripting/ecs_update.zsh$pkgs[idx]||breakdone

Another variation of the above script:

$for((idx=31; idx<=48; idx++))docase$pkgs[idx]in"tomcat" |"salesforce" |"redis")echo"-- skip$pkgs[idx] --";;"websphere_application_server" |"system" |"statsd_input" |"sql_input" |"springboot" |"rabbitmq" |"postgresql" |"prometheus_input" |"oracle_weblogic" |"php_fpm" |"oracle")echo"-- done$pkgs[idx] --";;*) scripting/ecs_update.zsh$pkgs[idx]||break;;esacdone

As I am creating a batch of 18 packages here, I am using a for loop. In case the migration is not possible for some packages, I put it in the first case to skip them.

Proposed commit message

Migrate packages owned by obs-infraobs-integrations but few couldn't be migrated. Marking them as follows:

oracle
oracle_weblogic
php_fpm
postgresql
prometheus_input
rabbitmq
redis
redisenterprise
salesforce
spring_boot
sql_input
statsd_input
system
tomcat
traefik
vsphere
websphere_application_server
zookeeper

Please see the comment#8203 (comment) in this PR itself to understand why some packages couldn't be migrated and also learn about the challenges faced when migrating.

Checklist

I have reviewedtips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package'schangelog.yml file.
I have verified that Kibana version constraints are current according toguidelines.

Author's Checklist

How to test this PR locally

Related issues

Relates to#8170
Relates to#8171

Screenshots

shmsr added30 commits

October 16, 2023 00:06

[oracle]: migration with yq

b7e22cd

[oracle_weblogic]: migration with yq

7800451

[postgresql]: migration with yq

05120ab

[rabbitmq]: migration with yq

16f5961

[redisenterprise]: migration with yq

d2a6874

[salesforce]: migration with yq

fc8d56d

[spring_boot]: migration with yq

70652ea

[redis]: migration with yq

0e6704d

[system]: migration with yq

0829c81

[tomcat]: migration with yq

325e8aa

[traefik]: migration with yq

238c030

[vsphere]: migration with yq

98e56a5

[websphere_application_server]: migration with yq

8d50af2

[zookeeper]: migration with yq

b0f3dc8

[oracle] - removed dotted YAML keys from manifest

e5ce8c2

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/oracle

[oracle_weblogic] - removed dotted YAML keys from manifest

1cbfa00

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/oracle_weblogic

[php_fpm] - removed dotted YAML keys from manifest

adccf8a

The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/php_fpm

[postgresql] - removed dotted YAML keys from manifest

d7729ac

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/postgresql

[prometheus_input] - removed dotted YAML keys from manifest

03acbf9

The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/prometheus_input

[prometheus_input]: fix test-default-config.yml

bc6d80d

[rabbitmq] - removed dotted YAML keys from manifest

e4d3ded

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/rabbitmq

[redisenterprise] - removed dotted YAML keys from manifest

6c4e288

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/redisenterprise

[spring_boot] - removed dotted YAML keys from manifest

fb7b7b2

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/spring_boot

[sql] - removed dotted YAML keys from manifest

f2b375b

The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/sql_input

[statsd_input] - removed dotted YAML keys from manifest

5e0b310

The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/statsd_input

[system] - change to format_version 3.0.0

de691ef

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Added'owner.type: elastic' to package manifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/system

[traefik] - removed dotted YAML keys from manifest

67d7674

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/traefik

[vsphere] - removed dotted YAML keys from manifest

cd51286

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/vsphere

[websphere_application_server] - removed dotted YAML keys from manifest

a3cf8b0

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/websphere_application_server

[zookeeper] - removed dotted YAML keys from manifest

7d5ba36

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removeddotted YAML keys from package manifest. Added 'owner.type: elastic' to packagemanifest.[git-generate]go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/zookeeper

play with stack version

970d849

shmsr commented

Oct 19, 2023

View reviewed changes

packages/spring_boot/_dev/deploy/docker/DockerfileShow resolvedHide resolved

shmsr commented

Oct 19, 2023

View reviewed changes

packages/spring_boot/manifest.ymlShow resolvedHide resolved

Copy link

MemberAuthor

shmsr commentedOct 25, 2023

Pulling outstatsd_input andsql_input because of the mapping-related ambiguities. Need to discuss the mappings in more detail and then commit the changes for both packages.