elastic/integrationsPublic

NotificationsYou must be signed in to change notification settings
Fork545
Star318

Comments

[apache_spark][application] Add Apache Spark package with Application data stream#2941

Merged

yug-rajani merged 7 commits intoelastic:mainfrom

yug-rajani:package_apache_spark_applications

Apr 21, 2022

Merged

[apache_spark][application] Add Apache Spark package with Application data stream#2941
yug-rajani merged 7 commits intoelastic:mainfrom
yug-rajani:package_apache_spark_applications

Conversation

Copy link

Contributor

yug-rajani commentedMar 30, 2022•
edited
Loading

What does this PR do?

Generated the skeleton of Apache Spark integration package.
Added 1 data stream ( Application )
Added data collection logic.
Added the ingest pipelines.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.

Checklist

I have reviewedtips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package'schangelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package'smanifest.yml file to point to the latest Elastic stack release (e.g.^7.13.0).

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/apache_spark directory.
Run the following command to run tests.

elastic-package test

Add applications data stream for Apache Spark

691005b

yug-rajani requested a review froma team as acode owner

March 30, 2022 17:28

Copy link

ContributorAuthor

yug-rajani commentedMar 30, 2022

This PR is a split of#2811 as discussed over the comment#2811 (comment)

Copy link

elasticmachine commentedMar 30, 2022•
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-04-11T09:53:52.587+0000
Duration: 14 min 10 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6
Skipped	0
Total	6

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

yug-rajani commented

Mar 30, 2022

View reviewed changes

packages/apache_spark/manifest.yml OutdatedShow resolvedHide resolved

yug-rajani self-assigned this

Mar 30, 2022

yug-rajani mentioned this pull request

Mar 30, 2022

[apache_spark] Add Apache Spark package#2811

Closed

4 tasks

yug-rajani requested a review frommtojek

March 30, 2022 19:19

yug-rajani added enhancement

New feature or request

Team:Integrations

Label for the Integrations team

New IntegrationIssue or pull request for creating a new integration package. labels

Mar 30, 2022

Copy link

elasticmachine commentedMar 30, 2022

Pinging @elastic/integrations (Team:Integrations)

mtojek reviewed

Mar 31, 2022

View reviewed changes

packages/apache_spark/_dev/build/docs/README.md OutdatedShow resolvedHide resolved

packages/apache_spark/changelog.yml OutdatedShow resolvedHide resolved

packages/apache_spark/data_stream/applications/fields/fields.yml OutdatedShow resolvedHide resolved

packages/apache_spark/data_stream/applications/manifest.yml OutdatedShow resolvedHide resolved

packages/apache_spark/data_stream/applications/sample_event.json OutdatedShow resolvedHide resolved

yug-rajani added3 commits

April 1, 2022 18:13

Resolve review commeents

78411d7

Update Kibana version constraint

4e5ece9

Resolve merge conflicts

6a6af33

yug-rajani requested a review froma team as acode owner

April 6, 2022 09:28

Add system test cases

ec18af2

yug-rajani requested a review frommtojek

April 6, 2022 11:50

mtojek reviewed

Apr 6, 2022

View reviewed changes

Copy link

Contributor

mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I commented about the Docker entrypoint in the other PR.

packages/apache_spark/docs/README.md OutdatedShow resolvedHide resolved

yug-rajani mentioned this pull request

Apr 6, 2022

[apache spark] Add visualizations and dashboard#3020

Merged

4 tasks

Update the Docker image reference to a specific SHA

9944672

yug-rajani changed the title~~[apache_spark][applications] Add Apache Spark package with Applications data stream~~[apache_spark][application] Add Apache Spark package with Application data stream

Apr 11, 2022

Change naming convention of datastream

24ef8d4

yug-rajani mentioned this pull request

Apr 12, 2022

[apache_spark][node] Change naming convention of the node data stream#3070

Merged

4 tasks

yug-rajani requested review fromjsoriano andmtojek

April 12, 2022 05:42

mtojek approved these changes

Apr 12, 2022

View reviewed changes

lalit-satapathy approved these changes

Apr 21, 2022

View reviewed changes

yug-rajani merged commitbb0eee1 intoelastic:main

Apr 21, 2022

v1v added a commit to v1v/integrations that referenced this pull request

Apr 25, 2022

'feature/labels-cloud-ephemeral-components' of github.com:v1v/integr…

02b4c7f

…ations into feature/labels-cloud-ephemeral-components* 'feature/labels-cloud-ephemeral-components' of github.com:v1v/integrations: (23 commits)  use branch  standardise labels/tags  Update .ci/Jenkinsfile  [ci][terraform][aws] tags with metadata  Bump github.com/elastic/elastic-package from 0.46.0 to 0.47.0 (elastic#3182)  Spring boot package [Memory - data stream] (elastic#2979)  [cisco_ise] Add Cisco ISE package (elastic#2855)  [apache_spark][executor] Add Apache Spark package with Executor data stream (elastic#2943)  [apache_spark][driver] Add Apache Spark package with Driver data stream (elastic#2945)  cisco_duo: simplify grok expression for handling ports (elastic#3170)  cisco_duo: fix handling of IP addresses with port numbers (elastic#3117)  [sophos] Various improvements and log samples from 18.5 (elastic#3127)  [cisco_asa] fix visualizations (elastic#3146)  [apache_spark][application] Add Apache Spark package with Application data stream (elastic#2941)  Fix pagination bug that skipped events when more than one page is present. (elastic#3140)  Cis k8s name migration (elastic#3113)  remove exported fields; these will be added to Kibana docs instead (elastic#3093)  Exclude ecosystem as owner of subdirectories of packages (elastic#3132)  [awsfargate] bump package version (elastic#3130)  add first csp rule template (elastic#3081)  ...