Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit523d481

Browse files
added section for hpa (codefresh-io#229)
* added section for hpa* Formatting fixes and whats new entryCo-authored-by: Kostis Kapelonis <kostis@codefresh.io>
1 parentbd476d3 commit523d481

File tree

2 files changed

+284
-2
lines changed

2 files changed

+284
-2
lines changed

‎_docs/administration/codefresh-on-prem.md‎

Lines changed: 282 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -722,7 +722,287 @@ consul:
722722
enabled: false
723723
```
724724

725-
## Upgrade the Codefresh platform
725+
## App Cluster Autoscaling
726+
727+
Autoscaling in Kubernetes is implemented as an interaction between Cluster Autoscaler and Horizontal Pod Autoscaler
728+
729+
{: .table .table-bordered .table-hover}
730+
| | Scaling Target| Trigger | Controller | How it Works |
731+
| ----------- | ------------- | ------- | --------- | --------- |
732+
| [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)| Nodes | **Up:** Pending pod <br/> **Down:** Node resource allocations is low | On GKE we can turn on/off autoscaler and configure min/max per node group can be also installed separately | Listens on pending pods for scale up and node allocations for scaledown. Should have permissions to call cloud api. Considers pod affinity, pdb, storage, special annotations |
733+
| [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) | replicas on deployments or StatefulSets | metrics value thresholds defined in HPA object | part of Kubernetes controller | Controller gets metrics from "metrics.k8s.io/v1beta1" , "custom.metrics.k8s.io/v1beta1", "external.metrics.k8s.io/v1beta1" requires [metrics-server](https://github.com/kubernetes-sigs/metrics-server) and custom metrics adapters ([prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter), [stackdriver-adapter](https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/custom-metrics-stackdriver-adapter)) to listen on this API (see note (1) below) and adjusts deployment or sts replicas according to definitions in HorizontalPodAutocaler <br/> There are v1 and beta api versions for HorizontalPodAutocaler: <br/> [v1](https://github.com/kubernetes/api/blob/master/autoscaling/v1/types.go) - supports for resource metrics (cpu, memory) - `kubect get hpa` <br/> [v2beta2](https://github.com/kubernetes/api/blob/master/autoscaling/v2beta2/types.go) and [v2beta1](https://github.com/kubernetes/api/blob/master/autoscaling/v2beta1/types.go) - supports for both resource and custom metrics - `kubectl get hpa.v2beta2.autoscaling` **The metric value should decrease on adding new pods.** <br/> *Wrong metrics Example:* request rate <br/> *Right metrics Example:* average request rate per pod |
734+
735+
Note (1)
736+
```
737+
kubectl get apiservices | awk 'NR==1 || $1 ~ "metrics"'
738+
NAME SERVICE AVAILABLE AGE
739+
v1beta1.custom.metrics.k8s.io monitoring/prom-adapter-prometheus-adapter True 60d
740+
v1beta1.metrics.k8s.io kube-system/metrics-server True 84d
741+
```
742+
743+
744+
**Implementation in Codefresh**
745+
746+
* Default “Enable Autoscaling” settings for GKE
747+
* Using [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter) with custom metrics
748+
749+
We define HPA for cfapi and pipeline-manager services
750+
751+
**CFapi HPA object**
752+
753+
It's based on three metrics (HPA controller scales of only one of the targetValue reached):
754+
755+
```
756+
kubectl get hpa.v2beta1.autoscaling cf-cfapi -oyaml
757+
```
758+
759+
{% highlight yaml %}
760+
{% raw %}
761+
apiVersion: autoscaling/v2beta1
762+
kind: HorizontalPodAutoscaler
763+
metadata:
764+
annotations:
765+
meta.helm.sh/release-name: cf
766+
meta.helm.sh/release-namespace: default
767+
labels:
768+
app.kubernetes.io/managed-by: Helm
769+
name: cf-cfapi
770+
namespace: default
771+
spec:
772+
maxReplicas: 16
773+
metrics:
774+
- object:
775+
metricName: requests_per_pod
776+
target:
777+
apiVersion: v1
778+
kind: Service
779+
name: cf-cfapi
780+
targetValue: "10"
781+
type: Object
782+
- object:
783+
metricName: cpu_usage_avg
784+
target:
785+
apiVersion: apps/v1
786+
kind: Deployment
787+
name: cf-cfapi-base
788+
targetValue: "1"
789+
type: Object
790+
- object:
791+
metricName: memory_working_set_bytes_avg
792+
target:
793+
apiVersion: apps/v1
794+
kind: Deployment
795+
name: cf-cfapi-base
796+
targetValue: 3G
797+
type: Object
798+
minReplicas: 2
799+
scaleTargetRef:
800+
apiVersion: apps/v1
801+
kind: Deployment
802+
name: cf-cfapi-base
803+
{% endraw%}
804+
{% endhighlight %}
805+
806+
* `requests_per_pod` is based on `rate(nginx_ingress_controller_requests)` metric ingested from nginx-ingress-controller
807+
* `cpu_usage_avg` based on cadvisor (from kubelet) rate `(rate(container_cpu_user_seconds_total)`
808+
* `memory_working_set_bytes_avg` based on cadvisor `container_memory_working_set_bytes`
809+
810+
**pipeline-manager HPA**
811+
812+
based on `cpu_usage_avg`
813+
814+
{% highlight yaml %}
815+
{% raw %}
816+
apiVersion: autoscaling/v2beta1
817+
kind: HorizontalPodAutoscaler
818+
metadata:
819+
annotations:
820+
meta.helm.sh/release-name: cf
821+
meta.helm.sh/release-namespace: default
822+
labels:
823+
app.kubernetes.io/managed-by: Helm
824+
name: cf-pipeline-manager
825+
spec:
826+
maxReplicas: 8
827+
metrics:
828+
- object:
829+
metricName: cpu_usage_avg
830+
target:
831+
apiVersion: apps/v1
832+
kind: Deployment
833+
name: cf-pipeline-manager-base
834+
targetValue: 400m
835+
type: Object
836+
minReplicas: 2
837+
scaleTargetRef:
838+
apiVersion: apps/v1
839+
kind: Deployment
840+
name: cf-pipeline-manager-base
841+
{% endraw%}
842+
{% endhighlight %}
843+
844+
**prometheus-adapter configuration**
845+
846+
Reference: [https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config.md](https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config.md
847+
)
848+
849+
{% highlight yaml %}
850+
{% raw %}
851+
Rules:
852+
- metricsQuery: |
853+
kube_service_info{<<.LabelMatchers>>} * on() group_right(service)
854+
(sum(rate(nginx_ingress_controller_requests{<<.LabelMatchers>>}[2m]))
855+
/ on() kube_deployment_spec_replicas{deployment='<<index .LabelValuesByName "service">>-base',namespace='<<index .LabelValuesByName "namespace">>'})
856+
name:
857+
as: requests_per_pod
858+
matches: ^(.*)$
859+
resources:
860+
overrides:
861+
namespace:
862+
resource: namespace
863+
service:
864+
resource: service
865+
seriesQuery: kube_service_info{service=~".*cfapi.*"}
866+
- metricsQuery: |
867+
kube_deployment_labels{<<.LabelMatchers>>} * on(label_app) group_right(deployment)
868+
(label_replace(
869+
avg by (container) (rate(container_cpu_user_seconds_total{container=~"cf-(tasker-kubernetes|cfapi.*|pipeline-manager.*)", job="kubelet", namespace='<<index .LabelValuesByName "namespace">>'}[15m]))
870+
, "label_app", "$1", "container", "(.*)"))
871+
name:
872+
as: cpu_usage_avg
873+
matches: ^(.*)$
874+
resources:
875+
overrides:
876+
deployment:
877+
group: apps
878+
resource: deployment
879+
namespace:
880+
resource: namespace
881+
seriesQuery: kube_deployment_labels{label_app=~"cf-(tasker-kubernetes|cfapi.*|pipeline-manager.*)"}
882+
- metricsQuery: "kube_deployment_labels{<<.LabelMatchers>>} * on(label_app) group_right(deployment)\n
883+
\ (label_replace(\n avg by (container) (avg_over_time (container_memory_working_set_bytes{container=~\"cf-.*\",
884+
job=\"kubelet\", namespace='<<index .LabelValuesByName \"namespace\">>'}[15m]))\n
885+
\ , \"label_app\", \"$1\", \"container\", \"(.*)\"))\n \n"
886+
name:
887+
as: memory_working_set_bytes_avg
888+
matches: ^(.*)$
889+
resources:
890+
overrides:
891+
deployment:
892+
group: apps
893+
resource: deployment
894+
namespace:
895+
resource: namespace
896+
seriesQuery: kube_deployment_labels{label_app=~"cf-.*"}
897+
- metricsQuery: |
898+
kube_deployment_labels{<<.LabelMatchers>>} * on(label_app) group_right(deployment)
899+
label_replace(label_replace(avg_over_time(newrelic_apdex_score[15m]), "label_app", "cf-$1", "exported_app", '(cf-api.*|pipeline-manager|tasker-kuberentes)\\[kubernetes\\]'), "label_app", "$1cfapi$3", "label_app", '(cf-)(cf-api)(.*)')
900+
name:
901+
as: newrelic_apdex
902+
matches: ^(.*)$
903+
resources:
904+
overrides:
905+
deployment:
906+
group: apps
907+
resource: deployment
908+
namespace:
909+
resource: namespace
910+
seriesQuery: kube_deployment_labels{label_app=~"cf-(tasker-kubernetes|cfapi.*|pipeline-manager)"}
911+
{% endraw%}
912+
{% endhighlight %}
913+
914+
**How to define HPA in Codefresh installer (kcfi) config**
915+
916+
Most of Codefresh's Microservices subcharts contain `templates/hpa.yaml`:
917+
918+
{% highlight yaml %}
919+
{% raw %}
920+
{{- if .Values.HorizontalPodAutoscaler }}
921+
apiVersion: autoscaling/v2beta1
922+
kind: HorizontalPodAutoscaler
923+
metadata:
924+
name: {{ template "cfapi.fullname" . }}
925+
spec:
926+
scaleTargetRef:
927+
apiVersion: apps/v1
928+
kind: Deployment
929+
name: {{ template "cfapi.fullname" . }}-{{ .version | default "base" }}
930+
minReplicas: {{ coalesce .Values.HorizontalPodAutoscaler.minReplicas .Values.replicaCount 1 }}
931+
maxReplicas: {{ coalesce .Values.HorizontalPodAutoscaler.maxReplicas .Values.replicaCount 2 }}
932+
metrics:
933+
{{- if .Values.HorizontalPodAutoscaler.metrics }}
934+
{{ toYaml .Values.HorizontalPodAutoscaler.metrics | indent 4 }}
935+
{{- else }}
936+
- type: Resource
937+
resource:
938+
name: cpu
939+
targetAverageUtilization: 60
940+
{{- end }}
941+
{{- end }}
942+
{% endraw%}
943+
{% endhighlight %}
944+
945+
To configure HPA for CFapi add `HorizontalPodAutoscaler` values to config.yaml, for example:
946+
947+
(assuming that we already have prometheus adapter configured for metrics `requests_per_pod`, `cpu_usage_avg`, `memory_working_set_bytes_avg`)
948+
949+
{% highlight yaml %}
950+
{% raw %}
951+
cfapi:
952+
replicaCount: 4
953+
resources:
954+
requests:
955+
memory: "4096Mi"
956+
cpu: "1100m"
957+
limits:
958+
memory: "4096Mi"
959+
cpu: "2200m"
960+
HorizontalPodAutoscaler:
961+
minReplicas: 2
962+
maxReplicas: 16
963+
metrics:
964+
- type: Object
965+
object:
966+
metricName: requests_per_pod
967+
target:
968+
apiVersion: "v1"
969+
kind: Service
970+
name: cf-cfapi
971+
targetValue: 10
972+
- type: Object
973+
object:
974+
metricName: cpu_usage_avg
975+
target:
976+
apiVersion: "apps/v1"
977+
kind: Deployment
978+
name: cf-cfapi-base
979+
targetValue: 1
980+
- type: Object
981+
object:
982+
metricName: memory_working_set_bytes_avg
983+
target:
984+
apiVersion: "apps/v1"
985+
kind: Deployment
986+
name: cf-cfapi-base
987+
targetValue: 3G
988+
{% endraw%}
989+
{% endhighlight %}
990+
991+
**Querying metrics (for debugging)**
992+
993+
CPU Metric API Call
994+
995+
```
996+
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/codefresh/pods/cf-cfapi-base-****-/ | jq
997+
```
998+
999+
Custom Metrics Call
1000+
1001+
```
1002+
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/codefresh/services/cf-cfapi/requests_per_pod | jq
1003+
```
1004+
1005+
## Upgrade the Codefresh Platform
7261006
7271007
To upgrade Codefresh to a newer version
7281008
@@ -741,7 +1021,7 @@ Notice that only `kfci` should be used for Codefresh upgrades. If you still have
7411021
7421022
#### Mongo
7431023
744-
All services using the MongoDB are dependent on the `mongo` pod being up and running.If the `mongo` pod is down, the following dependencies will not work:
1024+
All services using the MongoDB are dependent on the `mongo` pod being up and running. If the `mongo` pod is down, the following dependencies will not work:
7451025
7461026
- `runtime-environment-manager`
7471027
- `pipeline-manager`

‎_docs/whats-new/whats-new.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,11 @@ toc: true
1818

1919
###February 2021
2020

21+
2122
- Concurrency behavior for pending builds -[documentation]({{site.baseurl}}/docs/codefresh-yaml/steps/approval/#define-concurrency-limits)
2223
- Jira integration -[documentation]({{site.baseurl}}/docs/yaml-examples/examples/sending-the-notification-to-jira/)
2324
- SLA details -[documentation]({{site.baseurl}}/docs/terms-and-privacy-policy/sla/)
25+
- Autoscaling recommendations for Codefresh on-prem -[documentation]({{site.baseurl}}/docs/administration/codefresh-on-prem/#app-cluster-autoscaling)
2426
- Hide inaccessible clusters in the Kubernetes dashboard -[documentation]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/#accessing-the-kubernetes-dashboard)
2527
- Define access for non-admins in Helm repositories and shared config -[documentation]({{site.baseurl}}/docs/configure-ci-cd-pipeline/shared-configuration/#level-of-access)
2628
- Okta auto-sync of teams -[documentation]({{site.baseurl}}/docs/administration/single-sign-on/sso-okta/#syncing-of-teams-after-initial-sso-setup)

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp