Keda ScaleJob parallelism hard-coded to "1"

[Bastichou]

Hi,

In the Helm chart Keda ScaleJob's parallelism attribute is hard-coded to one. This is preventing us to run several jobs in parallel.

While testing my pipeline implementation, the agent was running only one job (pod) at the time increasing queue and waiting time a lot.
Is this an expected behaviour, perhaps parallelism must be handled somewhere else for this use-case (still discovering Azure Pipeline) ? Or should we able to turn on parallelism setting this attribute as a Helm value?

(I've got a small PR to turn this attribute as a values around the corner)

Cheers!

[clemlesne]

Autoscaling policy is managed by the autoscaling.maxReplicas option, default to 100.

The parallelism option from Keda is "how much instances should I scale for one trigger event", in that case a trigger event is "a job is queued on Azure DevOps". We want 1 instance for each event.

But what you saw is important, did the agent succeed to scale? If not, you can create an issue with logs, installation steps, and values used.

Note: Keda polls Azure DevOps every 15 secs to fetch the queue details, and trigger instances if needed. It means the delay between pipeline queued and instance ran (not ready, this depends from your Kubernetes context) is 1-16 secs. I done it to not overflow the server and avoid trigger API limits. If you want this to be customized, it could easily.

Edit: Created an issue to enhance documentation on scaling topic.

[Bastichou]

So Keda is watching for event "a pipeline is queued" while I though "a job is queued". This explains why I've only one agent running, I only triggering one pipeline with severals jobs. But I'm wondering how to deal with parallel jobs in the a same pipeline now.

I'll dig more, I'll be glad to add a bunch of docs once I got it right :D

I'll keep you posted (probably end of the week), thanks a lot for your quick answer.

[clemlesne]

You’re right, KEDA is well watching for “a job is queued”, and not “a pipeline is queued”. An instance should ran for each job you run in Azure DevOps!

[Bastichou]

Hi,

So I moved forward on the PoC implementation and it looks like the issue was on my side (as expected hehe).
Everything is working much better once I set up the placeholder agent which allow scaling to 0: https://keda.sh/blog/2021-05-27-azure-pipelines-scaler/#placeholder-agent

Now, scaling is working fine up to the parallel jobs available on the Azure Devops configuration.

Note: Parallelism attribute is the number of job a new event will trigger, so for one job scheduled would spin-up x jobs according the parallelism value.

I'm still trying to understand why having the placeholder agent (offline) help on the scaling of this. This is most likely something related to your note:

Note: Keda polls Azure DevOps every 15 secs to fetch the queue details, and trigger instances if needed. It means the delay between pipeline queued and instance ran (not ready, this depends from your Kubernetes context) is 1-16 secs

I identified an issue scaling down jobs to 0 when there is no event to deal with, I'll create an issue :)

Thanks you for you help

[clemlesne]

Oh indeed the placeholder agent (I called “template” in the code) should be kept forever.

Template job is already created:

blue-agent/src/helm/blue-agent/templates/pod.yaml

Lines 15 to 19 in 80158ef

	{{- $data := dict
	"restartPolicy" "Never"
	"azpAgentName" (dict "value" (printf "%s-%s" (include "blue-agent.fullname" .) "template"))
	}}
	{{- include "blue-agent.podSharedTemplate" (merge (dict "Args" $data) . ) | nindent 2 }}

But, all agents are made to be cleaned up after run:

blue-agent/src/docker/start.sh

Lines 120 to 125 in 80158ef

	# Unregister on success
	trap 'unregister; exit 0' EXIT
	# Unregister on Ctrl+C
	trap 'unregister; exit 130' INT
	# Unregister on SIGTERM
	trap 'unregister; exit 143' TERM

What it could be implemented is to add a flag to ignore the cleanup process for the template instance. Accordingly to the doc, it should work?

[clemlesne]

Just created a bug report: #294 (comment)