Add Pipeline design #21

Yancey1989 · 2020-03-23T12:10:41Z

Add API design for Tekton Pipeline.

Yancey1989 · 2020-03-23T14:52:46Z

typhoonzero · 2020-03-24T04:59:26Z

doc/design.md

@@ -101,7 +101,7 @@ spec:
 We hope Fluid users could represent it by the following line.

 ```python
-skaffold_git = fluid.Git(
+skaffold_git = fluid.git_resource(


Please also update below descriptions: Please be aware that the call to fluid.Git doesn't include the name

Why not use fluid.git for simple?

Why not use fluid.git for simple?

Just tracing the current implementation. I think we can update the API and design in another PR if need.

I remember that Tekton has a limited number of pre-defined resource types and git is one of them. I would suggest we keep it Git other than git_resource, because git_resouce is not a fullname; git_pipeline_resource is. But git_pipeline_resource is too long. It seems reasonable to use a short name Git for one of a few pre-defined types.

doc/design.md

typhoonzero · 2020-03-24T05:04:17Z

doc/design.md

+
+``` python
+@fluid.pipeline
+def tutorial(source_repo:"resource,git", web_image="resource,image"):


source_repo="resource,git"

I think it's okay, this is the parameter annotation instead of the default value, ref: https://www.python.org/dev/peps/pep-3107/#id29

typhoonzero · 2020-03-24T05:09:07Z

doc/design.md

+    build_skaffold_web = build_docker_image_from_git_source(source_repo, web_image)
+
+    deploy_web = deploy_using_kubectl(source_repo, web_image)
+    deploy_web.web_image.from(build_skaffold_web)


How to define dependency by not using input/output?

Can use the runAfter keyword, and I added a section PiepeLine with DAG to introduce how to construct the DAG.

tonyyang-svail · 2020-03-25T00:10:57Z

doc/design.md

+
+``` yaml
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline


What is the lifecycle of a pipeline object?

Pipeline object is like function definition in Python which includes Pipeline Tasks
A PipelineRun object would invoke the Pipeline Tasks as the dependency, can find some information from Task Status

doc/design.md

wangkuiyi · 2020-03-28T23:11:11Z

doc/design.md

+
+This will result in the following execution graph:
+
+``` text


No whitespace between ``` and text

wangkuiyi · 2020-03-29T04:29:01Z

doc/design.md

@@ -101,7 +101,7 @@ spec:
 We hope Fluid users could represent it by the following line.

 ```python
-skaffold_git = fluid.Git(
+skaffold_git = fluid.git_resource(


I remember that Tekton has a limited number of pre-defined resource types and git is one of them. I would suggest we keep it Git other than git_resource, because git_resouce is not a fullname; git_pipeline_resource is. But git_pipeline_resource is too long. It seems reasonable to use a short name Git for one of a few pre-defined types.

wangkuiyi · 2020-03-29T04:29:38Z

doc/design.md

@@ -125,7 +125,7 @@ spec:
 We hope Fluid users could represent the above YAML file by the following line.

 ```python
-skaffold_image_leeroy_web = fluid.Image(
+skaffold_image_leeroy_web = fluid.image_resource(


Similarly, "image" is one of a few pre-defined resource types. How about we keep the name Image.

wangkuiyi · 2020-03-29T04:29:49Z

doc/design.md

@@ -142,7 +142,7 @@ According to the [document](https://github.com/tektoncd/pipeline/blob/master/doc
 The following example from the [Tekton tutorial](https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md#task-inputs-and-outputs) takes an input resource, an output resource, and two input parameters.

 ```yaml
-goapiVersion: tekton.dev/v1alpha1
+apiVersion: tekton.dev/v1alpha1


Thanks for pointing this out!

wangkuiyi · 2020-03-29T04:35:15Z

doc/design.md

+
+### Pipeline
+
+A `Pipeline` object is like function declaration, according to the [definition](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md).


In the above text, we stated that a Task is like a function. Here we state the same with Pipeline. What is the difference between these two types of "functions"?

wangkuiyi · 2020-03-29T04:36:31Z

doc/design.md

+          value: "spec.template.spec.containers[0].image"
+```
+
+The above `Pipeline` is referencing a `Task` called `deploy-using-kubectl` defined as follows:


is referencing => refers to

wangkuiyi · 2020-03-29T04:38:38Z

doc/design.md

+    build_skaffold_web = build_docker_image_from_git_source(source_repo, web_image)
+
+    deploy_web = deploy_using_kubectl(source_repo, web_image)
+    deploy_web.web_image.from(build_skaffold_web)


This deploy_web.web_image.from syntax looks confusing. Python programmer do not do this with function parameters.

I am afraid that this weird design might come from the fact that a Pipeline is NOT similar to a function definition.

After reading more about Pipeline, I see it describes a DAG of tasks, where edges are data dependencies between tasks.

Thinking about the following example from the Tekton tutorial https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#from:

- name: build-app taskRef: name: build-push resources: outputs: - name: image resource: my-image - name: deploy-app taskRef: name: deploy-kubectl resources: inputs: - name: image resource: my-image from: - build-app

Using programming language idiom, it is simply function calls

deploy_kubectl(image=build_push(my_image))

It seems that what we expect users to write is

@fluid.pipeline def build_and_deploy(image): deploy_kubectl(image=build_push(my_image))

where @fluid.pipeline should dry-run the function build_and_deploy to analysis the function dependencies, which is deploy_kubectl.image <- build_push, and generate the YAML definition of the Pipeline object.

I am not sure if the above suggestion is correct, or how reasonable it is. It has been a while I haven't use Tekton.

wangkuiyi · 2020-03-29T04:39:06Z

doc/design.md

+As the following example of `Pipeline spec` comes from Tekton Pipeline tutorials:
+
+``` yaml
+- name: lint-repo


This doesn't look like a complete Kubernetes YAML. What is its kind?

wangkuiyi · 2020-03-29T04:39:53Z

doc/design.md

+- `from`:  clauses on the PipelineResources needed by a Task.
+- `runAfter`: clauses on the Pipeline Tasks.
+
+As the following example of `Pipeline spec` comes from Tekton Pipeline tutorials:


Which tutorial? We need a URL here.

Yancey1989 added 2 commits March 23, 2020 20:09

add pipline design

4f55d49

add pipeline design

e52bd0a

Yancey1989 force-pushed the pipeline_design branch from b6daece to e52bd0a Compare March 23, 2020 14:50

Yancey1989 changed the title ~~[wip]Add Pipeline design~~ Add Pipeline design Mar 23, 2020

Yancey1989 requested review from wangkuiyi and tonyyang-svail March 23, 2020 14:51

typhoonzero reviewed Mar 24, 2020

View reviewed changes

tonyyang-svail reviewed Mar 25, 2020

View reviewed changes

update

bcd2f8e

wangkuiyi reviewed Mar 29, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Pipeline design #21

Add Pipeline design #21

Yancey1989 commented Mar 23, 2020 •

edited

Loading

Yancey1989 commented Mar 23, 2020

typhoonzero Mar 24, 2020

typhoonzero Mar 24, 2020

Yancey1989 Mar 27, 2020

wangkuiyi Mar 29, 2020

typhoonzero Mar 24, 2020

Yancey1989 Mar 27, 2020 •

edited

Loading

typhoonzero Mar 24, 2020

Yancey1989 Mar 27, 2020 •

edited

Loading

tonyyang-svail Mar 25, 2020

Yancey1989 Mar 27, 2020

wangkuiyi Mar 28, 2020

wangkuiyi Mar 29, 2020

wangkuiyi Mar 29, 2020

wangkuiyi Mar 29, 2020

wangkuiyi Mar 29, 2020

wangkuiyi Mar 29, 2020

wangkuiyi Mar 29, 2020

wangkuiyi Mar 29, 2020

wangkuiyi Mar 29, 2020

wangkuiyi Mar 29, 2020


		### Pipeline

		A `Pipeline` object is like function declaration, according to the [definition](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md).

Add Pipeline design #21

Are you sure you want to change the base?

Add Pipeline design #21

Conversation

Yancey1989 commented Mar 23, 2020 • edited Loading

Yancey1989 commented Mar 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yancey1989 Mar 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yancey1989 Mar 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yancey1989 commented Mar 23, 2020 •

edited

Loading

Yancey1989 Mar 27, 2020 •

edited

Loading

Yancey1989 Mar 27, 2020 •

edited

Loading