From 55c74d79aa61850371d244fb981f85df8a9b60af Mon Sep 17 00:00:00 2001 From: kthfre Date: Thu, 29 Aug 2024 15:29:31 +0200 Subject: [PATCH 01/18] week 2: demo proposal --- .../demo/week2/mqnguyen-golman/README.md | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 contributions/demo/week2/mqnguyen-golman/README.md diff --git a/contributions/demo/week2/mqnguyen-golman/README.md b/contributions/demo/week2/mqnguyen-golman/README.md new file mode 100644 index 0000000000..c3efc45dda --- /dev/null +++ b/contributions/demo/week2/mqnguyen-golman/README.md @@ -0,0 +1,41 @@ +# Assignment Proposal + +## Title + +Continuous integretion testing of node and service availability of promox instances + +## Names and KTH ID + + - Minh Quang Nguyen (mqnguyen@kth.se) + - Fredrik Gölman (golman@kth.se) + +## Deadline +- Week 2 + +## Category +- Demo + +## Description + +Our demo will be performed in the context of physical servers that we access through a VPN. + +We want to test the following +- Availablity of different nodes in a proxmox cluster +- Availabilty of different services on these nodes + +Tools: +- Tailscale (VPN service/mesh network) +- Proxmox (VM virtualization) +- Jenkins (Automation Testing and Integration Testing) + +- Option Currently Deciding +- Ansible (Testing Node instance, Update version ...etc) +- Terraform (Create Template for VM/docker/Container) + +## Backup plan + +In case we run into difficulties we deem unfeasible to overcome (the environment is not fully set up yet, and we are sort of short of time given the schedule) we discussed with the TA to create a rather simple web application involving either authentication functionality and a web form with input where validation would occur at the backend and perform integration tests by interacting with the browser and test frontend, backend, and potentially some database. We (and the TA) were unsure on whether the complexity level of this would be sufficient, but were asked to include it to have it commented on. + +**Relevance** + +We believe this is very rare compared to more conventional integration tests that may, for example, include a frontend and backend the the interactions between those two components. We also believe it is comparatively complex as it involves a lot of steps both in terms of setting up the environment and performing the tests. It also seem to fulfill a task that frequently would need to be performed manually by a system administrator otherwise. It is of course highly relevant to DevOps in general as it involves several CI tools such as Jenkins. \ No newline at end of file From 230a55ec402d14334fbafdfd3dff745968b0a84c Mon Sep 17 00:00:00 2001 From: kthfre Date: Thu, 29 Aug 2024 15:34:45 +0200 Subject: [PATCH 02/18] week 2: demo proposal --- contributions/demo/week2/mqnguyen-golman/README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/contributions/demo/week2/mqnguyen-golman/README.md b/contributions/demo/week2/mqnguyen-golman/README.md index c3efc45dda..ebb4f00596 100644 --- a/contributions/demo/week2/mqnguyen-golman/README.md +++ b/contributions/demo/week2/mqnguyen-golman/README.md @@ -32,8 +32,7 @@ Tools: - Ansible (Testing Node instance, Update version ...etc) - Terraform (Create Template for VM/docker/Container) -## Backup plan - +Backup plan: In case we run into difficulties we deem unfeasible to overcome (the environment is not fully set up yet, and we are sort of short of time given the schedule) we discussed with the TA to create a rather simple web application involving either authentication functionality and a web form with input where validation would occur at the backend and perform integration tests by interacting with the browser and test frontend, backend, and potentially some database. We (and the TA) were unsure on whether the complexity level of this would be sufficient, but were asked to include it to have it commented on. **Relevance** From 562fe78bd8e5cd1283e695fecc7ea56eaf40649d Mon Sep 17 00:00:00 2001 From: kthfre Date: Thu, 5 Sep 2024 15:06:32 +0200 Subject: [PATCH 03/18] week 3: presentation proposal --- .../week3/mqnguyen-golman/README.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 contributions/presentation/week3/mqnguyen-golman/README.md diff --git a/contributions/presentation/week3/mqnguyen-golman/README.md b/contributions/presentation/week3/mqnguyen-golman/README.md new file mode 100644 index 0000000000..6f2a15926c --- /dev/null +++ b/contributions/presentation/week3/mqnguyen-golman/README.md @@ -0,0 +1,24 @@ +# Assignment Proposal + +## Title + +Rainbow deployments and its role in modern CD + +## Names and KTH ID + + - Minh Quang Nguyen (mqnguyen@kth.se) + - Fredrik Gölman (golman@kth.se) + +## Deadline +- Week 3 + +## Category +- Presentation + +## Description + +A brief introduction of the problem it attempts to solve when deploying new software versions. We would try to briefly cover previous (or "lesser") relevant similar methodologies such as blue/green deployment and discuss some limitations that motivate the extension, which is rainbow deployment. Such limitations may include multiple backend services serving an application or running long-term jobs that may not be interrupted blocking deployment of new versions, both cases advocating the need for more colors. + +**Relevance** + +We believe it's relevant to DevOps in general and in particular to the subject of week 3 as it deals with continuous deployment. We also believe no one has covered it previously in the course, although there is some coverage of blue/green deployment. From a60ff82efb25e7d3c24a69031a96da8c69f0d37c Mon Sep 17 00:00:00 2001 From: kthfre Date: Wed, 11 Sep 2024 13:22:08 +0200 Subject: [PATCH 04/18] merge --- contributions/demo/week2/mqnguyen-golman/README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/contributions/demo/week2/mqnguyen-golman/README.md b/contributions/demo/week2/mqnguyen-golman/README.md index 9249f21077..a444cb5c5d 100644 --- a/contributions/demo/week2/mqnguyen-golman/README.md +++ b/contributions/demo/week2/mqnguyen-golman/README.md @@ -2,11 +2,7 @@ ## Title -<<<<<<< HEAD -Continuous integretion testing of node and service availability of promox instances -======= Continuous integration testing of node and service availability of promox instances ->>>>>>> upstream/2024 ## Names and KTH ID From 5ebc6aac52d9b308dc57634df7a0c5266e43bd7a Mon Sep 17 00:00:00 2001 From: kthfre Date: Sat, 21 Sep 2024 19:16:32 +0200 Subject: [PATCH 05/18] executable tutorial proposal --- .../executable-tutorial/golman/README.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 contributions/executable-tutorial/golman/README.md diff --git a/contributions/executable-tutorial/golman/README.md b/contributions/executable-tutorial/golman/README.md new file mode 100644 index 0000000000..e6868289d5 --- /dev/null +++ b/contributions/executable-tutorial/golman/README.md @@ -0,0 +1,30 @@ +# Assignment Proposal + +## Title + +End-to-end training of a neural network to deployment in a live application + +## Names and KTH ID + + - Fredrik Gölman (golman@kth.se) + +## Deadline +- Asynchronous deadline goes here? + +## Category +- Executable Tutorial + +## Description + +I'd like to make an executable tutorial that goes through the training of a neural network in a Jupyter notebook on Colab, handling the intermediary steps, and deployment to some live application, so the end-to-end process. I'd put limited focus on the ML aspects and greater focus on the DevOps aspects. I'd like to whip together my own functionality for the DevOps parts, if I may, as it's a fun learning experience and could be meaningful scripts for future usage. The deployment criteria for the model could be to exceed previous test data accuracy, but there could also be any other reasonable criteria. I haven't fully decided on the functionality for the MLops/DevOps part. The bare minimum is actually deploying the model live when fulfilling the criteria. Other things being considered are model storage/rollback, job scheduling/queue in running notebooks, monitoring of multiple notebooks, etc. + +Architecture wise there would be: +- The Colab notebook running the ML stuff (and some network connectivity). +- An MLops tool consisting of a backend and a corresponding GUI (web). +- A demo web application that uses the model on the backend. + +I asked TA about this briefly in a lab session (not previous, but one before that) and it sounded OK. I meant to register it earlier, but other coursework came in between. I think it's still OK to register an MLops task since it's asynchronous and there is no "week" folder structure in the directory tree. So if it is, and the proposal sounds OK, is all I have to do commit to a deadline and deliver? + +**Relevance** + +Jupyter Notebook/Lab is often used for processing, preparing, and visualizing data, as well as subsequently training machine learning models. The process of deriving a model is often an iterative process to determine suitable model architectures and optimal hyperparameters. Models may furthermore require continuous altering after deployment as more data becomes available or use cases change. This process is presumably often done manually, particularly as data scientists and conventional developers may be different teams, but there are clear benefits in automating the process. From 9c7af465006a440cb329bad761fcdb631b77a02e Mon Sep 17 00:00:00 2001 From: kthfre Date: Thu, 26 Sep 2024 15:45:14 +0200 Subject: [PATCH 06/18] merge --- contributions/executable-tutorial/golman/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contributions/executable-tutorial/golman/README.md b/contributions/executable-tutorial/golman/README.md index e6868289d5..c3c8011949 100644 --- a/contributions/executable-tutorial/golman/README.md +++ b/contributions/executable-tutorial/golman/README.md @@ -9,10 +9,10 @@ End-to-end training of a neural network to deployment in a live application - Fredrik Gölman (golman@kth.se) ## Deadline -- Asynchronous deadline goes here? +- Task 1 ## Category -- Executable Tutorial +- Executable tutorial ## Description From 5e754cbb683c180c40aad9023d64ec60a50e2219 Mon Sep 17 00:00:00 2001 From: kthfre Date: Sat, 28 Sep 2024 18:02:30 +0200 Subject: [PATCH 07/18] update --- contributions/executable-tutorial/golman/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributions/executable-tutorial/golman/README.md b/contributions/executable-tutorial/golman/README.md index c3c8011949..252f4393ea 100644 --- a/contributions/executable-tutorial/golman/README.md +++ b/contributions/executable-tutorial/golman/README.md @@ -9,7 +9,7 @@ End-to-end training of a neural network to deployment in a live application - Fredrik Gölman (golman@kth.se) ## Deadline -- Task 1 +- Task 2 ## Category - Executable tutorial From 62ae443c59266c1cc208a76f8c18c62d473f1be6 Mon Sep 17 00:00:00 2001 From: kthfre Date: Sat, 28 Sep 2024 18:12:34 +0200 Subject: [PATCH 08/18] update --- contributions/executable-tutorial/golman/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributions/executable-tutorial/golman/README.md b/contributions/executable-tutorial/golman/README.md index c3c8011949..252f4393ea 100644 --- a/contributions/executable-tutorial/golman/README.md +++ b/contributions/executable-tutorial/golman/README.md @@ -9,7 +9,7 @@ End-to-end training of a neural network to deployment in a live application - Fredrik Gölman (golman@kth.se) ## Deadline -- Task 1 +- Task 2 ## Category - Executable tutorial From 1574721d7035e828d21658fcd529b57383d1d68f Mon Sep 17 00:00:00 2001 From: kthfre Date: Sat, 5 Oct 2024 17:16:37 +0200 Subject: [PATCH 09/18] merge --- contributions/feedback/golman/README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 contributions/feedback/golman/README.md diff --git a/contributions/feedback/golman/README.md b/contributions/feedback/golman/README.md new file mode 100644 index 0000000000..e9a83b25bd --- /dev/null +++ b/contributions/feedback/golman/README.md @@ -0,0 +1,23 @@ +# Assignment Proposal + +## Title + +Feedback for the executable tutorial "Infrastructure as Code with Pulumi". + +## Names and KTH ID + +- Fredrik Gölman (golman@kth.se) + +## Deadline + +- Task 3 + +## Category + +- Feedback + +## Description + +Feedback for [this executable tutorial](https://github.com/KTH/devops-course/tree/2024/contributions/executable-tutorial/mqnguyen) by mqnguyen@kth.se. + +I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? \ No newline at end of file From 47436268dec0dba3fb33db27ec4802abefa432c7 Mon Sep 17 00:00:00 2001 From: kthfre Date: Tue, 8 Oct 2024 20:13:00 +0200 Subject: [PATCH 10/18] merge --- contributions/executable-tutorial/golman/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/contributions/executable-tutorial/golman/README.md b/contributions/executable-tutorial/golman/README.md index 252f4393ea..72bcdee67a 100644 --- a/contributions/executable-tutorial/golman/README.md +++ b/contributions/executable-tutorial/golman/README.md @@ -28,3 +28,5 @@ I asked TA about this briefly in a lab session (not previous, but one before tha **Relevance** Jupyter Notebook/Lab is often used for processing, preparing, and visualizing data, as well as subsequently training machine learning models. The process of deriving a model is often an iterative process to determine suitable model architectures and optimal hyperparameters. Models may furthermore require continuous altering after deployment as more data becomes available or use cases change. This process is presumably often done manually, particularly as data scientists and conventional developers may be different teams, but there are clear benefits in automating the process. + +__Submission__: [Github repository](https://github.com/kthfre/dd2482-executable-tutorial) From b4f6af3ae374fa1076c5041c6324f5f97b2534f2 Mon Sep 17 00:00:00 2001 From: kthfre Date: Tue, 8 Oct 2024 20:16:31 +0200 Subject: [PATCH 11/18] merge --- contributions/executable-tutorial/golman/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/contributions/executable-tutorial/golman/README.md b/contributions/executable-tutorial/golman/README.md index 72bcdee67a..ae740eb932 100644 --- a/contributions/executable-tutorial/golman/README.md +++ b/contributions/executable-tutorial/golman/README.md @@ -29,4 +29,6 @@ I asked TA about this briefly in a lab session (not previous, but one before tha Jupyter Notebook/Lab is often used for processing, preparing, and visualizing data, as well as subsequently training machine learning models. The process of deriving a model is often an iterative process to determine suitable model architectures and optimal hyperparameters. Models may furthermore require continuous altering after deployment as more data becomes available or use cases change. This process is presumably often done manually, particularly as data scientists and conventional developers may be different teams, but there are clear benefits in automating the process. -__Submission__: [Github repository](https://github.com/kthfre/dd2482-executable-tutorial) +**Submission** + +[Github repository](https://github.com/kthfre/dd2482-executable-tutorial) From 1bc82469bf2499deed076092a45da639f7fb6d79 Mon Sep 17 00:00:00 2001 From: kthfre Date: Tue, 8 Oct 2024 20:25:57 +0200 Subject: [PATCH 12/18] merge --- contributions/executable-tutorial/golman/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/contributions/executable-tutorial/golman/README.md b/contributions/executable-tutorial/golman/README.md index 252f4393ea..ae740eb932 100644 --- a/contributions/executable-tutorial/golman/README.md +++ b/contributions/executable-tutorial/golman/README.md @@ -28,3 +28,7 @@ I asked TA about this briefly in a lab session (not previous, but one before tha **Relevance** Jupyter Notebook/Lab is often used for processing, preparing, and visualizing data, as well as subsequently training machine learning models. The process of deriving a model is often an iterative process to determine suitable model architectures and optimal hyperparameters. Models may furthermore require continuous altering after deployment as more data becomes available or use cases change. This process is presumably often done manually, particularly as data scientists and conventional developers may be different teams, but there are clear benefits in automating the process. + +**Submission** + +[Github repository](https://github.com/kthfre/dd2482-executable-tutorial) From d81c0a7c7db5d3cfd379c49c1011778e9730bdcb Mon Sep 17 00:00:00 2001 From: kthfre Date: Wed, 9 Oct 2024 06:05:04 +0200 Subject: [PATCH 13/18] update --- contributions/feedback/{golman => golman-cnra}/README.md | 1 + 1 file changed, 1 insertion(+) rename contributions/feedback/{golman => golman-cnra}/README.md (92%) diff --git a/contributions/feedback/golman/README.md b/contributions/feedback/golman-cnra/README.md similarity index 92% rename from contributions/feedback/golman/README.md rename to contributions/feedback/golman-cnra/README.md index e9a83b25bd..e1bd15ad5c 100644 --- a/contributions/feedback/golman/README.md +++ b/contributions/feedback/golman-cnra/README.md @@ -7,6 +7,7 @@ Feedback for the executable tutorial "Infrastructure as Code with Pulumi". ## Names and KTH ID - Fredrik Gölman (golman@kth.se) +- Chandni Narottambhai Rakhashiya (cnra@kth.se) ## Deadline From 1c17178757966363b16b51450a33759c79131de0 Mon Sep 17 00:00:00 2001 From: kthfre Date: Thu, 17 Oct 2024 15:48:02 +0200 Subject: [PATCH 14/18] update --- contributions/feedback/golman-cnra/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/contributions/feedback/golman-cnra/README.md b/contributions/feedback/golman-cnra/README.md index e1bd15ad5c..2ec90103bd 100644 --- a/contributions/feedback/golman-cnra/README.md +++ b/contributions/feedback/golman-cnra/README.md @@ -21,4 +21,6 @@ Feedback for the executable tutorial "Infrastructure as Code with Pulumi". Feedback for [this executable tutorial](https://github.com/KTH/devops-course/tree/2024/contributions/executable-tutorial/mqnguyen) by mqnguyen@kth.se. -I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? \ No newline at end of file +I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? + +This feedback assignment has been [submitted](https://github.com/KTH/devops-course/pull/2540#issuecomment-2419590447). \ No newline at end of file From 58d2ffd935b9c35cdba0fbdf5a65f53f91abbde4 Mon Sep 17 00:00:00 2001 From: kthfre Date: Mon, 21 Oct 2024 18:28:12 +0200 Subject: [PATCH 15/18] update --- contributions/feedback/golman-cnra/README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/contributions/feedback/golman-cnra/README.md b/contributions/feedback/golman-cnra/README.md index 2ec90103bd..e1bd15ad5c 100644 --- a/contributions/feedback/golman-cnra/README.md +++ b/contributions/feedback/golman-cnra/README.md @@ -21,6 +21,4 @@ Feedback for the executable tutorial "Infrastructure as Code with Pulumi". Feedback for [this executable tutorial](https://github.com/KTH/devops-course/tree/2024/contributions/executable-tutorial/mqnguyen) by mqnguyen@kth.se. -I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? - -This feedback assignment has been [submitted](https://github.com/KTH/devops-course/pull/2540#issuecomment-2419590447). \ No newline at end of file +I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? \ No newline at end of file From 1f863d39ea179ad253f75b44748c0c7ccd01bd19 Mon Sep 17 00:00:00 2001 From: kthfre Date: Tue, 22 Oct 2024 18:06:14 +0200 Subject: [PATCH 16/18] update feedback --- contributions/feedback/golman-cnra/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/contributions/feedback/golman-cnra/README.md b/contributions/feedback/golman-cnra/README.md index e1bd15ad5c..2ec90103bd 100644 --- a/contributions/feedback/golman-cnra/README.md +++ b/contributions/feedback/golman-cnra/README.md @@ -21,4 +21,6 @@ Feedback for the executable tutorial "Infrastructure as Code with Pulumi". Feedback for [this executable tutorial](https://github.com/KTH/devops-course/tree/2024/contributions/executable-tutorial/mqnguyen) by mqnguyen@kth.se. -I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? \ No newline at end of file +I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? + +This feedback assignment has been [submitted](https://github.com/KTH/devops-course/pull/2540#issuecomment-2419590447). \ No newline at end of file From a4511f57072f0041eb181cf943a59ee9c1735447 Mon Sep 17 00:00:00 2001 From: kthfre Date: Tue, 22 Oct 2024 18:10:00 +0200 Subject: [PATCH 17/18] update --- contributions/feedback/golman-cnra/README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/contributions/feedback/golman-cnra/README.md b/contributions/feedback/golman-cnra/README.md index 2ec90103bd..e1bd15ad5c 100644 --- a/contributions/feedback/golman-cnra/README.md +++ b/contributions/feedback/golman-cnra/README.md @@ -21,6 +21,4 @@ Feedback for the executable tutorial "Infrastructure as Code with Pulumi". Feedback for [this executable tutorial](https://github.com/KTH/devops-course/tree/2024/contributions/executable-tutorial/mqnguyen) by mqnguyen@kth.se. -I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? - -This feedback assignment has been [submitted](https://github.com/KTH/devops-course/pull/2540#issuecomment-2419590447). \ No newline at end of file +I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? \ No newline at end of file From 49b3593006a5b5a1fb9c6f384a05b087b526d15d Mon Sep 17 00:00:00 2001 From: kthfre Date: Tue, 22 Oct 2024 18:33:13 +0200 Subject: [PATCH 18/18] update --- contributions/feedback/golman-cnra/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/contributions/feedback/golman-cnra/README.md b/contributions/feedback/golman-cnra/README.md index e1bd15ad5c..2ec90103bd 100644 --- a/contributions/feedback/golman-cnra/README.md +++ b/contributions/feedback/golman-cnra/README.md @@ -21,4 +21,6 @@ Feedback for the executable tutorial "Infrastructure as Code with Pulumi". Feedback for [this executable tutorial](https://github.com/KTH/devops-course/tree/2024/contributions/executable-tutorial/mqnguyen) by mqnguyen@kth.se. -I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? \ No newline at end of file +I've received approval from the author to leave feedback on this assignment. Am I making the correct assumption that the deadline for such feedback would be "Task 3" + 48h? + +This feedback assignment has been [submitted](https://github.com/KTH/devops-course/pull/2540#issuecomment-2419590447). \ No newline at end of file