diff --git a/docs/sphinx/source/deploy-docker.ipynb b/docs/sphinx/source/deploy-docker.ipynb index 8a6d119f..2c4eeca6 100644 --- a/docs/sphinx/source/deploy-docker.ipynb +++ b/docs/sphinx/source/deploy-docker.ipynb @@ -6,7 +6,26 @@ "source": [ "# Deploy with Docker \n", "\n", - "The simplest way to deploy a Vespa app." + "The simplest way to deploy a Vespa app.\n", + "\n", + "Install `pyvespa` and make sure Docker is running with minimum 4G:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Total Memory: 11.7GiB\r\n" + ] + } + ], + "source": [ + "!docker info | grep \"Total Memory\"" ] }, { @@ -67,15 +86,8 @@ "import os\n", "from vespa.deployment import VespaDocker\n", "\n", - "disk_folder = os.path.join(os.getenv(\"WORK_DIR\"), \"sample_application\") # specify your desired absolute path here\n", - "vespa_docker = VespaDocker(\n", - " port=8080,\n", - " disk_folder=disk_folder\n", - ")\n", - "\n", - "app = vespa_docker.deploy(\n", - " application_package = app_package,\n", - ")" + "vespa_docker = VespaDocker()\n", + "app = vespa_docker.deploy(application_package=app_package)" ] }, { @@ -91,11 +103,8 @@ "metadata": {}, "outputs": [], "source": [ - "from shutil import rmtree\n", - "\n", "vespa_docker.container.stop()\n", - "vespa_docker.container.remove()\n", - "rmtree(disk_folder, ignore_errors=True)" + "vespa_docker.container.remove()" ] }, { @@ -109,14 +118,17 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "It is important to know that `pyvespa` simply provides a convenient API to define Vespa application packages from python. `vespa_docker.deploy` export Vespa configuration files to the `disk_folder` defined above. Going through those files is a nice way to start learning about Vespa syntax." + "It is important to know that `pyvespa` simply provides a convenient API\n", + "to define Vespa application packages from python.\n", + "Going through those files is a nice way to start learning about Vespa syntax." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "It is also possible to export the Vespa configuration files representing an application package created with `pyvespa` without deploying the application by using the `export_application_package` method: " + "Use [to_files](reference-api.rst#vespa.package.ApplicationPackage.to_files)\n", + "to export the application package package created with `pyvespa` from code to files:" ] }, { @@ -125,17 +137,16 @@ "metadata": {}, "outputs": [], "source": [ - "vespa_docker.export_application_package(\n", - " application_package=app_package, \n", - ")" + "from pathlib import Path\n", + "Path(\"mydir\").mkdir(parents=True, exist_ok=True)\n", + "app_package.to_files(\"mydir\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "This will export the application files to an `application` folder within the `disk_folder`.\n", - "Remove the files after use:" + "Inspect the exported files:" ] }, { @@ -144,48 +155,60 @@ "metadata": {}, "outputs": [], "source": [ - "rmtree(disk_folder, ignore_errors=True)" + "!find mydir" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Deploy application package from config files" + "It is also possible to export the application package as a zipped file\n", + "using [to_zipfile](reference-api.rst#vespa.package.ApplicationPackage.to_zipfile).\n", + "The zipfile can later be deployed with pyvespa or the [Vespa CLI](https://docs.vespa.ai/en/vespa-cli.html):" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "app_package.to_zipfile(\"myzip.zip\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The `pyvespa` API provides a subset of the functionality available in Vespa. The reason is that `pyvespa` is meant to be used as an experimentation tool for Information Retrieval (IR) and not for building production-ready applications. So, the python API expands based on the needs to replicate common use cases that often require IR experimentation.\n", - "\n", - "If the application requires functionality or fine-tuning not available in `pyvespa`, simply build it directly using Vespa configuration files as shown in [many examples](https://docs.vespa.ai/en/getting-started.html) on Vespa docs. But even in this case, one can still get value out of `pyvespa` by deploying it from python based on the Vespa configuration files stored on disk.\n", - "\n", - "Make sure Docker is running with minimum 4G:" + "Remove the files after use:" ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " Total Memory: 11.7GiB\r\n" - ] - } - ], + "outputs": [], "source": [ - "!docker info | grep \"Total Memory\"" + "from shutil import rmtree\n", + "rmtree(\"mydir\", ignore_errors=True)\n", + "rmtree(\"myzip.zip\", ignore_errors=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ + "## Deploy application package from config files" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `pyvespa` API provides a subset of the functionality available in Vespa. The reason is that `pyvespa` is meant to be used as an experimentation tool for Information Retrieval (IR) and not for building production-ready applications. So, the python API expands based on the needs to replicate common use cases that often require IR experimentation.\n", + "\n", + "If the application requires functionality or fine-tuning not available in `pyvespa`, simply build it directly using Vespa configuration files as shown in [many examples](https://docs.vespa.ai/en/getting-started.html) on Vespa docs. But even in this case, one can still get value out of `pyvespa` by deploying it from python based on the Vespa configuration files stored on disk.\n", + "\n", "Clone and deploy the news search app covered in this [Vespa tutorial](https://docs.vespa.ai/en/tutorials/news-3-searching.html):" ] }, @@ -199,17 +222,17 @@ "output_type": "stream", "text": [ "Cloning into 'sample-apps'...\n", - "remote: Enumerating objects: 1138, done.\u001b[K\n", - "remote: Counting objects: 100% (1138/1138), done.\u001b[K\n", - "remote: Compressing objects: 100% (754/754), done.\u001b[K\n", - "remote: Total 1138 (delta 189), reused 889 (delta 109), pack-reused 0\u001b[K\n", + "remote: Enumerating objects: 1138, done.\u001B[K\n", + "remote: Counting objects: 100% (1138/1138), done.\u001B[K\n", + "remote: Compressing objects: 100% (754/754), done.\u001B[K\n", + "remote: Total 1138 (delta 189), reused 889 (delta 109), pack-reused 0\u001B[K\n", "Receiving objects: 100% (1138/1138), 18.94 MiB | 9.38 MiB/s, done.\n", "Resolving deltas: 100% (189/189), done.\n" ] } ], "source": [ - "!git clone --depth 1 https://github.com/vespa-engine/sample-apps.git $WORK_DIR/sample-apps" + "!git clone --depth 1 https://github.com/vespa-engine/sample-apps.git sample-apps" ] }, { @@ -228,8 +251,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "\u001b[01;34msample-apps/news/app-3-searching/\u001b[00m\r\n", - "├── \u001b[01;34mschemas\u001b[00m\r\n", + "\u001B[01;34msample-apps/news/app-3-searching/\u001B[00m\r\n", + "├── \u001B[01;34mschemas\u001B[00m\r\n", "│   └── news.sd\r\n", "└── services.xml\r\n", "\r\n", @@ -245,8 +268,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Deploy to a Docker container from disk.\n", - "Note that `disk_folder` must be an absolute path, for Docker to bind the volume:" + "Deploy to a Docker container from disk:" ] }, { @@ -270,12 +292,10 @@ "import os\n", "from vespa.deployment import VespaDocker\n", "\n", - "disk_folder=os.getenv(\"WORK_DIR\") + \"/sample-apps/news/app-3-searching\"\n", - "vespa_docker_news = VespaDocker(\n", - " disk_folder=disk_folder,\n", - " port=8090\n", - ")\n", - "app = vespa_docker_news.deploy_from_disk(application_name=\"news\")" + "vespa_docker_news = VespaDocker(port=8080)\n", + "app = vespa_docker_news.deploy_from_disk(\n", + " application_name=\"news\",\n", + " application_folder=\"sample-apps/news/app-3-searching\")" ] }, { @@ -291,11 +311,9 @@ "metadata": {}, "outputs": [], "source": [ - "from shutil import rmtree\n", - "\n", - "rmtree(os.getenv(\"WORK_DIR\") + \"/sample-apps\", ignore_errors=True)\n", "vespa_docker_news.container.stop()\n", - "vespa_docker_news.container.remove()" + "vespa_docker_news.container.remove()\n", + "rmtree(\"sample-apps\", ignore_errors=True)" ] } ], @@ -320,4 +338,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/docs/sphinx/source/exchange-data-with-app.ipynb b/docs/sphinx/source/exchange-data-with-app.ipynb index 60c8ddd3..53147053 100644 --- a/docs/sphinx/source/exchange-data-with-app.ipynb +++ b/docs/sphinx/source/exchange-data-with-app.ipynb @@ -29,12 +29,7 @@ "from vespa.gallery import QuestionAnswering\n", "\n", "app_package = QuestionAnswering()\n", - "\n", - "disk_folder = os.path.join(os.getenv(\"WORK_DIR\"), \"sample_application\")\n", - "vespa_docker = VespaDocker(\n", - " port=8081, \n", - " disk_folder=disk_folder # requires absolute path\n", - ")\n", + "vespa_docker = VespaDocker()\n", "app = vespa_docker.deploy(application_package=app_package)" ] }, @@ -631,9 +626,7 @@ "outputs": [], "source": [ "# this is a hidden cell. It will not show on the documentation HTML.\n", - "from shutil import rmtree\n", "\n", - "rmtree(disk_folder, ignore_errors=True)\n", "vespa_docker.container.stop()\n", "vespa_docker.container.remove()" ] @@ -660,4 +653,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/docs/sphinx/source/getting-started-pyvespa.ipynb b/docs/sphinx/source/getting-started-pyvespa.ipynb index aa52e938..bfb47c4b 100644 --- a/docs/sphinx/source/getting-started-pyvespa.ipynb +++ b/docs/sphinx/source/getting-started-pyvespa.ipynb @@ -28,7 +28,7 @@ "name": "stdout", "output_type": "stream", "text": [ - " Total Memory: 11.7GiB\r\n" + " Total Memory: 15.64GiB\r\n" ] } ], @@ -58,12 +58,12 @@ "source": [ "## Create the application package\n", "\n", - "Create an [application package](https://pyvespa.readthedocs.io/en/latest/use_cases/text_search/text-search-quick-start.html):" + "Create an [application package](https://docs.vespa.ai/en/cloudconfig/application-packages.html):" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 3, "metadata": {}, "outputs": [], "source": [ @@ -177,17 +177,24 @@ }, { "cell_type": "code", - "execution_count": 2, - "metadata": {}, + "execution_count": 4, + "metadata": { + "scrolled": true + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Waiting for configuration server.\n", - "Waiting for configuration server.\n", - "Waiting for application status.\n", - "Waiting for application status.\n", + "Waiting for configuration server, 0/300 seconds...\n", + "Waiting for configuration server, 5/300 seconds...\n", + "Waiting for application status, 0/300 seconds...\n", + "Waiting for application status, 5/300 seconds...\n", + "Waiting for application status, 10/300 seconds...\n", + "Waiting for application status, 15/300 seconds...\n", + "Waiting for application status, 20/300 seconds...\n", + "Waiting for application status, 25/300 seconds...\n", + "Waiting for application status, 30/300 seconds...\n", "Finished deployment.\n" ] } @@ -196,8 +203,7 @@ "import os\n", "from vespa.deployment import VespaDocker\n", "\n", - "disk_folder = os.path.join(os.getenv(\"WORK_DIR\"), \"sample_application\")\n", - "vespa_docker = VespaDocker(port=8081, disk_folder=disk_folder)\n", + "vespa_docker = VespaDocker()\n", "app = vespa_docker.deploy(application_package=app_package)" ] }, @@ -205,44 +211,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "As part of deploying, pyvespa will export the configuration to an\n", - "[application package](https://docs.vespa.ai/en/reference/application-packages-reference.html) on disk.\n", - "This set of files can be deployed using [Vespa CLI](https://docs.vespa.ai/en/vespa-cli.html),\n", - "and can be useful to check into the source code repository.\n", - "As the application package was named \"qa\" in the code above, look for files in that directory:" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "qa/application/hosts.xml\r\n", - "qa/application/services.xml\r\n", - "qa/application/schemas/context.sd\r\n", - "qa/application/schemas/sentence.sd\r\n", - "qa/application/search/query-profiles/types/root.xml\r\n", - "qa/application/search/query-profiles/default.xml\r\n" - ] - } - ], - "source": [ - "!find qa -type f" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Use [disk_folder](reference-api.rst#vespadocker) to configure the working directory.\n", - "Use [export_application_package](reference-api.rst#vespa.deployment.VespaDocker.export_application_package)\n", - "to export the application package from code to files.\n", + "The above deploys the application package to a running Vespa instance.\n", + "Before moving on, inspect the generated configuration that was deployed.\n", "\n", - "Remember to [clean up](#Cleanup) after deploying to a local Docker container." + "See [deploy-docker](deploy-docker.ipynb) for how to export the application package to files,\n", + "and how to deploy after modifying the application package." ] }, { @@ -254,7 +227,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 10, "metadata": {}, "outputs": [ { @@ -263,7 +236,7 @@ "['text', 'dataset', 'questions', 'context_id', 'sentence_embedding']" ] }, - "execution_count": 5, + "execution_count": 10, "metadata": {}, "output_type": "execute_result" } @@ -286,7 +259,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 11, "metadata": {}, "outputs": [], "source": [ @@ -308,7 +281,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 12, "metadata": {}, "outputs": [], "source": [ @@ -326,7 +299,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 13, "metadata": {}, "outputs": [], "source": [ @@ -341,19 +314,19 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'id': 'index:qa_content/0/a87ff679ab8603b42a4ffde2',\n", - " 'relevance': 11.194862200830393,\n", + " 'relevance': 11.19486220083039,\n", " 'source': 'qa_content',\n", " 'fields': {'text': 'Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\".'}}" ] }, - "execution_count": 9, + "execution_count": 14, "metadata": {}, "output_type": "execute_result" } @@ -373,7 +346,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 15, "metadata": {}, "outputs": [], "source": [ @@ -383,18 +356,18 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "[,\n", - " ,\n", - " ]" + "[,\n", + " ,\n", + " ]" ] }, - "execution_count": 11, + "execution_count": 16, "metadata": {}, "output_type": "execute_result" } @@ -405,7 +378,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 17, "metadata": {}, "outputs": [ { @@ -932,7 +905,7 @@ " 'questions': [4]}}" ] }, - "execution_count": 12, + "execution_count": 17, "metadata": {}, "output_type": "execute_result" } @@ -952,7 +925,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 18, "metadata": {}, "outputs": [], "source": [ @@ -968,7 +941,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 19, "metadata": {}, "outputs": [], "source": [ @@ -986,7 +959,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 20, "metadata": {}, "outputs": [], "source": [ @@ -1003,15 +976,12 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 21, "metadata": {}, "outputs": [], "source": [ - "from shutil import rmtree\n", - "\n", "vespa_docker.container.stop()\n", - "vespa_docker.container.remove()\n", - "rmtree(disk_folder, ignore_errors=True)" + "vespa_docker.container.remove()" ] } ], diff --git a/docs/sphinx/source/use_cases/cord19/cord19_download_parse_trec_covid.ipynb b/docs/sphinx/source/use_cases/cord19/cord19_download_parse_trec_covid.ipynb index 98e9e60f..fdbb6a4e 100644 --- a/docs/sphinx/source/use_cases/cord19/cord19_download_parse_trec_covid.ipynb +++ b/docs/sphinx/source/use_cases/cord19/cord19_download_parse_trec_covid.ipynb @@ -31,8 +31,8 @@ "metadata": {}, "outputs": [], "source": [ - "!wget https://data.vespa.oath.cloud/blog/cord19/topics-rnd5.xml\n", - "!wget https://data.vespa.oath.cloud/blog/cord19/qrels-covid_d5_j0.5-5.txt" + "!curl -fsSLO https://data.vespa.oath.cloud/blog/cord19/topics-rnd5.xml\n", + "!curl -fsSLO https://data.vespa.oath.cloud/blog/cord19/qrels-covid_d5_j0.5-5.txt" ] }, { @@ -347,4 +347,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file diff --git a/docs/sphinx/source/use_cases/qa/semantic-retrieval-for-question-answering-applications.ipynb b/docs/sphinx/source/use_cases/qa/semantic-retrieval-for-question-answering-applications.ipynb index d8710739..682d3fbc 100644 --- a/docs/sphinx/source/use_cases/qa/semantic-retrieval-for-question-answering-applications.ipynb +++ b/docs/sphinx/source/use_cases/qa/semantic-retrieval-for-question-answering-applications.ipynb @@ -596,11 +596,7 @@ "import os\n", "from vespa.deployment import VespaDocker\n", "\n", - "disk_folder = os.path.join(os.getenv(\"WORK_DIR\"), \"sample_application\")\n", - "vespa_docker = VespaDocker(\n", - " port=8081, \n", - " disk_folder=disk_folder # requires absolute path\n", - ")\n", + "vespa_docker = VespaDocker()\n", "app = vespa_docker.deploy(application_package=app_package)" ] }, @@ -905,9 +901,6 @@ "metadata": {}, "outputs": [], "source": [ - "from shutil import rmtree\n", - "\n", - "rmtree(disk_folder, ignore_errors=True)\n", "vespa_docker.container.stop()\n", "vespa_docker.container.remove()" ] @@ -915,7 +908,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -929,7 +922,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.7" + "version": "3.9.9" } }, "nbformat": 4, diff --git a/docs/sphinx/source/use_cases/sequence-classification-task.ipynb b/docs/sphinx/source/use_cases/sequence-classification-task.ipynb index b6ac0144..a3e42bb1 100644 --- a/docs/sphinx/source/use_cases/sequence-classification-task.ipynb +++ b/docs/sphinx/source/use_cases/sequence-classification-task.ipynb @@ -24,7 +24,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 11, "id": "dedicated-shoot", "metadata": {}, "outputs": [], @@ -47,7 +47,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 12, "id": "cloudy-refund", "metadata": {}, "outputs": [], @@ -84,7 +84,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 13, "id": "spread-transport", "metadata": { "nbsphinx": "hidden" @@ -93,7 +93,7 @@ "source": [ "import os\n", "\n", - "os.environ[\"DISK_FOLDER\"] = os.path.join(os.getenv(\"WORK_DIR\"), \"sample_application\")\n", + "os.environ[\"DISK_FOLDER\"] = os.path.join(os.getenv(\"WORK_DIR\", \"\"), \"sample_application\")\n", "disk_folder = os.getenv(\"DISK_FOLDER\")\n", "\n", "os.environ[\"TENANT_NAME\"] = \"vespa-team\"\n", @@ -108,7 +108,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 14, "id": "fantastic-google", "metadata": {}, "outputs": [], @@ -139,14 +139,58 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 15, "id": "furnished-diana", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading tokenizer.\n", + "Downloading model.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Some weights of the model checkpoint at google/bert_uncased_L-2_H-128_A-2 were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.bias', 'cls.seq_relationship.weight']\n", + "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", + "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n", + "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at google/bert_uncased_L-2_H-128_A-2 and are newly initialized: ['classifier.bias', 'classifier.weight']\n", + "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Model loaded.\n", + "Using framework PyTorch: 1.10.0\n", + "Found input input_ids with shape: {0: 'batch', 1: 'sequence'}\n", + "Found input token_type_ids with shape: {0: 'batch', 1: 'sequence'}\n", + "Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}\n", + "Found output output_0 with shape: {0: 'batch'}\n", + "Ensuring inputs are in correct order\n", + "position_ids is not present in the generated input list.\n", + "Generated inputs order: ['input_ids', 'attention_mask', 'token_type_ids']\n", + "Waiting for configuration server, 0/300 seconds...\n", + "Waiting for configuration server, 5/300 seconds...\n", + "Waiting for application status, 0/300 seconds...\n", + "Waiting for application status, 5/300 seconds...\n", + "Waiting for application status, 10/300 seconds...\n", + "Waiting for application status, 15/300 seconds...\n", + "Waiting for application status, 20/300 seconds...\n", + "Waiting for application status, 25/300 seconds...\n", + "Finished deployment.\n" + ] + } + ], "source": [ "from vespa.deployment import VespaDocker\n", "\n", - "vespa_docker = VespaDocker(disk_folder=disk_folder, port=8081)\n", + "vespa_docker = VespaDocker()\n", "app = vespa_docker.deploy(application_package=model_server)" ] }, @@ -162,17 +206,17 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 7, "id": "lovely-scale", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "{'bert_tiny': 'http://localhost:8081/model-evaluation/v1/bert_tiny'}" + "{'bert_tiny': 'http://localhost:8080/model-evaluation/v1/bert_tiny'}" ] }, - "execution_count": 6, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } @@ -191,7 +235,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 8, "id": "digital-sewing", "metadata": {}, "outputs": [ @@ -200,14 +244,14 @@ "text/plain": [ "{'model': 'bert_tiny',\n", " 'functions': [{'function': 'output_0',\n", - " 'info': 'http://localhost:8081/model-evaluation/v1/bert_tiny/output_0',\n", - " 'eval': 'http://localhost:8081/model-evaluation/v1/bert_tiny/output_0/eval',\n", + " 'info': 'http://localhost:8080/model-evaluation/v1/bert_tiny/output_0',\n", + " 'eval': 'http://localhost:8080/model-evaluation/v1/bert_tiny/output_0/eval',\n", " 'arguments': [{'name': 'input_ids', 'type': 'tensor(d0[],d1[])'},\n", " {'name': 'attention_mask', 'type': 'tensor(d0[],d1[])'},\n", " {'name': 'token_type_ids', 'type': 'tensor(d0[],d1[])'}]}]}" ] }, - "execution_count": 7, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } @@ -228,17 +272,17 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 9, "id": "significant-great", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "[0.053629081696271896, -0.01650623418390751]" + "[0.016708193346858025, -0.2681295573711395]" ] }, - "execution_count": 8, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } @@ -257,7 +301,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 10, "id": "6a7dee01", "metadata": {}, "outputs": [], diff --git a/docs/sphinx/source/use_cases/text_search/text-search-quick-start.ipynb b/docs/sphinx/source/use_cases/text_search/text-search-quick-start.ipynb index a97dba03..17a4bc6f 100644 --- a/docs/sphinx/source/use_cases/text_search/text-search-quick-start.ipynb +++ b/docs/sphinx/source/use_cases/text_search/text-search-quick-start.ipynb @@ -17,7 +17,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 1, "id": "03f3d0f2", "metadata": {}, "outputs": [ @@ -25,7 +25,7 @@ "name": "stdout", "output_type": "stream", "text": [ - " Total Memory: 11.7GiB\r\n" + " Total Memory: 15.64GiB\r\n" ] } ], @@ -45,7 +45,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 2, "id": "bd5c2629", "metadata": {}, "outputs": [], @@ -68,7 +68,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 3, "id": "f2d0bea7", "metadata": {}, "outputs": [], @@ -109,7 +109,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 4, "id": "d0ecbb27", "metadata": {}, "outputs": [], @@ -135,7 +135,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 5, "id": "9bb438f4", "metadata": {}, "outputs": [], @@ -168,7 +168,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 6, "id": "canadian-blood", "metadata": {}, "outputs": [ @@ -176,10 +176,16 @@ "name": "stdout", "output_type": "stream", "text": [ - "Waiting for configuration server.\n", - "Waiting for configuration server.\n", - "Waiting for application status.\n", - "Waiting for application status.\n", + "Waiting for configuration server, 0/300 seconds...\n", + "Waiting for configuration server, 5/300 seconds...\n", + "Waiting for application status, 0/300 seconds...\n", + "Waiting for application status, 5/300 seconds...\n", + "Waiting for application status, 10/300 seconds...\n", + "Waiting for application status, 15/300 seconds...\n", + "Waiting for application status, 20/300 seconds...\n", + "Waiting for application status, 25/300 seconds...\n", + "Waiting for application status, 30/300 seconds...\n", + "Waiting for application status, 35/300 seconds...\n", "Finished deployment.\n" ] } @@ -188,8 +194,7 @@ "import os\n", "from vespa.deployment import VespaDocker\n", "\n", - "disk_folder = os.path.join(os.getenv(\"WORK_DIR\"), \"sample_application\")\n", - "vespa_docker = VespaDocker(port=8089, disk_folder=disk_folder)\n", + "vespa_docker = VespaDocker()\n", "app = vespa_docker.deploy(application_package=app_package)" ] }, @@ -217,7 +222,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 7, "id": "executed-reservoir", "metadata": {}, "outputs": [ @@ -298,7 +303,7 @@ "4 22b Cotton and African American Life Two thi... " ] }, - "execution_count": 21, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } @@ -322,7 +327,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 8, "id": "bottom-memorabilia", "metadata": {}, "outputs": [], @@ -344,7 +349,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 9, "id": "82b5f6fe", "metadata": {}, "outputs": [], @@ -360,7 +365,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 10, "id": "c930ef28", "metadata": {}, "outputs": [ @@ -368,7 +373,7 @@ "data": { "text/plain": [ "{'id': 'id:textsearch:textsearch::D1871659',\n", - " 'relevance': 25.629646778721725,\n", + " 'relevance': 25.62964677872173,\n", " 'source': 'textsearch_content',\n", " 'fields': {'sddocname': 'textsearch',\n", " 'documentid': 'id:textsearch:textsearch::D1871659',\n", @@ -377,7 +382,7 @@ " 'body': 'Answers com Wiki Answers Categories Cars Vehicles Airplanes and Aircraft What keeps airplanes in the air Flag What keeps airplanes in the air Answer by Karin L Confidence votes 95 0KThere s more to raising cattle than throwing them out to pasture Know your soil and plants to earn profit above ground and wealth below It is the combined forces of lift thrust and weight that keeps an airplane in the air Lift happens to be the largest force in this equation and is dependent on the speed of the wing or how fast an airplane is going vertical velocity of air and air density Well the elevator the rudder will help and something else I forgot what it was but don t judge me for that And that s how you be a bow done Like a boss Boss 15 people found this useful Was this answer useful Yes Somewhat No How do airplane windows keep out the cold Airplane windows The only way that heat can escape the warm cabin is to travel through something or radiate outward Since the windows are so small the radiation through Karin L There s more to raising cattle than throwing them out to pasture Know your soil and plants to earn profit above ground and wealth below Does speed keep an airplane in the air Yes to a degree speed is part of the equation Speed thrust and combine that with lift and weight though weight has to be smaller than lift and thrust combined then y Bala Surya 152 866 Contributions Adventurous Fun Dreaming High How does bernoulli s principle keep airplanes in the air Bernoulli s principle is that there is a region of high pressure under the wing So air rushes under the plane So it creates lift which in turn keeps the airplane in the air How airplane can fly in the air The airplane fly on the air by 4 main forces drag lift thrust and weight all these forces affect of the performances of the airplane to fly the high power of the e David Bäckman 388 346 Contributions Knowledge is a thing you can both share and keep Is the force that keeps an airplane in the air called lift or levitation Lift Sadia rulez 1 Contribution How does air help an airplane fly Air Helps An Aeroplane Fly Because Of The Up Thurst Up Thrust Is A Sort Of A Gravity That Pulls You Up Like A Float Floats In Water The Upthrust Pulls It Up But The Gravity Pull Djlax97 3 Contributions How do you keep your ears from popping on an airplane All you have to do is chew gum and swallow a lot Doing this has something to do with the place of your throat And yes it does work What does it mean when the air in airplanes are pressurized The air in an aircraft needs to be pressurised so that the people within the cabin don t pass out from oxygen starvation at higher altitudes The atmosphere can be described a Richard Loberger 26 278 Contributions Airplane can stop in the air It would depend on what you mean by stop in the air An airplane can have 0 MPH ground speed while in the air only IF the wind is going faster then the stall speed of the g How does an airplane stay stable in the air to keep a plane stable in the air it has different control surfaces or panels to allow the pilot to adjust the position of the plane in the air Some modern fighter jets such How do you recharge an airplane Air Conditioner An airplane airconditioner is completely different than the one in your house or car It doesn t rely on a refrigerant Rather it takes hot high pressure air from the hot comp What keeps an airplane up in the sky Bernoulli s Principle the statement that an increase in the speed of a fluid produces a decrease in pressure and a decrease in the speed produces an increase in pressure Win Karin L There s more to raising cattle than throwing them out to pasture Know your soil and plants to earn profit above ground and wealth below Answered In Physics What keeps the airplane from rolling unexpectedly On the tarmac there are triangular blocks that are placed in front and behind each wheel of the airplane called wheel chocks In the air a steady hand on the control sti David Bäckman 388 346 Contributions Knowledge is a thing you can both share and keep Answered In Airplanes and Aircraft What is a machine that keeps an airplane on course An auto pilot Charlie N 122 923 Contributions I have spent many years renovating buildings and leading a commercial handyman crew Answered In Airbus Machine that keeps an airplane on course Autopilot keeps an aircraft on course In modern times autopilot is assisted by GPS and radar Answered In Airplanes and Aircraft What keeps an airplane moving forward An engine producing THRUST keeps an airplane moving forward Types of engines used by airplanes include reciprocating engines turbo prop engines turbojet and turbofan engin Levyharaivan 396 Contributions Answered In Airplanes and Aircraft What keeps a airplane from rolling unexpectedly Brakes just like any other vehicle'}}" ] }, - "execution_count": 24, + "execution_count": 10, "metadata": {}, "output_type": "execute_result" } @@ -402,7 +407,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 11, "id": "d2ddde50", "metadata": {}, "outputs": [], @@ -418,7 +423,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 12, "id": "pretty-boost", "metadata": {}, "outputs": [], @@ -428,7 +433,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 13, "id": "chubby-caribbean", "metadata": {}, "outputs": [ @@ -436,16 +441,16 @@ "name": "stdout", "output_type": "stream", "text": [ - "{'id': 'D1871659', 'title': 'What keeps airplanes in the air ', 'relevance': 25.629646778721725}\n", - "{'id': 'D684487', 'title': 'MP02 Motion Diagrams', 'relevance': 10.530621046528196}\n", - "{'id': 'D254873', 'title': ' ', 'relevance': 9.360245799821822}\n", - "{'id': 'D1765332', 'title': 'Fast Furious 6', 'relevance': 8.074330525519944}\n", - "{'id': 'D1631044', 'title': 'First Cell', 'relevance': 7.633614797739622}\n", - "{'id': 'D3434656', 'title': 'A Global Guide to Pet Relocation Costs', 'relevance': 6.924114436169097}\n", - "{'id': 'D3157544', 'title': 'Thomas Cook to axe 2 600 jobs', 'relevance': 6.620220097056055}\n", - "{'id': 'D209851', 'title': 'The USS Scorpion Buried at Sea', 'relevance': 6.406117915168797}\n", - "{'id': 'D958861', 'title': 'George Harrison close to death ', 'relevance': 6.00868191627094}\n", - "{'id': 'D1529248', 'title': 'The airport shopping challenge Left buying Christmas presents til the last minute like I did You CAN do it all in two hours and I m living proof', 'relevance': 5.991353085441339}\n" + "{'id': 'D1871659', 'title': 'What keeps airplanes in the air ', 'relevance': 25.62964677872173}\n", + "{'id': 'D684487', 'title': 'MP02 Motion Diagrams', 'relevance': 10.530621046528191}\n", + "{'id': 'D254873', 'title': ' ', 'relevance': 9.360245799821817}\n", + "{'id': 'D1765332', 'title': 'Fast Furious 6', 'relevance': 8.074330525519938}\n", + "{'id': 'D1631044', 'title': 'First Cell', 'relevance': 7.633614797739615}\n", + "{'id': 'D3434656', 'title': 'A Global Guide to Pet Relocation Costs', 'relevance': 6.92411443616909}\n", + "{'id': 'D3157544', 'title': 'Thomas Cook to axe 2 600 jobs', 'relevance': 6.620220097056048}\n", + "{'id': 'D209851', 'title': 'The USS Scorpion Buried at Sea', 'relevance': 6.406117915168791}\n", + "{'id': 'D958861', 'title': 'George Harrison close to death ', 'relevance': 6.008681916270933}\n", + "{'id': 'D1529248', 'title': 'The airport shopping challenge Left buying Christmas presents til the last minute like I did You CAN do it all in two hours and I m living proof', 'relevance': 5.991353085441332}\n" ] } ], @@ -468,16 +473,13 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 14, "id": "e5064bd2", "metadata": {}, "outputs": [], "source": [ - "from shutil import rmtree\n", - "\n", "vespa_docker.container.stop()\n", - "vespa_docker.container.remove()\n", - "rmtree(disk_folder, ignore_errors=True)" + "vespa_docker.container.remove()" ] } ], diff --git a/screwdriver.yaml b/screwdriver.yaml index 2de4a606..acbb4996 100644 --- a/screwdriver.yaml +++ b/screwdriver.yaml @@ -93,10 +93,10 @@ jobs: # # Exclude non-testable notebooks below # - dnf install -y tree wget + dnf install -y tree echo "All non-cloud notebooks:" find docs -name '*.ipynb' ! -name '*cloud*.ipynb' - for notebook in `find docs -name '*.ipynb' -and \ + for notebook in `find docs/sphinx/source -name '*.ipynb' -and \ ! -name '*cloud*.ipynb' -and \ ! -path '*/.ipynb_checkpoints/*' -and \ ! -name 'image-search-scratch.ipynb'`; \ diff --git a/vespa/deployment.py b/vespa/deployment.py index 79e82829..c975169c 100644 --- a/vespa/deployment.py +++ b/vespa/deployment.py @@ -86,7 +86,6 @@ def from_container_name_or_id( container = client.containers.get(name_or_id) except docker.errors.NotFound: raise ValueError("The container does not exist.") - disk_folder = container.attrs["Mounts"][0]["Source"] port = int( container.attrs["HostConfig"]["PortBindings"]["8080/tcp"][0]["HostPort"] ) @@ -94,7 +93,6 @@ def from_container_name_or_id( container_image = container.image.tags[0] # vespaengine/vespa:latest return VespaDocker( - disk_folder=disk_folder, port=port, container_memory=container_memory, output_file=output_file, @@ -186,7 +184,7 @@ def export_application_package( :param application_package: Application package to export. :return: None. Application package file will be stored on `disk_folder`. """ - # ToDo: remove this method, not needed with ApplicationPackage::tofiles + # ToDo: remove this method, not needed with ApplicationPackage::to_files if not self.disk_folder: self.disk_folder = os.path.join(os.getcwd(), application_package.name) @@ -239,8 +237,6 @@ def export_application_package( "w", ) as f: f.write(application_package.query_profile_type_to_text) - with open(os.path.join(self.disk_folder, "application/hosts.xml"), "w") as f: - f.write(application_package.hosts_to_text) with open(os.path.join(self.disk_folder, "application/services.xml"), "w") as f: f.write(application_package.services_to_text) @@ -332,17 +328,7 @@ def deploy( :param application_package: ApplicationPackage to be deployed. :return: a Vespa connection instance. """ - if not self.disk_folder: - return self._deploy_data(application_package.name, application_package.to_zip()) - - self.export_application_package(application_package=application_package) - return self._execute_deployment( - application_name=application_package.name, - disk_folder=self.disk_folder, - container_memory=self.container_memory, - application_folder="application", - application_package=application_package, - ) + return self._deploy_data(application_package, application_package.to_zip()) def deploy_from_disk( self, @@ -357,16 +343,8 @@ def deploy_from_disk( If None, we assume `disk_folder` to be the application folder. :return: a Vespa connection instance. """ - # ToDo: Remove this method in a later release - deploy_zipped_from_disk will replace this / take its place - if not self.disk_folder: - self.disk_folder = os.path.join(os.getcwd(), application_name) - - return self._execute_deployment( - application_name=application_name, - disk_folder=self.disk_folder, - container_memory=self.container_memory, - application_folder=application_folder, - ) + # ToDo: Merge this method in a later release with deploy_zipped_from_disk + return self.deploy_zipped_from_disk(app_name=application_name, app_root=Path(application_folder)) def deploy_zipped_from_disk(self, app_name: str, app_root: Path) -> Vespa: """ @@ -391,10 +369,11 @@ def deploy_zipped_from_disk(self, app_name: str, app_root: Path) -> Vespa: data = f.read() os.remove(TMP_ZIP) - return self._deploy_data(app_name, data) + app = ApplicationPackage(name=app_name) + return self._deploy_data(app, data) - def _deploy_data(self, app_name: str, data) -> Vespa: + def _deploy_data(self, application: ApplicationPackage, data) -> Vespa: """ Deploys an Application Package as zipped data @@ -403,7 +382,7 @@ def _deploy_data(self, app_name: str, data) -> Vespa: :return: A Vespa connection instance """ self._run_vespa_engine_container( - application_name=app_name, + application_name=application.name, container_memory=self.container_memory, ) self.wait_for_config_server_start(max_wait=CFG_SERVER_START_TIMEOUT) @@ -419,6 +398,7 @@ def _deploy_data(self, app_name: str, data) -> Vespa: app = Vespa( url=self.url, port=self.local_port, + application_package=application ) app.wait_for_application_up(max_wait=APP_INIT_TIMEOUT) diff --git a/vespa/package.py b/vespa/package.py index 8ba68e2d..48af63e3 100644 --- a/vespa/package.py +++ b/vespa/package.py @@ -1350,22 +1350,6 @@ def query_profile_type_to_text(self): query_profile_type=self.query_profile_type ) - @property - def hosts_to_text(self): - # ToDo: Remove, not needed anymore - env = Environment( - loader=PackageLoader("vespa", "templates"), - autoescape=select_autoescape( - disabled_extensions=("txt",), - default_for_string=True, - default=True, - ), - ) - env.trim_blocks = True - env.lstrip_blocks = True - schema_template = env.get_template("hosts.xml") - return schema_template.render() - @property def services_to_text(self): env = Environment( @@ -1432,21 +1416,21 @@ def to_zip(self) -> BytesIO: "schemas/{}.sd".format(schema.name), schema.schema_to_text, ) - - for model in schema.models: - zip_archive.write( - model.model_file_path, - os.path.join("files", model.model_file_name), - ) - if self.models: - for model_id, model in self.models.items(): - temp_model_file = "{}.onnx".format(model_id) - model.export_to_onnx(output_path=temp_model_file) - zip_archive.write( - temp_model_file, - "models/{}.onnx".format(model_id), - ) - os.remove(temp_model_file) + for model in schema.models: + zip_archive.write( + model.model_file_path, + os.path.join("files", model.model_file_name), + ) + + if self.models: + for model_id, model in self.models.items(): + temp_model_file = "{}.onnx".format(model_id) + model.export_to_onnx(output_path=temp_model_file) + zip_archive.write( + temp_model_file, + "models/{}.onnx".format(model_id), + ) + os.remove(temp_model_file) if self.query_profile: zip_archive.writestr( diff --git a/vespa/test_integration_docker.py b/vespa/test_integration_docker.py index 0414483c..980e09e1 100644 --- a/vespa/test_integration_docker.py +++ b/vespa/test_integration_docker.py @@ -2,7 +2,6 @@ import pytest import os import re -import shutil import asyncio import json from pandas import DataFrame @@ -132,70 +131,25 @@ def create_sequence_classification_task(): class TestDockerCommon(unittest.TestCase): - def deploy(self, application_package, disk_folder, container_image=None): + def deploy(self, application_package, container_image=None): if container_image: - self.vespa_docker = VespaDocker(port=8089, disk_folder=disk_folder, container_image=container_image) + self.vespa_docker = VespaDocker(port=8089, container_image=container_image) else: - self.vespa_docker = VespaDocker(port=8089, disk_folder=disk_folder) + self.vespa_docker = VespaDocker(port=8089) - app = self.vespa_docker.deploy(application_package=application_package) - # - # Test deployment - # - self.assertTrue( - any(re.match("Generation: [0-9]+", line) for line in app.deployment_message) - ) - self.assertEqual(app.get_application_status().status_code, 200) - # - # Test VespaDocker serialization - # - self.assertEqual( - repr(self.vespa_docker), repr(VespaDocker.from_dict(self.vespa_docker.to_dict)) - ) - - def deploy_without_disk_folder(self, application_package): - self.vespa_docker = VespaDocker(port=8089) try: - app = self.vespa_docker.deploy(application_package=application_package) + self.vespa_docker.deploy(application_package=application_package) except RuntimeError as e: assert False, "Deployment error: {}".format(e) - - def deploy_from_disk_with_disk_folder(self, application_package, disk_folder): - self.vespa_docker = VespaDocker(port=8089, disk_folder=disk_folder) - self.vespa_docker.export_application_package( - application_package=application_package - ) - # - # Disk folder as the application folder - # - self.vespa_docker.disk_folder = os.path.join(disk_folder, "application") - app = self.vespa_docker.deploy_from_disk( - application_name=application_package.name, - ) - self.assertTrue( - any(re.match("Generation: [0-9]+", line) for line in app.deployment_message) - ) - - def deploy_from_disk_with_application_folder( - self, application_package, disk_folder - ): - self.vespa_docker = VespaDocker(port=8089, disk_folder=disk_folder) - self.vespa_docker.export_application_package( - application_package=application_package - ) # - # Application folder inside disk folder + # Test VespaDocker serialization # - app = self.vespa_docker.deploy_from_disk( - application_name=application_package.name, - application_folder="application", - ) - self.assertTrue( - any(re.match("Generation: [0-9]+", line) for line in app.deployment_message) + self.assertEqual( + repr(self.vespa_docker), repr(VespaDocker.from_dict(self.vespa_docker.to_dict)) ) def create_vespa_docker_from_container_name_or_id( - self, application_package, disk_folder + self, application_package ): # # Raises ValueError if container does not exist @@ -205,31 +159,28 @@ def create_vespa_docker_from_container_name_or_id( # # Test VespaDocker instance created from container # - self.vespa_docker = VespaDocker(port=8089, disk_folder=disk_folder) + self.vespa_docker = VespaDocker(port=8089) _ = self.vespa_docker.deploy(application_package=application_package) vespa_docker_from_container = VespaDocker.from_container_name_or_id( application_package.name ) self.assertEqual(self.vespa_docker, vespa_docker_from_container) - def redeploy_with_container_stopped(self, application_package, disk_folder): - self.vespa_docker = VespaDocker(port=8089, disk_folder=disk_folder) - app = self.vespa_docker.deploy(application_package=application_package) - self.assertTrue( - any(re.match("Generation: [0-9]+", line) for line in app.deployment_message) - ) + def redeploy_with_container_stopped(self, application_package): + self.vespa_docker = VespaDocker(port=8089) + self.vespa_docker.deploy(application_package=application_package) self.vespa_docker.container.stop() app = self.vespa_docker.deploy(application_package=application_package) self.assertEqual(app.get_application_status().status_code, 200) def redeploy_with_application_package_changes( - self, application_package, disk_folder + self, application_package ): - self.vespa_docker = VespaDocker(port=8089, disk_folder=disk_folder) + self.vespa_docker = VespaDocker(port=8089) app = self.vespa_docker.deploy(application_package=application_package) res = app.query( body={ - "yql": "select * from sources * where default contains 'music';", + "yql": "select * from sources * where default contains 'music'", "ranking": "new-rank-profile", } ).json @@ -247,14 +198,14 @@ def redeploy_with_application_package_changes( app = self.vespa_docker.deploy(application_package=application_package) res = app.query( body={ - "yql": "select * from sources * where default contains 'music';", + "yql": "select * from sources * where default contains 'music'", "ranking": "new-rank-profile", } ).json self.assertTrue("errors" not in res["root"]) - def trigger_start_stop_and_restart_services(self, application_package, disk_folder): - self.vespa_docker = VespaDocker(port=8089, disk_folder=disk_folder) + def trigger_start_stop_and_restart_services(self, application_package): + self.vespa_docker = VespaDocker(port=8089) with self.assertRaises(RuntimeError): self.vespa_docker.stop_services() with self.assertRaises(RuntimeError): @@ -291,7 +242,7 @@ def execute_data_operations( :param fields_to_send: Dict where keys are field names and values are field values. Must contain 'id' field :param field_to_update: Dict where keys are field names and values are field values. :param expected_fields_from_get_operation: Dict containing fields as returned by Vespa get operation. - There are cases where fields returned from Vespa are different than inputs, e.g. when dealing with Tensors. + There are cases where fields returned from Vespa are different from inputs, e.g. when dealing with Tensors. :return: """ assert "id" in fields_to_send, "fields_to_send must contain 'id' field." @@ -455,7 +406,7 @@ async def execute_async_data_operations( contain 'id' field. :param field_to_update: Dict where keys are field names and values are field values. :param expected_fields_from_get_operation: Dict containing fields as returned by Vespa get operation. - There are cases where fields returned from Vespa are different than inputs, e.g. when dealing with Tensors. + There are cases where fields returned from Vespa are different from inputs, e.g. when dealing with Tensors. :return: """ async with app.asyncio(connections=120, total_timeout=50) as async_app: @@ -629,7 +580,7 @@ def batch_operations_synchronous_mode( :param fields_to_send: List of Dicts where keys are field names and values are field values. Must contain 'id' field. :param expected_fields_from_get_operation: Dict containing fields as returned by Vespa get operation. - There are cases where fields returned from Vespa are different than inputs, e.g. when dealing with Tensors. + There are cases where fields returned from Vespa are different from inputs, e.g. when dealing with Tensors. :param fields_to_update: Dict where keys are field names and values are field values. :param query_batch: Optional list of query strings. :param query_model: Optional QueryModel to use with query_batch. @@ -737,7 +688,7 @@ def batch_operations_asynchronous_mode( :param fields_to_send: List of Dicts where keys are field names and values are field values. Must contain 'id' field. :param expected_fields_from_get_operation: Dict containing fields as returned by Vespa get operation. - There are cases where fields returned from Vespa are different than inputs, e.g. when dealing with Tensors. + There are cases where fields returned from Vespa are different from inputs, e.g. when dealing with Tensors. :param fields_to_update: Dict where keys are field names and values are field values. :param query_batch: Optional list of query strings. :param query_model: Optional QueryModel to use with query_batch. @@ -846,7 +797,7 @@ def batch_operations_default_mode_with_one_schema( :param fields_to_send: List of Dicts where keys are field names and values are field values. Must contain 'id' field. :param expected_fields_from_get_operation: Dict containing fields as returned by Vespa get operation. - There are cases where fields returned from Vespa are different than inputs, e.g. when dealing with Tensors. + There are cases where fields returned from Vespa are different from inputs, e.g. when dealing with Tensors. :param fields_to_update: Dict where keys are field names and values are field values. :return: """ @@ -1030,47 +981,24 @@ def bert_model_input_and_output( class TestMsmarcoDockerDeployment(TestDockerCommon): def setUp(self) -> None: self.app_package = create_msmarco_application_package() - self.disk_folder = os.path.join(os.getenv("WORK_DIR"), "sample_application") def test_deploy(self): - self.deploy(application_package=self.app_package, disk_folder=self.disk_folder) - - def test_deploy_without_disk_folder(self): - self.deploy_without_disk_folder(application_package=self.app_package) - - def test_deploy_from_disk_with_disk_folder(self): - self.deploy_from_disk_with_disk_folder( - application_package=self.app_package, disk_folder=self.disk_folder - ) - - def test_deploy_from_disk_with_application_folder(self): - self.deploy_from_disk_with_application_folder( - application_package=self.app_package, disk_folder=self.disk_folder - ) + self.deploy(application_package=self.app_package) def test_instantiate_vespa_docker_from_container_name_or_id(self): - self.create_vespa_docker_from_container_name_or_id( - application_package=self.app_package, disk_folder=self.disk_folder - ) + self.create_vespa_docker_from_container_name_or_id(application_package=self.app_package) @pytest.mark.skip(reason="Works locally but fails on Screwdriver") def test_redeploy_with_container_stopped(self): - self.redeploy_with_container_stopped( - application_package=self.app_package, disk_folder=self.disk_folder - ) + self.redeploy_with_container_stopped(application_package=self.app_package) def test_redeploy_with_application_package_changes(self): - self.redeploy_with_application_package_changes( - application_package=self.app_package, disk_folder=self.disk_folder - ) + self.redeploy_with_application_package_changes(application_package=self.app_package) def test_trigger_start_stop_and_restart_services(self): - self.trigger_start_stop_and_restart_services( - application_package=self.app_package, disk_folder=self.disk_folder - ) + self.trigger_start_stop_and_restart_services(application_package=self.app_package) def tearDown(self) -> None: - shutil.rmtree(self.disk_folder, ignore_errors=True) self.vespa_docker.container.stop() self.vespa_docker.container.remove() @@ -1078,13 +1006,11 @@ def tearDown(self) -> None: class TestCord19DockerDeployment(TestDockerCommon): def setUp(self) -> None: self.app_package = create_cord19_application_package() - self.disk_folder = os.path.join(os.getenv("WORK_DIR"), "sample_application") def test_deploy(self): - self.deploy(application_package=self.app_package, disk_folder=self.disk_folder) + self.deploy(application_package=self.app_package) def tearDown(self) -> None: - shutil.rmtree(self.disk_folder, ignore_errors=True) self.vespa_docker.container.stop() self.vespa_docker.container.remove() @@ -1092,29 +1018,23 @@ def tearDown(self) -> None: class TestQaDockerDeployment(TestDockerCommon): def setUp(self) -> None: self.app_package = create_qa_application_package() - self.disk_folder = os.path.join(os.getenv("WORK_DIR"), "sample_application") def test_deploy(self): - self.deploy(application_package=self.app_package, disk_folder=self.disk_folder) + self.deploy(application_package=self.app_package) self.vespa_docker.container.stop() self.vespa_docker.container.remove() def test_deploy_image(self): self.deploy(application_package=self.app_package, - disk_folder=self.disk_folder, container_image="vespaengine/vespa:7.566.21") self.vespa_docker.container.stop() self.vespa_docker.container.remove() - def tearDown(self) -> None: - shutil.rmtree(self.disk_folder, ignore_errors=True) - class TestMsmarcoApplication(TestApplicationCommon): def setUp(self) -> None: self.app_package = create_msmarco_application_package() - self.disk_folder = os.path.join(os.getenv("WORK_DIR"), "sample_application") - self.vespa_docker = VespaDocker(port=8089, disk_folder=self.disk_folder) + self.vespa_docker = VespaDocker(port=8089) self.app = self.vespa_docker.deploy(application_package=self.app_package) self.fields_to_send = [ { @@ -1205,7 +1125,6 @@ def test_batch_operations_default_mode_with_one_schema(self): ) def tearDown(self) -> None: - shutil.rmtree(self.disk_folder, ignore_errors=True) self.vespa_docker.container.stop() self.vespa_docker.container.remove() @@ -1213,8 +1132,7 @@ def tearDown(self) -> None: class TestCord19Application(TestApplicationCommon): def setUp(self) -> None: self.app_package = create_cord19_application_package() - self.disk_folder = os.path.join(os.getenv("WORK_DIR"), "sample_application") - self.vespa_docker = VespaDocker(port=8089, disk_folder=self.disk_folder) + self.vespa_docker = VespaDocker(port=8089) self.app = self.vespa_docker.deploy(application_package=self.app_package) self.model_config = self.app_package.model_configs["pretrained_bert_tiny"] self.fields_to_send = [] @@ -1327,7 +1245,6 @@ def test_bert_model_input_and_output(self): ) def tearDown(self) -> None: - shutil.rmtree(self.disk_folder, ignore_errors=True) self.vespa_docker.container.stop() self.vespa_docker.container.remove() @@ -1341,8 +1258,7 @@ def setUp(self) -> None: self.app_package.get_schema("context").add_fields( Field(name="id", type="string", indexing=["attribute", "summary"]) ) - self.disk_folder = os.path.join(os.getenv("WORK_DIR"), "sample_application") - self.vespa_docker = VespaDocker(port=8089, disk_folder=self.disk_folder) + self.vespa_docker = VespaDocker(port=8089) self.app = self.vespa_docker.deploy(application_package=self.app_package) with open( os.path.join(os.environ["RESOURCES_DIR"], "qa_sample_sentence_data.json"), @@ -1440,7 +1356,6 @@ def test_batch_operations_asynchronous_mode(self): ) def tearDown(self) -> None: - shutil.rmtree(self.disk_folder, ignore_errors=True) self.vespa_docker.container.stop() self.vespa_docker.container.remove() @@ -1454,8 +1369,7 @@ def setUp(self) -> None: # # Deploy application # - self.disk_folder = os.path.join(os.getenv("WORK_DIR"), "sample_application") - self.vespa_docker = VespaDocker(port=8089, disk_folder=self.disk_folder) + self.vespa_docker = VespaDocker(port=8089) self.app = self.vespa_docker.deploy(application_package=self.app_package) # # Create a sample data frame @@ -1490,7 +1404,6 @@ def test_query(self): self.assertIn("fields", hit) def tearDown(self) -> None: - shutil.rmtree(self.disk_folder, ignore_errors=True) self.vespa_docker.container.stop() self.vespa_docker.container.remove() @@ -1498,8 +1411,7 @@ def tearDown(self) -> None: class TestSequenceClassification(TestApplicationCommon): def setUp(self) -> None: self.app_package = create_sequence_classification_task() - self.disk_folder = os.path.join(os.getenv("WORK_DIR"), "sample_application") - self.vespa_docker = VespaDocker(port=8089, disk_folder=self.disk_folder) + self.vespa_docker = VespaDocker(port=8089) self.app = self.vespa_docker.deploy(application_package=self.app_package) def test_model_endpoints(self): @@ -1514,6 +1426,5 @@ def test_prediction(self): ) def tearDown(self) -> None: - shutil.rmtree(self.disk_folder, ignore_errors=True) self.vespa_docker.container.stop() self.vespa_docker.container.remove() diff --git a/vespa/test_package.py b/vespa/test_package.py index 561965d4..406b66da 100644 --- a/vespa/test_package.py +++ b/vespa/test_package.py @@ -740,18 +740,6 @@ def test_schema_to_text(self): ) self.assertEqual(self.app_package.schema.schema_to_text, expected_result) - def test_hosts_to_text(self): - expected_result = ( - '\n' - "\n" - "\n" - ' \n' - " node1\n" - " \n" - "" - ) - self.assertEqual(self.app_package.hosts_to_text, expected_result) - def test_services_to_text(self): expected_result = ( '\n' @@ -1073,18 +1061,6 @@ def test_user_schema_to_text(self): self.app_package.get_schema("user").schema_to_text, expected_user_result ) - def test_hosts_to_text(self): - expected_result = ( - '\n' - "\n" - "\n" - ' \n' - " node1\n" - " \n" - "" - ) - self.assertEqual(self.app_package.hosts_to_text, expected_result) - def test_services_to_text(self): expected_result = ( '\n' @@ -1239,18 +1215,6 @@ def test_schema_to_text(self): ) self.assertEqual(self.app_package.schema.schema_to_text, expected_result) - def test_hosts_to_text(self): - expected_result = ( - '\n' - "\n" - "\n" - ' \n' - " node1\n" - " \n" - "" - ) - self.assertEqual(self.app_package.hosts_to_text, expected_result) - def test_services_to_text(self): expected_result = ( '\n' @@ -1530,18 +1494,6 @@ def test_schema_to_text(self): ) self.assertEqual(self.app_package.schema.schema_to_text, expected_result) - def test_hosts_to_text(self): - expected_result = ( - '\n' - "\n" - "\n" - ' \n' - " node1\n" - " \n" - "" - ) - self.assertEqual(self.app_package.hosts_to_text, expected_result) - def test_services_to_text(self): expected_result = ( '\n' @@ -1618,18 +1570,6 @@ def test_get_schema(self): self.assertIsNone(self.model_server.schema) self.assertEqual(self.model_server.schema, self.model_server.get_schema()) - def test_hosts_to_text(self): - expected_result = ( - '\n' - "\n" - "\n" - ' \n' - " node1\n" - " \n" - "" - ) - self.assertEqual(self.model_server.hosts_to_text, expected_result) - def test_services_to_text(self): expected_result = ( '\n'