bloom-harvester

Bloom Library Harvester

Basic Idea

This will be a process that can download books stored in Bloom Library and then upload products based on them so that they are available to users.

Development Dependencies

This program has various dependencies on DLLs and other outputs from BloomDesktop. Run build/getDependencies.ps1 using Powershell. This will download all the required dependencies to lib/dotnet/ as well as the Debug and Release build folders. Then open BloomHarvester.sln in Visual Studio

Local Bloom Dependencies

Normally, you can just use the getDependencies.ps1 script to pull in Bloom dependencies from the latest Bloom build. If you need to utilize your local Bloom build instead of an official build, you can use the build/copyBloomDependencies.sh script.

• Make sure you have a BloomDesktop repository and have built it.
• Change directories to the Harvester's "build" folder. (The script makes some assumptions as to the location of the Harvester's build directory).
• Run the copyBloomDependencies.sh script with the following arguments: "./copyBloomDependencies.sh [pathToBloomDesktopRepository]"
• This script will copy the Bloom.exe and the relevant files from Bloom's output/browser and DistFiles folders into BloomHarvester's build folder.

Setting Environment variables for runtime

You will need to set some or all of the following Environment variables to store some keys:

• BloomHarvesterAzureAppInsightsKey{Prod|Test|Dev}
  • Can be found in Azure Portal; find the dev-harvestAppInsights/test-harvestAppInsights/harvestAppInsights resource, and copy the Instrumentation Key from there.
• BloomHarvesterParseAppId{Prod|Test|Dev}
  • Parse server app ID.
• BloomHarvesterS3[Secret]Key{Prod|Test|Dev}.
  • This set is for a user that can write to bloomharvest, the harvester output bucket.
• BloomBooksS3[Secret]Key{Prod|Test|Dev}.
  • This set is for a user that can read from the bucket from which harvester downloads books.
• BloomHarvesterUserName
  • [email protected] - this is the parse server username.
• BloomHarvesterUserPassword{Prod|Test|Dev|Local}
  • This is the parse server password for BloomHarvesterUserName.

Cleaning the build

If you want to make sure your build is really clean, running Clean Build in Visual Studio doesn't suffice. (VS Clean Build removes any files that were directly generated by Visual Studio's build process. It doesn't clean any other files that might be there, e.g. as a result of running our getDependencies.ps1 script. You need to go into the directory it builds into and delete any or all unwanted files/subdirectories out of there.

Deployment

There are some scripts provided to help setup or update a Harvester instance on an environment designed for running the Harvester continually. You can use the provided build/updateDeployment.ps1 script for both. It will download the latest builds, setup the new folder, shutdown any existing Harvester-related processes, swap the folders, and then startup Harvester-related processes again.

• Copy the build/updateDeployment.ps1 and build/downloadAndExtractZip.ps1 scripts to the deployment machine.
• Optional: Modify the paths at the top of the updateDeployment.ps1 script to point to the location where the Harvester should be saved.
• Optional: You may want to download RestartOnCrash and have that set up to monitor and quickly restart the application if it goes down. See src/Monitoring/MonitoringReadme.txt for instructions if you wish to do so.
• For ad-hoc setup or update to the latest binaries, you can just run the updateDeployment.ps1 script using Powershell.
• If you want to run this at regular intervals, you can use the Windows Task Scheduler to setup a scheduled task to run this at the desired time intervals (e.g. once a day, once an hour, etc.). The command you should ask Task Scheduler to run is "Powershell -ExecutionPolicy Bypass -File [pathToUpdateDeploymentScript]".

Azure

Azure permissions: If you want to view the logs or status in Azure Portal, you will obviously also need access to the specified resources.

Searching for specific log text

1. portal.azure.com
2. All resources.
3. Find the appropriate resource. (It should be an ApplicationInsights resource). Probably named "harvestAppInsights" or "dev-harvestAppInsights"
4. Open up the "Search" blade.
5. You can type the text you want to search and it'll look through all the traces, events, etc. for it

Viewing metrics

1. portal.azure.com
2. All resources.
3. Find the appropriate resource. (It should be an ApplicationInsights resource). Probably named "harvestAppInsights" or "dev-harvestAppInsights"
4. Open up the "Metrics" blade.
5. Ensure the appropriate resource is selected. You can select a Metric such as "Events" and Aggregation=Sum to see the total number of events in each time period.
6. To get the count for a specific event, you can click "Add filter" and Property EventName = [the event name in question]

Querying tables via SQL-like Syntax

1. portal.azure.com
2. All resources.
3. Find the appropriate resource. (It should be an ApplicationInsights resource). Probably named "harvestAppInsights" or "dev-harvestAppInsights"
4. Open up the "Servers" blade.
5. Click the "Servers Logs / analytics" button.
6. On the left, you will see a list of tables. Find the relevant table, hover over it, and press the eye icon in the right side.
7. You can write various SQL-like queries. The language is called "Kusto", you can search for the language documentation for that to perform more sophisticated queries.