Merge branch 'master' into feature_teams

README.md

@@ -6,43 +6,47 @@ Further usage documentation can be found at my blog page [Labbots.com](https://l
 
 ## Dependencies
 
-This script does not have very many dependencies. Most of the dependencies are available by default in most linux platforms. This script requires the following packages
+This script does not have very many dependencies. Most of the dependencies are available by default in most linux platforms. This script explicitly requires the following packages:
 
+- Bash ( 4.x + )
 - Curl
-- sed (Stream editor)
-- find command
-- awk
-- getopts ( bash builtin )
-- xargs
+- grep
+- sed ( Stream editor )
+- file/mimetype ( generating mimetype for extensionless files )
+- find command ( for recursive uploading )
+- xargs ( for parallel uploading )
 
 ## Features
 
-- No dependencies at all.
-- Upload files and folders.
-- Upload files in parallel.
+- Upload files and folders normally/parallelly.
 - Upload sub-folders and content inside it hierarchically.
+- Upload multiple files / folders with single command.
+- Update (overwrite) your files/folder, like synchronisation.
 - Config file support ( easy to use script on multiple machines ).
 - Uses latest gdrive v3 api.
-- Share files after uploading.
+- Share files after uploading ( to an email or just anyone ).
 - Pretty Logging.
 
 ## Usage
 
 When the script is executed for the first time. It asks for few configuration variables interactively to connect with google APIs. The script requires Client ID and Client secret to access the APIs which can be generated at [google console].
 
 ### Generating Google OAuth2 Credentials
+
 - Log into google developer console at [google console].
 - Create new Project or use existing project if you don't have a project already created.
 - Create new OAuth 2.0 Credentials
-    - Select Application type "other"
-    - Provide name for the new credentials
-    - This would provide a new client ID and Client Secret
+  - Select Application type "other"
+  - Provide name for the new credentials
+  - This would provide a new client ID and Client Secret
 - Enable Google Drive API for the project under "Library"
 
 *Note:* When the script is run for the first time. The script must be authorized in browser. This may show a certificate warning if the application authorized domain is not configured on google console project.
 
 ### Running the script
-Script also asks for root folder to be set to which the script uploads documents by default. The default folder will be the root of your google drive. If you want to upload documents to any specific directory by default then provide the google folder ID or URL of that folder when root folder ID/URL is requested.
+
+Script also asks for root folder to be set to which the script uploads documents by default.
+The default folder will be the root of your google drive. If you want to upload documents to any specific directory by default then provide the google folder ID or URL of that folder when root folder ID/URL is requested.
 
 For example
 `https://drive.google.com/drive/folders/8ZzWiO_pMAtBrLpARBZJNV09TVmM`
@@ -60,21 +64,51 @@ Above command will create a folder under the pre-configured root directory and u
 
 Other Options available are
 
-    -C | --create-dir <foldername> - option to create directory. Will provide folder id.
-    -r | --root-dir <google_folderid> - google folder id to which the file/directory to upload.
+    -C | --create-dir <foldername> - option to create directory. Will provide folder id. Can be used to specify workspace folder for uploading files/folders.
+
+    -r | --root-dir <google_folderid> - google folder id or url to which the file/directory to upload.
+
     -s | --skip-subdirs - Skip creation of sub folders and upload all files inside the INPUT folder/sub-folders in the INPUT folder, use this along with -p/--parallel option to speed up the uploads.
-    -p | --parallel <no_of_files_to_parallely_upload> - Upload multiple files in parallel, only works along with --skip-subdirs/-s option, Max value = 10, low value are recommended.
+
+    -p | --parallel <no_of_files_to_parallely_upload> - Upload multiple files in parallel, Max value = 10, use with folders.
+        
+        Note :
+          This command is only helpful if you are uploading many files which aren't big enough to utilise your full bandwidth, using it otherwise will not speed up your upload and even error sometimes,
+          1 - 6 value is recommended, but can use upto 10. If errors with a high value, use smaller number.
+          Beaware, this isn't magic, obviously it comes at a cost of increased cpu/ram utilisation as it forks multiple bash processes to upload ( google how xargs works with -P option ).              
+
+    -o | --overwrite - Overwrite the files with the same name, if present in the root folder/input folder, also works with recursive folders and single/multiple files.
+
+    -f | --[file/folder] - Specify files and folders explicitly in one command, use multiple times for multiple folder/files.   
+
+        For uploading multiple input into the same folder:
+           1. Use -C / --create-dir ( e.g `./upload.sh -f file1 -f folder1 -f file2 -C <folder_wherw_to_upload>` ) option.
+           2. Give two initial arguments which will use the second argument as the folder you wanna upload ( e.g: `./upload.sh filename <folder_where_to_upload> -f filename -f foldername` ).
+           
+        This flag can also be used for uploading files/folders which have `-` character in their name, normally it won't work, because of the flags, but using `-f -[file|folder]namewithhyphen` works. Applies for -C/--create-dir too.
+        Also, as specified by longflags ( `--[file|folder]` ), you can simultaneously upload a folder and a file.
+        Incase of multiple -f flag having duplicate arguments, it takes the last duplicate of the argument to upload, in the same order provided.
+                            
     -S | --share <optional_email_address>- Share the uploaded input file/folder, grant reader permission to provided email address or to everyone with the shareable link.
-    -v | --verbose - Display detailed message.
-    -V | --verbose-progress - Display detailed message and detailed upload progress( curl normal progress info ).
-    -i | --save-info <file_to_save_info> - Save uploaded files info to the given filename.
+
+    -q | --quiet - Supress the normal output, only show success/error upload messages for files, and one extra line at the beginning for folder showing no. of files and sub folders.
+
+    -v | --verbose - Display detailed message (only for non-parallel uploads).
+
+    -V | --verbose-progress - Display detailed message and detailed upload progress(only for non-parallel uploads).
+
+    -i | --save-info <file_to_save_info> - Save uploaded files info to the given filename."
+
     -h | --help - Display usage instructions.
+
     -z | --config - Override default config file with custom config file.
-    -D | --debug - Display script command trace.
+
+    -D | --debug - Display script command trace."
 
 To create a folder:
 
     ./upload.sh -C <foldername> -r <optional-root-dir-id> 
+
 This will give the folder id of the newly created folder which can be used to upload files to specific directory in google drive.
 
 To Upload file to specific google folder
@@ -92,6 +126,6 @@ The script also allows to upload directories. If directory path is provided as a
 
 MIT
 
-[github-bashutils]:<https://github.com/soulseekah/bash-utils>
+[github-bashutils]: <https://github.com/soulseekah/bash-utils>
 [deanet-gist]:<https://gist.github.com/deanet/3427090>
 [google console]:<https://console.developers.google.com>

google-oauth2.sh

@@ -9,86 +9,99 @@
 # Set CLIENT_ID and CLIENT_SECRET and SCOPE
 # See SCOPES at https://developers.google.com/identity/protocols/oauth2/scopes#docsv1
 
-short_help() {
-    echo -e "\nNo valid arguments provided."
-    echo -e "Usage:\n"
-    echo -e " ./google-oauth2.sh create - authenticates a user."
-    echo -e " ./google-oauth2.sh refresh - gets a new access token."
+shortHelp() {
+    printf "
+No valid arguments provided.
+Usage:
+
+ ./%s create - authenticates a user.
+ ./%s refresh - gets a new access token.\n" "$0" "$0"
     exit 0
 }
 
-[ "$#" = "0" ] && short_help
+[[ $1 = create ]] || [[ $1 = refresh ]] || shortHelp
 
-# Clear nth no. of line to the beginning of the line.
-clear_line() {
-    echo -en "\033[""$1""A"
-    echo -en "\033[2K"
+# Move cursor to nth no. of line and clear it to the begining.
+clearLine() {
+    printf "\033[%sA\033[2K" "$1"
 }
 
-[ "$1" = create ] || [ "$1" = refresh ] || short_help
+# Method to extract data from json response.
+# Usage: jsonValue key < json ( or use with a pipe output ).
+jsonValue() {
+    [[ $# = 0 ]] && printf "%s: Missing arguments\n" "${FUNCNAME[0]}" && return 1
+    declare LC_ALL=C num="${2:-1}"
+    grep -o "\"""$1""\"\:.*" | sed -e "s/.*\"""$1""\": //" -e 's/[",]*$//' -e 's/["]*$//' -e 's/[,]*$//' -e "s/\"//" -n -e "${num}"p
+}
+
+# Update Config. Incase of old value, update, for new value add.
+# Usage: updateConfig valuename value configpath
+updateConfig() {
+    [[ $# -lt 3 ]] && printf "%s: Missing arguments\n" "${FUNCNAME[0]}" && return 1
+    declare VALUE_NAME="$1" VALUE="$2" CONFIG_PATH="$3" FINAL=()
+    declare -A Aseen
+    printf "" >> "$CONFIG_PATH" # If config file doesn't exist.
+    mapfile -t VALUES < "$CONFIG_PATH" && VALUES+=("$VALUE_NAME=$VALUE")
+    for i in "${VALUES[@]}"; do
+        [[ $i =~ $VALUE_NAME\= ]] && FINAL+=("$VALUE_NAME=$VALUE") || FINAL+=("$i")
+    done
+    for i in "${FINAL[@]}"; do
+        [[ ${Aseen[$i]} ]] && continue
+        printf "%s\n" "$i" && Aseen[$i]=x
+    done >| "$CONFIG_PATH"
+}
 
-echo "Starting script.."
+printf "Starting script..\n"
 
 CLIENT_ID=""
 CLIENT_SECRET=""
 SCOPE="https://www.googleapis.com/auth/drive"
 REDIRECT_URI="urn:ietf:wg:oauth:2.0:oob"
+TOKEN_URL="https://accounts.google.com/o/oauth2/token"
 
 # shellcheck source=/dev/null
-[ -e "$HOME"/.googledrive.conf ] && source "$HOME"/.googledrive.conf
+[[ -f $HOME/.googledrive.conf ]] && source "$HOME"/.googledrive.conf
 
-echo "Checking credentials.."
+printf "Checking credentials..\n"
 
-if [ -z "$CLIENT_ID" ]; then
+# Credentials
+if [[ -z $CLIENT_ID ]]; then
     read -r -p "Client ID: " CLIENT_ID
-    unset token
-    echo "CLIENT_ID=$CLIENT_ID" >> "$HOME"/.googledrive.conf
+    updateConfig CLIENT_ID "$CLIENT_ID" "$HOME"/.googledrive.conf
 fi
-
-if [ -z "$CLIENT_SECRET" ]; then
+if [[ -z $CLIENT_SECRET ]]; then
     read -r -p "Client Secret: " CLIENT_SECRET
-    unset token
-    echo "CLIENT_SECRET=$CLIENT_SECRET" >> "$HOME"/.googledrive.conf
+    updateConfig CLIENT_SECRET "$CLIENT_SECRET" "$HOME"/.googledrive.conf
 fi
 
-sleep 1
-clear_line 1
-clear_line 1
-echo "Required credentials set."
-sleep 1
+for _ in {1..2}; do clearLine 1; done
+printf "Required credentials set.\n"
 
-# Method to extract data from json response
-jsonValue() {
-    num="$2"
-    grep \""$1"\" | sed "s/\:/\n/" | grep -v \""$1"\" | sed -e "s/\"\,//g" -e 's/["]*$//' -e 's/[,]*$//' -e 's/^[ \t]*//' -e s/\"// | sed -n "${num}"p
-}
-
-if [ "$1" == "create" ]; then
-    echo "Visit the below URL, tap on allow and then enter the code obtained:"
-    sleep 1
+if [[ $1 = create ]]; then
+    printf "Visit the below URL, tap on allow and then enter the code obtained:\n"
     URL="https://accounts.google.com/o/oauth2/auth?client_id=$CLIENT_ID&redirect_uri=$REDIRECT_URI&scope=$SCOPE&response_type=code&prompt=consent"
-    echo -e """$URL""\n"
+    printf "%s\n\n" "$URL"
     read -r -p "Enter the authorization code: " CODE
 
-    CODE="$(echo "$CODE" | tr -d ' ' | tr -d '[:blank:]' | tr -d '[:space:]')"
-    if [ -n "$CODE" ]; then
-        RESPONSE="$(curl -s --request POST --data "code=$CODE&client_id=$CLIENT_ID&client_secret=$CLIENT_SECRET&redirect_uri=$REDIRECT_URI&grant_type=authorization_code" https://accounts.google.com/o/oauth2/token)"
+    CODE="${CODE//[[:space:]]/}"
+    if [[ -n $CODE ]]; then
+        RESPONSE="$(curl --compressed -s -X POST --data "client_id=$CLIENT_ID&client_secret=$CLIENT_SECRET&refresh_token=$REFRESH_TOKEN&grant_type=refresh_token" $TOKEN_URL)"
 
-        ACCESS_TOKEN="$(echo "$RESPONSE" | jsonValue access_token)"
-        REFRESH_TOKEN="$(echo "$RESPONSE" | jsonValue refresh_token)"
+        ACCESS_TOKEN="$(jsonValue access_token <<< "$RESPONSE")"
+        REFRESH_TOKEN="$(jsonValue refresh_token <<< "$RESPONSE")"
 
-        echo "Access Token: $ACCESS_TOKEN"
-        echo "Refresh Token: $REFRESH_TOKEN"
+        printf "Access Token: %s\n" "$ACCESS_TOKEN"
+        printf "Refresh Token: %s\n" "$REFRESH_TOKEN"
     else
-        echo -e "\nNo code provided, run the script and try again"
+        printf "\nNo code provided, run the script and try again.\n"
         exit 1
     fi
-elif [ "$1" == "refresh" ]; then
-    if [ -n "$REFRESH_TOKEN" ]; then
-        RESPONSE="$(curl -s --request POST --data "client_id=$CLIENT_ID&client_secret=$CLIENT_SECRET&refresh_token=$REFRESH_TOKEN&grant_type=refresh_token" https://accounts.google.com/o/oauth2/token)"
+elif [[ $1 = refresh ]]; then
+    if [[ -n $REFRESH_TOKEN ]]; then
+        RESPONSE="$(curl --compressed -s -X POST --data "client_id=$CLIENT_ID&client_secret=$CLIENT_SECRET&refresh_token=$REFRESH_TOKEN&grant_type=refresh_token" $TOKEN_URL)"
         ACCESS_TOKEN="$(echo "$RESPONSE" | jsonValue access_token)"
-        echo "Access Token: $ACCESS_TOKEN"
+        printf "Access Token: %s\n" "$ACCESS_TOKEN"
     else
-        echo "Refresh Token not set, use $0 create to generate one."
+        printf "Refresh Token not set, use %s create to generate one.\n" "$0"
     fi
 fi