Webscraping_Engineer_Peer_Review_Assignment.ipynb

{"cells":[{"cell_type":"markdown","metadata":{},"source":["<p style=\"text-align:center\">\n","    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0221ENSkillsNetwork23455645-2022-01-01\" target=\"_blank\">\n","    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n","    </a>\n","</p>\n"]},{"cell_type":"markdown","metadata":{},"source":["# Peer Review Assignment - Data Engineer - Webscraping\n"]},{"cell_type":"markdown","metadata":{},"source":["Estimated time needed: **20** minutes\n"]},{"cell_type":"markdown","metadata":{},"source":["## Objectives\n","\n","In this part you will:\n","\n","*   Use webscraping to get bank information\n"]},{"cell_type":"markdown","metadata":{},"source":["For this lab, we are going to be using Python and several Python libraries. Some of these libraries might be installed in your lab environment or in SN Labs. Others may need to be installed by you. The cells below will install these libraries when executed.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["#!mamba install pandas==1.3.3 -y\n","#!mamba install requests==2.26.0 -y\n","!mamba install bs4==4.10.0 -y\n","!mamba install html5lib==1.1 -y"]},{"cell_type":"markdown","metadata":{},"source":["## Imports\n","\n","Import any additional libraries you may need here.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["from bs4 import BeautifulSoup\n","import html5lib\n","import requests\n","import pandas as pd"]},{"cell_type":"markdown","metadata":{},"source":["## Extract Data Using Web Scraping\n"]},{"cell_type":"markdown","metadata":{},"source":["The wikipedia webpage [https://en.wikipedia.org/wiki/List_of_largest_banks](https://en.wikipedia.org/wiki/List_of_largest_banks?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0221ENSkillsNetwork23455645-2022-01-01) provides information about largest banks in the world by various parameters. Scrape the data from the table 'By market capitalization' and store it in a JSON file.\n"]},{"cell_type":"markdown","metadata":{},"source":["### Webpage Contents\n","\n","Gather the contents of the webpage in text format using the `requests` library and assign it to the variable <code>html_data</code>\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["#Write your code here\n"]},{"cell_type":"markdown","metadata":{},"source":["<b>Question 1</b> Print out the output of the following line, and remember it as it will be a quiz question:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["html_data[101:124]"]},{"cell_type":"markdown","metadata":{},"source":["### Scraping the Data\n","\n","<b> Question 2</b> Using the contents and `beautiful soup` load the data from the `By market capitalization` table into a `pandas` dataframe. The dataframe should have the bank `Name` and `Market Cap (US$ Billion)` as column names.  Display the first five rows using head.\n"]},{"cell_type":"markdown","metadata":{},"source":["Using BeautifulSoup parse the contents of the webpage.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["#Replace the dots below\n","soup=\"......\""]},{"cell_type":"markdown","metadata":{},"source":["Load the data from the `By market capitalization` table into a pandas dataframe. The dataframe should have the bank `Name` and `Market Cap (US$ Billion)` as column names. Using the empty dataframe `data` and the given loop extract the necessary data from each row and append it to the empty dataframe.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["data = pd.DataFrame(columns=[\"Name\", \"Market Cap (US$ Billion)\"])\n","\n","for row in soup.find_all('tbody')[3].find_all('tr'):\n","    col = row.find_all('td')\n","    #Write your code here"]},{"cell_type":"markdown","metadata":{},"source":["**Question 3** Display the first five rows using the `head` function.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["#Write your code here\n"]},{"cell_type":"markdown","metadata":{},"source":["### Loading the Data\n","\n","Usually you will Load the `pandas` dataframe created above into a JSON named `bank_market_cap.json` using the `to_json()` function, but this time the data will be sent to another team who will split the data file into two files and inspect it. If you save the data it will interfere with the next part of the assignment.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["#Write your code here\n"]},{"cell_type":"markdown","metadata":{},"source":["## Authors\n"]},{"cell_type":"markdown","metadata":{},"source":["Ramesh Sannareddy, Joseph Santarcangelo and Azim Hirjani\n"]},{"cell_type":"markdown","metadata":{},"source":["### Other Contributors\n"]},{"cell_type":"markdown","metadata":{},"source":["Rav Ahuja\n"]},{"cell_type":"markdown","metadata":{},"source":["## Change Log\n"]},{"cell_type":"markdown","metadata":{},"source":["| Date (YYYY-MM-DD) | Version | Changed By          | Change Description                 |\n","| ----------------- | ------- | ------------------- | ---------------------------------- |\n","| 2022-07-12        | 0.2     | Appalabhaktula Hema | Corrected the code and markdown    |\n","| 2020-11-25        | 0.1     | Ramesh Sannareddy   | Created initial version of the lab |\n"]},{"cell_type":"markdown","metadata":{},"source":["Copyright © 2020 IBM Corporation.\n"]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.0"}},"nbformat":4,"nbformat_minor":4}