Skip to content

Pierre #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
plermant wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Pierre #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
1ac1464
Add files via upload
plermant Sep 14, 2022
b8ca05d
Merge pull request #1 from plermant/pierre
plermant Sep 14, 2022
0e69b88
Add files via upload
plermant Sep 14, 2022
38fa2b6
Merge pull request #2 from plermant/pierre
plermant Sep 14, 2022
df01e75
Update Mini_Project_Data_Wrangling_Pandas.ipynb
plermant Sep 26, 2022
bef0bcc
Update Mini_Project_Data_Wrangling_Pandas.ipynb
plermant Sep 26, 2022
f945342
Update Mini_Project_Wrangling_Json_Exercise.ipynb
plermant Sep 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 156 additions & 79 deletions mec-3.4.1-api-mini-project/api_data_wrangling_mini_project.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,147 +2,147 @@
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This exercise will require you to pull some data from https://data.nasdaq.com/ (formerly Quandl API)."
],
"metadata": {}
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As a first step, you will need to register a free account on the https://data.nasdaq.com/ website."
],
"metadata": {}
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After you register, you will be provided with a unique API key, that you should store:\r\n",
"\r\n",
"*Note*: Use a `.env` file and put your key in there and `python-dotenv` to access it in this notebook. \r\n",
"\r\n",
"The code below uses a key that was used when generating this project but has since been deleted. Never submit your keys to source control. There is a `.env-example` file in this repository to illusrtate what you need. Copy that to a file called `.env` and use your own api key in that `.env` file. Make sure you also have a `.gitignore` file with a line for `.env` added to it. \r\n",
"\r\n",
"After you register, you will be provided with a unique API key, that you should store:\n",
"\n",
"*Note*: Use a `.env` file and put your key in there and `python-dotenv` to access it in this notebook. \n",
"\n",
"The code below uses a key that was used when generating this project but has since been deleted. Never submit your keys to source control. There is a `.env-example` file in this repository to illusrtate what you need. Copy that to a file called `.env` and use your own api key in that `.env` file. Make sure you also have a `.gitignore` file with a line for `.env` added to it. \n",
"\n",
"The standard Python gitignore is [here](https://github.com/github/gitignore/blob/master/Python.gitignore) you can just copy that. "
],
"metadata": {}
]
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# get api key from your .env file\n",
"import os\n",
"from dotenv import load_dotenv # if missing this module, simply run `pip install python-dotenv`\n",
"from dotenv import load_dotenv # if missin this module, simply run `pip install python-dotenv`\n",
"\n",
"load_dotenv()\n",
"API_KEY = os.getenv('NASDAQ_API_KEY')\n",
"pierreKey = os.getenv('NASDAQ_API_KEY')\n",
"\n",
"print(API_KEY)"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"KRfk96yoWvruWZ-LjPbo\n"
]
}
],
"metadata": {}
"#print(pierreKey)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nasdaq Data has a large number of data sources, but, unfortunately, most of them require a Premium subscription. Still, there are also a good number of free datasets."
],
"metadata": {}
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For this mini project, we will focus on equities data from the Frankfurt Stock Exhange (FSE), which is available for free. We'll try and analyze the stock prices of a company called Carl Zeiss Meditec, which manufactures tools for eye examinations, as well as medical lasers for laser eye surgery: https://www.zeiss.com/meditec/int/home.html. The company is listed under the stock ticker AFX_X."
],
"metadata": {}
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can find the detailed Nasdaq Data API instructions here: https://docs.data.nasdaq.com/docs/in-depth-usage"
],
"metadata": {}
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"While there is a dedicated Python package for connecting to the Nasdaq API, we would prefer that you use the *requests* package, which can be easily downloaded using *pip* or *conda*. You can find the documentation for the package here: http://docs.python-requests.org/en/master/ "
],
"metadata": {}
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, apart from the *requests* package, you are encouraged to not use any third party Python packages, such as *pandas*, and instead focus on what's available in the Python Standard Library (the *collections* module might come in handy: https://pymotw.com/3/collections/).\n",
"Also, since you won't have access to DataFrames, you are encouraged to us Python's native data structures - preferably dictionaries, though some questions can also be answered using lists.\n",
"You can read more on these data structures here: https://docs.python.org/3/tutorial/datastructures.html"
],
"metadata": {}
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Keep in mind that the JSON responses you will be getting from the API map almost one-to-one to Python's dictionaries. Unfortunately, they can be very nested, so make sure you read up on indexing dictionaries in the documentation provided above."
],
"metadata": {}
]
},
{
"cell_type": "code",
"execution_count": 6,
"source": [
"# First, import the relevant modules"
],
"execution_count": 4,
"metadata": {},
"outputs": [],
"metadata": {}
"source": [
"# First, import the relevant modules\n",
"import json\n",
"import requests"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note: API's can change a bit with each version, for this exercise it is reccomended to use the nasdaq api at `https://data.nasdaq.com/api/v3/`. This is the same api as what used to be quandl so `https://www.quandl.com/api/v3/` should work too.\r\n",
"\r\n",
"Note: API's can change a bit with each version, for this exercise it is reccomended to use the nasdaq api at `https://data.nasdaq.com/api/v3/`. This is the same api as what used to be quandl so `https://www.quandl.com/api/v3/` should work too.\n",
"\n",
"Hint: We are looking for the `AFX_X` data on the `datasets/FSE/` dataset."
],
"metadata": {}
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\"dataset_data\":{\"limit\":null,\"transform\":null,\"column_index\":null,\"column_names\":[\"Date\",\"Open\",\"High\",\"Low\",\"Close\",\"Change\",\"Traded Volume\",\"Turnover\",\"Last Price of the Day\",\"Daily Traded Units\",\"Daily Turnover\"],\"start_date\":\"2020-12-01\",\"end_date\":\"2020-12-01\",\"frequency\":\"daily\",\"data\":[[\"2020-12-01\",112.2,112.2,111.5,112.0,null,51.0,5703.0,null,null,null]],\"collapse\":null,\"order\":null}}\n"
]
}
],
"source": [
"# Now, call the Nasdaq API and pull out a small sample of the data (only one day) to get a glimpse\n",
"# into the JSON structure that will be returned"
],
"outputs": [],
"metadata": {}
"# into the JSON structure that will be returned\n",
"re='https://data.nasdaq.com/api/v3/datasets/FSE/AFX_X/data.json?start_date=2020-12-01&end_date=2020-12-03&api_key='\n",
"req=re+pierreKey\n",
"#print(re)\n",
"r=requests.get(req)\n",
"print(r.text)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Inspect the JSON structure of the object you created, and take note of how nested it is,\n",
"# as well as the overall structure"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"{'dataset': {'id': 10095370, 'dataset_code': 'AFX_X', 'database_code': 'FSE', 'name': 'Carl Zeiss Meditec (AFX_X)', 'description': 'Stock Prices for Carl Zeiss Meditec (2020-11-02) from the Frankfurt Stock Exchange.<br><br>Trading System: Xetra<br><br>ISIN: DE0005313704', 'refreshed_at': '2020-12-01T14:48:09.907Z', 'newest_available_date': '2020-12-01', 'oldest_available_date': '2000-06-07', 'column_names': ['Date', 'Open', 'High', 'Low', 'Close', 'Change', 'Traded Volume', 'Turnover', 'Last Price of the Day', 'Daily Traded Units', 'Daily Turnover'], 'frequency': 'daily', 'type': 'Time Series', 'premium': False, 'limit': None, 'transform': None, 'column_index': None, 'start_date': '2021-01-03', 'end_date': '2020-12-01', 'data': [], 'collapse': None, 'order': None, 'database_id': 6129}}\n"
]
}
],
"metadata": {}
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These are your tasks for this mini project:\n",
"\n",
Expand All @@ -153,28 +153,108 @@
"5. What was the largest change between any two days (based on Closing Price)?\n",
"6. What was the average daily trading volume during this year?\n",
"7. (Optional) What was the median trading volume during this year. (Note: you may need to implement your own function for calculating the median.)"
],
"metadata": {}
]
},
{
"cell_type": "code",
"execution_count": null,
"source": [],
"outputs": [],
"metadata": {}
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"For request = https://data.nasdaq.com/api/v3/datasets/FSE/AFX_X/data.json?start_date=2017-01-01&end_date=2017-12-31&api_key= we got 255 data points\n",
"Minimum opening price is: 34.0 , max opening price is: 53.11\n",
"Maximum price difference in a day is: 2.81\n",
"Maximum price difference between 2 consecutive days is: 2.56\n",
"Average daily trading volume is: 89124.34\n",
"Median daily trading volume is: 76286.0\n"
]
}
],
"source": [
"re='https://data.nasdaq.com/api/v3/datasets/FSE/AFX_X/data.json?start_date=2017-01-01&end_date=2017-12-31&api_key='\n",
"req=re+pierreKey\n",
"t=requests.get(req)\n",
"# for testing only t='''{\"dataset_data\":{\"limit\":null,\"transform\":null,\"column_index\":null,\"column_names\":[\"Date\",\"Open\",\"High\",\"Low\",\"Close\",\"Change\",\"Traded Volume\",\"Turnover\",\"Last Price of the Day\",\"Daily Traded Units\",\"Daily Turnover\"],\"start_date\":\"2020-12-01\",\"end_date\":\"2020-12-01\",\"frequency\":\"daily\",\"data\":[[\"2020-12-01\",112.2,112.2,111.5,112.0,null,53.0,5703.0,null,null,null],[\"2020-12-02\",110.0,114.2,109.5,112.2,null,51.0,5703.0,null,null,null]],\"collapse\":null,\"order\":null}}'''\n",
"o=json.loads(t.text) #main dictionary object\n",
"d=o['dataset_data']['data'] #daily list information\n",
"#print(json.dumps(d))\n",
"\n",
"#NOTE, we perform separate loops for each question below.\n",
"#We could optimize the code to only parse the data once as a future exercice\n",
"\n",
"#3- let's compute the lowest and highest opening prices for the time range.\n",
"min=-1\n",
"max=0\n",
"for i in d:\n",
" if i[1]: #test for null values\n",
" if min == -1:\n",
" min = i[1] #opening price is the second element\n",
" max = i[1]\n",
" else:\n",
" if i[1] < min: min=i[1]\n",
" if i[1] > max: max=i[1]\n",
"print(\"For request =\",re,\"we got\",len(d),\"data points\")\n",
"print(\"Minimum opening price is:\",str(min),\", max opening price is:\",str(max))\n",
"\n",
"#4-computing max difference in one day, high is index 2, low is 3\n",
"diff=-1\n",
"for i in d:\n",
" if i[2] and i[3]:\n",
" if diff == -1: \n",
" diff = i[2]-i[3]\n",
" else:\n",
" if i[2]-i[3] > diff: diff= i[2]-i[3]\n",
"print(\"Maximum price difference in a day is:\",str(round(diff,2)))\n",
"\n",
"#5-computing max difference between 2 consecutive days, based on closing price\n",
"diff=0\n",
"for i in range(0,len(d)-2):\n",
" if d[i+1][4] and d[i][4]:\n",
" if diff < d[i+1][4]-d[i][4]: \n",
" diff = d[i+1][4]-d[i][4]\n",
"print(\"Maximum price difference between 2 consecutive days is:\",str(round(diff,2)))\n",
"\n",
"#6-Compute the average daily trading volume during this year - index 6 in data\n",
"sum=0\n",
"tot=0\n",
"for i in d:\n",
" if i[6]:\n",
" sum=sum+i[6]\n",
" tot=tot+1\n",
"print(\"Average daily trading volume is:\",str(round(sum/tot,2)))\n",
"\n",
"#7-Compute the median trading volume during this year\n",
"def median(l):\n",
" n = len(l)\n",
" s = sorted(l)\n",
" return (s[n//2-1]/2.0+s[n//2]/2.0, s[n//2])[n % 2] if n else None\n",
"\n",
"vol=[]\n",
"for i in d:\n",
" if i[6]:\n",
" vol.append(i[6])\n",
"print(\"Median daily trading volume is:\",str(round(median(vol),2)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"source": [],
"metadata": {},
"outputs": [],
"metadata": {}
"source": []
}
],
"metadata": {
"interpreter": {
"hash": "4885f37acae9217c235118400878352aafa7b76e66df698a1f601374f86939a7"
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3.7.9 64-bit ('springboard': conda)"
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
Expand All @@ -186,12 +266,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
},
"interpreter": {
"hash": "4885f37acae9217c235118400878352aafa7b76e66df698a1f601374f86939a7"
"version": "3.9.12"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
}
Loading