API’s amaze me. Short for Application Programming Interface, an API is how software communicates with other software. API’s are part of the plumbing that connects our digital world together. Every tech-forward company is developing internal and external API’s to connect.
API’s also allow developers to abstract away parts of a program that already exist by plugging in an API to connect with a vendor who handles the process. If you are building a new e-commerce site today, instead of spending months writing code to process payments, you could connect to Stripe’s API in a few lines of code. Your application now has access to all of the functionality that Stripe has spent years developing.
Building connections to an API is a key part of a data engineer’s role as it allows them to extract data from one source to use elsewhere. This process can be scaled and automated.
In my previous job in real estate investing, I routinely updated our economic reports with a very manual process. I downloaded economic data from Moody’s, saved it in a .csv (Excel) file, copied and pasted into the data worksheet on a dashboard (Excel spreadsheet), and updated the cell links and analysis calculations to connect the new data. Some dashboards — like monthly job growth in 10 different MSA’s — required doing this process each time for each city. There are three major flaws in this process that using an API would fix:
1. Does not scale — If each city takes N minutes to update, adding more cities required the number of cities x N. This was a linear process.
2. Manual process — An automated process would refresh the dashboard every month when the data is released and remove all of my work.
3. Prone to human error — There is always the potential for human error like forgetting to update a formula.
Moody’s has a well-documented API for subscribers which would have automated much of this process and saved hundreds of hours over the years. While I didn’t know how to use it at the time, I learned during my first two months in Jigsaw Labs’ Data Engineering school how easily I could build this.
Extracting data from an API and creating objects is very useful. For this example, I am going to use the OpenWeather API since I no longer have a Moody’s subscription.
1. Read the API Documentation
2. Set up authorization and URL to query. Create a function request_weather() that calls the API and returns JSON data
3. Create a class City. This is an object with the attributes we will want to load the data into
5. Create a class CityBuilder that inputs the API’s weather data when initialized and loads that data into an instance of the City object
6. Finally, set up the instructions to run the program. For this I created separate modules for each class and imported them into an index.py file. First you connect to the API, then you request the weather data, then you load that data into an object and finally you’re returned an object that you can do anything with.
Now loaded with real time data, there is unlimited potential for usage. For example, you could build a new weather app by layering on a UI, study weather patterns over time, or load the data into a machine learning model with other data and forecast the weather’s impact.