No (public) API? No problem!
Intro
Recently, I bought my first home and 3 months in couldn’t be happier. When we went to view the house for the second time, the previous owner made an off-hand comment about how she thought the heating system (EPH Ember) is internet connected but hadn’t bothered setting it up. So it wasn’t too long before I tired of taking the labourous few steps to the thermostat to adjust heating and decied to have a go at setting it up, and before long it was ready to go.
Another month of using the app passed and I started to notice features missing. Among others, I wanted a way to view historical temperature data, or to link with Google Home. Before the idea existed long enough to even know whether it was a good idea or not, I was set on reverse engineering the API.
In this post, I’ll show you how you can reverse engineer a propietry API using Charles and Postman in order to leverage functionality beyond the original purpose.
Monitoring the network
So we have an app that we know sends traffic over the network. But how can we see what that traffic looks like? We need to capture that network traffic and see what it looks like.
A widely known tool here is Wireshark, which allows you to monitor all traffic going over the network. If the app worked via LAN this would be useful, however I’ve confirmed that the system operates over the internet (a simple test is to switch to mobile data and confirmed that the app still works) so I suspect there’s a better tool for the job here.
Enter Charles. Charles acts as a HTTP/HTTPS proxy. Every HTTP request made on the device running Charles will be routed via Charles, which will log all the essentials for examination. Most importantly, it allows usage of its own certificates making HTTPS traffic visible within the app (the certificate is unique to your copy of Charles so your traffic is still safe).
There’s a desktop version which you can run, and route your mobile traffic through. But here I’m going to give the iOS version a go. The plan here is to enable Charles, perform an action in the thermostat app and then jump back to Charles to analyse the data.
After doing so we’re greeted with this:
Notice how there’s nothing particularly useful here and we can’t make head or tail of the data. That’s because we didn’t enable SSL proxying. So let’s do that for this URL, and… success!
We have a series of endpoints. First of all, we want to find temperature data. Scanning through the list /zones/polling looks most promising. So let’s go ahead and look at the response body. Lucky guess; it contains the information we want. Charles should contain all the information we need to recreate this. So let’s take a look at the headers first.
Everything here is more or less “default” (most libraries will send those headers without you instructing it to). There are 2 things that stand out. The first is the User-Agent. It probably isn’t important but some APIs or WAFs (the application firewall that sits in front of the API to help protect it from malicious requests) will be configured to reject requests that aren’t from the expected User-Agent. But more importantly is Authorization. This looks like a token or key. We’ll need to figure out how to get this, but let’s look at the body before we get distracted.
So knowing a little bit about the system, the documentation all refers to the gateway as a device you plug-in that sits between your boiler and internet, reading the values and passing them onto the API over the Wi-Fi.
Interestingly, this is stated on the back of the device, so we could skip the API endpoint discovery that returns this and hardcode it. So about that token. After flicking through the requests there is only 1 that is of interest which is a ‘refreshToken’ endpoint. However that needs… a refresh token. So this time I repeat the entire process but logging out first, with the goal to capture the login attempt. And… success! A much more promising endpoint called ‘appLogin/login’, which takes a userName and password and gives us a token.
Now we should have all the parts we need, let’s jump onto a computer and see if we can recreate a simple flow.
Sending requests with Postman
There’s a few ways we can make requests. Simple GET requests can be made through a browser. However most APIs will also make use of POST, PUT, DELETE etc. We could make these requests in Terminal using curl, but it can be a little bit fiddly. This is where Postman comes in.
Postman is a standalone tool, formerly a browser extension, that lets you easily send requests for all HTTP methods and configure the body and headers.
The interface looks something like this. Pretty self explanatory, with the method sent to POST and our URL entered.
But before we can hit that big shiny Send button, we need to configure the body of our request.
Notice that I choose ‘raw’ for the format and then chose JSON as the format from the dropdown. How did I know to do this? There are different ways of encoding data, and usually the request header tells you what format the data is in. If you take another look at the Charles headers you’ll notice the format is ‘application/json’. So let’s see what happens…
What looks like a valid response first time, and it contains our token that we’ll need to use for our other request. Should I not have got lucky, the initial troubleshooting steps I would have tried would be to enter an invalid username/password. Make all of the headers exactly what I observed in Charles. Add a random letter to the end of the endpoint so it is no longer valid. In each of these scenarios I’m trying to see “do I get a different response?” as this will help narrow down whether something is the problem.
So let’s move on to the poll request. We setup the body much the same way with the gateWayId value we discussed earlier, but we have one extra step. Let’s look at the headers tab. Here you can see a list of all the headers that Postman is going to send and modify them. In our case we want to just add one called ‘Authorization’ with our token we just got.
We can now examine the body and see that we got all the useful information. We’ve successfully created a flow where we can login and read data.
Putting it into code
Before I call it a day, after 30 minutes of hacking away, let’s see if I can actually achieve my original aim. My first step is to start writing it in my chosen language (Node.js).
Another cool feature about Postman is the ‘Code’ button. Hit it and you’ll be presented with a list of dozens of different languages/libraries and it will generate code that matches exactly your request.
This can make it super easy to get something up and running, however I recommend taking with a pinch of salt. For example, there is shorthand for POST in axios that is axios.post, which simplies this code. I also use async/await throughout my code. So I’m going to tweak it a bit.
Here we have 3 different functions that perform roughly the flow above. As you can see it’s just a few lines. Then to put them all together:
Pretty simple, right. Give it a run and it prints out the zone data as expected. With a few more modifications I have a tiny application that pulls my username and password from AWS Parameter Store, reads the data and stores it in DynamoDB, that I can run in a lambda on AWS.
Conclusion
All of this took a couple of hours maximum. Sure, I had some lucky guesses where I followed my nose that may not be so quick if you this was your first rodeo, but hopefully this article documents my adventures whilst providing you with a rough guide to how you might go about doing something similar yourself.