Organising Our Code

Build a Twitter Analytics App

1 Introduction: Start Here

2 The First Step: Design Your Solution

3 In Which I Rant a Little

4  Design Solution

5 Writing Great Code

6 Writing the Backend Twitter Server

Writing the Code in Small Parts: Part 1, The Basic App

Part 2: Adding a Counter to Exit

Part 3: Adding Language and Retweet Count

Part 4: Organising Our Code

7 Adding the Data to a Database

8 Testing: What and How to Test

8.1 Testing Our Frontend

8.2 Testing Our Backend

9 Displaying our Data using the Flask Webserver

9.1 Introduction to Flask

9.2 Adding templates to our Flask app

9.3 Displaying our Tweets in the Flask Web Server

10 Future Work and Improvements

So, our code is starting to get messy, and we need to start organising it. There are two main tasks:

1 We have functions spread all over the place. Put them in a single class. This will make them easy to manage, and more importantly, easy to test.

2 At the moment, we are just printing the statistics. We need a way to pass them to the main function (so we can store, manipulate it etc).

Remember, global variables are evil. They cause unintended consequences, are hard to track, and make maintenance a nightmare. So how will we pass our data from the Twitter streaming API back to our main function? Remember the streaming API is real time, and we don’t want to do any processing (like writing to a database) in it.

Our solution will be to create an another class to store the statistics. This will make it easy to access it from multiple places, but still easy to test. This class will be light, as it will store the data as Python objects in memory, so it won’t take too much time to read/write from it (as it would from a database).

Let’s look at each of the above in detail.

A Twitter Class

First, let’s create a twitter class to encapsulate the multiple functions we have.  I’m going to simplify the code so we only have 2 functions- one to read streaming api, the other to read the trends, as that’s what our original use case required.

Let’s go over our class step by step. The code is here:

We set up the authorisation in the initialisation function. We also setup the other variables we need, like num_tweets_to_grab.

After that, we bring in our existing functions:

A Statistics class

As discussed, we need a lightweight statistics class. The code for it is actually quite simple. It is taken from here:

As we can see, all we are doing is storing the data in Python dictionaries. So why use this method?

Since this is now an object, we can use it from multiple places. We will create it once, and pass the object around. That means, unlike global variables, if we need to change the code, it only needs to be done once.

Let’s see how the class is used. It will be created in the TwitterMain class’s init() function. Only showing new code:

We are going to pass this object to the twitter_listener class, which actually grabs the streaming data:

If you remember, originally, we were printing the language data. We will now pass the data to our stats object:

Instead of:

we’ll do:

This will store the language in the lang list in our stats class. We will do the same for the other values we want to store.

Then, back in the main code, once we have gathered our streaming data, we can read back what we have stored by

The get_stats() function returns everything we have stored, which we now print.

So what was the advantage of doing it this way?

  • We have passed data from the *twitter_listener* to the *TwitterMain()* class without using global variables
  • The method is quite quick, as we are just storing the objects in memory
  • We can now read and write the stats in multiple places, which means one function could be updating them, while another could be reading them.

Make sure you look at the whole code to understand what is going on.

Next: Enough printing on the screen, we need to start saving the code in a database.