Twitter

You can fetch, store and display data about your own Tweets and Liked Tweets for one or more Twitter accounts. By default the included models and templates will link to image and video files at Twitter, but you can also download the original files to store locally. These can then be used to serve different sizes of images locally. See the Fetch Files management command below.

Set-up

Go to https://developer.twitter.com/portal. You’ll need to sign up for a Developer Account, and then create a new App for your Twitter account. Note down the three credentials it says that you should note down.

When viewing your App’s details, click the “Edit” button, then the “Keys and tokens” tab.

In the “Access Token and Secret” section, tap the “Generate” button. Make a note of the Access Token and Access Token Secret.

Django Ditto currently uses the v1.1 Twitter API rather than the latest v2. Unfortunately, accessing v1.1 requires that you apply for “Elevated” access before the API calls used will work. You can do that here: https://developer.twitter.com/en/portal/products/elevated You will then need to wait for approval.

Once you have that, go to the Django admin, and add a new Account in the Twitter app. Copy the API Key, API Key Secret, Access Token and Access Token Secret from your Twitter App into the Django admin (ignore the other fields), and save the account. A new User object will be created, reflecting the Twitter user your API credentials are associated with. If it’s worked your Account should have a title like @philgyford instead of a number like 1.

Once this is done you’ll need to import a downloaded archive of Tweets, and/or fetch Tweets from the Twitter API. See Management commands below.

Models

The models available in ditto.twitter.models are:

Account: Representing a Twitter account that has API credentials and that we fetch Tweets and/or Likes for. It has a one-to-one relationship with a User model.
Media: A photo, video or Animated GIF that was attached to one or more Tweets. Its media_type property differentiates between 'photo', 'video' and 'animated_gif'. (Yes, the plural model name is a bit confusing, sorry.) The files for photos and animated GIFs can be fetched from Twitter and used locally, although this isn’t the default. See the Fetch Files management command below.
Tweet: A Tweet. It may have been posted by one of the Users with an Account that we fetch Twets for. Or it might have been posted by a different User and favorited/liked by one of the Users with an Account.
User: A Twitter user. Every time a Tweet is fetched a new User object is created/updated for the person who posted it. One or more of these Users will have an associated Account, as these are the user(s) we fetch data for.

So, you will end up with many User objects in the system. But only one (or a few) will be associated with Account objects – these are the Users for which you fetch Tweets or favorited/liked Tweets.

Managers

Media

For Media objects there is only the standard manager available:

Media.objects.all(): Fetches all Media objects, attached to Tweets by anyone, whether public or private.

There is no concept in Twitter of a ‘private’ photo or video. The same photo could be posted by different public and private Twitter users. Exercise caution when displaying Media objects to public webpages.

Tweet

Tweet models have several managers. There are two main differentiators we want to restrict things by:

Public vs private: Whether a Tweet was posted by a public or private Twitter user.
Posted Tweet vs Favorited Tweet: Sometimes we’ll want to only fetch Tweets posted by one of the Users with an API-credentialed Account. Sometimes we’ll want to only fetch Tweets those Accounts have favorited/liked. Sometimes we’ll want to fetch ALL Tweets, whether posted or favorited.

On public-facing web pages you should only use the managers starting Tweet.public_ as these will filter out all Tweets posted by private Twitter users.

Tweet.objects.all(): The default manager fetches all Tweets in the system, posted by all Users, whether they’re public or private, and whether posted by an Account’s User or favorited by them.
Tweet.public_objects.all(): Only gets Tweets posted by any Users who are not private. This includes Tweets that have simply been favorited/liked by one of your Accounts.
Tweet.tweet_objects.all(): Only gets Tweets posted by a User with an Account, not favorited by them. And from both public and private Users.
Tweet.public_tweet_objects.all(): Only gets Tweets posted by a User with an Account, not favorited by them, but only for Users who aren’t private. For use on public pages.
Tweet.favorite_objects.all(): Only gets Tweets favorited by a User with an Account, whether those Tweets are posted by public or private Users.
Tweet.public_favorite_objects.all(): Only gets Tweets favorited by a User with an Account, and only Tweets posted by Users who aren’t private.

Of course, these can all be filtered as usual. So, if you wanted to get all the public Tweets posted by a particular User:

from ditto.twitter.models import User, Tweet

user = User.objects.get(screen_name='philgyford')
tweets = Tweet.public_tweet_objects.filter(user=user)

And to get public Tweets that user has favorited:

favorites = Tweet.public_favorite_objects.filter(user=user)

Template tags

There are four assigment template tags available for displaying Tweets in your templates.

Annual Favorite Counts

Get the number of favorites per year by all or one of the non-private Users-with-Accounts (only counting public Tweets):

{% load ditto_twitter %}

{% annual_favorite_counts as counts %}

{% for row in counts %}
    <p>
        {{ row.year }}: {{ row.count }}
    </p>
{% endfor %}

Both the year and count in each row are integers.

Or we can restrict this to favorites by a single User-with-an-Account:

{% annual_favorite_counts screen_name='philgyford' as counts %}

NOTE: The date used is the date the Tweets were posted on, not the date on which they were favorited.

Annual Tweet Counts

Get the number of Tweets per year by all or one of the non-private Users-with-Accounts:

{% load ditto_twitter %}

{% annual_tweet_counts as counts %}

{% for row in counts %}
    <p>
        {{ row.year }}: {{ row.count }}
    </p>
{% endfor %}

Both the year and count in each row are integers.

Or we can restrict this to Tweets posted by a single User-with-an-Account:

{% annual_tweet_counts screen_name='philgyford' as counts %}

Day Favorites

Use this to fetch Tweets posted on a particular day by non-private Users, which have been favorited/liked by any of the Users-with-Accounts. Again, my_date is a datetime.datetime.date type:

{% load ditto_twitter %}

{% day_favorites my_date as favorites %}

NOTE: The date is the date the Tweets were posted on, not the date on which they were favorited.

Again, we can restrict this to Tweets favorited by a single User-with-an-Account:

{% day_favorites my_date screen_name='philgyford' as favorites %}

Day Tweets

Gets Tweets posted on a particular day by any of the non-private Users-with-Accounts. In this example, my_date is a datetime.datetime.date type:

{% load ditto_twitter %}

{% day_tweets my_date as tweets %}

Or we can restrict this to Tweets posted by a single User-with-an-Account:

{% day_tweets my_date screen_name='philgyford' as tweets %}

Recent Favorites

This works like recent_tweets except it only fetches Tweets favorited/liked by our Users-with-Accounts, which were posted by public Twitter users:

{% load ditto_twitter %}

{% recent_favorites as favorites %}

Similarly, we can change the number of Tweets returned (10 by default), and only return Tweets favorited by a particular User:

{% recent_favorites screen_name='philgyford' limit=5 as favorites %}

Recent Tweets

To display the most recent Tweets posted by any of the non-private Users with an API-credentialled Account. By default the most recent 10 are fetched:

{% load ditto_twitter %}

{% recent_tweets as tweets %}

{% for tweet in tweets %}
    <p>
        <b>{{ tweet.user.name }} (@{{ tweet.user.screen_name }})</b><br>
        {{ tweet.text_html|safe }}
    </p>
{% endfor %}

(There’s a lot more to displaying Tweets in full; see the included templates for examples.)

The tag can also fetch a different number of Tweets and/or only get Tweets from a single User-with-an-Account. Here we only get the 5 most recent Tweets posted by the User with a screen_name of 'philgyford':

{% recent_tweets screen_name='philgyford' limit=5 as tweets %}

Management commands

Import Tweets

If you have more than 3,200 Tweets, you can only include older Tweets by downloading your archive and importing it. To do so, request your archive at https://twitter.com/settings/download_your_data . When you’ve downloaded it, do:

$ ./manage.py import_twitter_tweets --path=/path/to/twitter-2022-01-31-123456abcdef

using the correct path to the directory you’ve downloaded and unzipped (in this case, the unzipped directory is twitter-2022-01-031-123456abcdef). This will import all of the Tweets found in the archive and all of the images and animated GIFs’ MP4 files (other videos aren’t currently imported).

NOTE: If you have an archive of data you downloaded before sometime in early 2019 it will be a different format. You can tell because the folder will contain five directories, and three files: index.html, README.txt, and tweets.csv. If you want to import an archive in this format add the –archive-version=v1 argument:

$ ./manage.py import_twitter_tweets --archive-version=v1 --path=/path/to/123456_abcdef

using the correct path to the directory (in this case, the unzipped directory is 123456_abcdef). This will import Tweet data but no images etc, because these files weren’t included in the older archive format.

Update Tweets

If you’ve imported your Tweets (above), you won’t yet have complete data about each one. To fully-populate those Tweets you should run this (replacing philgyford with your Twitter screen name):

$ ./manage.py update_twitter_tweets --account=philgyford

This will fetch data for up to 6,000 Tweets. If you have more than that in your archive, run it every 15 minutes to avoid rate limiting, until you’ve fetched all of the Tweets. This command will fetch data for the least-recently fetched. It’s worth running every so often in the future, to fetch the latest data (such as Retweet and Like counts).

Fetch Tweets

Periodically you’ll want to fetch the latest Tweets. This will fetch only those Tweets posted since you last fetched any:

$ ./manage.py fetch_twitter_tweets --account=philgyford --recent=new

You might also, or instead, want to fetch more than that. Here we fetch the most recent 200:

$ ./manage.py fetch_twitter_tweets --account=philgyford --recent=200

This would update data such as the Retweet and Like counts for all of the 200 fetched Tweets, even if they’re older than your last fetch. It’s fine to fetch data about Tweets you’ve already fetched; their data will be updated.

If you have more than one Twitter Account in Ditto, the above commands can be run across all of them by omitting the --account option. eg:

$ ./manage.py fetch_twitter_tweets --recent=new

Fetch Favorites

To fetch recent Tweets that all your Accounts have favorited/liked, run one of these:

$ ./manage.py fetch_twitter_favorites --recent=new
$ ./manage.py fetch_twitter_favorites --recent=200

Or restrict to Tweets favorited/liked by a single Account:

$ ./manage.py fetch_twitter_favorites --account=philgyford --recent=new
$ ./manage.py fetch_twitter_favorites --account=philgyford --recent=200

Update Users

When a Tweet of any kind is fetched, its User data is also stored, and the User’s profile photo (avatar) is downloaded and stored in your project’s MEDIA_ROOT directory. You can optionally set the DITTO_TWITTER_DIR_BASE setting to change the location. The default is:

DITTO_TWITTER_DIR_BASE = 'twitter'

If your MEDIA_ROOT was set to /var/www/example.com/media/ then the above setting would save the profile image for the user with a Twitter ID 12345678 to something like this:

/var/www/example.com/media/twitter/avatars/56/78/12345678/my_avatar.jpg

You may periodically want to update the stored data about all the Twitter users stored in Ditto. (quantity of Tweets, descriptions, etc). Do it like this:

$ ./manage.py update_twitter_users --account=philgyford

This requires an account as the data is fetched from that Twitter user’s point of view, when it comes to privacy etc.

Fetch Files (Media)

Fetching Tweets (whether your own or your favorites/likes) fetches all the data about them, but does not fetch any media files uploaded with them. There’s a separate command for fetching images and the MP4 video files created from animated GIFs. (There’s no way to fetch the videos, or original GIF files.)

You must first download the Tweet data (above), and then you can fetch the files for all those Tweets:

$ ./manage.py fetch_twitter_files

This will fetch the files for all Tweets whose files haven’t already been fetched. So, the first time, it’s all the Tweets’ files, which can take a while.

If you want to force a re-fetching of all files, whether they’ve already been downloaded or not:

$ ./manage.py fetch_twitter_files --all

Each image/MP4 is associated with the relevant Tweet(s) and saved within your project’s MEDIA_ROOT directory, as defined in settings.py. There’s one optional setting to customise the directory in which the files are saved. Its default value is as shown here:

DITTO_TWITTER_DIR_BASE = 'twitter'

Files are organised into separate directories according to the final characters of their file names (so as not to have too many in one directory). eg, an image might be saved in:

/var/www/example.com/media/twitter/media/6T/ay/CRXEfBEWUAA6Tay.png

Every uploaded image, animated GIF and video should have a single image. Animated GIFs will also have an MP4 file.

Once you’ve downloaded the original files, you can use these to generate all the different sizes of image required for your site, instead of linking direct to the image files on Twitter. To do this, ensure imagekit is in your INSTALLED_APPS setting:

INSTALLED_APPS = (
    # ...
    'imagekit',
    # ...
)

And add this to your settings.py (its default value is False):

DITTO_TWITTER_USE_LOCAL_MEDIA = True

Any requests in your templates for the URLs of photo files of any size will now use resized versions of your downloaded original files, generated by Imagekit. The first time you load a page (especially if it lists many images) it will be slow, but the images are cached (in a CACHE directory in your media folder).

For example, before changing this setting, the URL of small image (Media.small_url) would be something like this:

https://pbs.twimg.com/media/CjuCDVLXIAALhYz.jpg:small

After choosing to use local photos, it would be something like this:

/media/CACHE/images/twitter/media/Lh/Yz/CjuCDVLXIAALhYz/5a726ea25d3bbd1b35b21b8b61b98c4c.jpg

If you change your mind you can switch back to using the images hosted on Twitter by removing the DITTO_TWITTER_USE_LOCAL_MEDIA setting or changing it to False.

Animated GIFs are converted into MP4 videos when first uploaded to Twitter. Ditto downloads and uses these in a similar way to images. ie, by default the video_url property of a Media object that’s an Animated GIF would be like:

https://pbs.twimg.com/tweet_video/CRXEfBEWUAA6Tay.mp4

If it’s been downloaded and DITTO_TWITTER_USE_LOCAL_MEDIA is True then calling video_url would return a URL like:

/media/twitter/media/6T/ay/CRXEfBEWUAA6Tay.mp4

However, there’s no way to download actual videos that were uploaded to Twitter, and so Ditto will always try to use videos hosted on Twitter, no matter what the value of DITTO_TWITTER_USE_LOCAL_MEDIA.

Fetch Accounts

Deprecated: This only needs to be used whenever a new Account is added to the system. It fetches the User data for each Account that has API credentials, and associates the two objects.

$ ./manage.py fetch_twitter_accounts

I don’t think this is needed now.