Twitter
You can fetch, store and display data about your own Tweets and Liked Tweets for one or more Twitter accounts. By default the included models and templates will link to image and video files at Twitter, but you can also download the original files to store locally. These can then be used to serve different sizes of images locally. See the Fetch Files management command below.
Set-up
Go to https://developer.twitter.com/portal. You’ll need to sign up for a Developer Account, and then create a new App for your Twitter account. Note down the three credentials it says that you should note down.
When viewing your App’s details, click the “Edit” button, then the “Keys and tokens” tab.
In the “Access Token and Secret” section, tap the “Generate” button. Make a note of the Access Token and Access Token Secret.
Django Ditto currently uses the v1.1 Twitter API rather than the latest v2. Unfortunately, accessing v1.1 requires that you apply for “Elevated” access before the API calls used will work. You can do that here: https://developer.twitter.com/en/portal/products/elevated You will then need to wait for approval.
Once you have that, go to the Django admin, and add a new Account
in the Twitter app. Copy the API Key, API Key Secret, Access Token and Access Token Secret from your Twitter App into the Django admin (ignore the other fields), and save the account. A new User
object will be created, reflecting the Twitter user your API credentials are associated with. If it’s worked your Account should have a title like @philgyford instead of a number like 1.
Once this is done you’ll need to import a downloaded archive of Tweets, and/or fetch Tweets from the Twitter API. See Management commands below.
Models
The models available in ditto.twitter.models
are:
Account
Representing a Twitter account that has API credentials and that we fetch Tweets and/or Likes for. It has a one-to-one relationship with a
User
model.Media
A photo, video or Animated GIF that was attached to one or more Tweets. Its
media_type
property differentiates between'photo'
,'video'
and'animated_gif'
. (Yes, the plural model name is a bit confusing, sorry.) The files for photos and animated GIFs can be fetched from Twitter and used locally, although this isn’t the default. See the Fetch Files management command below.Tweet
A Tweet. It may have been posted by one of the Users with an
Account
that we fetch Twets for. Or it might have been posted by a differentUser
and favorited/liked by one of the Users with anAccount
.User
A Twitter user. Every time a Tweet is fetched a new
User
object is created/updated for the person who posted it. One or more of these Users will have an associatedAccount
, as these are the user(s) we fetch data for.
So, you will end up with many User
objects in the system. But only one (or a few) will be associated with Account
objects – these are the Users for which you fetch Tweets or favorited/liked Tweets.
Managers
Media
For Media
objects there is only the standard manager available:
Media.objects.all()
Fetches all
Media
objects, attached to Tweets by anyone, whether public or private.
There is no concept in Twitter of a ‘private’ photo or video. The same photo
could be posted by different public and private Twitter users. Exercise caution
when displaying Media
objects to public webpages.
Tweet
Tweet
models have several managers. There are two main differentiators we want to restrict things by:
Public vs private: Whether a Tweet was posted by a public or private Twitter user.
Posted Tweet vs Favorited Tweet: Sometimes we’ll want to only fetch Tweets posted by one of the Users with an API-credentialed Account. Sometimes we’ll want to only fetch Tweets those Accounts have favorited/liked. Sometimes we’ll want to fetch ALL Tweets, whether posted or favorited.
On public-facing web pages you should only use the managers starting Tweet.public_
as these will filter out all Tweets posted by private Twitter users.
Tweet.objects.all()
The default manager fetches all Tweets in the system, posted by all Users, whether they’re public or private, and whether posted by an Account’s User or favorited by them.
Tweet.public_objects.all()
Only gets Tweets posted by any Users who are not private. This includes Tweets that have simply been favorited/liked by one of your Accounts.
Tweet.tweet_objects.all()
Only gets Tweets posted by a User with an Account, not favorited by them. And from both public and private Users.
Tweet.public_tweet_objects.all()
Only gets Tweets posted by a User with an Account, not favorited by them, but only for Users who aren’t private. For use on public pages.
Tweet.favorite_objects.all()
Only gets Tweets favorited by a User with an Account, whether those Tweets are posted by public or private Users.
Tweet.public_favorite_objects.all()
Only gets Tweets favorited by a User with an Account, and only Tweets posted by Users who aren’t private.
Of course, these can all be filtered as usual. So, if you wanted to get all the public Tweets posted by a particular User
:
from ditto.twitter.models import User, Tweet
user = User.objects.get(screen_name='philgyford')
tweets = Tweet.public_tweet_objects.filter(user=user)
And to get public Tweets that user has favorited:
favorites = Tweet.public_favorite_objects.filter(user=user)
Management commands
Import Tweets
If you have more than 3,200 Tweets, you can only include older Tweets by downloading your archive and importing it. To do so, request your archive at https://twitter.com/settings/download_your_data . When you’ve downloaded it, do:
$ ./manage.py import_twitter_tweets --path=/path/to/twitter-2022-01-31-123456abcdef
using the correct path to the directory you’ve downloaded and unzipped (in this case, the unzipped directory is twitter-2022-01-031-123456abcdef
). This will import all of the Tweets found in the archive and all of the images and animated GIFs’ MP4 files (other videos aren’t currently imported).
NOTE: If you have an archive of data you downloaded before sometime in early 2019 it will be a different format. You can tell because the folder will contain five directories, and three files: index.html
, README.txt
, and tweets.csv
. If you want to import an archive in this format add the –archive-version=v1 argument:
$ ./manage.py import_twitter_tweets --archive-version=v1 --path=/path/to/123456_abcdef
using the correct path to the directory (in this case, the unzipped directory is 123456_abcdef
). This will import Tweet data but no images etc, because these files weren’t included in the older archive format.
Update Tweets
If you’ve imported your Tweets (above), you won’t yet have complete data about each one. To fully-populate those Tweets you should run this (replacing philgyford
with your Twitter screen name):
$ ./manage.py update_twitter_tweets --account=philgyford
This will fetch data for up to 6,000 Tweets. If you have more than that in your archive, run it every 15 minutes to avoid rate limiting, until you’ve fetched all of the Tweets. This command will fetch data for the least-recently fetched. It’s worth running every so often in the future, to fetch the latest data (such as Retweet and Like counts).
Fetch Tweets
Periodically you’ll want to fetch the latest Tweets. This will fetch only those Tweets posted since you last fetched any:
$ ./manage.py fetch_twitter_tweets --account=philgyford --recent=new
You might also, or instead, want to fetch more than that. Here we fetch the most recent 200:
$ ./manage.py fetch_twitter_tweets --account=philgyford --recent=200
This would update data such as the Retweet and Like counts for all of the 200 fetched Tweets, even if they’re older than your last fetch. It’s fine to fetch data about Tweets you’ve already fetched; their data will be updated.
If you have more than one Twitter Account in Ditto, the above commands can be run across all of them by omitting the --account
option. eg:
$ ./manage.py fetch_twitter_tweets --recent=new
Fetch Favorites
To fetch recent Tweets that all your Accounts have favorited/liked, run one of these:
$ ./manage.py fetch_twitter_favorites --recent=new
$ ./manage.py fetch_twitter_favorites --recent=200
Or restrict to Tweets favorited/liked by a single Account:
$ ./manage.py fetch_twitter_favorites --account=philgyford --recent=new
$ ./manage.py fetch_twitter_favorites --account=philgyford --recent=200
Update Users
When a Tweet of any kind is fetched, its User data is also stored, and the User’s profile photo (avatar) is downloaded and stored in your project’s MEDIA_ROOT
directory. You can optionally set the DITTO_TWITTER_DIR_BASE
setting to change the location. The default is:
DITTO_TWITTER_DIR_BASE = 'twitter'
If your MEDIA_ROOT
was set to /var/www/example.com/media/
then the above setting would save the profile image for the user with a Twitter ID 12345678
to something like this:
/var/www/example.com/media/twitter/avatars/56/78/12345678/my_avatar.jpg
You may periodically want to update the stored data about all the Twitter users stored in Ditto. (quantity of Tweets, descriptions, etc). Do it like this:
$ ./manage.py update_twitter_users --account=philgyford
This requires an account
as the data is fetched from that Twitter user’s point of view, when it comes to privacy etc.
Fetch Files (Media)
Fetching Tweets (whether your own or your favorites/likes) fetches all the data about them, but does not fetch any media files uploaded with them. There’s a separate command for fetching images and the MP4 video files created from animated GIFs. (There’s no way to fetch the videos, or original GIF files.)
You must first download the Tweet data (above), and then you can fetch the files for all those Tweets:
$ ./manage.py fetch_twitter_files
This will fetch the files for all Tweets whose files haven’t already been fetched. So, the first time, it’s all the Tweets’ files, which can take a while.
If you want to force a re-fetching of all files, whether they’ve already been downloaded or not:
$ ./manage.py fetch_twitter_files --all
Each image/MP4 is associated with the relevant Tweet(s) and saved within your project’s MEDIA_ROOT
directory, as defined in settings.py
. There’s one optional setting to customise the directory in which the files are saved. Its default value is as shown here:
DITTO_TWITTER_DIR_BASE = 'twitter'
Files are organised into separate directories according to the final characters of their file names (so as not to have too many in one directory). eg, an image might be saved in:
/var/www/example.com/media/twitter/media/6T/ay/CRXEfBEWUAA6Tay.png
Every uploaded image, animated GIF and video should have a single image. Animated GIFs will also have an MP4 file.
Once you’ve downloaded the original files, you can use these to generate all the different sizes of image required for your site, instead of linking direct to the image files on Twitter. To do this, ensure imagekit
is in your INSTALLED_APPS
setting:
INSTALLED_APPS = (
# ...
'imagekit',
# ...
)
And add this to your settings.py (its default value is False
):
DITTO_TWITTER_USE_LOCAL_MEDIA = True
Any requests in your templates for the URLs of photo files of any size will now use resized versions of your downloaded original files, generated by Imagekit. The first time you load a page (especially if it lists many images) it will be slow, but the images are cached (in a CACHE
directory in your media folder).
For example, before changing this setting, the URL of small image (Media.small_url
) would be something like this:
https://pbs.twimg.com/media/CjuCDVLXIAALhYz.jpg:small
After choosing to use local photos, it would be something like this:
/media/CACHE/images/twitter/media/Lh/Yz/CjuCDVLXIAALhYz/5a726ea25d3bbd1b35b21b8b61b98c4c.jpg
If you change your mind you can switch back to using the images hosted on Twitter by removing the DITTO_TWITTER_USE_LOCAL_MEDIA
setting or changing it to False
.
Animated GIFs are converted into MP4 videos when first uploaded to Twitter. Ditto downloads and uses these in a similar way to images. ie, by default the video_url
property of a Media
object that’s an Animated GIF would be like:
https://pbs.twimg.com/tweet_video/CRXEfBEWUAA6Tay.mp4
If it’s been downloaded and DITTO_TWITTER_USE_LOCAL_MEDIA
is True
then
calling video_url
would return a URL like:
/media/twitter/media/6T/ay/CRXEfBEWUAA6Tay.mp4
However, there’s no way to download actual videos that were uploaded to Twitter, and so Ditto will always try to use videos hosted on Twitter, no matter what the value of DITTO_TWITTER_USE_LOCAL_MEDIA
.
Fetch Accounts
Deprecated: This only needs to be used whenever a new Account
is added to the system. It fetches the User
data for each Account
that has API credentials, and associates the two objects.
$ ./manage.py fetch_twitter_accounts
I don’t think this is needed now.