CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutSign UpSign In
veeralakrishna

Real-time collaboration for Jupyter Notebooks, Linux Terminals, LaTeX, VS Code, R IDE, and more,
all in one place. Commercial Alternative to JupyterHub.

GitHub Repository: veeralakrishna/DataCamp-Project-Solutions-Python
 Path: blob/master/Real-time Insights from Social Media Data/notebook.ipynb
Views: 1229
Kernel: Python 3

1. Local and global thought patterns

While we might not be Twitter fans, we have to admit that it has a huge influence on the world (who doesn't know about Trump's tweets). Twitter data is not only gold in terms of insights, but Twitter-storms are available for analysis in near real-time. This means we can learn about the big waves of thoughts and moods around the world as they arise.

As any place filled with riches, Twitter has security guards blocking us from laying our hands on the data right away ⛔️ Some authentication steps (really straightforward) are needed to call their APIs for data collection. Since our goal today is learning to extract insights from data, we have already gotten a green-pass from security ✅ Our data is ready for usage in the datasets folder — we can concentrate on the fun part! 🕵️‍♀️🌎


Twitter provides both global and local trends. Let's load and inspect data for topics that were hot worldwide (WW) and in the United States (US) at the moment of query — snapshot of JSON response from the call to Twitter's GET trends/place API.

Note: Here is the documentation for this call, and here a full overview on Twitter's APIs.

# Loading json module
import json

# Load WW_trends and US_trends data into the the given variables respectively
WW_trends = json.loads(open('datasets/WWTrends.json').read())
US_trends = json.loads(open('datasets/USTrends.json').read())

# Inspecting data by printing out WW_trends and US_trends variables
print(WW_trends)
print(US_trends)
[{'trends': [{'name': '#BeratKandili', 'url': 'http://twitter.com/search?q=%23BeratKandili', 'promoted_content': None, 'query': '%23BeratKandili', 'tweet_volume': 46373}, {'name': '#GoodFriday', 'url': 'http://twitter.com/search?q=%23GoodFriday', 'promoted_content': None, 'query': '%23GoodFriday', 'tweet_volume': 81891}, {'name': '#WeLoveTheEarth', 'url': 'http://twitter.com/search?q=%23WeLoveTheEarth', 'promoted_content': None, 'query': '%23WeLoveTheEarth', 'tweet_volume': 159698}, {'name': '#195TLdenTTVerilir', 'url': 'http://twitter.com/search?q=%23195TLdenTTVerilir', 'promoted_content': None, 'query': '%23195TLdenTTVerilir', 'tweet_volume': None}, {'name': '#AFLNorthDons', 'url': 'http://twitter.com/search?q=%23AFLNorthDons', 'promoted_content': None, 'query': '%23AFLNorthDons', 'tweet_volume': None}, {'name': 'Shiv Sena', 'url': 'http://twitter.com/search?q=%22Shiv+Sena%22', 'promoted_content': None, 'query': '%22Shiv+Sena%22', 'tweet_volume': None}, {'name': 'Lyra McKee', 'url': 'http://twitter.com/search?q=%22Lyra+McKee%22', 'promoted_content': None, 'query': '%22Lyra+McKee%22', 'tweet_volume': 17606}, {'name': 'Priyanka Chaturvedi', 'url': 'http://twitter.com/search?q=%22Priyanka+Chaturvedi%22', 'promoted_content': None, 'query': '%22Priyanka+Chaturvedi%22', 'tweet_volume': 22342}, {'name': 'Derry', 'url': 'http://twitter.com/search?q=Derry', 'promoted_content': None, 'query': 'Derry', 'tweet_volume': 28234}, {'name': '池袋の事故', 'url': 'http://twitter.com/search?q=%E6%B1%A0%E8%A2%8B%E3%81%AE%E4%BA%8B%E6%95%85', 'promoted_content': None, 'query': '%E6%B1%A0%E8%A2%8B%E3%81%AE%E4%BA%8B%E6%95%85', 'tweet_volume': 34381}, {'name': 'プリウス', 'url': 'http://twitter.com/search?q=%E3%83%97%E3%83%AA%E3%82%A6%E3%82%B9', 'promoted_content': None, 'query': '%E3%83%97%E3%83%AA%E3%82%A6%E3%82%B9', 'tweet_volume': 22944}, {'name': 'Hemant Karkare', 'url': 'http://twitter.com/search?q=%22Hemant+Karkare%22', 'promoted_content': None, 'query': '%22Hemant+Karkare%22', 'tweet_volume': 24067}, {'name': '高齢者', 'url': 'http://twitter.com/search?q=%E9%AB%98%E9%BD%A2%E8%80%85', 'promoted_content': None, 'query': '%E9%AB%98%E9%BD%A2%E8%80%85', 'tweet_volume': 28382}, {'name': '브이알', 'url': 'http://twitter.com/search?q=%EB%B8%8C%EC%9D%B4%EC%95%8C', 'promoted_content': None, 'query': '%EB%B8%8C%EC%9D%B4%EC%95%8C', 'tweet_volume': 15490}, {'name': '刀ステ', 'url': 'http://twitter.com/search?q=%E5%88%80%E3%82%B9%E3%83%86', 'promoted_content': None, 'query': '%E5%88%80%E3%82%B9%E3%83%86', 'tweet_volume': None}, {'name': '免許返納', 'url': 'http://twitter.com/search?q=%E5%85%8D%E8%A8%B1%E8%BF%94%E7%B4%8D', 'promoted_content': None, 'query': '%E5%85%8D%E8%A8%B1%E8%BF%94%E7%B4%8D', 'tweet_volume': None}, {'name': 'Berat Kandilimiz', 'url': 'http://twitter.com/search?q=%22Berat+Kandilimiz%22', 'promoted_content': None, 'query': '%22Berat+Kandilimiz%22', 'tweet_volume': 10901}, {'name': 'örgütdeğil arkadaşgrubu', 'url': 'http://twitter.com/search?q=%22%C3%B6rg%C3%BCtde%C4%9Fil+arkada%C5%9Fgrubu%22', 'promoted_content': None, 'query': '%22%C3%B6rg%C3%BCtde%C4%9Fil+arkada%C5%9Fgrubu%22', 'tweet_volume': None}, {'name': 'グレア', 'url': 'http://twitter.com/search?q=%E3%82%B0%E3%83%AC%E3%82%A2', 'promoted_content': None, 'query': '%E3%82%B0%E3%83%AC%E3%82%A2', 'tweet_volume': 23485}, {'name': '東京・池袋衝突事故', 'url': 'http://twitter.com/search?q=%E6%9D%B1%E4%BA%AC%E3%83%BB%E6%B1%A0%E8%A2%8B%E8%A1%9D%E7%AA%81%E4%BA%8B%E6%95%85', 'promoted_content': None, 'query': '%E6%9D%B1%E4%BA%AC%E3%83%BB%E6%B1%A0%E8%A2%8B%E8%A1%9D%E7%AA%81%E4%BA%8B%E6%95%85', 'tweet_volume': None}, {'name': '重体の女性と女児', 'url': 'http://twitter.com/search?q=%E9%87%8D%E4%BD%93%E3%81%AE%E5%A5%B3%E6%80%A7%E3%81%A8%E5%A5%B3%E5%85%90', 'promoted_content': None, 'query': '%E9%87%8D%E4%BD%93%E3%81%AE%E5%A5%B3%E6%80%A7%E3%81%A8%E5%A5%B3%E5%85%90', 'tweet_volume': None}, {'name': 'Lil Dicky', 'url': 'http://twitter.com/search?q=%22Lil+Dicky%22', 'promoted_content': None, 'query': '%22Lil+Dicky%22', 'tweet_volume': 42461}, {'name': '歩行者', 'url': 'http://twitter.com/search?q=%E6%AD%A9%E8%A1%8C%E8%80%85', 'promoted_content': None, 'query': '%E6%AD%A9%E8%A1%8C%E8%80%85', 'tweet_volume': 25405}, {'name': 'Derrick White', 'url': 'http://twitter.com/search?q=%22Derrick+White%22', 'promoted_content': None, 'query': '%22Derrick+White%22', 'tweet_volume': 27104}, {'name': '十二国記', 'url': 'http://twitter.com/search?q=%E5%8D%81%E4%BA%8C%E5%9B%BD%E8%A8%98', 'promoted_content': None, 'query': '%E5%8D%81%E4%BA%8C%E5%9B%BD%E8%A8%98', 'tweet_volume': 46803}, {'name': '#KpuJanganCurang', 'url': 'http://twitter.com/search?q=%23KpuJanganCurang', 'promoted_content': None, 'query': '%23KpuJanganCurang', 'tweet_volume': 75384}, {'name': '#HayırlıCumalar', 'url': 'http://twitter.com/search?q=%23Hay%C4%B1rl%C4%B1Cumalar', 'promoted_content': None, 'query': '%23Hay%C4%B1rl%C4%B1Cumalar', 'tweet_volume': 19848}, {'name': '#HayırlıKandiller', 'url': 'http://twitter.com/search?q=%23Hay%C4%B1rl%C4%B1Kandiller', 'promoted_content': None, 'query': '%23Hay%C4%B1rl%C4%B1Kandiller', 'tweet_volume': None}, {'name': '#HanumanJayanti', 'url': 'http://twitter.com/search?q=%23HanumanJayanti', 'promoted_content': None, 'query': '%23HanumanJayanti', 'tweet_volume': 83138}, {'name': '#IndonesianElectionHeroes', 'url': 'http://twitter.com/search?q=%23IndonesianElectionHeroes', 'promoted_content': None, 'query': '%23IndonesianElectionHeroes', 'tweet_volume': 19664}, {'name': '#يوم_الجمعه', 'url': 'http://twitter.com/search?q=%23%D9%8A%D9%88%D9%85_%D8%A7%D9%84%D8%AC%D9%85%D8%B9%D9%87', 'promoted_content': None, 'query': '%23%D9%8A%D9%88%D9%85_%D8%A7%D9%84%D8%AC%D9%85%D8%B9%D9%87', 'tweet_volume': 80799}, {'name': '#NRLBulldogsSouths', 'url': 'http://twitter.com/search?q=%23NRLBulldogsSouths', 'promoted_content': None, 'query': '%23NRLBulldogsSouths', 'tweet_volume': None}, {'name': '#NikahUmurBerapa', 'url': 'http://twitter.com/search?q=%23NikahUmurBerapa', 'promoted_content': None, 'query': '%23NikahUmurBerapa', 'tweet_volume': None}, {'name': '#DragRace', 'url': 'http://twitter.com/search?q=%23DragRace', 'promoted_content': None, 'query': '%23DragRace', 'tweet_volume': 37166}, {'name': '#ViernesSanto', 'url': 'http://twitter.com/search?q=%23ViernesSanto', 'promoted_content': None, 'query': '%23ViernesSanto', 'tweet_volume': None}, {'name': '#HardikPatel', 'url': 'http://twitter.com/search?q=%23HardikPatel', 'promoted_content': None, 'query': '%23HardikPatel', 'tweet_volume': None}, {'name': '#BLACKPINKxCorden', 'url': 'http://twitter.com/search?q=%23BLACKPINKxCorden', 'promoted_content': None, 'query': '%23BLACKPINKxCorden', 'tweet_volume': 253605}, {'name': '#Ontas', 'url': 'http://twitter.com/search?q=%23Ontas', 'promoted_content': None, 'query': '%23Ontas', 'tweet_volume': 27924}, {'name': '#ConCalmaRemix', 'url': 'http://twitter.com/search?q=%23ConCalmaRemix', 'promoted_content': None, 'query': '%23ConCalmaRemix', 'tweet_volume': 37846}, {'name': '#ProtestoEdiyorum', 'url': 'http://twitter.com/search?q=%23ProtestoEdiyorum', 'promoted_content': None, 'query': '%23ProtestoEdiyorum', 'tweet_volume': None}, {'name': '#DinahJane1', 'url': 'http://twitter.com/search?q=%23DinahJane1', 'promoted_content': None, 'query': '%23DinahJane1', 'tweet_volume': 23757}, {'name': '#ShivSena', 'url': 'http://twitter.com/search?q=%23ShivSena', 'promoted_content': None, 'query': '%23ShivSena', 'tweet_volume': None}, {'name': '#DuyguAsena', 'url': 'http://twitter.com/search?q=%23DuyguAsena', 'promoted_content': None, 'query': '%23DuyguAsena', 'tweet_volume': None}, {'name': '#TheJudasInMyLife', 'url': 'http://twitter.com/search?q=%23TheJudasInMyLife', 'promoted_content': None, 'query': '%23TheJudasInMyLife', 'tweet_volume': None}, {'name': '#Jersey', 'url': 'http://twitter.com/search?q=%23Jersey', 'promoted_content': None, 'query': '%23Jersey', 'tweet_volume': 20509}, {'name': '#اغلاق_BBM', 'url': 'http://twitter.com/search?q=%23%D8%A7%D8%BA%D9%84%D8%A7%D9%82_BBM', 'promoted_content': None, 'query': '%23%D8%A7%D8%BA%D9%84%D8%A7%D9%82_BBM', 'tweet_volume': 17055}, {'name': '#19aprile', 'url': 'http://twitter.com/search?q=%2319aprile', 'promoted_content': None, 'query': '%2319aprile', 'tweet_volume': None}, {'name': '#CHIvLIO', 'url': 'http://twitter.com/search?q=%23CHIvLIO', 'promoted_content': None, 'query': '%23CHIvLIO', 'tweet_volume': None}, {'name': '#Karfreitag', 'url': 'http://twitter.com/search?q=%23Karfreitag', 'promoted_content': None, 'query': '%23Karfreitag', 'tweet_volume': None}, {'name': '#JunquerasACN', 'url': 'http://twitter.com/search?q=%23JunquerasACN', 'promoted_content': None, 'query': '%23JunquerasACN', 'tweet_volume': None}], 'as_of': '2019-04-19T08:43:43Z', 'created_at': '2019-04-19T08:39:15Z', 'locations': [{'name': 'Worldwide', 'woeid': 1}]}] [{'trends': [{'name': '#WeLoveTheEarth', 'url': 'http://twitter.com/search?q=%23WeLoveTheEarth', 'promoted_content': None, 'query': '%23WeLoveTheEarth', 'tweet_volume': 159698}, {'name': '#DragRace', 'url': 'http://twitter.com/search?q=%23DragRace', 'promoted_content': None, 'query': '%23DragRace', 'tweet_volume': 37166}, {'name': 'Lil Dicky', 'url': 'http://twitter.com/search?q=%22Lil+Dicky%22', 'promoted_content': None, 'query': '%22Lil+Dicky%22', 'tweet_volume': 42461}, {'name': 'Derrick White', 'url': 'http://twitter.com/search?q=%22Derrick+White%22', 'promoted_content': None, 'query': '%22Derrick+White%22', 'tweet_volume': 27104}, {'name': '#CUZILOVEYOU', 'url': 'http://twitter.com/search?q=%23CUZILOVEYOU', 'promoted_content': None, 'query': '%23CUZILOVEYOU', 'tweet_volume': None}, {'name': 'Kevin Durant', 'url': 'http://twitter.com/search?q=%22Kevin+Durant%22', 'promoted_content': None, 'query': '%22Kevin+Durant%22', 'tweet_volume': 21870}, {'name': '#StarTrekDiscovery', 'url': 'http://twitter.com/search?q=%23StarTrekDiscovery', 'promoted_content': None, 'query': '%23StarTrekDiscovery', 'tweet_volume': None}, {'name': '#GSWvsLAC', 'url': 'http://twitter.com/search?q=%23GSWvsLAC', 'promoted_content': None, 'query': '%23GSWvsLAC', 'tweet_volume': None}, {'name': 'Oshie', 'url': 'http://twitter.com/search?q=Oshie', 'promoted_content': None, 'query': 'Oshie', 'tweet_volume': None}, {'name': 'Seth Abramson', 'url': 'http://twitter.com/search?q=%22Seth+Abramson%22', 'promoted_content': None, 'query': '%22Seth+Abramson%22', 'tweet_volume': None}, {'name': 'Lyra McKee', 'url': 'http://twitter.com/search?q=%22Lyra+McKee%22', 'promoted_content': None, 'query': '%22Lyra+McKee%22', 'tweet_volume': 17606}, {'name': 'Silky', 'url': 'http://twitter.com/search?q=Silky', 'promoted_content': None, 'query': 'Silky', 'tweet_volume': 12881}, {'name': 'Kazuo Koike', 'url': 'http://twitter.com/search?q=%22Kazuo+Koike%22', 'promoted_content': None, 'query': '%22Kazuo+Koike%22', 'tweet_volume': None}, {'name': 'Game 6', 'url': 'http://twitter.com/search?q=%22Game+6%22', 'promoted_content': None, 'query': '%22Game+6%22', 'tweet_volume': None}, {'name': 'Yvie', 'url': 'http://twitter.com/search?q=Yvie', 'promoted_content': None, 'query': 'Yvie', 'tweet_volume': 10680}, {'name': 'Gallant', 'url': 'http://twitter.com/search?q=Gallant', 'promoted_content': None, 'query': 'Gallant', 'tweet_volume': None}, {'name': 'Lone Wolf and Cub', 'url': 'http://twitter.com/search?q=%22Lone+Wolf+and+Cub%22', 'promoted_content': None, 'query': '%22Lone+Wolf+and+Cub%22', 'tweet_volume': None}, {'name': 'George Conway', 'url': 'http://twitter.com/search?q=%22George+Conway%22', 'promoted_content': None, 'query': '%22George+Conway%22', 'tweet_volume': 27458}, {'name': 'David Fletcher', 'url': 'http://twitter.com/search?q=%22David+Fletcher%22', 'promoted_content': None, 'query': '%22David+Fletcher%22', 'tweet_volume': None}, {'name': 'Derry', 'url': 'http://twitter.com/search?q=Derry', 'promoted_content': None, 'query': 'Derry', 'tweet_volume': 28234}, {'name': 'Mike Anderson', 'url': 'http://twitter.com/search?q=%22Mike+Anderson%22', 'promoted_content': None, 'query': '%22Mike+Anderson%22', 'tweet_volume': None}, {'name': 'Shy Glizzy', 'url': 'http://twitter.com/search?q=%22Shy+Glizzy%22', 'promoted_content': None, 'query': '%22Shy+Glizzy%22', 'tweet_volume': None}, {'name': 'Tomas Hertl', 'url': 'http://twitter.com/search?q=%22Tomas+Hertl%22', 'promoted_content': None, 'query': '%22Tomas+Hertl%22', 'tweet_volume': None}, {'name': 'Servais', 'url': 'http://twitter.com/search?q=Servais', 'promoted_content': None, 'query': 'Servais', 'tweet_volume': None}, {'name': 'WE LOVE THE EARTH', 'url': 'http://twitter.com/search?q=%22WE+LOVE+THE+EARTH%22', 'promoted_content': None, 'query': '%22WE+LOVE+THE+EARTH%22', 'tweet_volume': None}, {'name': '"Earth"', 'url': 'http://twitter.com/search?q=%22Earth%22', 'promoted_content': None, 'query': '%22Earth%22', 'tweet_volume': 338417}, {'name': '#DinahJane1', 'url': 'http://twitter.com/search?q=%23DinahJane1', 'promoted_content': None, 'query': '%23DinahJane1', 'tweet_volume': 23757}, {'name': '#WhatStopsYouFromGoingHome', 'url': 'http://twitter.com/search?q=%23WhatStopsYouFromGoingHome', 'promoted_content': None, 'query': '%23WhatStopsYouFromGoingHome', 'tweet_volume': None}, {'name': '#MakeAMovieSensual', 'url': 'http://twitter.com/search?q=%23MakeAMovieSensual', 'promoted_content': None, 'query': '%23MakeAMovieSensual', 'tweet_volume': None}, {'name': '#DontChangeOutNow', 'url': 'http://twitter.com/search?q=%23DontChangeOutNow', 'promoted_content': None, 'query': '%23DontChangeOutNow', 'tweet_volume': None}, {'name': '#BLACKPINKxCorden', 'url': 'http://twitter.com/search?q=%23BLACKPINKxCorden', 'promoted_content': None, 'query': '%23BLACKPINKxCorden', 'tweet_volume': 253605}, {'name': '#WorldofWarcraftMains', 'url': 'http://twitter.com/search?q=%23WorldofWarcraftMains', 'promoted_content': None, 'query': '%23WorldofWarcraftMains', 'tweet_volume': None}, {'name': '#MyDrunkUncleSays', 'url': 'http://twitter.com/search?q=%23MyDrunkUncleSays', 'promoted_content': None, 'query': '%23MyDrunkUncleSays', 'tweet_volume': None}, {'name': '#WGAMIX', 'url': 'http://twitter.com/search?q=%23WGAMIX', 'promoted_content': None, 'query': '%23WGAMIX', 'tweet_volume': None}, {'name': '#Earth', 'url': 'http://twitter.com/search?q=%23Earth', 'promoted_content': None, 'query': '%23Earth', 'tweet_volume': 13655}, {'name': '#TheLegendOfVoxMachina', 'url': 'http://twitter.com/search?q=%23TheLegendOfVoxMachina', 'promoted_content': None, 'query': '%23TheLegendOfVoxMachina', 'tweet_volume': None}, {'name': '#AFLNorthDons', 'url': 'http://twitter.com/search?q=%23AFLNorthDons', 'promoted_content': None, 'query': '%23AFLNorthDons', 'tweet_volume': None}, {'name': '#FridayFeeling', 'url': 'http://twitter.com/search?q=%23FridayFeeling', 'promoted_content': None, 'query': '%23FridayFeeling', 'tweet_volume': 19510}, {'name': '#MyInnerDemonSaid', 'url': 'http://twitter.com/search?q=%23MyInnerDemonSaid', 'promoted_content': None, 'query': '%23MyInnerDemonSaid', 'tweet_volume': None}, {'name': '#rupaulsdragrace', 'url': 'http://twitter.com/search?q=%23rupaulsdragrace', 'promoted_content': None, 'query': '%23rupaulsdragrace', 'tweet_volume': None}, {'name': '#ConCalmaRemix', 'url': 'http://twitter.com/search?q=%23ConCalmaRemix', 'promoted_content': None, 'query': '%23ConCalmaRemix', 'tweet_volume': 37846}, {'name': '#TimeToImpeach', 'url': 'http://twitter.com/search?q=%23TimeToImpeach', 'promoted_content': None, 'query': '%23TimeToImpeach', 'tweet_volume': 21732}, {'name': '#NRLBulldogsSouths', 'url': 'http://twitter.com/search?q=%23NRLBulldogsSouths', 'promoted_content': None, 'query': '%23NRLBulldogsSouths', 'tweet_volume': None}, {'name': '#CriticalRoleSpoilers', 'url': 'http://twitter.com/search?q=%23CriticalRoleSpoilers', 'promoted_content': None, 'query': '%23CriticalRoleSpoilers', 'tweet_volume': None}, {'name': '#GossipShouldBe', 'url': 'http://twitter.com/search?q=%23GossipShouldBe', 'promoted_content': None, 'query': '%23GossipShouldBe', 'tweet_volume': None}, {'name': '#LilDicky', 'url': 'http://twitter.com/search?q=%23LilDicky', 'promoted_content': None, 'query': '%23LilDicky', 'tweet_volume': None}, {'name': '#RPDR', 'url': 'http://twitter.com/search?q=%23RPDR', 'promoted_content': None, 'query': '%23RPDR', 'tweet_volume': None}, {'name': '#WeirdDateStories', 'url': 'http://twitter.com/search?q=%23WeirdDateStories', 'promoted_content': None, 'query': '%23WeirdDateStories', 'tweet_volume': None}, {'name': '#HustleAndSoul', 'url': 'http://twitter.com/search?q=%23HustleAndSoul', 'promoted_content': None, 'query': '%23HustleAndSoul', 'tweet_volume': None}, {'name': '#fridaymotivation', 'url': 'http://twitter.com/search?q=%23fridaymotivation', 'promoted_content': None, 'query': '%23fridaymotivation', 'tweet_volume': None}], 'as_of': '2019-04-19T08:43:43Z', 'created_at': '2019-04-19T08:39:15Z', 'locations': [{'name': 'United States', 'woeid': 23424977}]}]

2. Prettifying the output

Our data was hard to read! Luckily, we can resort to the json.dumps() method to have it formatted as a pretty JSON string.

# Pretty-printing the results. First WW and then US trends.

print("WW trends:")
json.dumps(WW_trends, indent=1)

print("\n", "US trends:")
json.dumps(US_trends, indent=1)
WW trends: US trends:
'[\n {\n "trends": [\n {\n "name": "#WeLoveTheEarth",\n "url": "http://twitter.com/search?q=%23WeLoveTheEarth",\n "promoted_content": null,\n "query": "%23WeLoveTheEarth",\n "tweet_volume": 159698\n },\n {\n "name": "#DragRace",\n "url": "http://twitter.com/search?q=%23DragRace",\n "promoted_content": null,\n "query": "%23DragRace",\n "tweet_volume": 37166\n },\n {\n "name": "Lil Dicky",\n "url": "http://twitter.com/search?q=%22Lil+Dicky%22",\n "promoted_content": null,\n "query": "%22Lil+Dicky%22",\n "tweet_volume": 42461\n },\n {\n "name": "Derrick White",\n "url": "http://twitter.com/search?q=%22Derrick+White%22",\n "promoted_content": null,\n "query": "%22Derrick+White%22",\n "tweet_volume": 27104\n },\n {\n "name": "#CUZILOVEYOU",\n "url": "http://twitter.com/search?q=%23CUZILOVEYOU",\n "promoted_content": null,\n "query": "%23CUZILOVEYOU",\n "tweet_volume": null\n },\n {\n "name": "Kevin Durant",\n "url": "http://twitter.com/search?q=%22Kevin+Durant%22",\n "promoted_content": null,\n "query": "%22Kevin+Durant%22",\n "tweet_volume": 21870\n },\n {\n "name": "#StarTrekDiscovery",\n "url": "http://twitter.com/search?q=%23StarTrekDiscovery",\n "promoted_content": null,\n "query": "%23StarTrekDiscovery",\n "tweet_volume": null\n },\n {\n "name": "#GSWvsLAC",\n "url": "http://twitter.com/search?q=%23GSWvsLAC",\n "promoted_content": null,\n "query": "%23GSWvsLAC",\n "tweet_volume": null\n },\n {\n "name": "Oshie",\n "url": "http://twitter.com/search?q=Oshie",\n "promoted_content": null,\n "query": "Oshie",\n "tweet_volume": null\n },\n {\n "name": "Seth Abramson",\n "url": "http://twitter.com/search?q=%22Seth+Abramson%22",\n "promoted_content": null,\n "query": "%22Seth+Abramson%22",\n "tweet_volume": null\n },\n {\n "name": "Lyra McKee",\n "url": "http://twitter.com/search?q=%22Lyra+McKee%22",\n "promoted_content": null,\n "query": "%22Lyra+McKee%22",\n "tweet_volume": 17606\n },\n {\n "name": "Silky",\n "url": "http://twitter.com/search?q=Silky",\n "promoted_content": null,\n "query": "Silky",\n "tweet_volume": 12881\n },\n {\n "name": "Kazuo Koike",\n "url": "http://twitter.com/search?q=%22Kazuo+Koike%22",\n "promoted_content": null,\n "query": "%22Kazuo+Koike%22",\n "tweet_volume": null\n },\n {\n "name": "Game 6",\n "url": "http://twitter.com/search?q=%22Game+6%22",\n "promoted_content": null,\n "query": "%22Game+6%22",\n "tweet_volume": null\n },\n {\n "name": "Yvie",\n "url": "http://twitter.com/search?q=Yvie",\n "promoted_content": null,\n "query": "Yvie",\n "tweet_volume": 10680\n },\n {\n "name": "Gallant",\n "url": "http://twitter.com/search?q=Gallant",\n "promoted_content": null,\n "query": "Gallant",\n "tweet_volume": null\n },\n {\n "name": "Lone Wolf and Cub",\n "url": "http://twitter.com/search?q=%22Lone+Wolf+and+Cub%22",\n "promoted_content": null,\n "query": "%22Lone+Wolf+and+Cub%22",\n "tweet_volume": null\n },\n {\n "name": "George Conway",\n "url": "http://twitter.com/search?q=%22George+Conway%22",\n "promoted_content": null,\n "query": "%22George+Conway%22",\n "tweet_volume": 27458\n },\n {\n "name": "David Fletcher",\n "url": "http://twitter.com/search?q=%22David+Fletcher%22",\n "promoted_content": null,\n "query": "%22David+Fletcher%22",\n "tweet_volume": null\n },\n {\n "name": "Derry",\n "url": "http://twitter.com/search?q=Derry",\n "promoted_content": null,\n "query": "Derry",\n "tweet_volume": 28234\n },\n {\n "name": "Mike Anderson",\n "url": "http://twitter.com/search?q=%22Mike+Anderson%22",\n "promoted_content": null,\n "query": "%22Mike+Anderson%22",\n "tweet_volume": null\n },\n {\n "name": "Shy Glizzy",\n "url": "http://twitter.com/search?q=%22Shy+Glizzy%22",\n "promoted_content": null,\n "query": "%22Shy+Glizzy%22",\n "tweet_volume": null\n },\n {\n "name": "Tomas Hertl",\n "url": "http://twitter.com/search?q=%22Tomas+Hertl%22",\n "promoted_content": null,\n "query": "%22Tomas+Hertl%22",\n "tweet_volume": null\n },\n {\n "name": "Servais",\n "url": "http://twitter.com/search?q=Servais",\n "promoted_content": null,\n "query": "Servais",\n "tweet_volume": null\n },\n {\n "name": "WE LOVE THE EARTH",\n "url": "http://twitter.com/search?q=%22WE+LOVE+THE+EARTH%22",\n "promoted_content": null,\n "query": "%22WE+LOVE+THE+EARTH%22",\n "tweet_volume": null\n },\n {\n "name": "\\"Earth\\"",\n "url": "http://twitter.com/search?q=%22Earth%22",\n "promoted_content": null,\n "query": "%22Earth%22",\n "tweet_volume": 338417\n },\n {\n "name": "#DinahJane1",\n "url": "http://twitter.com/search?q=%23DinahJane1",\n "promoted_content": null,\n "query": "%23DinahJane1",\n "tweet_volume": 23757\n },\n {\n "name": "#WhatStopsYouFromGoingHome",\n "url": "http://twitter.com/search?q=%23WhatStopsYouFromGoingHome",\n "promoted_content": null,\n "query": "%23WhatStopsYouFromGoingHome",\n "tweet_volume": null\n },\n {\n "name": "#MakeAMovieSensual",\n "url": "http://twitter.com/search?q=%23MakeAMovieSensual",\n "promoted_content": null,\n "query": "%23MakeAMovieSensual",\n "tweet_volume": null\n },\n {\n "name": "#DontChangeOutNow",\n "url": "http://twitter.com/search?q=%23DontChangeOutNow",\n "promoted_content": null,\n "query": "%23DontChangeOutNow",\n "tweet_volume": null\n },\n {\n "name": "#BLACKPINKxCorden",\n "url": "http://twitter.com/search?q=%23BLACKPINKxCorden",\n "promoted_content": null,\n "query": "%23BLACKPINKxCorden",\n "tweet_volume": 253605\n },\n {\n "name": "#WorldofWarcraftMains",\n "url": "http://twitter.com/search?q=%23WorldofWarcraftMains",\n "promoted_content": null,\n "query": "%23WorldofWarcraftMains",\n "tweet_volume": null\n },\n {\n "name": "#MyDrunkUncleSays",\n "url": "http://twitter.com/search?q=%23MyDrunkUncleSays",\n "promoted_content": null,\n "query": "%23MyDrunkUncleSays",\n "tweet_volume": null\n },\n {\n "name": "#WGAMIX",\n "url": "http://twitter.com/search?q=%23WGAMIX",\n "promoted_content": null,\n "query": "%23WGAMIX",\n "tweet_volume": null\n },\n {\n "name": "#Earth",\n "url": "http://twitter.com/search?q=%23Earth",\n "promoted_content": null,\n "query": "%23Earth",\n "tweet_volume": 13655\n },\n {\n "name": "#TheLegendOfVoxMachina",\n "url": "http://twitter.com/search?q=%23TheLegendOfVoxMachina",\n "promoted_content": null,\n "query": "%23TheLegendOfVoxMachina",\n "tweet_volume": null\n },\n {\n "name": "#AFLNorthDons",\n "url": "http://twitter.com/search?q=%23AFLNorthDons",\n "promoted_content": null,\n "query": "%23AFLNorthDons",\n "tweet_volume": null\n },\n {\n "name": "#FridayFeeling",\n "url": "http://twitter.com/search?q=%23FridayFeeling",\n "promoted_content": null,\n "query": "%23FridayFeeling",\n "tweet_volume": 19510\n },\n {\n "name": "#MyInnerDemonSaid",\n "url": "http://twitter.com/search?q=%23MyInnerDemonSaid",\n "promoted_content": null,\n "query": "%23MyInnerDemonSaid",\n "tweet_volume": null\n },\n {\n "name": "#rupaulsdragrace",\n "url": "http://twitter.com/search?q=%23rupaulsdragrace",\n "promoted_content": null,\n "query": "%23rupaulsdragrace",\n "tweet_volume": null\n },\n {\n "name": "#ConCalmaRemix",\n "url": "http://twitter.com/search?q=%23ConCalmaRemix",\n "promoted_content": null,\n "query": "%23ConCalmaRemix",\n "tweet_volume": 37846\n },\n {\n "name": "#TimeToImpeach",\n "url": "http://twitter.com/search?q=%23TimeToImpeach",\n "promoted_content": null,\n "query": "%23TimeToImpeach",\n "tweet_volume": 21732\n },\n {\n "name": "#NRLBulldogsSouths",\n "url": "http://twitter.com/search?q=%23NRLBulldogsSouths",\n "promoted_content": null,\n "query": "%23NRLBulldogsSouths",\n "tweet_volume": null\n },\n {\n "name": "#CriticalRoleSpoilers",\n "url": "http://twitter.com/search?q=%23CriticalRoleSpoilers",\n "promoted_content": null,\n "query": "%23CriticalRoleSpoilers",\n "tweet_volume": null\n },\n {\n "name": "#GossipShouldBe",\n "url": "http://twitter.com/search?q=%23GossipShouldBe",\n "promoted_content": null,\n "query": "%23GossipShouldBe",\n "tweet_volume": null\n },\n {\n "name": "#LilDicky",\n "url": "http://twitter.com/search?q=%23LilDicky",\n "promoted_content": null,\n "query": "%23LilDicky",\n "tweet_volume": null\n },\n {\n "name": "#RPDR",\n "url": "http://twitter.com/search?q=%23RPDR",\n "promoted_content": null,\n "query": "%23RPDR",\n "tweet_volume": null\n },\n {\n "name": "#WeirdDateStories",\n "url": "http://twitter.com/search?q=%23WeirdDateStories",\n "promoted_content": null,\n "query": "%23WeirdDateStories",\n "tweet_volume": null\n },\n {\n "name": "#HustleAndSoul",\n "url": "http://twitter.com/search?q=%23HustleAndSoul",\n "promoted_content": null,\n "query": "%23HustleAndSoul",\n "tweet_volume": null\n },\n {\n "name": "#fridaymotivation",\n "url": "http://twitter.com/search?q=%23fridaymotivation",\n "promoted_content": null,\n "query": "%23fridaymotivation",\n "tweet_volume": null\n }\n ],\n "as_of": "2019-04-19T08:43:43Z",\n "created_at": "2019-04-19T08:39:15Z",\n "locations": [\n {\n "name": "United States",\n "woeid": 23424977\n }\n ]\n }\n]'

🕵️‍♀️ From the pretty-printed results (output of the previous task), we can observe that:

  • We have an array of trend objects having: the name of the trending topic, the query parameter that can be used to search for the topic on Twitter-Search, the search URL and the volume of tweets for the last 24 hours, if available. (The trends get updated every 5 mins.)

  • At query time #BeratKandili, #GoodFriday and #WeLoveTheEarth were trending WW.

  • "tweet_volume" tell us that #WeLoveTheEarth was the most popular among the three.

  • Results are not sorted by "tweet_volume".

  • There are some trends which are unique to the US.

It’s easy to skim through the two sets of trends and spot common trends, but let's not do "manual" work. We can use Python’s set data structure to find common trends — we can iterate through the two trends objects, cast the lists of names to sets, and call the intersection method to get the common names between the two sets.

# Extracting all the WW trend names from WW_trends
world_trends = set([trend['name'] for trend in WW_trends[0]['trends']])

# Extracting all the US trend names from US_trends
us_trends = set([trend['name'] for trend in US_trends[0]['trends']]) 

# Getting the intersection of the two sets of trends
common_trends = world_trends.intersection(us_trends)

# Inspecting the data
print(world_trends, "\n")
print(us_trends, "\n")
print (len(common_trends), "common trends:", common_trends)
{'#IndonesianElectionHeroes', '#WeLoveTheEarth', 'グレア', '#JunquerasACN', '#195TLdenTTVerilir', '#BeratKandili', '#GoodFriday', '#ViernesSanto', '브이알', '#HayırlıCumalar', '刀ステ', 'Priyanka Chaturvedi', '#AFLNorthDons', '#19aprile', '#BLACKPINKxCorden', '#HardikPatel', '#ShivSena', 'Lyra McKee', '#NikahUmurBerapa', '高齢者', '#HayırlıKandiller', '#Ontas', '免許返納', '#Karfreitag', 'Derry', '東京・池袋衝突事故', '#CHIvLIO', 'Derrick White', '十二国記', '#KpuJanganCurang', '#ConCalmaRemix', '#NRLBulldogsSouths', 'Berat Kandilimiz', '#Jersey', '#اغلاق_BBM', 'Hemant Karkare', 'Shiv Sena', '#DuyguAsena', '#TheJudasInMyLife', '#يوم_الجمعه', 'プリウス', '重体の女性と女児', '#DragRace', '歩行者', '#HanumanJayanti', 'Lil Dicky', '池袋の事故', '#ProtestoEdiyorum', '#DinahJane1', 'örgütdeğil arkadaşgrubu'} {'WE LOVE THE EARTH', '"Earth"', '#WeLoveTheEarth', '#GossipShouldBe', '#LilDicky', 'Lone Wolf and Cub', 'Mike Anderson', '#MakeAMovieSensual', '#HustleAndSoul', 'Game 6', '#AFLNorthDons', '#StarTrekDiscovery', '#GSWvsLAC', '#BLACKPINKxCorden', 'Shy Glizzy', 'Tomas Hertl', 'Lyra McKee', '#TimeToImpeach', '#Earth', '#WGAMIX', 'Seth Abramson', '#TheLegendOfVoxMachina', '#FridayFeeling', 'George Conway', '#RPDR', '#fridaymotivation', 'Derry', 'Derrick White', 'Servais', '#DontChangeOutNow', '#ConCalmaRemix', '#NRLBulldogsSouths', 'Kevin Durant', '#CriticalRoleSpoilers', 'Oshie', '#MyDrunkUncleSays', '#WeirdDateStories', '#MyInnerDemonSaid', 'David Fletcher', 'Gallant', '#WorldofWarcraftMains', '#DragRace', '#WhatStopsYouFromGoingHome', 'Lil Dicky', '#CUZILOVEYOU', '#DinahJane1', 'Kazuo Koike', '#rupaulsdragrace', 'Silky', 'Yvie'} 11 common trends: {'Derry', '#WeLoveTheEarth', '#BLACKPINKxCorden', 'Derrick White', '#ConCalmaRemix', '#NRLBulldogsSouths', '#DragRace', 'Lyra McKee', 'Lil Dicky', '#DinahJane1', '#AFLNorthDons'}

4. Exploring the hot trend

🕵️‍♀️ From the intersection (last output) we can see that, out of the two sets of trends (each of size 50), we have 11 overlapping topics. In particular, there is one common trend that sounds very interesting: #WeLoveTheEarth — so good to see that Twitteratis are unanimously talking about loving Mother Earth! 💚

Note: We could have had no overlap or a much higher overlap; when we did the query for getting the trends, people in the US could have been on fire obout topics only relevant to them.

Image Source:Official Music Video Cover: https://welovetheearth.org/video/

We have found a hot-trend, #WeLoveTheEarth. Now let's see what story it is screaming to tell us!
If we query Twitter's search API with this hashtag as query parameter, we get back actual tweets related to it. We have the response from the search API stored in the datasets folder as 'WeLoveTheEarth.json'. So let's load this dataset and do a deep dive in this trend.

# Loading the data
tweets = json.loads(open('datasets/WeLoveTheEarth.json').read())

# Inspecting some tweets
tweets[0:2]
[{'created_at': 'Fri Apr 19 08:46:48 +0000 2019', 'id': 1119160405270523904, 'id_str': '1119160405270523904', 'text': 'RT @lildickytweets: 🌎 out now #WeLoveTheEarth https://t.co/L22XsoT5P1', 'truncated': False, 'entities': {'hashtags': [{'text': 'WeLoveTheEarth', 'indices': [30, 45]}], 'symbols': [], 'user_mentions': [{'screen_name': 'lildickytweets', 'name': 'LD', 'id': 1209516660, 'id_str': '1209516660', 'indices': [3, 18]}], 'urls': [{'url': 'https://t.co/L22XsoT5P1', 'expanded_url': 'https://youtu.be/pvuN_WvF1to', 'display_url': 'youtu.be/pvuN_WvF1to', 'indices': [46, 69]}]}, 'metadata': {'iso_language_code': 'en', 'result_type': 'recent'}, 'source': '<a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 212375312, 'id_str': '212375312', 'name': 'fake smile', 'screen_name': 'Pati95Poland', 'location': 'SWAGLAND ', 'description': "''you just got knocked the fuck out''", 'url': 'https://t.co/nxzlSyYZSK', 'entities': {'url': {'urls': [{'url': 'https://t.co/nxzlSyYZSK', 'expanded_url': 'http://loveeeujdb.tumblr.com/', 'display_url': 'loveeeujdb.tumblr.com', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 2306, 'friends_count': 697, 'listed_count': 28, 'created_at': 'Fri Nov 05 22:25:29 +0000 2010', 'favourites_count': 5552, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': False, 'statuses_count': 185750, 'lang': 'pl', 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'FFFFFF', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme18/bg.gif', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme18/bg.gif', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1093929135183937537/hQuxtwKq_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1093929135183937537/hQuxtwKq_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/212375312/1522705183', 'profile_link_color': 'ABB8C2', 'profile_sidebar_border_color': 'FFFFFF', 'profile_sidebar_fill_color': 'F6F6F6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'regular'}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'retweeted_status': {'created_at': 'Fri Apr 19 04:22:29 +0000 2019', 'id': 1119093888524754946, 'id_str': '1119093888524754946', 'text': '🌎 out now #WeLoveTheEarth https://t.co/L22XsoT5P1', 'truncated': False, 'entities': {'hashtags': [{'text': 'WeLoveTheEarth', 'indices': [10, 25]}], 'symbols': [], 'user_mentions': [], 'urls': [{'url': 'https://t.co/L22XsoT5P1', 'expanded_url': 'https://youtu.be/pvuN_WvF1to', 'display_url': 'youtu.be/pvuN_WvF1to', 'indices': [26, 49]}]}, 'metadata': {'iso_language_code': 'en', 'result_type': 'recent'}, 'source': '<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 1209516660, 'id_str': '1209516660', 'name': 'LD', 'screen_name': 'lildickytweets', 'location': 'Earth', 'description': 'Rapper/Actor/Comedian/Model #WeLoveTheEarth', 'url': 'https://t.co/aFrPkkJKqs', 'entities': {'url': {'urls': [{'url': 'https://t.co/aFrPkkJKqs', 'expanded_url': 'https://LilDicky.lnk.to/Earth', 'display_url': 'LilDicky.lnk.to/Earth', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 503111, 'friends_count': 945, 'listed_count': 657, 'created_at': 'Fri Feb 22 19:06:15 +0000 2013', 'favourites_count': 7696, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': True, 'statuses_count': 14430, 'lang': 'en', 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '000000', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme14/bg.gif', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme14/bg.gif', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1119087366679846912/XSa4fpQA_normal.png', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1119087366679846912/XSa4fpQA_normal.png', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/1209516660/1555646206', 'profile_link_color': '858585', 'profile_sidebar_border_color': 'FFFFFF', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': False, 'default_profile': False, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none'}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 7482, 'favorite_count': 13317, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'en'}, 'is_quote_status': False, 'retweet_count': 7482, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'en'}, {'created_at': 'Fri Apr 19 08:46:48 +0000 2019', 'id': 1119160404876206080, 'id_str': '1119160404876206080', 'text': '💚🌎💚 #WeLoveTheEarth 👇🏼', 'truncated': False, 'entities': {'hashtags': [{'text': 'WeLoveTheEarth', 'indices': [5, 20]}], 'symbols': [], 'user_mentions': [], 'urls': []}, 'metadata': {'iso_language_code': 'und', 'result_type': 'recent'}, 'source': '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 72150460, 'id_str': '72150460', 'name': 'Alfonsina del Mar (Dianishka Prietishka)', 'screen_name': 'Diana____X', 'location': 'México', 'description': '☸︎ ⚓︎ 👩🏻\u200d✈️ ★★★★ ♒︎', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 223, 'friends_count': 513, 'listed_count': 0, 'created_at': 'Sun Sep 06 23:26:24 +0000 2009', 'favourites_count': 750, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': False, 'statuses_count': 1668, 'lang': 'es', 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '642D8B', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme10/bg.gif', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme10/bg.gif', 'profile_background_tile': True, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1072296818531278848/tgn0e2h4_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1072296818531278848/tgn0e2h4_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/72150460/1545198367', 'profile_link_color': '8400FF', 'profile_sidebar_border_color': '65B0DA', 'profile_sidebar_fill_color': '7AC3EE', 'profile_text_color': '3D1957', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none'}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 0, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'lang': 'und'}]

5. Digging deeper

🕵️‍♀️ Printing the first two tweet items makes us realize that there’s a lot more to a tweet than what we normally think of as a tweet — there is a lot more than just a short text!

But hey, let's not get overwhemled by all the information in a tweet object! Let's focus on a few interesting fields and see if we can find any hidden insights there. 

# Extracting the text of all the tweets from the tweet object
texts = [tweet['text'] for tweet in tweets]

# Extracting screen names of users tweeting about #WeLoveTheEarth
names = [hashtag['screen_name'] 
             for tweet in tweets
                 for hashtag in tweet['entities']['user_mentions']]

# Extracting all the hashtags being used when talking about this topic
hashtags = [hashtag['text'] 
             for tweet in tweets
                 for hashtag in tweet['entities']['hashtags']]

# Inspecting the first 10 results
print (json.dumps(texts[0:10], indent=1),"\n")
print (json.dumps(names[0:10], indent=1),"\n")
print (json.dumps(hashtags[0:10], indent=1),"\n")
[ "RT @lildickytweets: \ud83c\udf0e out now #WeLoveTheEarth https://t.co/L22XsoT5P1", "\ud83d\udc9a\ud83c\udf0e\ud83d\udc9a #WeLoveTheEarth \ud83d\udc47\ud83c\udffc", "RT @cabeyoomoon: Ta piosenka to bop, wpada w ucho i dochody z niej id\u0105 na dobry cel, warto s\u0142ucha\u0107 w k\u00f3\u0142ko i w k\u00f3\u0142ko gdziekolwiek si\u0119 ty\u2026", "#WeLoveTheEarth \nCzemu ja si\u0119 pop\u0142aka\u0142am", "RT @Spotify: This is epic. @lildickytweets got @justinbieber, @arianagrande, @halsey, @sanbenito, @edsheeran, @SnoopDogg, @ShawnMendes, @Kr\u2026", "RT @biebercentineo: Justin : are we gonna die? \nLil dicky: you know bieber we might die \n\nBTCH IM CRYING #EARTH #WeLoveTheEarth #WELOVEEART\u2026", "RT @dreamsiinflate: #WeLoveTheEarth \u201ci am a fat fucking pig\u201d okay brendon urie https://t.co/FdJmq31xZc", "Literally no one:\n\nMe in the past 4 hours:\n\nI'm a koala and I sleep all the time, so what, it's cute \ud83c\udfb6\n\n#WeLoveTheEarth #EdSheeranTheKoala", "RT @Yuuupthatsme: Mia\u0142e\u015b by\u0107 \u017cyraf\u0105 #WeLoveTheEarth https://t.co/0kNCpU8o6q", "RT @jaguareffects: eu prestando aten\u00e7\u00e3o no \u00e1udio pra identificar cada artista\n\n#WeLoveTheEarth https://t.co/0cDtiV2t1E" ] [ "lildickytweets", "cabeyoomoon", "Spotify", "lildickytweets", "justinbieber", "ArianaGrande", "halsey", "sanbenito", "edsheeran", "SnoopDogg" ] [ "WeLoveTheEarth", "WeLoveTheEarth", "WeLoveTheEarth", "EARTH", "WeLoveTheEarth", "WeLoveTheEarth", "WeLoveTheEarth", "EdSheeranTheKoala", "WeLoveTheEarth", "WeLoveTheEarth" ]

6. Frequency analysis

🕵️‍♀️ Just from the first few results of the last extraction, we can deduce that:

  • We are talking about a song about loving the Earth.
  • A lot of big artists are the forces behind this Twitter wave, especially Lil Dicky.
  • Ed Sheeran was some cute koala in the song — "EdSheeranTheKoala" hashtag! 🐨

Observing the first 10 items of the interesting fields gave us a sense of the data. We can now take a closer look by doing a simple, but very useful, exercise — computing frequency distributions. Starting simple with frequencies is generally a good approach; it helps in getting ideas about how to proceed further.

# Importing modules
from collections import Counter

# Counting occcurrences/ getting frequency dist of all names and hashtags
for item in [names, hashtags]:
    c = Counter(item) 
    # Inspecting the 10 most common items in c
    print (c.most_common(10), "\n")
[('lildickytweets', 102), ('LeoDiCaprio', 44), ('ShawnMendes', 33), ('halsey', 31), ('ArianaGrande', 30), ('justinbieber', 29), ('Spotify', 26), ('edsheeran', 26), ('sanbenito', 25), ('SnoopDogg', 25)] [('WeLoveTheEarth', 313), ('4future', 12), ('19aprile', 12), ('EARTH', 11), ('fridaysforfuture', 10), ('EarthMusicVideo', 3), ('ConCalmaRemix', 3), ('Earth', 3), ('aliens', 2), ('AvengersEndgame', 2)]

7. Activity around the trend

🕵️‍♀️ Based on the last frequency distributions we can further build-up on our deductions:

  • We can more safely say that this was a music video about Earth (hashtag 'EarthMusicVideo') by Lil Dicky.
  • DiCaprio is not a music artist, but he was involved as well (Leo is an environmentalist so not a surprise to see his name pop up here).
  • We can also say that the video was released on a Friday; very likely on April 19th.

We have been able to extract so many insights. Quite powerful, isn't it?!

Let's further analyze the data to find patterns in the activity around the tweets — did all retweets occur around a particular tweet?

If a tweet has been retweeted, the 'retweeted_status' field gives many interesting details about the original tweet itself and its author.

We can measure a tweet's popularity by analyzing the retweetcount and favoritecount fields. But let's also extract the number of followers of the tweeter — we have a lot of celebs in the picture, so can we tell if their advocating for #WeLoveTheEarth influenced a significant proportion of their followers?

Note: The retweet_count gives us the total number of times the original tweet was retweeted. It should be the same in both the original tweet and all the next retweets. Tinkering around with some sample tweets and the official documentaiton are the way to get your head around the mnay fields.

# Extracting useful information from retweets
retweets = [(tweet['retweet_count'], 
             tweet['retweeted_status']['favorite_count'], 
             tweet['retweeted_status']['user']['followers_count'], 
             tweet['retweeted_status']['user']['screen_name'],
             tweet['text']) for tweet in tweets if 'retweeted_status' in tweet]

8. A table that speaks a 1000 words

Let's manipulate the data further and visualize it in a better and richer way — "looks matter!"

# Importing modules
import matplotlib.pyplot as plt
import pandas as pd

# Create a DataFrame and visualize the data in a pretty and insightful format
df = pd.DataFrame(retweets, 
                  columns=['Retweets',
                           'Favorites',
                           'Followers', 
                           'ScreenName', 
                           'Text']).groupby(['ScreenName',
                                             'Text',
                                             'Followers']).sum().sort_values(by=['Followers'], ascending=False)

df.style.background_gradient()

9. Analyzing used languages

🕵️‍♀️ Our table tells us that:

  • Lil Dicky's followers reacted the most — 42.4% of his followers liked his first tweet.
  • Even if celebrities like Katy Perry and Ellen have a huuge Twitter following, their followers hardly reacted, e.g., only 0.0098% of Katy's followers liked her tweet.
  • While Leo got the most likes and retweets in terms of counts, his first tweet was only liked by 2.19% of his followers.

The large differences in reactions could be explained by the fact that this was Lil Dicky's music video. Leo still got more traction than Katy or Ellen because he played some major role in this initiative.

Can we find some more interesting patterns in the data? From the text of the tweets, we could spot different languages, so let's create a frequency distribution for the languages.

# Extracting language for each tweet and appending it to the list of languages
tweets_languages = []
for tweet in tweets:
        tweets_languages.append(tweet['lang'])



# Plotting the distribution of languages
%matplotlib inline
plt.hist(tweets_languages)
(array([303., 107., 22., 14., 36., 32., 3., 2., 1., 2.]), array([ 0. , 1.3, 2.6, 3.9, 5.2, 6.5, 7.8, 9.1, 10.4, 11.7, 13. ]), <a list of 10 Patch objects>)
Image in a Jupyter notebook

10. Final thoughts

🕵️‍♀️ The last histogram tells us that:

  • Most of the tweets were in English.
  • Polish, Italian and Spanish were the next runner-ups.
  • There were a lot of tweets with a language alien to Twitter (lang = 'und').

Why is this sort of information useful? Because it can allow us to get an understanding of the "category" of people interested in this topic (clustering). We could also analyze the device type used by the Twitteratis, tweet['source'], to answer questions like, "Does owning an Apple compared to Andorid influences people's propensity towards this trend?". I will leave that as a further exercise for you!

What an exciting journey it has been! We started almost clueless, and here we are.. rich in insights.

From location based comparisons to analyzing the activity around a tweet to finding patterns from languages and devices, we have covered a lot today — let's give ourselves a well-deserved pat on the back! ✋

Magic Formula = Data + Python + Creativity + Curiosity

# Congratulations!
print("High Five!!!")
High Five!!!