Reliable and Efficient Long-Term Social Media Monitoring

Social media data is now widely used by many academic researchers. However, long-term social media data collection projects, which most typically involve collecting data from public-use APIs, often encounter issues when relying on local area network servers (LANs) to collect high-volume streaming social media data over long periods of time. In this paper, we present a cloud-based data collection, pre-processing, and archiving infrastructure, and argue that this system mitigates or resolves the problems most typically encountered when running social media data collection projects on LANs at minimal cloud-computing costs. We show how this approach works in different cloud computing architectures, and how to adapt the method to collect streaming data from other social media platforms. The contribution of our research lies in the development of methodologies that researchers can use to monitor and analyze phenomena including how public opinion and public discourse change in response to events, monitoring the evolution and change of misinformation campaigns, and studying how organizations and entities change how they present and frame information online.