MediaCrawler

Product Introduction

1. Definition: MediaCrawler is an open-source, multi-platform social media data scraping and collection tool, categorized as a web crawler or data extraction framework. It is designed to automate the collection of publicly available content from major Chinese social media and content platforms.
2. Core Value Proposition: It exists to solve the efficiency bottleneck of manual data collection for researchers, market analysts, and content operators. Its primary value is providing a unified, automated solution for batch acquisition of cross-platform public content data, enabling downstream tasks like public opinion monitoring, competitive analysis, and content research.

Main Features

1. Multi-Platform Support: MediaCrawler supports automated crawling from Xiaohongshu (Little Red Book), Douyin (TikTok), Kuaishou, Bilibili (B站), Weibo, Baidu Tieba, and Zhihu. It handles diverse content types including图文笔记 (image-text notes), videos, posts, and their associated评论 (comments).
2. Flexible Data Storage: The tool offers multiple data persistence options to fit different workflows. It supports exporting to flat files (CSV and JSON) for simple analysis and to databases (SQLite and MySQL) for structured storage and complex querying. Database schema initialization is handled via a dedicated command-line argument (--init_db).
3. Configurable Crawling Modes: It provides two primary operational modes: search mode (crawling content based on configured keywords) and detail mode (crawling specific posts/videos based on a provided ID list). This allows for both broad discovery and targeted data collection.
4. Modern Technical Stack: The crawler utilizes Playwright for browser automation, providing robust handling of JavaScript-rendered content common on modern social platforms. It uses uv for fast and consistent Python dependency management and requires Node.js for platforms like Douyin and Zhihu that may rely on specific client-side logic.

Problems Solved

1. Pain Point: Eliminates the time-consuming, error-prone, and non-scalable process of manually copying, pasting, and organizing social media data from multiple sources. It directly addresses the challenge of gathering large-scale, structured datasets for analysis.
2. Target Audience: The primary user personas are Data Researchers and Academic Researchers studying social trends; Market Analysts and Competitive Intelligence Professionals monitoring brands and industry discourse; and Content Strategists or Operators researching content performance and audience engagement across platforms.
3. Use Cases: Essential for building datasets for sentiment analysis (舆情监控), tracking marketing campaign reach and user feedback, analyzing competitor content strategy, and conducting large-scale content studies for academic or business intelligence purposes.

Unique Advantages

1. Differentiation: Unlike single-platform scrapers or generic web crawlers, MediaCrawler is pre-configured for the specific anti-crawler mechanisms and data structures of major Chinese platforms. Compared to manual collection or using disparate tools, it offers a consolidated, automated workflow.
2. Key Innovation: Its architecture provides a unified framework for heterogeneous platforms. The abstraction of platform-specific crawlers under a common interface, combined with configurable storage backends and login strategies (like QR code authentication), makes it a versatile and developer-friendly toolkit for social media data acquisition.

Frequently Asked Questions (FAQ)

1. Is MediaCrawler legal to use? MediaCrawler is designed for collecting publicly available data for learning and research purposes. Users must comply with the target platforms' robots.txt files and Terms of Service, avoid excessive request rates that could burden servers, and must not use the data for commercial infringement or illegal activities. The project includes a免责声明 (disclaimer) highlighting these responsibilities.
2. How does MediaCrawler handle login-required data or platforms? The tool supports login mechanisms such as QR code scanning (--lt qrcode) for platforms like Xiaohongshu. This allows the crawler to access data visible to a logged-in user. Configuration for these methods is managed within config/base_config.py.
3. What are the system requirements to run MediaCrawler? It requires Python (3.11 is recommended), Node.js (>=16.0.0 for Douyin/Zhihu support), and a package manager like uv. Playwright browsers will be installed automatically. Sufficient network bandwidth and storage for the collected data are also necessary.
4. Can MediaCrawler download videos or images? Based on its description as a content爬虫 (crawler) that stores data in CSV, JSON, or databases, its primary function is to extract metadata (text, URLs, engagement stats, comments). Actual media file downloading is not explicitly listed as a core feature, though extracted URLs could be used for that purpose with additional tools.
5. What is the difference between --save_data_option sqlite and --save_data_option db? The sqlite option saves data to a local SQLite database file, ideal for individual users due to its zero setup. The db option (a legacy parameter) saves data to a MySQL database, which is suitable for team use or integration into larger applications but requires a pre-configured MySQL server.