WordPress数据采集实战指南：从零开始构建自动化采集系统

一、环境准备与核心工具选择

采集WordPress数据前需要准备两个基础环境：本地开发环境和目标网站访问环境。

1.1 PHP开发环境配置

安装PHP 7.4及以上版本，并启用以下扩展：

• curl
• dom
• fileinfo
• mbstring
• openssl
• xml

在Ubuntu系统上安装命令：

sudo apt update
sudo apt install php php-curl php-dom php-fileinfo php-mbstring php-openssl php-xml

1.2 必备工具包安装

使用Composer安装采集所需的核心库：

composer require guzzlehttp/guzzle
composer require symfony/dom-crawler
composer require symfony/css-selector
composer require league/html-to-markdown

二、目标网站分析与数据定位

2.1 分析WordPress网站结构

使用浏览器开发者工具（F12）分析目标网站：

1. 查看文章列表页URL模式，通常为：/page/{页码}/或/?paged={页码}
2. 识别文章容器元素的CSS选择器，常见的有：article.post、div.post、content article
3. 记录关键数据位置：标题（h1/h2）、发布时间（time）、正文（.entry-content）、分类（.cat-links）

2.2 构建请求头避免被屏蔽

创建headers.php配置文件：

 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8',
'Accept-Language' => 'zh-CN,zh;q=0.9,en;q=0.8',
'Accept-Encoding' => 'gzip, deflate',
'Connection' => 'keep-alive',
'Upgrade-Insecure-Requests' => '1',
'Cache-Control' => 'max-age=0',
];

三、核心采集代码实现

3.1 创建基础采集类

创建WordPressCrawler.php：

baseUrl = rtrim($baseUrl, '/');
$this->client = new Client([
'timeout' => 30,
'verify' => false,
'headers' => require 'headers.php'
]);
}
public function fetchPage($url) {
try {
$response = $this->client->get($url);
return (string)$response->getBody();
} catch (\Exception $e) {
echo "获取页面失败: {$url}\n错误: {$e->getMessage()}\n";
return null;
}
}
}

3.2 实现文章列表采集

在WordPressCrawler类中添加方法：

public function fetchArticleLinks($listUrl, $maxPages = 10) {
$articles = [];
for ($page = 1; $page <= $maxPages; $page++) {
$url = str_replace('{page}', $page, $listUrl);
$html = $this->fetchPage($url);
if (!$html) break;
$crawler = new Crawler($html);
// 提取文章链接，根据实际网站调整选择器
$links = $crawler->filter('article a.entry-title-link, h2.entry-title a, h2.post-title a');
if ($links->count() === 0) break;
$links->each(function (Crawler $node) use (&$articles) {
$href = $node->attr('href');
if ($href && !in_array($href, $articles)) {
$articles[] = [
'url' => $href,
'title' => trim($node->text())
];
}
});
echo "已采集第 {$page} 页，找到 {$links->count()} 篇文章\n";
sleep(rand(2, 5)); // 避免请求过快
}
return $articles;
}

3.3 实现文章详情采集

添加文章内容提取方法：

public function fetchArticleDetail($articleUrl) {
$html = $this->fetchPage($articleUrl);
if (!$html) return null;
$crawler = new Crawler($html);
$article = [];
// 提取标题
$title = $crawler->filter('h1.entry-title, h1.post-title, h1.page-title, header h1');
$article['title'] = $title->count() > 0 ? trim($title->text()) : '';
// 提取发布时间
$date = $crawler->filter('time.entry-date, .post-date, .published, meta[property="article:published_time"]');
if ($date->count() > 0) {
$article['publish_date'] = $date->attr('datetime') ?? trim($date->text());
}
// 提取正文内容
$content = $crawler->filter('.entry-content, .post-content, .article-content, content .content');
if ($content->count() > 0) {
$article['content'] = $content->html();
} else {
// 备用选择器
$content = $crawler->filter('article');
$article['content'] = $content->count() > 0 ? $content->html() : '';
}
// 提取分类标签
$categories = [];
$crawler->filter('.cat-links a, .post-categories a, .entry-categories a')->each(
function (Crawler $node) use (&$categories) {
$categories[] = trim($node->text());
}
);
$article['categories'] = $categories;
// 提取标签
$tags = [];
$crawler->filter('.tags-links a, .post-tags a, .entry-tags a')->each(
function (Crawler $node) use (&$tags) {
$tags[] = trim($node->text());
}
);
$article['tags'] = $tags;
$article['url'] = $articleUrl;
$article['crawled_at'] = date('Y-m-d H:i:s');
return $article;
}

四、数据处理与存储

4.1 数据清洗与格式化

创建DataProcessor.php处理采集到的数据：

]>(.?)<\/script>/is', '', $html);
// 移除style标签
$html = preg_replace('/]>(.?)<\/style>/is', '', $html);
// 移除注释
$html = preg_replace('//s', '', $html);
// 移除空白字符
$html = preg_replace('/\s+/', ' ', $html);
return trim($html);
}
public static function convertToMarkdown($html) {
$converter = new \League\HTMLToMarkdown\HtmlConverter();
$converter->getConfig()->setOption('strip_tags', true);
$converter->getConfig()->setOption('remove_nodes', 'script style');
return $converter->convert($html);
}
public static function generateSlug($title) {
$slug = preg_replace('/[^\p{L}\p{N}\s]/u', '', $title);
$slug = strtolower(trim($slug));
$slug = preg_replace('/\s+/', '-', $slug);
return $slug;
}
}

4.2 数据存储到数据库

创建数据库表结构：

CREATE TABLE crawled_articles (
id INT AUTO_INCREMENT PRIMARY KEY,
original_url VARCHAR(500) NOT NULL UNIQUE,
title VARCHAR(255) NOT NULL,
slug VARCHAR(255) NOT NULL,
content_html LONGTEXT,
content_markdown LONGTEXT,
publish_date DATETIME,
categories JSON,
tags JSON,
crawled_at DATETIME NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

创建数据存储类DatabaseStorage.php：

pdo = new PDO(
'mysql:host=localhost;dbname=your_database;charset=utf8mb4',
'your_username',
'your_password',
[PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION]
);
}
public function saveArticle($article) {
$sql = "INSERT INTO crawled_articles (
original_url, title, slug, content_html,
content_markdown, publish_date, categories,
tags, crawled_at
) VALUES (
:original_url, :title, :slug, :content_html,
:content_markdown, :publish_date, :categories,
:tags, :crawled_at
) ON DUPLICATE KEY UPDATE
title = VALUES(title),
content_html = VALUES(content_html),
content_markdown = VALUES(content_markdown),
crawled_at = VALUES(crawled_at)";
$stmt = $this->pdo->prepare($sql);
$stmt->execute([
':original_url' => $article['url'],
':title' => $article['title'],
':slug' => DataProcessor::generateSlug($article['title']),
':content_html' => DataProcessor::cleanHtmlContent($article['content'] ?? ''),
':content_markdown' => DataProcessor::convertToMarkdown($article['content'] ?? ''),
':publish_date' => $article['publish_date'] ?? null,
':categories' => json_encode($article['categories'] ?? [], JSON_UNESCAPED_UNICODE),
':tags' => json_encode($article['tags'] ?? [], JSON_UNESCAPED_UNICODE),
':crawled_at' => $article['crawled_at']
]);
return $this->pdo->lastInsertId();
}
}

五、完整采集流程集成

5.1 创建采集任务脚本

创建crawl.php主程序：

fetchArticleLinks($listUrlPattern, 5); // 采集5页
echo "\n找到 " . count($articles) . " 篇文章\n";
// 步骤2：逐篇采集详情
echo "\n=== 采集文章详情 ===\n";
$successCount = 0;
$failCount = 0;
foreach ($articles as $index => $articleInfo) {
echo "处理第 " . ($index + 1) . "/" . count($articles) . " 篇: {$articleInfo['title']}\n";
$article = $crawler->fetchArticleDetail($articleInfo['url']);
if ($article && !empty($article['content'])) {
try {
$id = $storage->saveArticle($article);
echo "  保存成功 (ID: {$id})\n";
$successCount++;
} catch (Exception $e) {
echo "  保存失败: " . $e->getMessage() . "\n";
$failCount++;
}
} else {
echo "  内容为空，跳过\n";
$failCount++;
}
// 随机延迟，避免请求过快
sleep(rand(3, 8));
}
echo "\n=== 采集完成 ===\n";
echo "成功: {$successCount} 篇\n";
echo "失败: {$failCount} 篇\n";

5.2 错误处理与日志记录

创建Logger.php：

getMessage();
}
$log .= PHP_EOL;
file_put_contents(self::$logFile, $log, FILE_APPEND);
echo "错误: " . $message . PHP_EOL;
}
}

六、高级优化与注意事项

6.1 请求频率控制

修改WordPressCrawler的fetchPage方法，添加速率限制：

private $lastRequestTime = 0;
private $minRequestInterval = 3; // 最小请求间隔（秒）
public function fetchPage($url) {
// 控制请求频率
$now = microtime(true);
$elapsed = $now - $this->lastRequestTime;
if ($elapsed < $this->minRequestInterval) {
$sleepTime = $this->minRequestInterval - $elapsed;
usleep($sleepTime  1000000);
}
try {
$response = $this->client->get($url);
$this->lastRequestTime = microtime(true);
return (string)$response->getBody();
} catch (\Exception $e) {
Logger::error("获取页面失败: {$url}", $e);
return null;
}
}

6.2 代理设置（如需）

在WordPressCrawler构造函数中添加代理支持：

public function __construct($baseUrl, $proxy = null) {
$this->baseUrl = rtrim($baseUrl, '/');
$config = [
'timeout' => 30,
'verify' => false,
'headers' => require 'headers.php'
];
if ($proxy) {
$config['proxy'] = $proxy;
}
$this->client = new Client($config);
}

6.3 断点续采实现

创建断点记录文件checkpoint.json：

{
"last_crawled_url": "",
"last_crawled_time": "",
"total_crawled": 0
}

在crawl.php中添加断点检查：

// 读取断点
$checkpointFile = 'checkpoint.json';
if (file_exists($checkpointFile)) {
$checkpoint = json_decode(file_get_contents($checkpointFile), true);
echo "从断点恢复，上次采集到: {$checkpoint['last_crawled_url']}\n";
}
// 在保存成功后更新断点
$checkpointData = [
'last_crawled_url' => $article['url'],
'last_crawled_time' => date('Y-m-d H:i:s'),
'total_crawled' => $successCount
];
file_put_contents($checkpointFile, json_encode($checkpointData, JSON_PRETTY_PRINT));