- 百度
- Bing
- 360
- 搜狗
- 神马
本系统采用微服务架构，通过四个核心模块实现数据采集、处理、存储和展示的完整闭环。每个模块职责清晰，便于后期扩展和维护。
系统由以下组件构成：
所有组件均使用Docker部署，确保环境一致性。以下为完整的安装脚本。
首先安装Docker和Docker Compose：
``` Ubuntu系统安装命令 sudo apt-get update sudo apt-get install docker.io docker-compose -y 启动Docker服务 sudo systemctl start docker sudo systemctl enable docker 验证安装 docker --version docker-compose --version ```创建docker-compose.yml文件，包含所有服务配置：
``` version: '3.7' services: zookeeper: image: confluentinc/cp-zookeeper:7.3.0 environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 ports: - "2181:2181" kafka: image: confluentinc/cp-kafka:7.3.0 depends_on: - zookeeper environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 ports: - "9092:9092" flink-jobmanager: image: flink:1.15.2-scala_2.12 command: jobmanager ports: - "8081:8081" environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: flink-jobmanager flink-taskmanager: image: flink:1.15.2-scala_2.12 depends_on: - flink-jobmanager command: taskmanager environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: flink-jobmanager taskmanager.numberOfTaskSlots: 2 clickhouse: image: clickhouse/clickhouse-server:22.8 ports: - "8123:8123" - "9000:9000" ulimits: nofile: soft: 262144 hard: 262144 grafana: image: grafana/grafana:9.1.0 ports: - "3000:3000" environment: - GF_SECURITY_ADMIN_PASSWORD=admin123 ```执行部署命令：
``` docker-compose up -d ```数据采集分为服务端日志和客户端行为两种方式，确保覆盖所有用户交互场景。
修改Nginx配置文件，添加用户行为字段：
``` http { log_format user_behavior '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time $upstream_response_time ' '"$http_x_forwarded_for" "$cookie_user_id" ' '"$arg_page_id" "$arg_action_type"'; access_log /var/log/nginx/access.log user_behavior; } ```创建tracker.js文件并部署到网站：
``` class BehaviorTracker { constructor(topic = 'user_behavior') { this.topic = topic; this.endpoint = 'http://your-kafka-proxy:8080/produce'; this.userId = this.getUserId(); this.sessionId = this.generateSessionId(); } track(event, properties = {}) { const data = { timestamp: Date.now(), userId: this.userId, sessionId: this.sessionId, event: event, properties: properties, page: window.location.pathname, userAgent: navigator.userAgent }; this.sendToKafka(data); } sendToKafka(data) { // 使用navigator.sendBeacon确保数据可靠发送 const blob = new Blob([JSON.stringify(data)], {type: 'application/json'}); navigator.sendBeacon(this.endpoint, blob); } getUserId() { return localStorage.getItem('user_id') || this.generateUserId(); } generateUserId() { const id = 'user_' + Math.random().toString(36).substr(2, 9); localStorage.setItem('user_id', id); return id; } generateSessionId() { return 'session_' + Date.now() + '_' + Math.random().toString(36).substr(2, 6); } } // 自动跟踪页面浏览事件 window.addEventListener('load', () => { window.tracker = new BehaviorTracker(); tracker.track('page_view', { referrer: document.referrer, screen_width: window.screen.width, screen_height: window.screen.height }); }); // 跟踪点击事件 document.addEventListener('click', (e) => { if (e.target.matches('[data-track]')) { const eventName = e.target.getAttribute('data-track'); tracker.track(eventName, { element_text: e.target.textContent.trim(), element_id: e.target.id }); } }); ```数据通过Kafka进入处理管道，经过Flink清洗和聚合后存入ClickHouse。
创建UserBehaviorJob.java处理程序：
``` import org.apache.flink.api.common.eventtime.WatermarkStrategy; import org.apache.flink.api.common.serialization.SimpleStringSchema; import org.apache.flink.connector.kafka.source.KafkaSource; import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer; import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.Table; import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; public class UserBehaviorJob { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env); // 创建Kafka数据源 KafkaSource
在ClickHouse中创建存储表：
``` CREATE DATABASE IF NOT EXISTS user_analytics; CREATE TABLE user_analytics.behavior_metrics ( window_start DateTime, window_end DateTime, page_url String, event_type String, user_count UInt64, event_count UInt64, avg_session_duration Float64 ) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(window_start) ORDER BY (window_start, page_url, event_type) TTL window_start + INTERVAL 30 DAY; ```使用Grafana创建实时监控仪表盘，直观展示分析结果。
登录Grafana（http://localhost:3000，用户名admin，密码admin123），添加ClickHouse数据源：
新建Dashboard并添加以下面板：
1. 实时用户活跃度面板
``` SELECT toStartOfMinute(window_start) as time, sum(user_count) as active_users FROM behavior_metrics WHERE window_start >= now() - INTERVAL 1 HOUR GROUP BY time ORDER BY time ```2. 页面访问热力图
``` SELECT page_url, sum(event_count) as page_views FROM behavior_metrics WHERE window_start >= today() GROUP BY page_url ORDER BY page_views DESC LIMIT 10 ```3. 用户行为漏斗分析
``` SELECT event_type, countDistinct(user_id) as user_count FROM user_behavior_parsed WHERE event_time >= now() - INTERVAL 24 HOUR AND event_type IN ('page_view', 'button_click', 'form_submit', 'purchase') GROUP BY event_type ORDER BY CASE event_type WHEN 'page_view' THEN 1 WHEN 'button_click' THEN 2 WHEN 'form_submit' THEN 3 WHEN 'purchase' THEN 4 END ```确保系统稳定运行的关键监控指标和优化策略。
调整Flink任务并行度：
``` flink-conf.yaml配置 taskmanager.numberOfTaskSlots: 4 parallelism.default: 2 ```优化ClickHouse查询性能：
``` users.xml配置创建数据质量监控SQL：
``` -- 检查数据完整性 SELECT toDate(window_start) as date, countIf(user_count = 0) as empty_records, countIf(event_count > 10000) as spike_records FROM behavior_metrics WHERE window_start >= today() - 7 GROUP BY date HAVING empty_records > 0 OR spike_records > 0 ```通过以上完整配置，系统能够实时处理每秒万级的用户行为事件，在5秒内完成从采集到可视化的全流程，为精准内容分发提供实时数据支撑。
下一篇： 精准软文推广：老韭菜带你避坑的搞钱真经
易频IT社区是综合性互联网IT技术门户网站，专注分享网络技术、服务器运维、网络安全、编程开发、系统架构、云计算、大数据等行业干货，实时更新IT行业资讯、零基础教程、实战案例，为IT从业者、技术爱好者提供专业的学习交流平台。
Copyright © 2021-2026 易频IT社区. All Rights Reserved. 备案号：闽ICP备2023013482号 网站地图
