夜夜操天天操,亚洲福利视频一区,免费观看黄色一级视频

http://blog.csdn.net/hijk139/article/details/8308224

業(yè)務系統(tǒng)需要收集監(jiān)控系統(tǒng)日志，想到了hadoop的flume。經(jīng)過試驗，雖說功能不算足夠強大，但基本上能夠滿足功能需求。Flume 是一個分布式、可靠和高可用的服務日志收集工具，能夠和hadoop,hive等配置完成日志收集，存儲，分析處理等工作，更詳細的介紹可以參見apache網(wǎng)站。下面介紹下簡單的安裝配置方法

1,網(wǎng)上下載flume-ng安裝包，分別部署在收集和接收日志文件的服務器上，服務器上需安裝jdk 1.6以上，
http://flume.apache.org/download.html
tar -zxvf apache-flume-1.3.0-bin.tar.gz
2, 日志文件接收端端新建conf/flume-conf.properties server端的具體配置如下

從avro source端接收數(shù)據(jù)，然后寫入到HDFS文件系統(tǒng)中

[html] ? view plain copy

[flume@?conf]$?cat??flume-conf.properties??
agent.sources?=? avrosrc??
agent.channels?=? memoryChanne3??
agent.sinks?=? hdfsSink??
??
#?For?each?one?of?the?sources,?the?type?is?defined??
agent.sources.avrosrc.type?=? avro??
agent.sources.avrosrc.bind?=? 172.16.251.1??
agent.sources.avrosrc.port?=? 44444??
??
#?The?channel?can?be?defined?as?follows.??
agent.sources.avrosrc.channels?=? memoryChanne3??
??
#?Each?channel's?type?is?defined.??
agent.channels.memoryChanne3.type?=? memory??
agent.channels.memoryChanne3.keep-alive?=? 10??
agent.channels.memoryChanne3.capacity?=? 100000??
agent.channels.memoryChanne3.transactionCapacity?= 100000??
??
#?Each?sink's?type?must?be?defined??
agent.sinks.hdfsSink.type?=? hdfs??
agent.sinks.hdfsSink.channel?=? memoryChanne3??
agent.sinks.hdfsSink.hdfs.path?=?/logdata/%{hostname}_linux/%Y%m%d_date??
agent.sinks.hdfsSink.hdfs.filePrefix?=?%{datacenter}_??
agent.sinks.hdfsSink.hdfs.rollInterval?=? 0??
agent.sinks.hdfsSink.hdfs.rollSize?=? 4000000??
agent.sinks.hdfsSink.hdfs.rollCount?=? 0??
agent.sinks.hdfsSink.hdfs.writeFormat?=? Text??
agent.sinks.hdfsSink.hdfs.fileType?=? DataStream??
agent.sinks.hdfsSink.hdfs.batchSize?=? 10??

如果flume和hadoop不是同一用戶，需要注意相關權限問題

3，日志收集端的conf/flume-conf.properties server文件配置，這里收集二個日志文件到收集端

[html] ? view plain copy

agent.sources?=? tailsource-1?tailsource-2??
agent.channels?=? memoryChannel-1?memoryChannel-2??
agent.sinks?=? remotesink?remotesink-2??
??
agent.sources.tailsource-1.type?=? exec??
agent.sources.tailsource-1.command?=? tail?-F?/tmp/linux2.log??
agent.sources.tailsource-1.channels?=? memoryChannel-1??
??
agent.sources.tailsource-2.type?=? exec??
agent.sources.tailsource-2.command?=? tail?-F?/tmp/linux2_2.log??
agent.sources.tailsource-2.channels?=? memoryChannel-2??
??
agent.sources.tailsource-1.interceptors?=? host_int?timestamp_int?inter1??
agent.sources.tailsource-1.interceptors.host_int.type?=? host??
agent.sources.tailsource-1.interceptors.host_int.hostHeader?=? hostname??
??
agent.sources.tailsource-1.interceptors.timestamp_int.type?=? org.apache.flume.interceptor.TimestampInterceptor$Builder??
??
# agent.sources.tailsource-1.interceptors?=? inter1??
agent.sources.tailsource-1.interceptors.inter1.type?=? static??
agent.sources.tailsource-1.interceptors.inter1.key?=? datacenter??
agent.sources.tailsource-1.interceptors.inter1.value?=? BEIJING??
??
agent.sources.tailsource-2.interceptors?=? host_int?timestamp_int?inter1??
agent.sources.tailsource-2.interceptors.host_int.type?=? host??
agent.sources.tailsource-2.interceptors.host_int.hostHeader?=? hostname??
??
agent.sources.tailsource-2.interceptors.timestamp_int.type?=? org.apache.flume.interceptor.TimestampInterceptor$Builder??
??
# agent.sources.tailsource-1.interceptors?=? inter1??
agent.sources.tailsource-2.interceptors.inter1.type?=? static??
agent.sources.tailsource-2.interceptors.inter1.key?=? datacenter??
agent.sources.tailsource-2.interceptors.inter1.value?=? linux2_2??
??
agent.channels.memoryChannel-1.type?=? memory??
agent.channels.memoryChannel-1.keep-alive?=? 10??
agent.channels.memoryChannel-1.capacity?=? 100000??
agent.channels.memoryChannel-1.transactionCapacity?= 100000??
??
agent.channels.memoryChannel-2.type?=? memory??
agent.channels.memoryChannel-2.keep-alive?=? 10??
agent.channels.memoryChannel-2.capacity?=? 100000??
agent.channels.memoryChannel-2.transactionCapacity?= 100000??
??
agent.sinks.remotesink.type?=? avro??
agent.sinks.remotesink.hostname?=? 172.16.251.1??
agent.sinks.remotesink.port?=? 44444??
agent.sinks.remotesink.channel?=? memoryChannel-1??
??
agent.sinks.remotesink-2.type?=? avro??
agent.sinks.remotesink-2.hostname?=? 172.16.251.1??
agent.sinks.remotesink-2.port?=? 44444??
agent.sinks.remotesink-2.channel?=? memoryChannel-2??

4，后臺運行
nohup bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties >1.log &

查看日志vi flume.log
端口連接情況 netstat -an|grep 44444
[flume@dtydb6 flume-1.4]$ netstat -an|grep 44444
tcp??????? 0????? 0 ::ffff:172.16.251.1:44444?? :::*??????????????????????? LISTEN ?

5,測試方法

可以使用如下類似的腳本，定期向日志文件寫入來進行測試

for i in {1..1000000}; do echo "LINUX2? PRESS ************* Flume log rotation $i" >> /tmp/linux3.log; sleep 0.0001; done

參考資料：
http://flume.apache.org/FlumeUserGuide.html

flume ng 1.3 安裝

更多文章、技術交流、商務合作、聯(lián)系博主

微信掃碼或搜索：z360901061

微信掃一掃加我為好友

QQ號聯(lián)系： 360901061

您的支持是博主寫作最大的動力，如果您喜歡我的文章，感覺我的文章對您有幫助，請用微信掃描下面二維碼支持博主2元、5元、10元、20元等您想捐的金額吧，狠狠點擊下面給點支持吧，站長非常感激您！手機微信長按不能支付解決辦法：請將微信支付二維碼保存到相冊，切換到微信，然后點擊微信右上角掃一掃功能，選擇支付二維碼完成支付。

【本文對您有幫助就好】元

2元

5元

10元

20元

自定義