爱华网公司一台3560G的cpu利用率这些天一直在80%左右，sh proc cpu sorted查看，IPInput最多，50%以上，处理过程如下：
1、使用sh controllers cpu-interface /sh platform port-asic statsdrop /sh ip traffic查看，发现host增长很快
bj-fb8-sw3560G-1#sh controllers cpu-interface
ASICRxbiterrRxunderFwdctfixTxbuflosRxbufloc Rxbufdrain
-------------------------------------------------------------------------
ASIC0000000
ASIC1000000
ASIC2000000
ASIC3000000
ASIC4000000
ASIC5000000
ASIC6000000
ASIC7000000
ASIC8000000
ASIC9000000
ASICA000000
ASICB000000
ASICC000000
cpu-queue-frames retrieveddroppedinvalidhol-block stray
----------------- ---------- ---------- ---------- --------------------
rpc10000
stp1055596070000
ipc00000
routing protocol 144674350004290
L2protocol343931830000
remoteconsole00000
swforwarding2775524800000
host40177563020000
broadcast165047268003760
cbt-to-spt00000
igmpsnooping1143240000
icmp7601247840000
logging00000
rpf-fail00000
queue1400000
cpuheartbeat13770773710000
Supervisor ASIC receive-queue parameters
----------------------------------------
queue 0 maxrecevsize 0 pakhead 0 paktail 0
queue 1 maxrecevsize 7E0 pakhead 2405438 paktail2406800
queue 2 maxrecevsize 7E0 pakhead 239DB40 paktail239FF84
queue 3 maxrecevsize 7E0 pakhead 25F1C60 paktail25F619C
queue 4 maxrecevsize 7E0 pakhead 2422658 paktail241F830
queue 5 maxrecevsize 7E0 pakhead 1E5E3B0 paktail1E5E3B0
queue 6 maxrecevsize 7E0 pakhead 25DFF94 paktail25DD804
queue 7 maxrecevsize 7E0 pakhead 2558548 paktail2541DEC
queue 8 maxrecevsize 7E0 pakhead 256598C paktail256AF44
queue 9 maxrecevsize 7E0 pakhead 24B4820 paktail24B5550
queue A maxrecevsize 7E0 pakhead 24A5460 paktail24A4DC8
queue B maxrecevsize 7E0 pakhead 2604CF4 paktail260538C
queue C maxrecevsize 7E0 pakhead 24BCF9C paktail24C0110
queue D maxrecevsize 7E0 pakhead 24B121C paktail24B4390
queue E maxrecevsize 0 pakhead 0 paktail 0
queue F maxrecevsize 7E0 pakhead 2492528 paktail24921DC
Supervisor ASIC exception status
--------------------------------
Receiveoverrun00000000 Transmit overrun00000000
FrameSignatureErr00000000MicInitialize00000000
BadFrameErr00000000LenExceededErr 00000000
BadJumboSegments 00000000
===========================================================
Complete Board Id:0x4023
===========================================================
bj-fb8-sw3560G-1#sh controllers cpu-interface
……
cpu-queue-frames retrieveddroppedinvalidhol-block stray
----------------- ---------- ---------- ---------- --------------------
rpc10000
stp1055596120000
ipc00000
routing protocol 144674375004290
L2protocol343931830000
remoteconsole00000
swforwarding2775524820000
host40177599940000
broadcast165047292003760
cbt-to-spt00000
igmpsnooping1143240000
icmp7601247900000
logging00000
rpf-fail00000
queue1400000
cpuheartbeat13770774020000
……
bj-fb8-sw3560G-1# sh platform port-asic stats drop
Port-asic Port Drop Statistics - Summary
========================================
RxQueue 0 Drop Stats: 0
RxQueue 1 Drop Stats: 0
RxQueue 2 Drop Stats: 0
RxQueue 3 Drop Stats: 0
Port 0 TxQueue Drop Stats:0
Port 1 TxQueue Drop Stats:245898
Port 2 TxQueue Drop Stats:343176
Port 3 TxQueue Drop Stats:45288
Supervisor TxQueue Drop Statistics
Queue 0: 0
Queue 1: 0
Queue 2: 0
Queue 3: 297994035
Queue 4: 34399724
Queue 5: 0
Queue 6: 50319920
Queue 7: 97399504
Queue 8: 522852659
Queue 9: 0
Queue 10:0
Queue 11:923919461
Queue 12:0
Queue 13:0
Queue 14:0
Queue 15:0
bj-fb8-sw3560G-1#sh iptraffic
IP statistics:
Rcvd: 4225595798 total,732594632 local destination
39370 format errors, 11961 checksum errors, 249965523 bad hopcount
17 unknown protocol, 12838076 not a gateway
0 security failures, 0 bad options, 32179 with options
Opts: 14179 end, 365 nop,12579 basic security, 180 loose source route
22 timestamp, 0 extended security, 363 record route
0 stream ID, 0 strict source route, 5042 alert, 0 cipso, 0ump
0 other
Frags: 72822 reassembled, 56 timeouts, 325couldn't reassemble
6228220 fragmented, 1 couldn't fragment
Bcast: 78949976 received, 9252 sent
Mcast: 35473413 received, 18448660 sent
Sent: 911515179 generated,2857320102 forwarded
Drop: 88849498 encapsulationfailed, 3 unresolved, 5902847 no adjacency
12303 no route, 0 unicast RPF, 0 forced drop
0 options denied, 1 source IP address zero
ICMP statistics:
Rcvd: 579 format errors, 105965 checksum errors,16787 redirects, 16769 unreachable
569637977 echo, 131965 echo reply, 0 mask requests, 1 mask replies,395 quench
31 parameter, 61 timestamp, 2 info request, 219 other
0 irdp solicitations, 0 irdp advertisements
Sent: 8940 redirects, 529891 unreachable, 131957echo, 569637977 echo reply
0 mask requests, 0 mask replies, 0 quench, 61 timestamp
0 info reply, 278073406 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements
TCP statistics:
Rcvd: 14035770 total, 136745 checksum errors,44261 no port
Sent: 6085025 total
UDP statistics:
Rcvd: 113010673 total, 253 checksum errors,70009403 no port
Sent: 38462217 total, 0 forwarded broadcasts
IP-EIGRP statistics:
Rcvd: 0 total
Sent: 0 total
BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0updates
0 keepalives, 0 route-refresh
PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 formaterrors
Registers: 0/0 (0 non-rp, 0 non-sm-group),Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts:0/0
Bootstraps: 0/0, Candidate_RP_Advertisements:0/0
State-Refresh: 0/0
IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors:0/0
Host Queries: 0/0, Host Reports: 0/0, HostLeaves: 0/0
DVMRP: 0/0, PIM: 0/0
OSPF statistics:
Rcvd: 35480830 total, 0 checksum errors
27704784 hello, 1217 database desc, 49 link state req
5984579 link state updates, 1790201 link state acks
Sent: 18462414 total
13851793 hello, 1156 database desc, 29 link state req
3461507 link state updates, 1147933 link state acks
ARP statistics:
Rcvd: 26534171 requests, 1904860 replies, 225reverse, 0 other
Sent: 81111265 requests, 7718826 replies (705906proxy), 0 reverse
Drop due to input queue full: 0
bj-fb8-sw3560G-1#sh ip traffic
IP statistics:
Rcvd: 4225685456 total,732595331 local destination
39370 format errors, 11961 checksum errors, 249965609 bad hopcount
17 unknown protocol, 12838076 not a gateway
0 security failures, 0 bad options, 32179 with options
Opts: 14179 end, 365 nop,12579 basic security, 180 loose source route
22 timestamp, 0 extended security, 363 record route
0 stream ID, 0 strict source route, 5042 alert, 0 cipso, 0ump
0 other
Frags: 72822 reassembled, 56 timeouts, 325couldn't reassemble
6228220 fragmented, 1 couldn't fragment
Bcast: 78950118 received, 9252 sent
Mcast: 35473429 received, 18448668 sent
Sent: 911515823 generated,2877687219 forwarded
Drop: 88849655 encapsulationfailed, 3 unresolved, 5902847 no adjacency
12303 no route, 0 unicast RPF, 0 forced drop
0 options denied, 1 source IP address zero
ICMP statistics:
Rcvd: 579 format errors, 105965 checksum errors,16787 redirects, 16769 unreachable
569638390 echo, 131965 echo reply, 0 mask requests, 1 mask replies,395 quench
31 parameter, 61 timestamp, 2 info request, 219 other
0 irdp solicitations, 0 irdp advertisements
Sent: 8940 redirects, 529891 unreachable, 131957echo, 569638390 echo reply
0 mask requests, 0 mask replies, 0 quench, 61 timestamp
0 info reply, 278073499 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements
TCP statistics:
Rcvd: 14035883 total, 136747 checksum errors,44261 no port
Sent: 6085047 total
UDP statistics:
Rcvd: 113010828 total, 253 checksum errors,70009444 no port
Sent: 38462325 total, 0 forwarded broadcasts
IP-EIGRP statistics:
Rcvd: 0 total
Sent: 0 total
BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0updates
0 keepalives, 0 route-refresh
PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 formaterrors
Registers: 0/0 (0 non-rp, 0 non-sm-group),Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts:0/0
Bootstraps: 0/0, Candidate_RP_Advertisements:0/0
State-Refresh: 0/0
IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors:0/0
Host Queries: 0/0, Host Reports: 0/0, HostLeaves: 0/0
DVMRP: 0/0, PIM: 0/0
OSPF statistics:
Rcvd: 35480834 total, 0 checksum errors
27704788 hello, 1217 database desc, 49 link state req
5984579 link state updates, 1790201 link state acks
Sent: 18462417 total
13851796 hello, 1156 database desc, 29 link state req
3461507 link state updates, 1147933 link state acks
ARP statistics:
Rcvd: 26534183 requests, 1904861 replies, 225reverse, 0 other
Sent: 81111337 requests, 7718829 replies (705906proxy), 0 reverse
Drop due to input queue full: 0
2、打开以下debug，发现log中大量关于g0/15的信息：
bj-fb8-sw3560G-1(config)#no logging console
bj-fb8-sw3560G-1(config)#logging buffered 128000
bj-fb8-sw3560G-1(config)#^Z
bj-fb8-sw3560G-1#debug platform cpu-queues?
broadcast-qDebug packets received by Broadcast Q
cbt-to-spt-qDebug packets received by cbt-to-spt Q
cpuhub-qDebug packets received by CPU heartbeat Q
host-qDebug packets received by host Q
icmp-qDebug packets received by ICMP Q
igmp-snooping-qDebug packets received by IGMP snooping Q
layer2-protocol-q Debug packetsreceived by layer2 protocol Q
logging-qDebug packets received by logging Q
remote-console-qDebug packets received by remote console Q
routing-protocol-q Debugpackets received by routing protocol Q
rpffail-qDebug packets received by RPF fail Q
software-fwd-qDebug packets received by software forward Q
stp-qDebug packets received by STP Q
bj-fb8-sw3560G-1#debug platform cpu-queues bro
bj-fb8-sw3560G-1#debug platform cpu-queues broadcast-q
debug platform cpu-queue broadcast-q debugging is on
bj-fb8-sw3560G-1#debug platform cpu-queuescbt
bj-fb8-sw3560G-1#debug platform cpu-queues cbt-to-spt-q
debug platform cpu-queue cbt-sbt-q debugging is on
bj-fb8-sw3560G-1#debug platform cpu-queuescpu
bj-fb8-sw3560G-1#debug platform cpu-queues cpuhub-q
debug platform cpu-queue cpuhb debugging is on
bj-fb8-sw3560G-1#debug platform cpu-queueshost-
bj-fb8-sw3560G-1#debug platform cpu-queues host-q
debug platform cpu-queue host-q debugging is on
bj-fb8-sw3560G-1#debug platform cpu-queuesic
bj-fb8-sw3560G-1#debug platform cpu-queues icmp-q
debug platform cpu-queue icmp-q debugging is on
bj-fb8-sw3560G-1#sh log
Syslog logging: enabled (1463080 messages dropped, 1 messagesrate-limited, 0 flushes, 0
Log Buffer (128000 bytes):
thernet0/15 DI:0x30, LT:7,Vlan:1033 SrcGPN:15, SrcGID:15,ACLLogIdx:0x0, MacDA:001e.796e.06d2, MacSA:e41f.132c.6c86IP_SA:*.*.217.165 IP_DA:123.130.26.201 IP_Proto:6
TPFFD:DC00000F_00000409_00A00596-00000030_87910000_00000000
Mar 22 10:29:33: L2B-Q:Dropped Duplicate Broadcast ARP/RARP:Local Port Fwding L3If:GigabitEthernet0/22 L2If:GigabitEthernet0/22DI:0x58F, LT:1, Vlan:1028SrcGPN:22, SrcGID:22, ACLLogIdx:0x0, MacDA:ffff.ffff.ffff, MacSA:0015.1753.00a8 ARP:00010800_06040001_00151753_00A8C0A8_00CB0000_00000000_C0A80065
TPFFD:E2D90016_04040404_00800040-0000058F_00000000_00000000
Mar 22 10:29:33: Host-Q:Queued L3If: Local Port FwdingL3If:GigabitEthernet0/15 L2If:GigabitEthernet0/15 DI:0x30, LT:7,Vlan:1033 SrcGPN:15, SrcGID:15,ACLLogIdx:0x0, MacDA:001e.796e.06d2, MacSA:e41f.132c.6c86IP_SA:*.*.217.165 IP_DA:115.149.120.51 IP_Proto:6
TPFFD:DC00000F_00000409_00A005C0-00000030_31E90000_00000000
Mar 22 10:29:33: Host-Q:Queued L3If: Local Port FwdingL3If:GigabitEthernet0/15 L2If:GigabitEthernet0/15 DI:0x30, LT:7,Vlan:1033 SrcGPN:15, SrcGID:15,ACLLogIdx:0x0, MacDA:001e.796e.06d2, MacSA:e41f.132c.6c86IP_SA:*.*.217.165 IP_DA:122.137.75.151 IP_Proto:6
TPFFD:DC00000F_00000409_00A00040-00000030_5D160000_00000000
Mar 22 10:29:33: Host-Q:Queued L3If: Local Port FwdingL3If:GigabitEthernet0/15 L2If:GigabitEthernet0/15 DI:0x30, LT:7,Vlan:1033 SrcGPN:15, SrcGID:15,ACLLogIdx:0x0, MacDA:001e.796e.06d2, MacSA:e41f.132c.6c86IP_SA:*.*.217.165 IP_DA:123.130.26.201 IP_Proto:6
TPFFD:DC00000F_00000409_00A0058A-00000030_879A0000_00000000
Mar 22 10:29:33: Host-Q:Queued L3If: Local Port FwdingL3If:GigabitEthernet0/15 L2If:GigabitEthernet0/15 DI:0x30, LT:7,Vlan:1033 SrcGPN:15, SrcGID:15,ACLLogIdx:0x0, MacDA:001e.796e.06d2, MacSA:0021.5e56.9a74IP_SA:*.*.217.169 IP_DA:124.42.124.83 IP_Proto:6
TPFFD:DC00000F_00000409_00A005EE-00000030_21790000_00000000
Mar 22 10:29:33: Host-Q:Queued L3If: Local Port FwdingL3If:GigabitEthernet0/15 L2If:GigabitEthernet0/15 DI:0x30, LT:7,Vlan:1033 SrcGPN:15, SrcGID:15,ACLLogIdx:0x0, MacDA:001e.796e.06d2, MacSA:0021.5e56.9a74IP_SA:*.*.217.169 IP_DA:124.42.124.83 IP_Proto:6
TPFFD:DC00000F_00000409_00A000C9-00000030_269D0000_00000000
Mar 22 10:29:33: Host-Q:Queued L3If: Local Port FwdingL3If:GigabitEthernet0/15 L2If:GigabitEthernet0/15 DI:0x30, LT:7,Vlan:1033 SrcGPN:15, SrcGID:15,ACLLogIdx:0x0, MacDA:001e.796e.06d2, MacSA:e41f.132c.6c86IP_SA:*.*.217.165 IP_DA:115.149.120.51 IP_Proto:6
TPFFD:DC00000F_00000409_00A005C0-00000030_31E60000_00000000
……
3、关闭g0/15口后，cpu下降至7%，再次打开g0/15，cpu正常。
g0/15的流量相对于其他端口并不大，这些天也没有异常变化，只是shutdown和noshutdown了一下，cpu就正常了，不知是何原因？
参考文档：
http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html
