CCIE DC: FCoE/FC Troubleshooting

Hi Guys

Hopefully this blog post will get more and more information as I go along, I am trying to get as much troubleshooting info as I can from FC ports so you can work out why ports are isolated, why ports are in initializing etc

The most common problem with FC ports being stuck in initilizing is normally nothing to do with the underlying SAN and almost always an error in your actual FCoE Config, a VLAN might be missing or something else

Here is a very useful amount of troubleshoot info from the very helpful show port internal info, i have deliberately in this example bound a VFC to a 1 gig interface, which will never ever come up

I have highlighted in red the sections i find interesting.

N5k(config-if)# show port internal info interface vfc 1


vfc1 - if_index: 0x1E000000, phy_port_index: 0x1000
     local_index: 0xffff
  Admin Config - state(up), mode(F), speed(auto), trunk(on)
    beacon(off), snmp trap(on), tem(false)

    description()
    port owner()
    Link failure syslog level (5)
    Link failure syslog level = 5
  Operational Info - state(trunking), mode(TF), speed(auto), trunk(on)



    32-bit state reason(2,2 - None), state reason(2,2 - None)
    phy port enable (1), phy layer (FC), port_osm_mode(OSM)
    participating(1), fcid(0x000000), bb_scn(0)
    config port vsan (1), oper port vsan (1)
    rx bb_credit(0), rx bb_credit multiplier(0)
    rx bb_credit performance buffers(0), tx bb_credit(0)
    not first time port bringup is 0
 



    Port WWN is 20:00:00:05:9b:20:fa:7f
    current state [PI_FSM_ST_TXPORT_INIT_TRUNKING_ENABLED]

 

    port_init_eval_flag(0x00000401), cfg wait for none
        FC_PORT_INIT_DATA_VALID (0)
        FC_PORT_INIT_ADMIN_ST_DOWN_TO_UP (10)
        FC_PORT_INIT_PHY_PORT_UP (49)
    Mts node id 0x0
    Lock Info: resource [vfc1]
      type[0] p_gwrap[(nil)]
          FREE @ 75967 usecs after Sun Jul 14 12:27:34 2013
      type[1] p_gwrap[(nil)]
          FREE @ 76754 usecs after Sun Jul 14 12:27:34 2013
      type[2] p_gwrap[(nil)]
          FREE @ 77158 usecs after Sun Jul 14 12:27:34 2013
    0x10000001e000000
    eport_init_flag(0x00000000), elp_chk_flag(0x00000000)
    elp_rcvd_fc2_handle(0x00000000), elp_sent_fc2_handle(0x00000000)
    esc_chk_flag(0x00000000), esc_fc2_handle(0x00000000)
    elp_flags(0x0000), classes_supported(F), tx bb_credit(0)
    Peer ELP Revision: 0
    cnt_link_failure(0), cnt_link_success(0), cnt_port_up(0)
    cnt_cfg_wait_timeout(0), cnt_port_cfg_failure(0), cnt_init_retry(0)
    nos link failures (0), debounce window (300 ms), elp wait tout (750 ms)
    link fail reason (0x0)    num_elps_before_up (0)
    internal_reinit_counter (0) external_reinit_counter (0)
 last_max_reinit_reach_time  (n/a, 0)
  Trunk Info -
    trunk vsans (allowed active) (1,100)
    trunk vsans (operational allowed) (1,100)
    trunk vsans (up) ()
    trunk vsans (isolated) ()

  TE port per vsan information
  vfc1, Vsan 1 - state(down), state reason(Vlan not FCoE Enabled), fcid(0x000000
)
    port init flag(0x0), rcvd_dm_eport_rdy 0
    Lock Info: resource [vfc1, vsan 1]
      type[0] p_gwrap[(nil)]
          FREE @ 75243 usecs after Sun Jul 14 12:27:34 2013
      type[1] p_gwrap[(nil)]
          FREE @ 75244 usecs after Sun Jul 14 12:27:34 2013
      type[2] p_gwrap[(nil)]
          FREE @ 75244 usecs after Sun Jul 14 12:27:34 2013
    0x40000011e000000
current state [TE_FSM_ST_VFC_PORT_BRINGUP_SEQ_FAIL]
    RNID info not found.
    first time elp: 1
    Peer ELP Revision: 0
    Peer sw_wwn runtime tlv: 00:00:00:00:00:00:00:00
    Peer port_wwn runtime tlv: 00:00:00:00:00:00:00:00
  vfc1, Vsan 100 - state(down), state reason(Error disabled), fcid(0x000000)
    port init flag(0x0), rcvd_dm_eport_rdy 0
    Lock Info: resource [vfc1, vsan 100]
      type[0] p_gwrap[(nil)]
          FREE @ 75267 usecs after Sun Jul 14 12:27:34 2013
      type[1] p_gwrap[(nil)]
          FREE @ 75268 usecs after Sun Jul 14 12:27:34 2013
      type[2] p_gwrap[(nil)]
          FREE @ 75268 usecs after Sun Jul 14 12:27:34 2013
    0x40000641e000000
current state [TE_FSM_ST_VFC_PORT_BRINGUP_SEQ_FAIL]
    RNID info not found.
    first time elp: 1
    Peer ELP Revision: 0
    Peer sw_wwn runtime tlv: 00:00:00:00:00:00:00:00
    Peer port_wwn runtime tlv: 00:00:00:00:00:00:00:00
  Present in index object table: yes
  Pacer Info -
    Pacer state: released credits due to timer expiry
PQP: not in progress




Here is another great troubleshoot when I have an isolated VSAN:



N5k(config-if)# show int fc2/7
fc2/7 is trunking
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:47:00:05:9b:20:fa:40
    Admin port mode is NP, trunk mode is on
    snmp link state traps are enabled
    Port mode is TNP
    Port vsan is 1
    Speed is 4 Gbps
    Transmit B2B Credit is 16
    Receive B2B Credit is 16
    Receive data field Size is 2112
    Beacon is turned off
    Trunk vsans (admin allowed and active) (1,100)
    Trunk vsans (up)                       (1)
    Trunk vsans (isolated)                 (100)
    Trunk vsans (initializing)             ()
    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      20843 frames input, 21729740 bytes
        17 discards, 0 errors
        0 CRC,  0 unknown class
        0 too long, 0 too short
      1209452 frames output, 2509138052 bytes
        0 discards, 0 errors
      3 input OLS, 3 LRR, 4 NOS, 0 loop inits
      6 output OLS, 6 LRR, 7 NOS, 0 loop inits
    last clearing of "show interface" counters never
      16 receive B2B credit remaining
      16 transmit B2B credit remaining
      0 low priority transmit B2B credit remaining
    Interface last changed at Sun Jul 14 13:13:27 2013





N5k(config-if)# show port internal info interface fc2/7


fc2/7 - if_index: 0x01086000, phy_port_index: 0x86
     local_index: 0xffff
  Admin Config - state(up), mode(NP), speed(auto), trunk(on)
    beacon(off), snmp trap(on), tem(false)
    rx bb_credit(default), rx bb_credit multiplier(default)
    rx bb_credit performance buffers(default)
    bb scn config(on)
    speed group(NA)
    ignore flags (ignore:none), service state(in service)
    rxbufsize(2112), encap(default), user_cfg_flag(0x3)
    description()
    port owner()
    admin rate-mode(default) port act license(eligible)
    congestion drop timeout mode F (500), congestion drop timeout mode E (500)
    no-credit-force mode F enable (0), no-credit-force mode F timeout (500)
    no-credit-force mode E enable (0)    no-credit-force mode E timeout (500)
    Link failure syslog level (5)
    Link failure syslog level = 5
   Port guard info -
     link failure state (disabled, 4) nt 0, dur 0
     tsv state (disabled, 4) nt 0, dur 0
     bit error state (disabled, 2) nt 0, dur 0
     sig loss state (disabled, 2) nt 0, dur 0
     sync loss state (disabled, 2) nt 0, dur 0
     link reset state (disabled, 2) nt 0, dur 0
     credit loss state (disabled, 2) nt 0, dur 0
  Operational Info - state(trunking), mode(TNP), speed(4 Gbps), trunk(on)
    32-bit state reason(2,2 - None), state reason(2,2 - None)
    phy port enable (1), phy layer (FC), port_rate_mode(dedicated)
    participating(1), fcid(0x000000), bb_scn(0)
    cached rate mode(Unknown(0))
    config port vsan (1), oper port vsan (1)
    rx bb_credit(16), rx bb_credit multiplier(0)
    rx bb_credit performance buffers(0), tx bb_credit(16)
    not first time port bringup is 1
    Port WWN is 20:47:00:05:9b:20:fa:40
    current state [PI_FSM_ST_TXPORT_INIT_TRUNKING_ENABLED]
    port_init_eval_flag(0x00002401), cfg wait for (0x200000)  n_port_multiplexer
        FC_PORT_INIT_DATA_VALID (0)
        FC_PORT_INIT_ADMIN_ST_DOWN_TO_UP (10)
        FC_PORT_INIT_FLOGI_ACC_SENT (13)
        FC_PORT_INIT_PHY_PORT_UP (49)
    Mts node id 0x202
    Lock Info: resource [fc2/7]
      type[0] p_gwrap[(nil)]
          FREE @ 818280 usecs after Sun Jul 14 13:13:27 2013
      type[1] p_gwrap[(nil)]
          FREE @ 818990 usecs after Sun Jul 14 13:13:27 2013
      type[2] p_gwrap[(nil)]
          FREE @ 840911 usecs after Sun Jul 14 13:13:27 2013
    0x100000001086000
    eport_init_flag(0x00000000), elp_chk_flag(0x00000000)
    elp_rcvd_fc2_handle(0x00000000), elp_sent_fc2_handle(0x00000000)
    esc_chk_flag(0x00000000), esc_fc2_handle(0x00000000)
    elp_flags(0x0000), classes_supported(F), tx bb_credit(0)
    Peer ELP Revision: 0
    Timer active for PI_FSM_EV_TE_2_E_TIMER_EXPIRED
    cnt_link_failure(1), cnt_link_success(6), cnt_port_up(0)
    cnt_cfg_wait_timeout(0), cnt_port_cfg_failure(0), cnt_init_retry(0)
    nos link failures (0), debounce window (300 ms), elp wait tout (750 ms)
    link fail reason (0x0)    num_elps_before_up (0)
    internal_reinit_counter (0) external_reinit_counter (0)
 last_max_reinit_reach_time  (n/a, 0)
  Port Capabilities -
    Modes: E,TE,F,SD
    Min Speed: 1000
    Max Speed: 4000
    Max Sourcable Pkt Size: 0
    Max Tx Bytes: 2112
    Max Rx Bytes: 2112
    Max Tx Buffer Credit: 64
    Rx Buffer Credit in FRM mode: Min(1), Max(240), Default(16)
    Rx Buffer Credit in OSM mode: Min(0), Max(0), Default(0)
    Rx Buffer Credit(ISL) in FRM mode: Min(1), Max(240), Default(16)
    Rx Buffer Credit(ISL) in OSM mode: Min(0), Max(0), Default(0)
    Rx Buffer Credit Multiplier in FRM mode: Min(0), Max(0), Default(0)
    Rx Buffer Credit Multiplier in OSM mode: Min(0), Max(0), Default(0)
    Rx Buffer Credit Multiplier(ISL) in FRM mode: Min(0), Max(0), Default(0)
    Rx Buffer Credit Multiplier(ISL) in OSM mode: Min(0), Max(0), Default(0)
    Rx Buffer Credit performance buffers in FRM mode: Min(1), Max(0), Default(0)
    Rx Buffer Credit performance buffers in OSM mode: Min(1), Max(0), Default(0)
    Rx Buffer Credit performance buffers(ISL) in FRM mode: Min(1), Max(0), Default(0)
    Rx Buffer Credit performance buffers(ISL) in OSM mode: Min(1), Max(0), Default(0)
    Rx Buffer Credit change allowed in FRM mode, not allowed in OSM mode
    Rx Buffer Credit performance buffer change not allowed in FRM mode, not allowed in OSM mode
    Port rate mode change not allowed
    BB_SCN processing not supported
    Max BB_SCN 0
    Max Private Devices: 0
    Hw Capabilities: 0x100
    Connector Type: 0x1
    On demand port activation license is not supported
    VF tagging capable
    first flogi from cisco (20:70:00:0d:ec:2b:3c:40)
    epp fc2 service (EPP_SERVICE_ELS_CL_3)
    is delayed trap timer active (0)
    Slowdrain not capable
  Port guard info -
    link failure ed FALSE, oper cnt 0 ts (n/a 0)
    tsv state ed FALSE, oper cnt 0 ts (n/a 0)
    bit error state ed FALSE, oper cnt 0 ts (n/a 0)
    sig loss state ed FALSE, oper cnt 0 ts (n/a 0)
    sync loss state ed FALSE, oper cnt 0 ts (n/a 0)
    link reset state ed FALSE, oper cnt 0 ts (n/a 0)
    credit loss state ed FALSE, oper cnt 0 ts (n/a 0)
    speed group convertible: 0x00 (   )
    SFP Initializing:0
  SFP info -
    Min Speed: 1000
    Max Speed: 4000
    Module Type: 8
    Connector Type: 7
    Gigabit Eth Compliance Codes: 0
    Diagmontype = 0x68
    FC Transmitter Type: Shortwave Laser(3)
    Vendor Name: CISCO-FINISAR  
    Vendor ID: 0:144:101
    Vendor Part Num: FTLF8524P2BNL-C2
    Vendor Revision Level: 0000
  Trunk Info -
    trunk vsans (allowed active) (1,100)
    trunk vsans (operational allowed) (1,100)
    trunk vsans (up) (1)
    trunk vsans (isolated) (100)
  TE port per vsan information
  fc2/7, Vsan 1 - state(up), state reason(None), fcid(0x000000)
    port init flag(0x0), rcvd_dm_eport_rdy 0
    Lock Info: resource [fc2/7, vsan 1]
      type[0] p_gwrap[(nil)]
          FREE @ 817623 usecs after Sun Jul 14 13:13:27 2013
      type[1] p_gwrap[(nil)]
          FREE @ 817624 usecs after Sun Jul 14 13:13:27 2013
      type[2] p_gwrap[(nil)]
          FREE @ 817625 usecs after Sun Jul 14 13:13:27 2013
    0x400000101086000
current state [TE_FSM_ST_F_PORT_UP]
    RNID info not found.
    first time elp: 1
    Peer ELP Revision: 0
    Peer sw_wwn runtime tlv: 00:00:00:00:00:00:00:00
    Peer port_wwn runtime tlv: 00:00:00:00:00:00:00:00
  fc2/7, Vsan 100 - state(down), state reason(Isolation due to vsan not configured on peer), fcid(0x000000)
    port init flag(0x0), rcvd_dm_eport_rdy 0
    Lock Info: resource [fc2/7, vsan 100]
      type[0] p_gwrap[(nil)]
          FREE @ 817649 usecs after Sun Jul 14 13:13:27 2013
      type[1] p_gwrap[(nil)]
          FREE @ 817650 usecs after Sun Jul 14 13:13:27 2013
      type[2] p_gwrap[(nil)]
          FREE @ 817651 usecs after Sun Jul 14 13:13:27 2013
    0x400006401086000
current state [TE_FSM_ST_ISOLATED_VSAN]
    RNID info not found.
    first time elp: 0
    Peer ELP Revision: 0
    Peer sw_wwn runtime tlv: 00:00:00:00:00:00:00:00
    Peer port_wwn runtime tlv: 00:00:00:00:00:00:00:00
  Pacer Info -
    Pacer state: released credits
PQP: not in progress



 Time for some FCoE Troubleshooting:


n5k1# show int vfc16 trunk vsan
vfc16 is trunking
    Vsan 1 is down (Vlan not FCoE Enabled)
    Vsan 11 is down (Error Disabled - VLAN L2 down on Eth interface)
    Vsan 12 is down (Initializing)


 This command let's you see what is up and what is down and why

This is just a general useful command:

n5k1# show fcoe
Global FCF details
        FCF-MAC is 00:05:73:cd:73:60
        FC-MAP is 0e:fc:00
        FCF Priority is 128
        FKA Advertisement period for FCF is 8 seconds

VFC MAC details
        vfc16 FCF-MAC is 00:05:73:cd:73:77



Here is some more good troubleshooting info:

n5k1# show platform software fcoe_mgr info global
FCOE-Mgr module
Low Priority Pending queue: len(0), max len(1) [Sat Jul 13 21:14:27 2013]
High Priority Pending queue: len(0), max len(32) [Sat Jul 13 21:14:27 2013]
Log Buffer:
Send fcoe_mgr_tx_fip_pkt for 1a00f000  with bpdu 0

FCOE-Mgr database
================
------ Global Config Data ------
================
    FCMAP: 0xefc00
    FCF Priority: 128
    FKA Adv Period: 8
    is ca system: 0
    VE Loopback : disabled
------VLAN Info------
================
Info for VLAN 11
fcoe_enabled 1
vsan_id: 11
Info for VLAN 12
fcoe_enabled 1
vsan_id: 12
------FCF Info------
================
FCF 
    FCF pointer Addr: 0x83c91ac
    FCF MAC Addr: 00:05:73:cd:73:60
    FCF Num Pinned by NPM: 0
    FCF Num Pinned by FIP: 0
    FCF Priority Offset : 0
    FCF disable-fka: 0
          List of Active VSANS
    FCF Vsan: 11
    FCF Switch WWN: 20:0b:00:05:73:cd:73:81
    FCF Fabric WWN: 20:0b:00:05:73:cd:73:81
    FCF Vsan: 12
    FCF Switch WWN: 20:0c:00:05:73:cd:73:81
    FCF Fabric WWN: 20:0c:00:05:73:cd:73:81
------Fex Info ------
======================
-----  Vfc DB ------
======================
vfc16(0x845dd54), if_index: 0x1e00000f, VFC RID vfc16
  FSM current state: FCOE_MGR_VFC_ST_PHY_UP
  PSS Runtime Config:-
      Type: 3
      Bound IF: Eth1/16
      FCF Priority: 128 (Global)
      Disable FKA: 0
  PSS Runtime Data:-
      IOD: 0x00000000, WWN: 20:0f:00:05:73:cd:73:bf
      Created at: Sat Jul 13 21:00:40 2013

      FC Admin State: up
      Oper State: up, Reason: down
      Eth IF Index: Eth1/16
      Port Vsan: 1
      Port Mode: E port
      Config Vsan: 1,11-12
      Oper Vsan: 12
      Solicits on vsan:
      Isolated Vsan:
      FIP Capable ? : TRUE
      UP using DCBX ? : FALSE
      Peer MAC : 00:00:00:00:00:00

  PSS VN Port data:-



If Peer MAC is down you know it's something to do with your VLAN not being trunked correctly between the E ports or something else that is upsetting the config


Here's how to troubleshoot F ports:



n5k1# show platform software fcoe_mgr info interface vfc1
vfc1(0x847bd44), if_index: 0x1e000000, VFC RID vfc1
  FSM current state: FCOE_MGR_VFC_ST_PHY_UP
 
PSS Runtime Config:-
      Type: 3
      Bound IF: Eth1/1
      FCF Priority: 128 (Global)
      Disable FKA: 0
  PSS Runtime Data:-
      IOD: 0x00000000, WWN: 20:00:00:05:73:cd:73:bf
      Created at: Sat Jul 13 21:38:49 2013

      FC Admin State: up
      Oper State: up, Reason: down
      Eth IF Index: Eth1/1
      Port Vsan: 11
      Port Mode: F port
      Config Vsan: 1,11-12
      Oper Vsan: 11
      Solicits on vsan: 11
      Isolated Vsan:
      FIP Capable ? : TRUE

      UP using DCBX ? : FALSE
      Peer MAC : 00:c0:dd:13:ce:c5
  PSS VN Port data:-
     FC ID 0x620000 -
     vfc  index 503316480 vfc name vfc1
     vsan id 11 
     enode_mac 00:c0:dd:13:ce:c5
     vfc wwn 21:00:00:c0:dd:13:ce:c5


>>>>FSM: vfc1> has 27 logged transitions<<<<<

1) FSM:vfc1> Transition at 511293 usecs after Sat Jul 13 21:38:49 2013
    Previous state: [FCOE_MGR_VFC_ST_INIT]
    Triggered event: [FCOE_MGR_VFC_EV_CREATE]
    Next state: [FCOE_MGR_VFC_ST_CREATE_WAIT]

2) FSM:vfc1> Transition at 515727 usecs after Sat Jul 13 21:38:49 2013
    Previous state: [FCOE_MGR_VFC_ST_CREATE_WAIT]
    Triggered event: [FCOE_MGR_VFC_EV_CREATE_SUCC]
    Next state: [FCOE_MGR_VFC_ST_CREATED]

3) FSM:vfc1> Transition at 564684 usecs after Sat Jul 13 21:38:50 2013
    Previous state: [FCOE_MGR_VFC_ST_CREATED]
    Triggered event: [FCOE_MGR_VFC_EV_BIND]
    Next state: [FCOE_MGR_VFC_ST_BIND_WAIT]

4) FSM:vfc1> Transition at 565140 usecs after Sat Jul 13 21:38:50 2013
    Previous state: [FCOE_MGR_VFC_ST_BIND_WAIT]
    Triggered event: [FCOE_MGR_VFC_EV_BIND_SUCC]
    Next state: [FCOE_MGR_VFC_ST_DOWN]

5) FSM:vfc1> Transition at 886640 usecs after Sat Jul 13 21:38:50 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

6) FSM:vfc1> Transition at 988430 usecs after Sat Jul 13 21:38:50 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

7) FSM:vfc1> Transition at 90261 usecs after Sat Jul 13 21:38:51 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

8) FSM:vfc1> Transition at 302365 usecs after Sat Jul 13 21:38:57 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

9) FSM:vfc1> Transition at 404137 usecs after Sat Jul 13 21:38:57 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

10) FSM:vfc1> Transition at 505973 usecs after Sat Jul 13 21:38:57 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

11) FSM:vfc1> Transition at 718044 usecs after Sat Jul 13 21:39:03 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

12) FSM:vfc1> Transition at 819860 usecs after Sat Jul 13 21:39:03 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

13) FSM:vfc1> Transition at 921695 usecs after Sat Jul 13 21:39:03 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

14) FSM:vfc1> Transition at 755521 usecs after Sat Jul 13 21:39:04 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_BRING_DOWN_EVAL]
    Next state: [FSM_ST_NO_CHANGE]

15) FSM:vfc1> Transition at 756358 usecs after Sat Jul 13 21:39:04 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_BRING_DOWN_EVAL]
    Next state: [FSM_ST_NO_CHANGE]

16) FSM:vfc1> Transition at 133805 usecs after Sat Jul 13 21:39:10 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

17) FSM:vfc1> Transition at 235569 usecs after Sat Jul 13 21:39:10 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

18) FSM:vfc1> Transition at 342696 usecs after Sat Jul 13 21:39:10 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]



Here it took a little while for the VLAN discovery to work but eventually it did

19) FSM: Transition at 347586 usecs after Sat Jul 13 21:39:12 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_BRING_UP_EVAL]
    Next state: [FSM_ST_NO_CHANGE]

20) FSM: Transition at 347624 usecs after Sat Jul 13 21:39:12 2013
    Previous state: [FCOE_MGR_VFC_ST_DOWN]
    Triggered event: [FCOE_MGR_VFC_EV_BRING_UP]
    Next state: [FCOE_MGR_VFC_ST_BRINGUP_WAIT]

21) FSM: Transition at 354862 usecs after Sat Jul 13 21:39:12 2013
    Previous state: [FCOE_MGR_VFC_ST_BRINGUP_WAIT]
    Triggered event: [FCOE_MGR_VFC_EV_BRING_UP_SUCC]
    Next state: [FCOE_MGR_VFC_ST_PHY_UP]

22) FSM: Transition at 363252 usecs after Sat Jul 13 21:39:12 2013
    Previous state: [FCOE_MGR_VFC_ST_PHY_UP]
    Triggered event: [FCOE_MGR_VFC_EV_BRING_UP_EVAL]
    Next state: [FSM_ST_NO_CHANGE]

23) FSM: Transition at 364318 usecs after Sat Jul 13 21:39:12 2013
    Previous state: [FCOE_MGR_VFC_ST_PHY_UP]
    Triggered event: [FCOE_MGR_VFC_EV_BRING_UP_EVAL]
    Next state: [FSM_ST_NO_CHANGE]

24) FSM: Transition at 364355 usecs after Sat Jul 13 21:39:12 2013
    Previous state: [FCOE_MGR_VFC_ST_PHY_UP]
    Triggered event: [FCOE_MGR_VFC_EV_BRING_UP]
    Next state: [FSM_ST_NO_CHANGE]

25) FSM: Transition at 368844 usecs after Sat Jul 13 21:39:12 2013
    Previous state: [FCOE_MGR_VFC_ST_PHY_UP]
    Triggered event: [FCOE_MGR_VFC_EV_BRING_UP_EVAL]
    Next state: [FSM_ST_NO_CHANGE]

26) FSM: Transition at 549515 usecs after Sat Jul 13 21:39:16 2013
    Previous state: [FCOE_MGR_VFC_ST_PHY_UP]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_VLAN_DISCOVERY]
    Next state: [FSM_ST_NO_CHANGE]

27) FSM:vfc1> Transition at 550376 usecs after Sat Jul 13 21:39:16 2013
    Previous state: [FCOE_MGR_VFC_ST_PHY_UP]
    Triggered event: [FCOE_MGR_VFC_EV_FIP_SOLICITATION]
    Next state: [FSM_ST_NO_CHANGE]


    Curr state: [FCOE_MGR_VFC_ST_PHY_UP]

PROTOS Info:
vfc1(0x847d9fc), if_index: 0x1e000000, Proto RID 11, 00:c0:dd:13:ce:c5
  FSM current state: FCOE_MGR_PROTO_ST_UP
  PSS Runtime Data:-
      Eth IF Index: Eth1/1
      Port Mode: Unknown(0)
      FKA check enabled ? : FALSE
      Recv Multicast solicitation from peer? : FALSE
      Recv Unicast advertisement from peer? : FALSE
      Advertisement period from peer? : 0 ms
      Proto number of  devices : 0
  FIP FKA event count : 0


>>>>FSM: 11, 00:c0:dd:13:ce:c5> has 3 logged transitions<<<<<

1) FSM:11, 00:c0:dd:13:ce:c5> Transition at 651964 usecs after Sat Jul 13 21:39
:16 2013
    Previous state: [FCOE_MGR_PROTO_ST_INIT]
    Triggered event: [FCOE_MGR_PROTO_EV_FIP_FLOGI]
    Next state: [FCOE_MGR_PROTO_ST_BRUP_WAIT]

2) FSM:11, 00:c0:dd:13:ce:c5> Transition at 652063 usecs after Sat Jul 13 21:39
:16 2013
    Previous state: [FCOE_MGR_PROTO_ST_BRUP_WAIT]
    Triggered event: [FCOE_MGR_PROTO_EV_FIP_FLOGI]
    Next state: [FCOE_MGR_PROTO_ST_FC2_SEND]

3) FSM:11, 00:c0:dd:13:ce:c5> Transition at 662480 usecs after Sat Jul 13 21:39
:16 2013
    Previous state: [FCOE_MGR_PROTO_ST_FC2_SEND]
    Triggered event: [FCOE_MGR_PROTO_EV_FC2_DONE]
    Next state: [FCOE_MGR_PROTO_ST_UP]


    Curr state: [FCOE_MGR_PROTO_ST_UP]


n5k1#      



OK let's talk a bit more about FC Ports, so another thing to see is:

Link not connected when your sure there is connectivity could potentially be a speed issue or one end is suspended, the OLS and NOR (NOR meaning stop operation) can both be quite telling



MDS2(config-if)# show int fc2/1
fc2/1 is down (Offline)
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:41:00:0d:ec:2b:3c:40
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port vsan is 1
    Receive data field Size is 2112
    Beacon is turned off
    5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    5 minutes output rate 16 bits/sec, 2 bytes/sec, 0 frames/sec
      10 frames input, 148 bytes
        0 discards, 0 errors
        0 CRC,  0 unknown class
        0 too long, 0 too short
      9 frames output, 860 bytes
        4 discards, 0 errors
      3 input OLS, 3 LRR, 0 NOS, 12 loop inits
      10 output OLS, 8 LRR, 1 NOS, 4 loop inits


In this exampleOffline state, this actually means that there is a mismatch between one end of the ports and the other, they where unable to negotiate a port mode, one of them is E and one of them is F.


1 comment:

  1. I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.view vmware openings in hyderabad for experienced.

    ReplyDelete

Popular old posts.