Hi,
as Jure mentioned my post (thnx!) , let me add my .02€:
I learned to see "no network HB" NOT to read as "the Network has a problem"
( where "Network" is "everything between the physical plugs of the network
cards")
instead see it as "there are 2 or more processes which missed to
communicate - and those processes are likely not on the same machine".
Of course, UDP, Multicast etc make it more complex to analyze - but
checking progress & error counters on network interfaces are for me a
reasonable method to identify where to investigate.
hth
Martin
2017-01-04 9:10 GMT+01:00 Jure Bratina <jure.bratina@xxxxxxxxx>:
Hi,
Martin Berger recently posted a blog entry which discusses an issue with
similar symptoms and its resolution: http://berxblog.blogspot.si/
2016/12/interconnect-fragmentation-kills-cluster.html
Regards,
Jure Bratina
On Wed, Jan 4, 2017 at 8:13 AM, Justin Mungal <justin@xxxxxxx> wrote:
"no network HB" means that the Network Heartbeat is failing for some
reason.
This is rather anecdotal, but a RAC that my co-worker is responsible for
was evicting nodes in a similar manner (similar environment as well)
without any evident network problems. His theory was that the heartbeats
were failing because the interconnect was not responding fast enough, due
to all of the existing database activity. He enabled jumbo frames and the
problem went away. So in other words the TCP/IP stack was busy
disassembling and reassembling frames and this caused heartbeat responses
to get slower, and enabling jumbo frames reduced that overhead.
Recommendation for the Real Application Cluster Interconnect and Jumbo
Frames (Doc ID 341788.1)
You can also investigate your timeout settings and adjust them, but
Oracle generally doesn't recommend this and will probably just tell you to
install 11.2.0.4 instead.
CSS Timeout Computation in Oracle Clusterware (Doc ID 294430.1)
On Tue, Jan 3, 2017 at 4:20 PM, Yong Huang <dmarc-noreply@xxxxxxxxxxxxx>
wrote:
Oracle and GI (grid infrastructure) 11.2.0.3 on 64-bit Red Hat Linux
6.6. Cisco UCS.
Node 2 of a 2-node RAC crashed. Log ocssd.log shows:
2016-12-18 02:03:06.307: [ CSSD][499648256]clssnmPollingThread: node
d1prpcrndb1a (1) at 50% heartbeat fatal, removal in 14.760 seconds
2016-12-18 02:03:06.307: [ CSSD][499648256]clssnmPollingThread: node
d1prpcrndb1a (1) is impending reconfig, flag 2493454, misstime 15240
2016-12-18 02:03:06.307: [ CSSD][499648256]clssnmPollingThread: local
diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending
reconfig status(1)
2016-12-18 02:03:06.307: [ CSSD][510686976]clssnmvDHBValidateNcopy:
node 1, d1prpcrndb1a, has a disk HB, but no network HB, DHB has rcfg
306434975, wrtcnt, 197140394, LATS 4040636964, lastSeqNo 185041690,
uniqueness 1468029747, timestamp 1482048185/1112586906
...[some lines snipped here]...
2016-12-18 02:03:28.094: [ CSSD][510686976]clssnmvDHBValidateNcopy:
node 1, d1prpcrndb1a, has a disk HB, but no network HB, DHB has rcfg
306434975, wrtcnt, 197140475, LATS 4040658754, lastSeqNo 197140472,
uniqueness 1468029747, timestamp 1482048207/1112608986
We installed Oracle's OSWatcher and enabled traceroute for the private
network, which shows no error during the time:
zzz ***Sun Dec 18 02:03:28 CST 2016
traceroute to dcprpcrndb1bic1 (10.114.21.3), 30 hops max, 60 byte packets
1 dcprpcrndb1bic1 (10.114.21.3) 0.020 ms 0.008 ms 0.004 ms
traceroute to dcprpcrndb1bic2 (10.114.21.67), 30 hops max, 60 byte
packets
1 dcprpcrndb1bic2 (10.114.21.67) 0.020 ms 0.006 ms 0.004 ms
traceroute to d1prpcrndb1aic1 (10.114.21.2), 30 hops max, 60 byte packets
1 d1prpcrndb1aic1 (10.114.21.2) 0.262 ms 0.259 ms 0.255 ms
traceroute to d1prpcrndb1aic2 (10.114.21.66), 30 hops max, 60 byte
packets
1 d1prpcrndb1aic2 (10.114.21.66) 0.135 ms 0.123 ms 0.110 ms
If traceroute never reports a problem, what does "no network HB" in
occsd.log mean? At 02:03:28, we see both "no network HB" and successful
traceroute pings. This is not the first time we have this problem. The
network team never finds any issue, consist with the traceroute report.
OSWatcher traceroute has only basic options:
traceroute -r -F <private network IP>
where -r means "Bypass the normal routing tables and send directly to a
host on an attached network". -F means "Do not fragment probe packets".
/var/log/messages reports no problem at the time. It only starts to show
problems after the cluster already decides on eviction.
Yong Huang
--
//www.freelists.org/webpage/oracle-l
