RE: Is filesystemio_options relevant when the database is on ASM ?
- From: "Hameed, Amir" <Amir.Hameed@xxxxxxxxx>
- To: Kevin Jernigan <kevin.jernigan@xxxxxxxxxx>, Don Seiler <don@xxxxxxxxx>
- Date: Fri, 17 Oct 2014 17:12:00 +0000
dNFS beats kNFS hands down; there is no question about it. dNFS also does a
good job of load-balancing load between the dNFS interfaces. It is the various
layers that are involved in the NAS infrastructure that makes implementing dNFS
very challanging. We have had our storage vendor engaged for a long time now
but have not had much luck which leads me to believe that NAS itself has its
limitations and does not scale well when compared to FC based connectivity.
From: Kevin Jernigan [mailto:kevin.jernigan@xxxxxxxxxx]
Sent: Friday, October 17, 2014 1:06 PM
To: Hameed, Amir; Don Seiler
Cc: fuzzy.graybeard@xxxxxxxxx; oracle-l@xxxxxxxxxxxxx
Subject: Re: Is filesystemio_options relevant when the database is on ASM ?
Direct NFS (dNFS) can lead to challenges for the underlying NAS infrastructure,
because it eliminates the performance bottlenecks inherent in most OS
implementations of the NFS client (what we usually call "kernel" NFS or kNFS).
For example, dNFS will allow each Oracle process to make a direct connection to
the NAS server, rather than funneling all I/O requests through a single
connection. This means that the NAS server(s) need to be able to handle the
increase in I/O traffic that the database will request from the storage.
So, it's not a flaw in dNFS that creates problems for some NAS setups - there
are often performance limitations in the NAS server that were masked by kNFS
limitations, and dNFS removes those limitations. dNFS does extremely well in
environments that have the right setup to handle higher IOPS etc - if the
application workload on top of the database requires a certain level of I/O
throughput and latency, it's much easier to get to those levels with dNFS than
with kNFS.
-KJ
--
Kevin Jernigan
Senior Director Product Management
Advanced Compression, Hybrid Columnar
Compression (HCC), Database File System
(DBFS), SecureFiles, Database Smart Flash
Cache, Total Recall, Database Resource
Manager (DBRM), Direct NFS Client (dNFS),
Continuous Query Notification (CQN),
Index Organized Tables (IOT), Information
Lifecycle Management (ILM)
+1-650-607-0392 (o)
+1-415-710-8828 (m)
On 10/17/14, 8:13 AM, Hameed, Amir wrote:
We started using DNFS about 1 ½ years ago and have been bleeding ever since.
One of the issues is with the NAS technology from our vendor which we have
found unstable. The IO timings fluctuate too much and the 10046 traces can
easily prove it. The other issue is most likely with our network
infrastructure, which we were told initially both by our data center folks and
by our network vendor that we did not need a separate dedicate network for the
private NAS traffic and that the existing switches would provide the QoS that
is needed but now both are proposing a dedicated set of switches to try and
resolve the issue.
DNFS works fine for environments that are not very demanding on the IOs and
latency but based on our experience, it does not do well with high IOPS and low
latency requirements.
From: Don Seiler [mailto:don@xxxxxxxxx]
Sent: Thursday, October 16, 2014 4:20 PM
To: Kevin Jernigan
Cc: Hameed, Amir; fuzzy.graybeard@xxxxxxxxx<mailto:fuzzy.graybeard@xxxxxxxxx>;
oracle-l@xxxxxxxxxxxxx<mailto:oracle-l@xxxxxxxxxxxxx>
Subject: Re: Is filesystemio_options relevant when the database is on ASM ?
For what it's worth, we have the 7420 (two of them). My criticism of them
doesn't come lightly. It's been a long year+ and the fact that these problems
have kept us from migrating to them has resulted us having to deal with other
emergencies from our current aging storage that we hoped to be off of nearly a
year ago.
Don.
On Thu, Oct 16, 2014 at 3:12 PM, Kevin Jernigan
<kevin.jernigan@xxxxxxxxxx<mailto:kevin.jernigan@xxxxxxxxxx>> wrote:
Don,
Based on your experiences I think anyone would understand your perspective. My
areas of responsibility within Oracle are all within the database team, so I am
not an expert on ZFSSA (ZS3) storage, but I will contact the ZS3 product team
to see if they have any comments.
Thanks for your feedback,
-Kevin J
--
Kevin Jernigan
Senior Director Product Management
Advanced Compression, Hybrid Columnar
Compression (HCC), Database File System
(DBFS), SecureFiles, Database Smart Flash
Cache, Total Recall, Database Resource
Manager (DBRM), Direct NFS Client (dNFS),
Continuous Query Notification (CQN),
Index Organized Tables (IOT), Information
Lifecycle Management (ILM)
+1-650-607-0392<tel:%2B1-650-607-0392> (o)
+1-415-710-8828<tel:%2B1-415-710-8828> (m)
On 10/16/14, 12:59 PM, Don Seiler wrote:
Of course, it's entirely possible that the two ZFSSA units that we received are
the only two lemons off of the assembly line. That would explain by the ZFSSA
support techs had so much trouble finding/fixing the problems (most of which
are still not fixed).
Last summer I couldn't have been more excited to get these units installed and
start using them. But it was clear from the start that things were going wrong,
and it was a series of problems from the start in various components of the
ZFSSA. Even better was the fact that the two units each had unique problems,
failing differently than their counterpart.
So, given the problems we've seen, considering both the quantity and severity,
and how completely unimpressed we were with the "one-stop shop" for support
that was one of the big selling points, we can in no way consider moving our
production databases onto it, and are looking for alternative storage to remove
our staging and unit test databases off of it. Even if/when the open bugs get
fixed and someone discovers the cause of the NFS hangs, the trust is completely
gone in these systems.
Don.
On Thu, Oct 16, 2014 at 2:41 PM, Don Seiler
<don@xxxxxxxxx<mailto:don@xxxxxxxxx>> wrote:
We've had nothing but problems with them from installation misconfigurations to
hardware failures (two silent NIC failures in addition to the disk failures we
get warnings about, and one instance of the entire appliance IO module
crashing). The last few months we'd see intermittent NFS hanging for 5-6
minutes to all mounts from the ZFSSA, resulting in those databases crashing.
That is the most worrisome. If the storage can't stay online, then we have
bigger problems than worrying about speed.
Add on to that the less than stellar support recommendations we've gotten
flip-flopping around Infiniband recommendations, write-bias setttings, bonding
configurations, etc. and it's been a complete nightmare that's left us still on
our old storage that is starting to fail. I have zero confidence in the ZFSSA
(at least the two machines that we've been sold) to run our database. This is
14 months after installation and they're still not in production and never will
be.
You say you have many customers on it, but we found this hard to believe given
all of the bugs that we tripped over along the way, including a couple that
were apparently discovered by us. We would have expected support to give us a
heads-up about all of the needed patches if they have so many customers that
have done the same thing. Other than the recommended Direct NFS patches MOS
note, we've been basically stumbling around in the dark. At various points,
support suggests patches that only *might* fix the problem ... as if they
aren't sure themselves. And those patches don't ever fix the problem.
Don.
On Thu, Oct 16, 2014 at 2:18 PM, Kevin Jernigan
<kevin.jernigan@xxxxxxxxxx<mailto:kevin.jernigan@xxxxxxxxxx>> wrote:
Don,
Why do you recommend not using ZFSSA for live databases yet? We (Oracle) have
many customers using ZFSSA for live production environments - including our own
IT department, with >200PBs of ZFSSA storage in place for applications that
support all aspects of our business...
-Kevin J
--
Kevin Jernigan
Senior Director Product Management
Advanced Compression, Hybrid Columnar
Compression (HCC), Database File System
(DBFS), SecureFiles, Database Smart Flash
Cache, Total Recall, Database Resource
Manager (DBRM), Direct NFS Client (dNFS),
Continuous Query Notification (CQN),
Index Organized Tables (IOT), Information
Lifecycle Management (ILM)
+1-650-607-0392<tel:%2B1-650-607-0392> (o)
+1-415-710-8828<tel:%2B1-415-710-8828> (m)
On 10/16/14, 11:32 AM, Don Seiler wrote:
Yes in 12c DNFS works on NFSv4. In fact, NFSv4 is required if you plan to use
OISP (Oracle Intelligent Storage Protocol) to talk to their ZFSSA. Although I
would not suggest using the ZFSSA to run live databases yet. Should be OK for
FRA uses.
Don.
On Thu, Oct 16, 2014 at 10:46 AM, Kevin Jernigan
<kevin.jernigan@xxxxxxxxxx<mailto:kevin.jernigan@xxxxxxxxxx>> wrote:
As of Oracle Database 12c, dNFS works with both NFSv3 and NFSv4...KJ
--
Kevin Jernigan
Senior Director Product Management
Advanced Compression, Hybrid Columnar
Compression (HCC), Database File System
(DBFS), SecureFiles, Database Smart Flash
Cache, Total Recall, Database Resource
Manager (DBRM), Direct NFS Client (dNFS),
Continuous Query Notification (CQN),
Index Organized Tables (IOT), Information
Lifecycle Management (ILM)
+1-650-607-0392<tel:%2B1-650-607-0392> (o)
+1-415-710-8828<tel:%2B1-415-710-8828> (m)
On 10/16/14, 8:37 AM, Hameed, Amir wrote:
I don’t believe DNFS is certified to work with NFSv4.
From: oracle-l-bounce@xxxxxxxxxxxxx<mailto:oracle-l-bounce@xxxxxxxxxxxxx>
[mailto:oracle-l-bounce@xxxxxxxxxxxxx] On Behalf Of Hans Forbrich
Sent: Thursday, October 16, 2014 11:30 AM
To: oracle-l@xxxxxxxxxxxxx<mailto:oracle-l@xxxxxxxxxxxxx>
Subject: Re: Is filesystemio_options relevant when the database is on ASM ?
On 16/10/2014 3:35 AM, Frits Hoogland wrote:
When using NFS underneath ASM, I've witnessed filesystemio_options being
honoured by the database, which means it needs setting it to 'setall' for the
combination AIO+DIO. Which makes sense, because you need to create a file on a
(NFS) filesystem to be used as ASM disk device.
Then it becomes important to know which NFS?
I believe DNFS behaves different than standard NFSv3 which may be different
again from NFSv4
/Hans
--
Don Seiler
http://www.seiler.us
--
Don Seiler
http://www.seiler.us
--
Don Seiler
http://www.seiler.us
--
Don Seiler
http://www.seiler.us
Other related posts:
