Can't you use ASM with DNFS? I wouldn't think it would be impossible. I really
like the ease of adding space to ASM.
Sent from my iPhone
On Mar 28, 2016, at 4:10 PM, Jeff Chirco <backseatdba@xxxxxxxxx> wrote:
It is funny I am in the opposite debate as you. I am switching from Windows
to Linux. Currently we have our luns mapped using ISCSI and not using ASM
with NetApp. I was thinking of going with dNFS on the new server and not
using ASM since I have no experience with it and not sure what benefit it
gives me and do I want to complicate my environment even more with the move.
I know that you can not compare the two directly as ASM is a LVM. So now I
am more confused. Do we stay with iscsi or go with dNFS and do we use ASM or
not. Currently running EE 11.2.0.4 (looking at 12c this year), single
instance although RAC is being debated for the future year or two.
On Wed, Feb 24, 2016 at 12:08 PM, Ruel, Chris <Chris.Ruel@xxxxxxx> wrote:
I want to thank everyone who took the time to respond. It was very
insightful information and I learned a few things. While researching this
myself, I did find that the dNFS vs. ASM debate is not a religious war like
many things. Not too many people seem to have a strong opinion yet one way
or another. Don't get me wrong, there are proponents for both sides but no
one seems ready to sacrifice their first born for their beliefs…strange
behavior for internet debate…I applaud it. It does make it easier for me to
see the facts without too much emotion behind them.
One thing I am beginning to wonder is I am not sure it is time for my
company to consider dNFS. For one, I have yet to find any really compelling
reasons to switch…key word is YET…I am still researching and have just begun
testing myself. One big reason to keep our current ASM configuration is
because that is what my company is very familiar with and it is used in
every database we have. Not sure I need to “fix what isn't broken”. Also,
moving to dNFS will mostly likely introduce 2-3 years of having both ASM and
dNFS as we migrate our systems from one to the other…this would require us
to support two configurations.
Let me give some feedback on some of the replies I got…
Mladen: That depends on how do you do ASM. If the drives are iSCSI on a
machine without the proper HBA, then dNFS is a clear choice, since it's much
easier to administer and will even perform better than iSCSI. For FC
connections and iSCSI with the proper HBA, ASM will perform better. Since
RAC is ALWAYS about performance, you should choose what performs better.
Generating an artificial load similar to your workload by Swingbench or
HammerOra should provide a good benchmark.
I will have to go back and review our HBA setup to make sure I understand it
and that it is "right". So far, in my testing using Swingbench, I have
found throughput to pretty even between both ASM and dNFS. However, I think
I need to drive more activity to push the underlying storage to see which
one falls off first.
Seth:
You do not need ASM for OCR and voting disks. These can be on a supported
cluster file system or over standard NFS.
I did not know this. I think non-ASM OCR/voting was only supported for
upgrades of the clusterware from <11.2 to 11.2. I have never tried
launching the GI installer without candidate ASM disks ready which the GI
installer will then launch into the CRS Disk Group setup screen. Even the
documentation says block and raw devices are no longer supported but I guess
NFS is not considered a block storage device?
http://docs.oracle.com/cd/E11882_01/install.112/e41961/storage.htm#CWLIN312
If I am interpreting this incorrectly, I am open to a learning moment here!
It is unclear to me why storage snapshots for ASM disk groups required you
to use RDMs. Could you not snap multiple VMDKs at the same time?
We tried this but had trouble getting it to work. Could be a problem with
our set up and not enough knowledge but we have some pretty good NetApp and
VMware folks.
All that being said, dNFS has lots of benefits over ASM as well and as I
assume you were alluding to, is not mutually exclusive to ASM. NFS in
general is obviously much more flexible than ASM including the ability to
use CloneDB.
Yes! And this is what attracts us to dNFS but we want to make sure we
understand what, if anything we are giving up by ditching ASM…especially in
terms of performance.
Amir: If you have an IO intensive system, you may want to stick with FC.
dNFS has been working fine for us for those systems that do not do a lot of
throughput, like SOA databases, etc. However, for heavy-duty ERP systems,
even though we have implemented 10gbe end-to-end (from hosts to switches to
NAS/heads), we are barely meeting the performance. All of our vendors,
including Oracle, storage vendor and network vendor looked at their
infrastructure for literally months but no one was able to pinpoint where
the bottleneck was coming from. We ended up moving two of our Oracle ERP
systems back to FC and will move the remaining ERP systems in the near
future.
This is the sort of thing we are afraid of encountering.
Kyle: NFS is the future, has larger bandwidth than FC, market is growing
faster than FC, cheaper, easier, more flexible, cloud ready and improving
faster than FC.
In my benchmarking, FC and NFS, throughput and latency are on par given
similar speed NICs and HBAs and properly setup network fabric.
Simple issues like having routers on the NFS path can kill performance.
Latency:
NFS has a longer code path than FC and with it comes some extra latency but
usually not that much. In my tests one could push 8K over 10GbE in about
200us with NFS where as over FC you can get it around 50us. Now that's 4x
slower on NFS but that's without any disk I/O. If disk I/O is 6.00 ms then
adding 0.15ms transfer time is lost in the wash. That on top of the issue
that FC is often not that tuned so what could be done in 50us ends up taking
100-150us and is alms the same as NFS.
I've heard of efforts are being made to shorten NFS code path, but don't
have details.
Throughput
NFS is awesome for throughput. It's easy to configure and on things like
VMware it is easy to bond multiple NICs. You can even change the config
dynamically while the VMs are running.
NFS is already has 100GbE NICs and is shooting for 200GbE next year.
FC on the other hand has just gotten 32G and doesn't look like that will
start to get deployed until next year and even then will be expensive.
Analyzing Performance on NFS
If you are having performance issues on NFS and can't figure out why, one
cool thing to do is take tcpdump on the receiver as well as sender side and
compare the timings. The problem is either the sender, network or receiver.
Once you know which the analysis can be dialed in.
Thanks for the links and the thoughtful insight…again, more of what I am
looking for. It does indeed sound like NFS will be a better choice in the
future…but, is it enough reason for us to consider switching right now? For
one, we just got 10gE…100gE is not even a twinkle in our infrastructure's
eye as far as I know. As long as it performs on par with FC and the
flexibility and available features with dNFS pan out, that could be reason
enough to switch…tough decisions ahead.
Stefan: My clients are using both ASM with FC and dNFS or kNFS for older
Oracle releases.
I recently did an I/O benchmark at a client environment (VSphere 6, OEL 6.7
as guest, Oracle 12c, NetApp NFS, 10GE, no Jumbo Frames, W-RSIZE 64k) with
SLOB and we reached out close to the max of 1GB/s by an average single block
I/O performance of 4 ms (if it was coming from disk it was round about
8-10 ms and the other stuff was coming from storage cache).
I just comment some of your points.
2a) You can do this with ASM or dNFS by RMAN. I highly recommend that you do
not rely on storage snapshot / backup mechanism only as you will not notice
any physical or logical block corruption until it may be too late. Trust me
i have seen more than enough of such cases.
4b) When you are using dNFS in a VMWare environment for Oracle you have no
VMDKs for the Oracle files (data,temp,control,redo,arch) at all. You map the
NFS share directly into the VM and access it via dNFS inside the VM. You
only have VMDKs for the OS (and Oracle software) for example. In addition to
scale with dNFS you may not do NIC teaming on VMware level, but rather put
each interface into the VM and let dNFS do all the load balancing, etc.
(e.g. ARP).
In sum nowadays there is no reason to demonize NFS for Oracle (with dNFS).
It works very well with good performance (FC like).
… i am a kid from the FC decade and i am saying this ;-)
Thanks for your experience and comments. We are not using Snaps for our
total backup solution…we still use RMAN as a first priority. Snaps are
there if we can use them and for cloning. However, I am glad you reminded
me of that fact as I have been considering coming up with a snap only
strategy for our larger databases (as long as we can mirror the snaps to a
geographically separate site). I see I will have to remember to run RMAN
commands (or DBV) to make sure corruption is not an issue.
Chris..
_____________________________________________________________________
Chris Ruel * Oracle Database Administrator * Lincoln Financial Group
cruel@xxxxxxx * Desk:317.759.2172 * Cell 317.523.8482
Notice of Confidentiality: **This E-mail and any of its attachments may
contain
Lincoln National Corporation proprietary information, which is privileged,
confidential,
or subject to copyright belonging to the Lincoln National Corporation family
of
companies. This E-mail is intended solely for the use of the individual or
entity to
which it is addressed. If you are not the intended recipient of this E-mail,
you are
hereby notified that any dissemination, distribution, copying, or action
taken in
relation to the contents of and attachments to this E-mail is strictly
prohibited
and may be unlawful. If you have received this E-mail in error, please
notify the
sender immediately and permanently delete the original and any copy of this
and any printout. Thank You.**
