Thank you Gabriel. I was trying to compare the current X5-2 VS the new X8
figures side by side. I got most of these for X8M-2from your suggested doc
but am not able to get a few IOPS figures for X5-2 as below. Would you
suggest some document which has this information available?
IOPS For Half Rack
Flash Read IOPS Flash Write IOPS Disk Read IOPS Disk Write IOPS
X5-2 High capacity
X8M-2 High capacity 6million 3.2million
Storage capacity for half rack
Raw Disk Storage Usable Disk Storage with High Redundancy Usable Disk
Storage with Normal Redundancy Flash Storage Number of Cores
X5-2 High capacity 672TB 210TB 280TB 42TB
X8M-2 High capacity 1176TB 349TB 477TB 179TB 224
Regards
Lok
On Wed, Feb 17, 2021 at 5:38 PM Gabriel Hanauer <gabriel.hanauer@xxxxxxxxx>
wrote:
Hello Lok,
You could look at the X8M-2 Datasheet.
https://www.oracle.com/a/ocom/docs/engineered-systems/exadata/exadata-x8m-2-ds.pdf
There is a handful of information in there.
Regards,
On Wed, Feb 17, 2021 at 7:17 AM Lok P <loknath.73@xxxxxxxxx> wrote:
Thank you so much Rajesh. It really helps.
My thought was that ~168TB Raw storage per server means 168/2= ~84TB
usable storage per cell server with Normal redundancy and 168/3=~56TB
usable storage per cell server with High redundancy. But it seems it's not
that simple and some additional amount of space is really going out of the
way.
In current X5-2 high capacity Half Rac , it looks like we are capped
around ~1 to 1.5million IOPS for flash and ~18K IOPS for hard disk. What is
the flash and hard disk IOPS limit for X8 High capacity half rac ?
And i confirmed with the infra team, we are planning to use the
19.0.0.0.0 ASM grid in new X8 keeping the Oracle database version at
11.2.0.4 only , so it means we are eligible for flex disk groups.
Regards
Lok
On Wed, Feb 17, 2021 at 1:52 AM Rajesh Aialavajjala <
r.aialavajjala@xxxxxxxxx> wrote:
Lok,
Here are the numbers as relates to "usable" storage when it comes to
X8-2 / X8M-2 Exadata
A 1/2 rack X8-2 will offer <this is 4 compute nodes + 7 storage cells>
approximately
Data Capacity (Usable) – Normal Redundancy = 477 TB
Data Capacity (Usable) – High Redundancy = 349 TB
What Grid Infrastructure version are you planning to use? You are
correct that FLEX disk groups/redundancy was introduced with GI/ASM version
12.2.0.1 and higher.
Thanks,
--Rajesh
On Mon, Feb 15, 2021 at 10:45 AM Lok P <loknath.73@xxxxxxxxx> wrote:
Thanks a lot Rajesh and Shane. It really gave valuable info about the
X8 configuration.
If I sum up the data file(v$datafile) + tempfile(v$tempfile)+log
file(v$log), it's coming as ~150TB now out of which ~45TB is showing as
free in dba_free_space. And the break up of the ~150TB is coming as ,
~144TB is data file , 1.2TB is tempfile, ~.5TB of log file. So it seems
this figure of ~150TB is the size of a single copy but not the sum of three
copies which are maintained for the HIGH redundancy. And with HIGH
redundancy we must be occupying ~150*3=~450TB of space from ASM or from
total 7 storage cells. Please correct me if I am wrong.
And as per the X8 configuration you mentioned, it looks like the new X8
half RAC, we will get ~25.6*7= ~180TB of flash (against ~41TB in current
X5) and ~168*7= ~1PB of hard disk(as opposed to ~614TB of hard disk in
current X5). So considering for large reads, we are touching max flash IOPS
of ~2million during peak load(opposed to ~1.3million IOPs limit in current
X5) but here in new X-8, we will have 4.5time more overall flash, so hope
things(both flash IOPS and Storage) will be lot better and well under the
limit/capacity on new X8.
We will try to push for making the ASM redundancy HIGH. But I was
thinking with high Redundancy what will be the maximum size of the data we
can hold with this X-8 half RAC. As in new X8 , we will be having ~1PB of
raw storage which is almost twice of current ~614TB in X-5, so what will
be the max usable storage available for our database if we go for HIGH
redundancy VS NORMAL redundancy with our new X-8 machine?
I was also checking regarding flex disk groups and seeing in a few docs
the RDBMS is asked to be minimum on 12.2, but we are currently on 11.2.0.4
, so perhaps that is not an option at this moment at least.
Regards
Lok
On Mon, Feb 15, 2021 at 6:44 PM Rajesh Aialavajjala <
r.aialavajjala@xxxxxxxxx> wrote:
Lok,
To try and answer your question about the percentage/share of flash vs
hard disk in the Exadata X8-2/X8M-2
HC Storage Cells are packaged with
12x 14 TB 7,200 RPM disks = 168 TB Raw
/ Hard Disk
4x 6.4TB NVMe PCIe 3.0 Flashcard = 25.6 TB Raw / Flash
So the ~166TB that Andy references is purely spinning drives/hard
disks - you have another ~25TB of Flash.
If your organization is acquiring the X8M-2 hardware - you will add on
1.5 TB of PMEM (Persistent Memory) to this.
You may certainly add more storage cells to your environment - elastic
configurations in Exadata is a common thing - expansions or starting with
elastic configurations is often seen. You might want to consider expanding
storage after you evaluate your new X8 configuration.
Your X5- 2 hardware has/had <the drive sizes did change mid-way
through the X5-2 generation - they started out with 12x4 TB drives and
that
was doubled to the 8 TB drives>
4 PCI Flash cards each with 1.6 TB (raw) Exadata Smart Flash Cache
12x 8 TB 7,200 RPM High Capacity disks
So you are looking at quite an increase in raw storage capacity and
Flash.
+1 to Shane's mention about not using NORMAL redundancy in a
Production configuration - FLEX disk groups are permitted in Exadata but
they are not widely used to the best of my understanding - the Oracle best
practices recommend HIGH redundancy - in fact I do not think you can
configure FLEX redundancy within OEDA as it falls outside best practices -
so you would have to tear down/recreate the disk groups manually. Or
customize the initial install prior to moving your database(s)
Thanks,
--Rajesh
On Mon, Feb 15, 2021 at 8:09 AM Shane Borden <
dmarc-noreply@xxxxxxxxxxxxx> wrote:
I think it's a mistake to go with NORMAL redundancy in a production
system. I could probably understand the argument for a test system, but
not production. How do you think a regular storage array is configured?
Likely not with a normal redundancy scheme. Aside from all of the other
things mentioned already, you are now also bound to offline patching or
if
you try to do rolling you run a risk being able to tolerate only 1 disk
failure or 1 storage server failure. Not only does this protect against
some sort of technical failure, but also the human factor that could
occur
during patching
If you must consider normal redundancy, I would go with FLEX
Diskgroups vs configuring the entire rack as normal redundancy. That
way,
if you must, then you can specify the redundancy at the table space level
rather than the disk group level. Should you change your mind later,
its a
simple alter command to change the redundancy rather than tearing down
the
entire rack and rebuilding it.
---
Thanks,
Shane Borden
sborden76@xxxxxxxxx
On Feb 15, 2021, at 1:55 AM, Lok P <loknath.73@xxxxxxxxx> wrote:
Thanks much Andy.
Yes we have an existing machine that is X5-2 half Rac. (It's
basically a full RAC machine logically splitted to two half RAC and we
have
only this database hosted on this half RAC). (And we are currently ~150TB
and keep growing so we're planning for NORMAL redundancy.)
In current X5 I am seeing its ~80TB hard disk+6TB flash disk per
storage cell. When you said *"The Exadata X8 and X8M storage cells
have 14TB disks. With 12 per cell, that's 168TB *per cell*." *Does
it mean the sum of flash+hard disk is ~168TB per cell? What is the
percentage/share of flash disk and hard disk in that?
Apart from current storage saturation, with regards to the IOPS
issue in our current X5 system, I am seeing in OEM the flash IOPS is
reaching to ~2000K for large reads and the max limit is showing somewhere
near ~1.3milion. Overall IO utilization for flash disk is showing ~75%.
Hard disk IO limit shows as ~20K and most of the time it looks to be both
small reads and large reads are staying below this limit. The overall IO
utilization is staying below ~30% for the hard disks.
Just got to know from the infra team the X8 to which we are planning
to move into is not extreme flash rather high capacity disk only(similar
to
what we have on current X5) , but considering more flash storage and hard
disk storage in each of the 7 - storage cells, we are expecting the new
X8
will satisfy the current capacity crunch both wrt space and IOPS.
And as you just mentioned in your explanation that adding more
storage cells will also help in bumping up the capacity. So should we
consider adding a few more new storage cells on and above half rac to
make
it 8 or 9 storage cells in total and if this is standard practice in the
exadata world?
Regards
Lok
On Sat, Feb 13, 2021 at 9:00 PM Andy Wattenhofer <watt0012@xxxxxxx>
wrote:
The Exadata X8 and X8M storage cells have 14TB disks. With 12 per
cell, that's 168TB *per cell*. You haven't mentioned which rack size
your
X5 machine is, but from the numbers you're showing it looks like maybe a
half rack. A half rack of X8M will come with 1PB of total disk, giving
you
over 300TB of usable space to divide between your RECO and DATA disk
groups
if you are using HIGH redundancy. That seems plenty for your 150TB
database. But if you need more, add another storage cell.
As for performance degradation from using HIGH redundancy, you need
to consider that the additional work of that extra write is being taken
on
by the storage cells. By definition the redundant block copies must go
to
separate cells. NORMAL redundancy writes to two cells and HIGH goes to
three. In aggregate, each write will be as fast as your slowest cell. So
any difference in write performance is more a function of the total
number
of cells you have to share the workload. That difference would be
diminished as you increase the number of cells in the cluster.
And of course that difference would be mitigated by the write back
cache too because writes to the flash cache are faster than writes to
disk.
Honestly, I can't imagine that Oracle would sell you an Exadata
machine where any of this would be a problem for you. It would be so
undersized from the beginning that your problems with it would be much
greater than any marginal difference in write performance from using
high
redundancy.
Andy
On Fri, Feb 12, 2021 at 10:31 AM Lok P <loknath.73@xxxxxxxxx> wrote:
Thanks Much.
I got some doc but missed. below URL also pointing to high
redundancy as requirement but may be its not compulsory as you stated.
We have the size of our database i.e. (~150TB), so we were thinking
to have some space reduction using double mirror rather triple
mirroring.
but i was not aware that the disk size itself is a lot bigger in X8
and as
you stated in X8 we have a bigger size disk and so the mirroring will
take
a lot of time(in case of crash/failure) and thus HIGH redundancy is
recommended. I think we have to relook into the same. Note- What I see
in
the current X-5 machine, we have ~6TB flash/storage server and ~80TB
hard
disk/storage server. Not sure what that is in case of X8 though.
And another doubt i had was, Is it also true that the IOPS will be
degraded by some percentage in case of tripple mirror as compared to
double
mirror as because it has to write one more additional copy of data
block
to flash/disk?
https://stefanpanek.wordpress.com/2017/10/20/exadata-flash-cache-enabled-for-write-back/
On Fri, Feb 12, 2021 at 8:52 PM Ghassan Salem <
salem.ghassan@xxxxxxxxx> wrote:
Please, can you point to where you saw that write-back is only
possible with high-redundancy?
High redundancy is very much recommended with X8 due to the size
of the disks, and the time it takes to re-mirror in case of disk
loss: if
you're in a normal redundancy, and you loose a disk, while
re-mirroring is
being done, you don't have any second copy of that data, and so, if
you
loose yet-another disk, you're in big trouble. With lower capacity
disks,
the re-mirroring takes much less time, and so the risk is lower.
regards
On Fri, Feb 12, 2021 at 3:57 PM Lok P <loknath.73@xxxxxxxxx>
wrote:
Basically i am seeing many doc stating triple mirroring is
recommended with "write back flash cache" and some other stating
write back
flash cache is not possible without HIGH redundancy/triple
mirroring. So
there is a difference between these two statements because if we
decide to
go for NORMAL redundancy to save some space and to have some IO
benefit(in
terms of not writing one more additional data block copy). But we
want to
utilize the "write back flash cache" option to get benefits on write
IOPS.
And in this case if restriction is put in place for "High
redundancy" we
won't be able to do that. Please Correct me if my understanding is
wrong?
On Fri, Feb 12, 2021 at 12:53 PM Lok P <loknath.73@xxxxxxxxx>
wrote:
Seeing in below doc which state its recommended to go for high
redundancy ASM disk group(i.e. triple mirroring) in case we are
using write
back flash cache as because the data will be first written/stay in
flash
cache and flushed to the disk later stage and in case of failure it
has to
be recovered from mirror copy. But i am wondering , is this not
possible
with double mirroring , will it not survive the data loss in case of
failure? Want to understand what is the suggested setup which will
give
optimal space usage without compromising IOPS and data loss.
https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-storage-server-software-introduction.html#GUID-E10F7A58-2B07-472D-BF31-28D6D0201D53
Regards
Lok
On Fri, Feb 12, 2021 at 10:42 AM Lok P <loknath.73@xxxxxxxxx>
wrote:
Hello Listers, We are moving from exadata X5 to X8 and there
are multiple reasons behind it. Few of them are , 1)we are almost
going to
saturate the existing storage capacity(DB size reaching ~150TB) in
current
X5. 2)And the current IOPS on X5 is also reaching its max while
the system
works during its peak load.
We are currently having HIGH redundancy(triple mirroring) for
our existing X5 machines for DATA and RECO disk group and DBFS is
kept as
NORMAL redundancy(double mirroring). Now few folks raised
questions on the
impact on IOPS and storage space consumption, if we use double
mirroring(NORMAl redundancy) vs triple mirroring(High redundancy)
in the
new X8 machine. I can see the benefit of double mirroring(Normal
redundancy) being saved in storage space(around 1/3rd in terms of
DATA and
REDO copies), but then what is the risk wrt data loss, is it okay
in a
production system? (Note- We do use ZDLRA backup for taking the DB
backup.
And for disaster recovery we have active data guard physical
standby in
place which runs in read only mode).
With regards to IOPS, we are going with default write back
flash cache enabled here. Is it correct that with double mirroring
we have
to write/read into two places VS in triple mirroring we have to do
that in
three places , so there will also be degradation in IOPS with
triple
mirroring/High redundancy as compared to double mirroring? if it's
true
then by what percentage the IOPS degradation will be there? And
then if
it's okay if we go for double mirroring as that will benefit us
wrt IOPS
and also saves a good amount of storage space?
Regards
Lok
--
Gabriel Hanauer
