Verifying disk subsystem performance   

Sorry about the crosspost - this forum looks a little more taylored to my 
question.  We've been having issues with the Outlook pop-up that talks about 
requesting 
information from the server, etc. and general slowness in Outlook such as 
moving between folders, etc.  I've read through the Optimizing Storage for 
Exchange 2003 whitepaper and have collected the information to the best of my 
knowledge.  I'd like to see if anyone can verify the information for me.  
We're using Exchange 2003 SP2 active/passive cluster on a NetApp FAS960 over 
iSCSI.

IOPS/sec = 507.8  (sum of Disk Transfer/sec counters from database luns that 
reside on the same volume.  These include Mailstore 1, 2, 3, and 4 from each 
corresponding Storage Group)  

DB IOPS/mailbox = 1.22  (IOPS/sec divided by 415 mailboxes)

Megacycles/mailbox = 2.17 ((Number of Processors multipled by speed of 
processors, multiplied by peak average CPU utilization(percentage)) divided 
by the number of mailboxes.

So,
Mailbox Count: 415
Peak DB IOPS: (415 × .70) + 20 % = 607.5
Peak Log IOPS: (DB IOPS/10) + 20% = 60.7
Peak megacycles: (415 × 2.17) + 20% = 1,080.66

My filer configuration can support 1,360 IOPS/sec:
16 * 85 = 1,360 (# of spindles * Disk speed = IOPS/Sec )

To me it looks like the storage is more than adequately sized, but there are 
a number of things that puzzle me such as how megacycles figure into the 
sizing equation.  What is this counter used for?

Also, it's confusing to me on how exactly the counter should be taken when 
there are multiple Storage Groups and Mailbox Stores that span several 
different filer volumes.  The equations start to make less sense to me when 
this is considered...

date: Tue, 9 May 2006 09:33:01 -0700   author:   SilverICE

Re: Verifying disk subsystem performance   

Megacycles comes into play when cizing the CPU of the host.

You do mention a 960, presumably a FC drives.  What version of data ONTAP, 
and is this an aggregate with flexvols or is it traditional volumes?  For 
the filer,  let's assume an aggregate with a single RAID DP group consisting 
of 18 10K spindles.  18-2 (we subtract the number of parity spindles to find 
the data spindles) = 16 * 85 =1360.

From this aggregate we carve flexvols.  For simplicity each flexvol contains 
a single LUN,

Database LUN    607
Log LUN 61 (I prefer to mease this and take the higher of the measured or 
10% of database IOPS)
SMTP
Snapinfo

The only reference to SMTP IOPS in Optimizing storage is 500.  I prefer to 
measure the peak plus 20%.

The snapinfo directory is used by the netapp backup integration piece, 
Snapmanager to:

1. preserve the state of the logs.
2. preserve the metadata for the snapshot (XML file for a transportable 
snapshot)

All VSS hardware providers are required to preserve the state of the logs 
after a backup.  Netapp does this by copying the logs to a sudirectory of 
the snapinfo directory specific to the snapshot.  When verification 
(ESEUtil) is run to truncate the logs, the snapinfo directory is mounted as 
a drive, and redirected to the logs that reside there.

IO against the snapinfo directory is primarially generated by the copy 
process, and when running verification.  If you are at Exchange 2003 SP2, 
verification can be throttled.  You will generally want to measure this.

Background processes such as replication (if you use snapmirror or snapvault 
for replication or backup use the -K parameter to throttle to a given 
transfer rate), ndmp copy to tape, etc, also require an IO allocation.  It's 
best to measure this from the filer.  Starting in 7.X, you can use perfmon 
to read the filer counters, so this isn't as daunting a tast as it used to 
be when you had to read the systat output.

So:

Database 607
Logs          61
SMTP     ???
Snapinfo  ???
Replication ???
Backup     ???

Fill in the blanks.




"SilverICE"  wrote in message 
news:A2B8CE0A-4D97-4AD5-880A-3E98968DBFF9@microsoft.com...
> Sorry about the crosspost - this forum looks a little more taylored to my
> question.  We've been having issues with the Outlook pop-up that talks 
> about
> requesting
> information from the server, etc. and general slowness in Outlook such as
> moving between folders, etc.  I've read through the Optimizing Storage for
> Exchange 2003 whitepaper and have collected the information to the best of 
> my
> knowledge.  I'd like to see if anyone can verify the information for me.
> We're using Exchange 2003 SP2 active/passive cluster on a NetApp FAS960 
> over
> iSCSI.
>
> IOPS/sec = 507.8  (sum of Disk Transfer/sec counters from database luns 
> that
> reside on the same volume.  These include Mailstore 1, 2, 3, and 4 from 
> each
> corresponding Storage Group)
>
> DB IOPS/mailbox = 1.22  (IOPS/sec divided by 415 mailboxes)
>
> Megacycles/mailbox = 2.17 ((Number of Processors multipled by speed of
> processors, multiplied by peak average CPU utilization(percentage)) 
> divided
> by the number of mailboxes.
>
> So,
> Mailbox Count: 415
> Peak DB IOPS: (415  .70) + 20 % = 607.5
> Peak Log IOPS: (DB IOPS/10) + 20% = 60.7
> Peak megacycles: (415  2.17) + 20% = 1,080.66
>
> My filer configuration can support 1,360 IOPS/sec:
> 16 * 85 = 1,360 (# of spindles * Disk speed = IOPS/Sec )
>
> To me it looks like the storage is more than adequately sized, but there 
> are
> a number of things that puzzle me such as how megacycles figure into the
> sizing equation.  What is this counter used for?
>
> Also, it's confusing to me on how exactly the counter should be taken when
> there are multiple Storage Groups and Mailbox Stores that span several
> different filer volumes.  The equations start to make less sense to me 
> when
> this is considered...
>
>

date: Wed, 10 May 2006 13:26:09 -0700   author:   John Fullbright [MVP]

Re: Verifying disk subsystem performance   

Exchange is iSCSI attached to the 960.

Data ONTAP v7.x

The particular data I have listed is from a traditional volume that contains 
one lun for each of the four storage groups.  Also, all numbers are averages 
over the peak time of the week as noted in the Optimization guide.  Are you 
saying that I should collect peaks instead?  Am I correct in assuming that 
the backup/verification times should be excluded since they are complete late 
at night when there are no people working in Outlook?

We have four other mailstores (one for each of the four Storage Groups) on 
this server that have luns on an aggragrate.  This aggragate is also shared 
by another server with identical storage groups/mailstores.  Do I need to do 
anything different when collecting this data because the spindles are shared 
by more than one server?  That's a bit confusin to me.


"John Fullbright [MVP]" wrote:

> Megacycles comes into play when cizing the CPU of the host.
> 
> You do mention a 960, presumably a FC drives.  What version of data ONTAP, 
> and is this an aggregate with flexvols or is it traditional volumes?  For 
> the filer,  let's assume an aggregate with a single RAID DP group consisting 
> of 18 10K spindles.  18-2 (we subtract the number of parity spindles to find 
> the data spindles) = 16 * 85 =1360.
> 
> From this aggregate we carve flexvols.  For simplicity each flexvol contains 
> a single LUN,
> 
> Database LUN    607
> Log LUN 61 (I prefer to mease this and take the higher of the measured or 
> 10% of database IOPS)
> SMTP
> Snapinfo
> 
> The only reference to SMTP IOPS in Optimizing storage is 500.  I prefer to 
> measure the peak plus 20%.
> 
> The snapinfo directory is used by the netapp backup integration piece, 
> Snapmanager to:
> 
> 1. preserve the state of the logs.
> 2. preserve the metadata for the snapshot (XML file for a transportable 
> snapshot)
> 
> All VSS hardware providers are required to preserve the state of the logs 
> after a backup.  Netapp does this by copying the logs to a sudirectory of 
> the snapinfo directory specific to the snapshot.  When verification 
> (ESEUtil) is run to truncate the logs, the snapinfo directory is mounted as 
> a drive, and redirected to the logs that reside there.
> 
> IO against the snapinfo directory is primarially generated by the copy 
> process, and when running verification.  If you are at Exchange 2003 SP2, 
> verification can be throttled.  You will generally want to measure this.
> 
> Background processes such as replication (if you use snapmirror or snapvault 
> for replication or backup use the -K parameter to throttle to a given 
> transfer rate), ndmp copy to tape, etc, also require an IO allocation.  It's 
> best to measure this from the filer.  Starting in 7.X, you can use perfmon 
> to read the filer counters, so this isn't as daunting a tast as it used to 
> be when you had to read the systat output.
> 
> So:
> 
> Database 607
> Logs          61
> SMTP     ???
> Snapinfo  ???
> Replication ???
> Backup     ???
> 
> Fill in the blanks.
> 
> 
> 
> 
> "SilverICE"  wrote in message 
> news:A2B8CE0A-4D97-4AD5-880A-3E98968DBFF9@microsoft.com...
> > Sorry about the crosspost - this forum looks a little more taylored to my
> > question.  We've been having issues with the Outlook pop-up that talks 
> > about
> > requesting
> > information from the server, etc. and general slowness in Outlook such as
> > moving between folders, etc.  I've read through the Optimizing Storage for
> > Exchange 2003 whitepaper and have collected the information to the best of 
> > my
> > knowledge.  I'd like to see if anyone can verify the information for me.
> > We're using Exchange 2003 SP2 active/passive cluster on a NetApp FAS960 
> > over
> > iSCSI.
> >
> > IOPS/sec = 507.8  (sum of Disk Transfer/sec counters from database luns 
> > that
> > reside on the same volume.  These include Mailstore 1, 2, 3, and 4 from 
> > each
> > corresponding Storage Group)
> >
> > DB IOPS/mailbox = 1.22  (IOPS/sec divided by 415 mailboxes)
> >
> > Megacycles/mailbox = 2.17 ((Number of Processors multipled by speed of
> > processors, multiplied by peak average CPU utilization(percentage)) 
> > divided
> > by the number of mailboxes.
> >
> > So,
> > Mailbox Count: 415
> > Peak DB IOPS: (415 × .70) + 20 % = 607.5
> > Peak Log IOPS: (DB IOPS/10) + 20% = 60.7
> > Peak megacycles: (415 × 2.17) + 20% = 1,080.66
> >
> > My filer configuration can support 1,360 IOPS/sec:
> > 16 * 85 = 1,360 (# of spindles * Disk speed = IOPS/Sec )
> >
> > To me it looks like the storage is more than adequately sized, but there 
> > are
> > a number of things that puzzle me such as how megacycles figure into the
> > sizing equation.  What is this counter used for?
> >
> > Also, it's confusing to me on how exactly the counter should be taken when
> > there are multiple Storage Groups and Mailbox Stores that span several
> > different filer volumes.  The equations start to make less sense to me 
> > when
> > this is considered...
> >
> > 
> 
> 
>

date: Mon, 15 May 2006 13:48:01 -0700   author:   SilverICE

Re: Verifying disk subsystem performance   

1.  "Optimizing Storage for Exchange Server 2003" in the Section titled "How
to calculate storage IO requirements using environmental data" it
specifically says peak (point 9).  Think about it; if you use an average you
will be underperforming half the time.

2.  With SME, the backup takes a negligable amount of IO.  in Data ONTAP, we
never overwrite, so all of the blocks are already there.  Taking a snapshot
is essentially copying an inode (block map). What does take a lot of IO is
Verification.  Verification first preserves that state of the logs by
copying them to a subdirectory in the snapinfo directory, then runs eseutil.
You can run verification immediately after a backup, or you can defer
verification to a later time.  You can run verification on the host from
which the backup was taken, or from a remote host.  With Exchange 2003 SP2,
you can throttle the IO consumed by verification (this does extend the
verification time proportionately).

"SilverICE"  wrote in message
news:BEE76E64-B992-477C-A116-26AE343EF9AA@microsoft.com...
> Exchange is iSCSI attached to the 960.
>
> Data ONTAP v7.x
>
> The particular data I have listed is from a traditional volume that
> contains
> one lun for each of the four storage groups.  Also, all numbers are
> averages
> over the peak time of the week as noted in the Optimization guide.  Are
> you
> saying that I should collect peaks instead?  Am I correct in assuming that
> the backup/verification times should be excluded since they are complete
> late
> at night when there are no people working in Outlook?
>
> We have four other mailstores (one for each of the four Storage Groups) on
> this server that have luns on an aggragrate.  This aggragate is also
> shared
> by another server with identical storage groups/mailstores.  Do I need to
> do
> anything different when collecting this data because the spindles are
> shared
> by more than one server?  That's a bit confusin to me.
>
>
> "John Fullbright [MVP]" wrote:
>
>> Megacycles comes into play when cizing the CPU of the host.
>>
>> You do mention a 960, presumably a FC drives.  What version of data
>> ONTAP,
>> and is this an aggregate with flexvols or is it traditional volumes?  For
>> the filer,  let's assume an aggregate with a single RAID DP group
>> consisting
>> of 18 10K spindles.  18-2 (we subtract the number of parity spindles to
>> find
>> the data spindles) = 16 * 85 =1360.
>>
>> From this aggregate we carve flexvols.  For simplicity each flexvol
>> contains
>> a single LUN,
>>
>> Database LUN    607
>> Log LUN 61 (I prefer to mease this and take the higher of the measured or
>> 10% of database IOPS)
>> SMTP
>> Snapinfo
>>
>> The only reference to SMTP IOPS in Optimizing storage is 500.  I prefer
>> to
>> measure the peak plus 20%.
>>
>> The snapinfo directory is used by the netapp backup integration piece,
>> Snapmanager to:
>>
>> 1. preserve the state of the logs.
>> 2. preserve the metadata for the snapshot (XML file for a transportable
>> snapshot)
>>
>> All VSS hardware providers are required to preserve the state of the logs
>> after a backup.  Netapp does this by copying the logs to a sudirectory of
>> the snapinfo directory specific to the snapshot.  When verification
>> (ESEUtil) is run to truncate the logs, the snapinfo directory is mounted
>> as
>> a drive, and redirected to the logs that reside there.
>>
>> IO against the snapinfo directory is primarially generated by the copy
>> process, and when running verification.  If you are at Exchange 2003 SP2,
>> verification can be throttled.  You will generally want to measure this.
>>
>> Background processes such as replication (if you use snapmirror or
>> snapvault
>> for replication or backup use the -K parameter to throttle to a given
>> transfer rate), ndmp copy to tape, etc, also require an IO allocation.
>> It's
>> best to measure this from the filer.  Starting in 7.X, you can use
>> perfmon
>> to read the filer counters, so this isn't as daunting a tast as it used
>> to
>> be when you had to read the systat output.
>>
>> So:
>>
>> Database 607
>> Logs          61
>> SMTP     ???
>> Snapinfo  ???
>> Replication ???
>> Backup     ???
>>
>> Fill in the blanks.
>>
>>
>>
>>
>> "SilverICE"  wrote in message
>> news:A2B8CE0A-4D97-4AD5-880A-3E98968DBFF9@microsoft.com...
>> > Sorry about the crosspost - this forum looks a little more taylored to
>> > my
>> > question.  We've been having issues with the Outlook pop-up that talks
>> > about
>> > requesting
>> > information from the server, etc. and general slowness in Outlook such
>> > as
>> > moving between folders, etc.  I've read through the Optimizing Storage
>> > for
>> > Exchange 2003 whitepaper and have collected the information to the best
>> > of
>> > my
>> > knowledge.  I'd like to see if anyone can verify the information for
>> > me.
>> > We're using Exchange 2003 SP2 active/passive cluster on a NetApp FAS960
>> > over
>> > iSCSI.
>> >
>> > IOPS/sec = 507.8  (sum of Disk Transfer/sec counters from database luns
>> > that
>> > reside on the same volume.  These include Mailstore 1, 2, 3, and 4 from
>> > each
>> > corresponding Storage Group)
>> >
>> > DB IOPS/mailbox = 1.22  (IOPS/sec divided by 415 mailboxes)
>> >
>> > Megacycles/mailbox = 2.17 ((Number of Processors multipled by speed of
>> > processors, multiplied by peak average CPU utilization(percentage))
>> > divided
>> > by the number of mailboxes.
>> >
>> > So,
>> > Mailbox Count: 415
>> > Peak DB IOPS: (415  .70) + 20 % = 607.5
>> > Peak Log IOPS: (DB IOPS/10) + 20% = 60.7
>> > Peak megacycles: (415  2.17) + 20% = 1,080.66
>> >
>> > My filer configuration can support 1,360 IOPS/sec:
>> > 16 * 85 = 1,360 (# of spindles * Disk speed = IOPS/Sec )
>> >
>> > To me it looks like the storage is more than adequately sized, but
>> > there
>> > are
>> > a number of things that puzzle me such as how megacycles figure into
>> > the
>> > sizing equation.  What is this counter used for?
>> >
>> > Also, it's confusing to me on how exactly the counter should be taken
>> > when
>> > there are multiple Storage Groups and Mailbox Stores that span several
>> > different filer volumes.  The equations start to make less sense to me
>> > when
>> > this is considered...
>> >
>> >
>>
>>
>>

date: Thu, 18 May 2006 07:46:36 -0700   author:   John Fullbright [MVP] fjohn@donotspamnetappdotcom