|
|
|
date: Thu, 15 Nov 2007 11:54:06 +0100,
group: microsoft.public.exchange.design
back
Storage problems in Exchange 2003
Hi,
I was recently put in charge of our Exchange servers and all the problems
around it (the biggest one is "Requesting data from the Exchange
server..."). I'm pretty sure it is because the storage so I have tried to
calculate the needed IOPS to show that what we have today is simply not
enough. We have a SAN and that is where I want to place the databases now
but it comes with some costs (buying HBAs), that is why I would like someone
to confirm my calculations are correct so I can get the money to connect the
servers to the SAN.
The servers have the following disk setup
2 SATA 7200rpm, RAID1 for OS and Transaction logs
4 SATA 7200rpm RAID5 for DB
Available disk in the SAN: 4 FC 10k rpm RAID10
Or 5 FC 10k rpm RAID5
Other numbers of interest from perfmon during peak hours
Physical Disk
Avg disk sec/read: average - 0,084 ; max - 8,521
Avg disk sec/transfer: average - 0,053 ; max - 7,299
Avg disk sec/write: average - 0,060 ; max - 8,048
Disk transfers/sec: average - 366 ; max 6541
MSExchangeIS
Active User Count: average - 1116
This is what I have come up with (assuming 50% read and 50% write) and these
are the sites I have used to do the math.
http://storageadvisors.adaptec.com/2007/03/20/sata-iops-measurement/
http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance
SATA 7200 rpm:
Avg seek time 8,5ms, avg write time +2ms, so avg time 9,5ms
9,5ms + 4,1ms (half rotation to position the head) = 13,6ms or 73,5 IOPS
FC 10k rpm:
Avg seek time 4,2ms, avg write time +2ms, so avg time 5,2ms
5,2ms + 3ms (half rotation to position the head) = 8,2ms or 121,9 IOPS
SATA RAID5 (4 disks)
73,5 * 0,57 (RAID5 penalty) * 4 = 167,58 IOPS
FC RAID5 (5 disks)
121,9 * 0,57 * 5 = 347,41 IOPS
FC RAID10 (4 disks)
121,9 * 0,8 (RAID10 penalty) * 4 = 390,08 IOPS
I know I haven't accounted for Exchange DB is usually 2 or 3 reads for every
write but there should not be a large increase in the amount of IOPS per
RAID.
From what I can find on various sites the average of Disk transfers/sec is
the number of IOPS I need, So the RAID10 in the SAN is the only solution
that can provide enough IOPS at the moment (I have disregarded the
controller cache).
Thanks in advance.
date: Thu, 15 Nov 2007 11:54:06 +0100
author: Qualidus
Re: Storage problems in Exchange 2003
Hi,
Calculating storage is fun huh :-)
Your current storage is definitely not upto the job.
<snip>
>
> Avg disk sec/read: average - 0,084 ; max - 8,521
>
> Avg disk sec/transfer: average - 0,053 ; max - 7,299
>
> Avg disk sec/write: average - 0,060 ; max - 8,048
>
> Disk transfers/sec: average - 366 ; max 6541
</snip>
From the above we can see your average disk sec/read and /write are far
above the 0,020 recommended limit.
You can use Disk transfers/sec, or you can add together Disk Reads/sec +
Disk Writes/sec to get your current IOPS per Diskset/LUN.
Your calculation for your IOPS per disk appear correct, taking your 10k FC
calculations
<snip>
> FC 10k rpm:
>
> Avg seek time 4,2ms, avg write time +2ms, so avg time 5,2ms
>
> 5,2ms + 3ms (half rotation to position the head) = 8,2ms or 121,9 IOPS
</snip>
This is correct as 1000/8,2 = 121.95
The only thing that worries me is it looks like you are stating you only
have 5 disk slots free on your current SAN? Is this correct? You may find
you need more disks
Looking at:
> Disk transfers/sec: average - 366 ; max 6541 (I prefer as stated earlier
> to add Disk Write/Sec and Disk Read/Sec) doesn't give me what I need to
> truly answer you question. If you've been doing this counter over a period
> of time the average will be artificial as there will be periods of time
> where hardly any access is logged (night for example).
What you should do is take the desired IOP per user and multiple it by your
actual users. I see you have taken into consideration your average active
mailbox logon, what % is this in the total number of users that are mailbox
enabled on your server? I usually always design to 100% concurrency. This is
mostly due to the nature of Exchange where I work, but you should be looking
at 75% concurrency or above.
You also need to take a fluff factor into account. Microsoft recommend the
IOPS you need + 20% overheard, + another percentage for AV, or BlackBerry
interaction etc. There's also the IO factor of taking into consideration
cached mode clients, if you force this on all users in your Org you can
lower the IOP per user calculation when deciding on storage IO.
I am probably blabbering now, take into consideration what I have said
above, look at your total amount of users and design for 100% concurrency -
unless you 100% know this will never be the case. Takes these figures and
then input them into the HP Storage Planning Calculator for Microsoft
Exchange Server 2003, available here:
http://www.msexchange.org/tutorials/Art-Science-Sizing-Exchange-2003-Part3.html,
as at this stage it's easier than working this out yourself.
Oliver
date: Thu, 15 Nov 2007 14:46:05 -0000
author: Oliver Moazzezi [MVP]
Re: Storage problems in Exchange 2003
Hi Oliver and thanks for your reply
As long as I get proof of my thoery I can do some calculating, only this was
my first time but now I know the basics :)
"Oliver Moazzezi [MVP]" skrev i meddelandet
news:e7CPCZ5JIHA.3356@TK2MSFTNGP02.phx.gbl...
> Hi,
>
> Calculating storage is fun huh :-)
>
> Your current storage is definitely not upto the job.
>
> <snip>
>
>>
>> Avg disk sec/read: average - 0,084 ; max - 8,521
>>
>> Avg disk sec/transfer: average - 0,053 ; max - 7,299
>>
>> Avg disk sec/write: average - 0,060 ; max - 8,048
>>
>> Disk transfers/sec: average - 366 ; max 6541
>
> </snip>
>
> From the above we can see your average disk sec/read and /write are far
> above the 0,020 recommended limit.
>
> You can use Disk transfers/sec, or you can add together Disk Reads/sec +
> Disk Writes/sec to get your current IOPS per Diskset/LUN.
>
> Your calculation for your IOPS per disk appear correct, taking your 10k FC
> calculations
>
> <snip>
>
>> FC 10k rpm:
>>
>> Avg seek time 4,2ms, avg write time +2ms, so avg time 5,2ms
>>
>> 5,2ms + 3ms (half rotation to position the head) = 8,2ms or 121,9 IOPS
>
> </snip>
>
> This is correct as 1000/8,2 = 121.95
>
>
> The only thing that worries me is it looks like you are stating you only
> have 5 disk slots free on your current SAN? Is this correct? You may find
> you need more disks
<comment>
Sorry I wasn't clear enough regarding the available disks. Today I have 4
free disks not assigned to any RAID-group and this is the disks I was
planning on doing a RAID10 of. The 5 disk RAID5 is already in use but it's
data that is seldom accessed so that would be a valid option if the penalty
for RAID5 was lower. I know now that I will need more disks but the problem
is that there is no room in the budget until next year, that's why I have to
use the available disk in the SAN since it will still be a vast improvement
compared to todays solution.
The best I can do for now is the RAID10 in the SAN for DB and create a new
RAID10 of the SATA-disk for Transaction Logs and pray to a higher power that
it will calm down the users enough so I get them of my back until we get the
new arrays. :)
</comment>
>
> Looking at:
>
>> Disk transfers/sec: average - 366 ; max 6541 (I prefer as stated earlier
>> to add Disk Write/Sec and Disk Read/Sec) doesn't give me what I need to
>> truly answer you question. If you've been doing this counter over a
>> period of time the average will be artificial as there will be periods of
>> time where hardly any access is logged (night for example).
<comment>
I ran perfmon for 2 hours (sample every 1 second) during peak time so the
numbers should be about as high as it gets, but of course there might be
spikes if everyone opens their mailbox at the exact time...
I've checked the server and it has 3599 mailboxes, the average mailboxs is
65MB in size and contains 605 objects. Here comes the fun part... about half
of the users have below 150 objects in their mailbox and their average
mailbox size is below 15MB...if I remove them I get an average size of 115MB
and 1129 objects and most of these users are online during peak hours.
</comment>
> What you should do is take the desired IOP per user and multiple it by
> your actual users. I see you have taken into consideration your average
> active mailbox logon, what % is this in the total number of users that are
> mailbox enabled on your server? I usually always design to 100%
> concurrency. This is mostly due to the nature of Exchange where I work,
> but you should be looking at 75% concurrency or above.
<comment>
An average IOPS per mailbox should end up somewhere around 0.5, and we have
about 55-60% of the users online at the same time so I'll probably should
calculate for 80% because most users are heavy users. That gives
3600*0,5*0,8*1,3 = 1872 IOPS
Using 15k rpm disks that would leave me with 14 disks in a RAID10 for DB
Then I need log-disks as well as disk to the other Exchange-server we have
(which is not that heavily loaded but still more than it can handle). All in
all I end up with 3 full arrays (total 42 disks + 1 spare in each array)...
Doubt I'll have enough money for that though.... I guess I will have to wait
until I know how much I can spend before I get someone who has done this a
few times to help me but to help the situation right now I move the DB to
the SAN and take it from there.
</comment>
> You also need to take a fluff factor into account. Microsoft recommend the
> IOPS you need + 20% overheard, + another percentage for AV, or BlackBerry
> interaction etc. There's also the IO factor of taking into consideration
> cached mode clients, if you force this on all users in your Org you can
> lower the IOP per user calculation when deciding on storage IO.
>
> I am probably blabbering now, take into consideration what I have said
> above, look at your total amount of users and design for 100%
> concurrency - unless you 100% know this will never be the case. Takes
> these figures and then input them into the HP Storage Planning Calculator
> for Microsoft Exchange Server 2003, available here:
> http://www.msexchange.org/tutorials/Art-Science-Sizing-Exchange-2003-Part3.html,
> as at this stage it's easier than working this out yourself.
>
> Oliver
All in all I have two responses to the whole situation... "Damn" and
"becoming a farmer sounds pretty okay now" ;)
I want to thank you very much for your input Oliver as well as for the link
to the HP-tool, you've helped me with the ammunition I need to ask for more
money (and if it doesn't work I just point all the angry users towards the
people in charge of the budget) :)
date: Fri, 16 Nov 2007 11:59:34 +0100
author: Qualidus
Re: Storage problems in Exchange 2003
SATA.. eeeew!
A rough rule of thumb is 40 IO operations per second per sata disk and 100
to 120 IOPS per SCSI/FC disk.
Disk Transfers per Second as measured by perfmon is a direct equivalent to
IOPS. Based on the numbers you quoted, the SATA drives clearly cannot handle
the IO load. 6541 IOPS is quite high.
If you want justification for the expense, measure RPC Latency. I think it
is a counter under the MSExchangeIS object in perfmon. 20ms is good, 30 ms
is poor and 50ms is unacceptable. Ifyou see 50ms latency in exchange's
ability to respond to RPC requests, I'll bet that the users are getting the
outlook popup baloons.
As an aside, does the "outlook is requesting data from..." balloon refer to
an Exchange server or a GC? If it references the name of a GC, your problem
is with the exchange server being able to get info from the GC in a timely
manner. This is why MS is suggesting 64bit GC's now.
Keep other disk intensive stuff off the edb and log file drives too. GFI
kills those drives.
Good luck
-Tim-
"Qualidus" wrote in message
news:Om5GWW3JIHA.3848@TK2MSFTNGP05.phx.gbl...
> Hi,
>
>
>
> I was recently put in charge of our Exchange servers and all the problems
> around it (the biggest one is "Requesting data from the Exchange
> server..."). I'm pretty sure it is because the storage so I have tried to
> calculate the needed IOPS to show that what we have today is simply not
> enough. We have a SAN and that is where I want to place the databases now
> but it comes with some costs (buying HBAs), that is why I would like
> someone to confirm my calculations are correct so I can get the money to
> connect the servers to the SAN.
>
>
>
> The servers have the following disk setup
>
> 2 SATA 7200rpm, RAID1 for OS and Transaction logs
>
> 4 SATA 7200rpm RAID5 for DB
>
>
>
> Available disk in the SAN: 4 FC 10k rpm RAID10
>
> Or 5 FC 10k rpm RAID5
>
>
>
> Other numbers of interest from perfmon during peak hours
>
> Physical Disk
>
> Avg disk sec/read: average - 0,084 ; max - 8,521
>
> Avg disk sec/transfer: average - 0,053 ; max - 7,299
>
> Avg disk sec/write: average - 0,060 ; max - 8,048
>
> Disk transfers/sec: average - 366 ; max 6541
>
>
>
> MSExchangeIS
>
> Active User Count: average - 1116
>
>
>
> This is what I have come up with (assuming 50% read and 50% write) and
> these are the sites I have used to do the math.
>
> http://storageadvisors.adaptec.com/2007/03/20/sata-iops-measurement/
>
> http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance
>
>
>
> SATA 7200 rpm:
>
> Avg seek time 8,5ms, avg write time +2ms, so avg time 9,5ms
>
> 9,5ms + 4,1ms (half rotation to position the head) = 13,6ms or 73,5 IOPS
>
>
>
> FC 10k rpm:
>
> Avg seek time 4,2ms, avg write time +2ms, so avg time 5,2ms
>
> 5,2ms + 3ms (half rotation to position the head) = 8,2ms or 121,9 IOPS
>
>
>
> SATA RAID5 (4 disks)
>
> 73,5 * 0,57 (RAID5 penalty) * 4 = 167,58 IOPS
>
>
>
> FC RAID5 (5 disks)
>
> 121,9 * 0,57 * 5 = 347,41 IOPS
>
>
>
> FC RAID10 (4 disks)
>
> 121,9 * 0,8 (RAID10 penalty) * 4 = 390,08 IOPS
>
>
>
> I know I haven't accounted for Exchange DB is usually 2 or 3 reads for
> every write but there should not be a large increase in the amount of IOPS
> per RAID.
>
> From what I can find on various sites the average of Disk transfers/sec is
> the number of IOPS I need, So the RAID10 in the SAN is the only solution
> that can provide enough IOPS at the moment (I have disregarded the
> controller cache).
>
>
>
> Thanks in advance.
>
>
date: Fri, 16 Nov 2007 16:40:13 -0500
author: Tim Hollingworth
Re: Storage problems in Exchange 2003
8 second reads...
A 7200 RPM SATA disk is good for about 40 IOPS/spindle @ 20ms response as
seen by the application when accessing data on an NTFS partition.
A 10K FC SCSI disk gets about 90 IOPS in the same situation. A 15K FC SCSI
disk gets about 130ish. Yes, I know the number a lot of folks like to quote
is 125 for 10K and 200 for 15K, however this is raw IOPS and does not
account for the filesystem overhead or metadata maintained by the
filesystem. On my list of top 10 storage sizing mistakes, overly optimistic
IOPS/spindle numbers ranks in the top 3.
RAID 5 is sizing mistake number 1. It takes twice as many spindles in RAID
5 to achieve equitable performance to RAID 1/10. In any event, you don't
have enough spindles available on the SAN to meet your peak IOPS (6541). If
you assume a 2:1 read write ratio, You need about a hundred spindles in
RAID 10. You never mention what your active user count peak is. You do
state an average of 1116.
"Qualidus" wrote in message
news:Om5GWW3JIHA.3848@TK2MSFTNGP05.phx.gbl...
> Hi,
>
>
>
> I was recently put in charge of our Exchange servers and all the problems
> around it (the biggest one is "Requesting data from the Exchange
> server..."). I'm pretty sure it is because the storage so I have tried to
> calculate the needed IOPS to show that what we have today is simply not
> enough. We have a SAN and that is where I want to place the databases now
> but it comes with some costs (buying HBAs), that is why I would like
> someone to confirm my calculations are correct so I can get the money to
> connect the servers to the SAN.
>
>
>
> The servers have the following disk setup
>
> 2 SATA 7200rpm, RAID1 for OS and Transaction logs
>
> 4 SATA 7200rpm RAID5 for DB
>
>
>
> Available disk in the SAN: 4 FC 10k rpm RAID10
>
> Or 5 FC 10k rpm RAID5
>
>
>
> Other numbers of interest from perfmon during peak hours
>
> Physical Disk
>
> Avg disk sec/read: average - 0,084 ; max - 8,521
>
> Avg disk sec/transfer: average - 0,053 ; max - 7,299
>
> Avg disk sec/write: average - 0,060 ; max - 8,048
>
> Disk transfers/sec: average - 366 ; max 6541
>
>
>
> MSExchangeIS
>
> Active User Count: average - 1116
>
>
>
> This is what I have come up with (assuming 50% read and 50% write) and
> these are the sites I have used to do the math.
>
> http://storageadvisors.adaptec.com/2007/03/20/sata-iops-measurement/
>
> http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance
>
>
>
> SATA 7200 rpm:
>
> Avg seek time 8,5ms, avg write time +2ms, so avg time 9,5ms
>
> 9,5ms + 4,1ms (half rotation to position the head) = 13,6ms or 73,5 IOPS
>
>
>
> FC 10k rpm:
>
> Avg seek time 4,2ms, avg write time +2ms, so avg time 5,2ms
>
> 5,2ms + 3ms (half rotation to position the head) = 8,2ms or 121,9 IOPS
>
>
>
> SATA RAID5 (4 disks)
>
> 73,5 * 0,57 (RAID5 penalty) * 4 = 167,58 IOPS
>
>
>
> FC RAID5 (5 disks)
>
> 121,9 * 0,57 * 5 = 347,41 IOPS
>
>
>
> FC RAID10 (4 disks)
>
> 121,9 * 0,8 (RAID10 penalty) * 4 = 390,08 IOPS
>
>
>
> I know I haven't accounted for Exchange DB is usually 2 or 3 reads for
> every write but there should not be a large increase in the amount of IOPS
> per RAID.
>
> From what I can find on various sites the average of Disk transfers/sec is
> the number of IOPS I need, So the RAID10 in the SAN is the only solution
> that can provide enough IOPS at the moment (I have disregarded the
> controller cache).
>
>
>
> Thanks in advance.
>
>
date: Sun, 18 Nov 2007 14:15:06 -0800
author: John Fullbright fjohn@donotspamnetappdotcom
|
|
