Exchange 2003 slow for users ("Retriving information.." popup in Outlook)   

We have two Exchange 2003 servers SP2, each with about 2000 users on them. 
The one server has two databases of about 100G each, each on it's own 3 
drive RAID5, and the other has a single database of about 160GB on an 8 
drive RAID10.   Each database has it's own RAID1 pair of drives for log 
files.    The drives are all 15K Ultra SCSI.

One server is an HP Proliant GL570G2, four 2.7Ghz CPU's.  The other is a 
570G4 with a four dual core CPU's.   The servers both have 4GB of RAM, and 
have the 3GB and USRVA set.  The systems are running native mode Windows 
2003 w/SP2 (one is R2), on member servers in a Windows 2003 native mode A/D. 
There are four domain controllers, three of which are global catalog 
servers.  The servers are all connected on a gigabit Ethernet network. 
We're running Microsoft's A/D integrated DNS (no WNS)

We've had the systems set up for about 3 years, and are hoping soon to be 
migrating to Exchange 2007, but it probably won't be for a couple of months 
at least (we have the Exchange 2007 systems set up, but are waiting on some 
licensing purchases before we can migrate everyone over).

It seems like the last six months or so, we're getting more and more reports 
of "slowness" on from the users.  They get the "retrieving information." 
popup in Outlook.   Usually it's only 20-30 seconds, but sometimes it will 
take three or four minutes to free up, even when they're just clicking on 
different folders in their mailbox.

There are no obvious entries in the event log (system or app) that would 
account for this.  The NIC statistics don't show errors with the network 
I/O.  The Exchange Best Practices Analyzer doesn't flag anything 
significant.

We do an offline defrag of the databases about twice a year, the last was in 
April.

I've done some internet searching, but most of the troubleshooting tips seem 
to be based on entries in the event log.

Any suggestions on where we can look to pin down this issue would be 
appreciated.

Mike O.

date: Mon, 16 Jun 2008 22:53:09 -0400   author:   Mike O

Re: Exchange 2003 slow for users ("Retriving information.." popup in Outlook)   

If I had to guess, you are probably taking a serious performance hit on 
those 3 disk RAID arrays.
You might want to take a look at this:
http://www.msexchange.org/tutorials/Optimize-Exchange-2003-Performance-Part1.html
and this:
http://www.computerperformance.co.uk/exchange2003/exchange2003_performance_disks.htm

Additionally you don't mention if your Outlook clients are using Cached 
Mode. If they are not, they should be. That will help reduce the disk load 
and give a more consistent end user experience.


"Mike O"  wrote in message 
news:%23$FPKVC0IHA.2384@TK2MSFTNGP04.phx.gbl...
> We have two Exchange 2003 servers SP2, each with about 2000 users on them. 
> The one server has two databases of about 100G each, each on it's own 3 
> drive RAID5, and the other has a single database of about 160GB on an 8 
> drive RAID10.   Each database has it's own RAID1 pair of drives for log 
> files.    The drives are all 15K Ultra SCSI.
>
> One server is an HP Proliant GL570G2, four 2.7Ghz CPU's.  The other is a 
> 570G4 with a four dual core CPU's.   The servers both have 4GB of RAM, and 
> have the 3GB and USRVA set.  The systems are running native mode Windows 
> 2003 w/SP2 (one is R2), on member servers in a Windows 2003 native mode 
> A/D. There are four domain controllers, three of which are global catalog 
> servers.  The servers are all connected on a gigabit Ethernet network. 
> We're running Microsoft's A/D integrated DNS (no WNS)
>
> We've had the systems set up for about 3 years, and are hoping soon to be 
> migrating to Exchange 2007, but it probably won't be for a couple of 
> months at least (we have the Exchange 2007 systems set up, but are waiting 
> on some licensing purchases before we can migrate everyone over).
>
> It seems like the last six months or so, we're getting more and more 
> reports of "slowness" on from the users.  They get the "retrieving 
> information." popup in Outlook.   Usually it's only 20-30 seconds, but 
> sometimes it will take three or four minutes to free up, even when they're 
> just clicking on different folders in their mailbox.
>
> There are no obvious entries in the event log (system or app) that would 
> account for this.  The NIC statistics don't show errors with the network 
> I/O.  The Exchange Best Practices Analyzer doesn't flag anything 
> significant.
>
> We do an offline defrag of the databases about twice a year, the last was 
> in April.
>
> I've done some internet searching, but most of the troubleshooting tips 
> seem to be based on entries in the event log.
>
> Any suggestions on where we can look to pin down this issue would be 
> appreciated.
>
> Mike O.
>
>

date: Mon, 16 Jun 2008 21:16:32 -0700   author:   Martin Blackstone

Re: Exchange 2003 slow for users ("Retriving information.." popup in Outlook)   

I know the RAID5 configuration isn't the best, and the new 2007 system 
config is going to address that, but it has been working OK since 2005. 
It's only been the last six months or so that we're getting the slowness 
issues.

Also, one of the servers is a newer model and uses an 8 drive RAID 1/0 array 
and we're also getting calls from users on that server, too.


"Martin Blackstone"  wrote in message 
news:8DC5A0EF-EEDF-4ACF-ADC5-575658DC9690@microsoft.com...
> If I had to guess, you are probably taking a serious performance hit on 
> those 3 disk RAID arrays.
> You might want to take a look at this:
> http://www.msexchange.org/tutorials/Optimize-Exchange-2003-Performance-Part1.html
> and this:
> http://www.computerperformance.co.uk/exchange2003/exchange2003_performance_disks.htm
>
> Additionally you don't mention if your Outlook clients are using Cached 
> Mode. If they are not, they should be. That will help reduce the disk load 
> and give a more consistent end user experience.
>
>
> "Mike O"  wrote in message 
> news:%23$FPKVC0IHA.2384@TK2MSFTNGP04.phx.gbl...
>> We have two Exchange 2003 servers SP2, each with about 2000 users on 
>> them. The one server has two databases of about 100G each, each on it's 
>> own 3 drive RAID5, and the other has a single database of about 160GB on 
>> an 8 drive RAID10.   Each database has it's own RAID1 pair of drives for 
>> log files.    The drives are all 15K Ultra SCSI.
>>
>> One server is an HP Proliant GL570G2, four 2.7Ghz CPU's.  The other is a 
>> 570G4 with a four dual core CPU's.   The servers both have 4GB of RAM, 
>> and have the 3GB and USRVA set.  The systems are running native mode 
>> Windows 2003 w/SP2 (one is R2), on member servers in a Windows 2003 
>> native mode A/D. There are four domain controllers, three of which are 
>> global catalog servers.  The servers are all connected on a gigabit 
>> Ethernet network. We're running Microsoft's A/D integrated DNS (no WNS)
>>
>> We've had the systems set up for about 3 years, and are hoping soon to be 
>> migrating to Exchange 2007, but it probably won't be for a couple of 
>> months at least (we have the Exchange 2007 systems set up, but are 
>> waiting on some licensing purchases before we can migrate everyone over).
>>
>> It seems like the last six months or so, we're getting more and more 
>> reports of "slowness" on from the users.  They get the "retrieving 
>> information." popup in Outlook.   Usually it's only 20-30 seconds, but 
>> sometimes it will take three or four minutes to free up, even when 
>> they're just clicking on different folders in their mailbox.
>>
>> There are no obvious entries in the event log (system or app) that would 
>> account for this.  The NIC statistics don't show errors with the network 
>> I/O.  The Exchange Best Practices Analyzer doesn't flag anything 
>> significant.
>>
>> We do an offline defrag of the databases about twice a year, the last was 
>> in April.
>>
>> I've done some internet searching, but most of the troubleshooting tips 
>> seem to be based on entries in the event log.
>>
>> Any suggestions on where we can look to pin down this issue would be 
>> appreciated.
>>
>> Mike O.
>>
>>
>

date: Tue, 17 Jun 2008 06:48:07 -0400   author:   Mike O

Re: Exchange 2003 slow for users ("Retriving information.." popup in Outlook)   

Start with the disk tests in the second link I sent you.
Then go from there.

"Mike O"  wrote in message 
news:Of0vkeG0IHA.4416@TK2MSFTNGP05.phx.gbl...
>I know the RAID5 configuration isn't the best, and the new 2007 system 
>config is going to address that, but it has been working OK since 2005. 
>It's only been the last six months or so that we're getting the slowness 
>issues.
>
> Also, one of the servers is a newer model and uses an 8 drive RAID 1/0 
> array and we're also getting calls from users on that server, too.
>
>
> "Martin Blackstone"  wrote in message 
> news:8DC5A0EF-EEDF-4ACF-ADC5-575658DC9690@microsoft.com...
>> If I had to guess, you are probably taking a serious performance hit on 
>> those 3 disk RAID arrays.
>> You might want to take a look at this:
>> http://www.msexchange.org/tutorials/Optimize-Exchange-2003-Performance-Part1.html
>> and this:
>> http://www.computerperformance.co.uk/exchange2003/exchange2003_performance_disks.htm
>>
>> Additionally you don't mention if your Outlook clients are using Cached 
>> Mode. If they are not, they should be. That will help reduce the disk 
>> load and give a more consistent end user experience.
>>
>>
>> "Mike O"  wrote in message 
>> news:%23$FPKVC0IHA.2384@TK2MSFTNGP04.phx.gbl...
>>> We have two Exchange 2003 servers SP2, each with about 2000 users on 
>>> them. The one server has two databases of about 100G each, each on it's 
>>> own 3 drive RAID5, and the other has a single database of about 160GB on 
>>> an 8 drive RAID10.   Each database has it's own RAID1 pair of drives for 
>>> log files.    The drives are all 15K Ultra SCSI.
>>>
>>> One server is an HP Proliant GL570G2, four 2.7Ghz CPU's.  The other is a 
>>> 570G4 with a four dual core CPU's.   The servers both have 4GB of RAM, 
>>> and have the 3GB and USRVA set.  The systems are running native mode 
>>> Windows 2003 w/SP2 (one is R2), on member servers in a Windows 2003 
>>> native mode A/D. There are four domain controllers, three of which are 
>>> global catalog servers.  The servers are all connected on a gigabit 
>>> Ethernet network. We're running Microsoft's A/D integrated DNS (no WNS)
>>>
>>> We've had the systems set up for about 3 years, and are hoping soon to 
>>> be migrating to Exchange 2007, but it probably won't be for a couple of 
>>> months at least (we have the Exchange 2007 systems set up, but are 
>>> waiting on some licensing purchases before we can migrate everyone 
>>> over).
>>>
>>> It seems like the last six months or so, we're getting more and more 
>>> reports of "slowness" on from the users.  They get the "retrieving 
>>> information." popup in Outlook.   Usually it's only 20-30 seconds, but 
>>> sometimes it will take three or four minutes to free up, even when 
>>> they're just clicking on different folders in their mailbox.
>>>
>>> There are no obvious entries in the event log (system or app) that would 
>>> account for this.  The NIC statistics don't show errors with the network 
>>> I/O.  The Exchange Best Practices Analyzer doesn't flag anything 
>>> significant.
>>>
>>> We do an offline defrag of the databases about twice a year, the last 
>>> was in April.
>>>
>>> I've done some internet searching, but most of the troubleshooting tips 
>>> seem to be based on entries in the event log.
>>>
>>> Any suggestions on where we can look to pin down this issue would be 
>>> appreciated.
>>>
>>> Mike O.
>>>
>>>
>>
>

date: Tue, 17 Jun 2008 04:44:27 -0700   author:   Martin Blackstone

Re: Exchange 2003 slow for users ("Retriving information.." popup in Outlook)   

Use cached mode.

You mention "Usually it's only 20-30 seconds, but
>>> sometimes it will take three or four minutes to free up, even when 
>>> they're just clicking on different folders in their mailbox."

The behavior is caused when there is not enough db cache to cache the 11 
default views for a folder, and the views are generated on the fly.  As the 
number of items items you have in default folders increases, the time and IO 
required to generate the 11 views also increases.  That's why there is a 
recommended limit of 5000 items per folder.

In Exchange 2003, the amount of cache available is generally less than 900MB 
for a server.  Moving to Exchange 2007 helps because you have more db cache. 
Moving to cached mode clients helps because it shifts the IO from the server 
to the client.

John

"Mike O"  wrote in message 
news:Of0vkeG0IHA.4416@TK2MSFTNGP05.phx.gbl...
>I know the RAID5 configuration isn't the best, and the new 2007 system 
>config is going to address that, but it has been working OK since 2005. 
>It's only been the last six months or so that we're getting the slowness 
>issues.
>
> Also, one of the servers is a newer model and uses an 8 drive RAID 1/0 
> array and we're also getting calls from users on that server, too.
>
>
> "Martin Blackstone"  wrote in message 
> news:8DC5A0EF-EEDF-4ACF-ADC5-575658DC9690@microsoft.com...
>> If I had to guess, you are probably taking a serious performance hit on 
>> those 3 disk RAID arrays.
>> You might want to take a look at this:
>> http://www.msexchange.org/tutorials/Optimize-Exchange-2003-Performance-Part1.html
>> and this:
>> http://www.computerperformance.co.uk/exchange2003/exchange2003_performance_disks.htm
>>
>> Additionally you don't mention if your Outlook clients are using Cached 
>> Mode. If they are not, they should be. That will help reduce the disk 
>> load and give a more consistent end user experience.
>>
>>
>> "Mike O"  wrote in message 
>> news:%23$FPKVC0IHA.2384@TK2MSFTNGP04.phx.gbl...
>>> We have two Exchange 2003 servers SP2, each with about 2000 users on 
>>> them. The one server has two databases of about 100G each, each on it's 
>>> own 3 drive RAID5, and the other has a single database of about 160GB on 
>>> an 8 drive RAID10.   Each database has it's own RAID1 pair of drives for 
>>> log files.    The drives are all 15K Ultra SCSI.
>>>
>>> One server is an HP Proliant GL570G2, four 2.7Ghz CPU's.  The other is a 
>>> 570G4 with a four dual core CPU's.   The servers both have 4GB of RAM, 
>>> and have the 3GB and USRVA set.  The systems are running native mode 
>>> Windows 2003 w/SP2 (one is R2), on member servers in a Windows 2003 
>>> native mode A/D. There are four domain controllers, three of which are 
>>> global catalog servers.  The servers are all connected on a gigabit 
>>> Ethernet network. We're running Microsoft's A/D integrated DNS (no WNS)
>>>
>>> We've had the systems set up for about 3 years, and are hoping soon to 
>>> be migrating to Exchange 2007, but it probably won't be for a couple of 
>>> months at least (we have the Exchange 2007 systems set up, but are 
>>> waiting on some licensing purchases before we can migrate everyone 
>>> over).
>>>
>>> It seems like the last six months or so, we're getting more and more 
>>> reports of "slowness" on from the users.  They get the "retrieving 
>>> information." popup in Outlook.   Usually it's only 20-30 seconds, but 
>>> sometimes it will take three or four minutes to free up, even when 
>>> they're just clicking on different folders in their mailbox.
>>>
>>> There are no obvious entries in the event log (system or app) that would 
>>> account for this.  The NIC statistics don't show errors with the network 
>>> I/O.  The Exchange Best Practices Analyzer doesn't flag anything 
>>> significant.
>>>
>>> We do an offline defrag of the databases about twice a year, the last 
>>> was in April.
>>>
>>> I've done some internet searching, but most of the troubleshooting tips 
>>> seem to be based on entries in the event log.
>>>
>>> Any suggestions on where we can look to pin down this issue would be 
>>> appreciated.
>>>
>>> Mike O.
>>>
>>>
>>
>

date: Tue, 17 Jun 2008 08:17:56 -0700   author:   John Fullbright fjohn@donotspamnetappdotcom

Re: Exchange 2003 slow for users ("Retriving information.." popup in Outlook)   

"John Fullbright" <fjohn@donotspamnetappdotcom> wrote in message 
news:%23MiwT1I0IHA.3920@TK2MSFTNGP02.phx.gbl...
> Use cached mode.
>
> You mention "Usually it's only 20-30 seconds, but
>>>> sometimes it will take three or four minutes to free up, even when 
>>>> they're just clicking on different folders in their mailbox."
>
> The behavior is caused when there is not enough db cache to cache the 11 
> default views for a folder, and the views are generated on the fly.  As 
> the number of items items you have in default folders increases, the time 
> and IO required to generate the 11 views also increases.  That's why there 
> is a recommended limit of 5000 items per folder.
>
> In Exchange 2003, the amount of cache available is generally less than 
> 900MB for a server.  Moving to Exchange 2007 helps because you have more 
> db cache. Moving to cached mode clients helps because it shifts the IO 
> from the server to the client.
>
> John
>

We're looking at doing that, unfortunately most of our clients are running 
Outlook XP that doesn't have cache mode.

I've been looking into tuning the server parameters.  I was going through 
the article KB 815372 about optimizing memory for Exchange 2003.  We had 
previously applied most of the settings in that article, but I did see the 
one parameter "mxExchESEParamCacheSizeMax" to update the DB Cache to 
1,200GB, I think we're going to try that this weekend.

I also came across the setting to control the number of views that Exchange 
caches ("msExchMaxCachedViews").  I'm wondering if it would help to lower 
that value to something like 5 or 6 (apparently the ExBPA complains if you 
go less than 5), instead of the default 11.  Do you think that would help?

We're pushing our ideal solution to migrate to Exchange 2007, but the 
funding is tight and even if it's approved, it will be a few months until we 
can get it.  I'm hoping to just get through until then.


Mike O.



> "Mike O"  wrote in message 
> news:Of0vkeG0IHA.4416@TK2MSFTNGP05.phx.gbl...
>>I know the RAID5 configuration isn't the best, and the new 2007 system 
>>config is going to address that, but it has been working OK since 2005. 
>>It's only been the last six months or so that we're getting the slowness 
>>issues.
>>
>> Also, one of the servers is a newer model and uses an 8 drive RAID 1/0 
>> array and we're also getting calls from users on that server, too.
>>
>>
>> "Martin Blackstone"  wrote in message 
>> news:8DC5A0EF-EEDF-4ACF-ADC5-575658DC9690@microsoft.com...
>>> If I had to guess, you are probably taking a serious performance hit on 
>>> those 3 disk RAID arrays.
>>> You might want to take a look at this:
>>> http://www.msexchange.org/tutorials/Optimize-Exchange-2003-Performance-Part1.html
>>> and this:
>>> http://www.computerperformance.co.uk/exchange2003/exchange2003_performance_disks.htm
>>>
>>> Additionally you don't mention if your Outlook clients are using Cached 
>>> Mode. If they are not, they should be. That will help reduce the disk 
>>> load and give a more consistent end user experience.
>>>
>>>
>>> "Mike O"  wrote in message 
>>> news:%23$FPKVC0IHA.2384@TK2MSFTNGP04.phx.gbl...
>>>> We have two Exchange 2003 servers SP2, each with about 2000 users on 
>>>> them. The one server has two databases of about 100G each, each on it's 
>>>> own 3 drive RAID5, and the other has a single database of about 160GB 
>>>> on an 8 drive RAID10.   Each database has it's own RAID1 pair of drives 
>>>> for log files.    The drives are all 15K Ultra SCSI.
>>>>
>>>> One server is an HP Proliant GL570G2, four 2.7Ghz CPU's.  The other is 
>>>> a 570G4 with a four dual core CPU's.   The servers both have 4GB of 
>>>> RAM, and have the 3GB and USRVA set.  The systems are running native 
>>>> mode Windows 2003 w/SP2 (one is R2), on member servers in a Windows 
>>>> 2003 native mode A/D. There are four domain controllers, three of which 
>>>> are global catalog servers.  The servers are all connected on a gigabit 
>>>> Ethernet network. We're running Microsoft's A/D integrated DNS (no WNS)
>>>>
>>>> We've had the systems set up for about 3 years, and are hoping soon to 
>>>> be migrating to Exchange 2007, but it probably won't be for a couple of 
>>>> months at least (we have the Exchange 2007 systems set up, but are 
>>>> waiting on some licensing purchases before we can migrate everyone 
>>>> over).
>>>>
>>>> It seems like the last six months or so, we're getting more and more 
>>>> reports of "slowness" on from the users.  They get the "retrieving 
>>>> information." popup in Outlook.   Usually it's only 20-30 seconds, but 
>>>> sometimes it will take three or four minutes to free up, even when 
>>>> they're just clicking on different folders in their mailbox.
>>>>
>>>> There are no obvious entries in the event log (system or app) that 
>>>> would account for this.  The NIC statistics don't show errors with the 
>>>> network I/O.  The Exchange Best Practices Analyzer doesn't flag 
>>>> anything significant.
>>>>
>>>> We do an offline defrag of the databases about twice a year, the last 
>>>> was in April.
>>>>
>>>> I've done some internet searching, but most of the troubleshooting tips 
>>>> seem to be based on entries in the event log.
>>>>
>>>> Any suggestions on where we can look to pin down this issue would be 
>>>> appreciated.
>>>>
>>>> Mike O.
>>>>
>>>>
>>>
>>
>
>

date: Tue, 24 Jun 2008 23:04:59 -0400   author:   Mike O