ME Enterprise and NVMe

[R4LRetro]

Hello all,

This past weekend I migrated our MailEnable instance to much more capable hardware and wanted to share my setup and my thoughts overall. We service about 230 mailboxes on one domain, using 1 SBL, 1 UBL, DKIM and SPF, and an enterprise XDR/AV solution (with exceptions in place for ME).

First, a breakdown of the old hardware vs. the new:

Old Server

Code: Select all

- Chassis: Cisco UCS 200 M2
- Processor: 2x Intel Xeon E5649 @ 2.53 GHz
- RAM: 64GB DDR3 ECC
- Storage: 2TB Toshiba SAS 7.5k RAID1

On this hardware if a couple of users archived, deleted or moved many emails at once, this would cause the server to hit a bottleneck on the drives (Avg. Disk Queue Length threshold is about 8, ours was something like 16-18). This would cause the entire server to essentially freeze until it could move onto the next thing to process. This would be felt by our clients as things like sending and receiving would also hang on their end. This would cause timeouts in Thunderbird and Outlook depending on the load on the server.

As an admin, I could not do anything to help as Web Admin and the server itself would also be hanging. We could not run all ME services. The indexer service in particular was disabled as it would destroy the server's performance. Some logs were disabled to help with read/write performance. I use WizTree to check the sizes of mailboxes. This would only be possible under off-peak hours as it would also destroy the server's performance. IMAP users were suffering the most I think with all the overhead on the server.

We had a permissions issue recently and fixing permissions on the message store took 3 days for ~230 mailboxes, about 7 million emails total using the Common Installation option from MEInstaller. RAID1 would take around 48 hours to rebuild, incremental backup of the server took about the same time. Very slow business overall.

New Server

Code: Select all

- Chassis: Dell PowerEdge R740xd
- Processor: 2x Intel Xeon Gold 6132 @ 2.60 GHz
- RAM: 256GB DDR4 ECC
- Storage: 1.2TB Dell SAS 7.5k RAID1 for the operating system, 3.5TB Micron U.2 Enterprise NVMe in software RAID1 for MailEnable

I've been running this hardware for about a month. Migration to this server took a full work day, a considerable improvement over the original migration. All services are running, all logs are enabled and yet the server is completely functional. No slowness running WizTree on the entire MailEnable directory (it scans the entire installation in < 5 mins), no slowness browsing the server, no slowness loading WebMail or WebAdmin pages. RAID1 rebuilds in about an hour.

--------------------

Notes

I know it's common sense that new hardware will be faster (duh) but I've seen posts on here of people running small and large ME deployments with very little information on the hardware stating issues with performance. I've also seen the opposite on here where someone had a better setup than what I listed here and they were not seeing good performance. I wanted to list my experience for others who may be running into performance issues. Upgrading the hardware has worked for us so far and really I can't stress enough how important it was for us to move the message store onto a separate drive away from the OS.

Other configurations

I'm sure if I ran a RAID10 with 8 disks this server would fly too. My next step is to move our server into our actual DMZ to help out with IMAP for our LAN. Currently our setup exists on a separate ISP line run to our building. I want to move the ME server onto our LAN's ISP into the DMZ. This should make IMAP clients much, much faster overall.

I'm interested in other people's setups for ME and what issues they face regarding performance. Please share your experiences!

[RSN]

Our server :
AMD 7313P (16 cores)
384GB RAM
ZFS Cluster with 8TB SAS SSDs
The server is virtualized with VMWare

The mailenbale VM has the following resources
4 CPUs (cores)
16 GB RAM
100GB system HDD (on the ZFS cluster)
2TB Mailenable data HDD (on the ZFS cluster)

approx. 1000 mailboxes with POP3, IMAP, ActiveSync and DAVx connections.
The mailenable data is stored in an SQL database and not in the file system.

The CPU utilization of the complete server (with a few more VMs) is between 20% and 50%.
The CPU utilization of the Mailenable VM is between 10% and 30%

The ZFS utilization is also minimal.



However, we also have performance problems with 2 or three of our customers. These customers have mailboxes with 30GB or 70GB.
With all our other customers there are virtually no problems or crashes (they have smaller mailboxes)
I have already read elsewhere in the forum that this problem is known and affects all users of large mailboxes, as the index service has problems with the large number of mails.

[kiamori]

AMD Epyc 64 cores
2TB RAM
2x 4TB m.2 in Raid 1 (system)
24x 2TB U.2 Drives in Hardware Raid 10 (data)
10Gbps Fiber

No performance issues,
23,975 users with ~ 4,000 active connections at any given time.

[R4LRetro]

However, we also have performance problems with 2 or three of our customers. These customers have mailboxes with 30GB or 70GB.
With all our other customers there are virtually no problems or crashes (they have smaller mailboxes)
I have already read elsewhere in the forum that this problem is known and affects all users of large mailboxes, as the index service has problems with the large number of mails.

I'm reading about this and having this fear as well but I have a few mailboxes well in the 50-60GB range that appear to be doing well still. The indexer isn't spitting out any errors at least.

Still, one of my goals is to maintain inboxes over time to prevent them from getting that large. One of the ideas I had was to use something like Cryoserver or MailPiler to push mail to automatically and set up retention policies on clients.