William Strucke

One of the things Apple does quite well is stick to standards when feasible and that usually means there is a plethora of options available for all services, just like there are in any other BSD or Linux installation. The problem we encounter here is that what they do slightly better is hide these advanced options so that the "No IT department required" slogan holds some grain of truth. Specifically in this case I'm referring to an item that debuted with OS X Leopard Server, the brand new "Team Server", or what is more commonly referred to as Wiki/Blog/WebCalender.

Naturally, apple does not provide extensive options in the server admin or workgroup manager GUI's to manage these services, either as a whole or for individual users and groups. We are basically expected to either turn them on or off and leave it at that. But did you know that you can control much more granular settings for the team server, using the inspector tool in workgroup manager? (and perhaps a bit of terminal too!)

First, a bit of background on how the team server works -- for this section you will want to reference Server Admin (you should probably not do any of this if you are using Server Preferences to manage the box) and make sure your web server is bound to an Open Directory (or Active Directory with apple extensions). You will notice that there is no "team server" button and corresponding "light" in server admin anywhere -- that means you can not check a box to enable the service nor can you view the status of it.
Instead, the "Team Server" is one of the completely hidden from view "just works" services in OS X. The only way to check it's status is via the command line on the server:

web-1:~ strucke1$ sudo serveradmin status teams
teams:state = "STOPPED"

The server will automatically start it up if you are using it with one or more of your web sites. To enable the team server for a site, you simply check the "Wiki and Blog" for groups or "Blog" for users option in a web site configuration:


This will appear in the system.log:

Apr 22 10:56:53 web-1 servermgrd[54]: servermgr_web: Teams service started

and you can check the status of the service:

web-1:~ strucke1$ sudo serveradmin status teams
teams:state = "RUNNING"


Now when you initially join a server to an open directory, OS X generates a Unique ID (henceforth referred to as 'GUID') for the server and registers that in the directory record. If you open workgroup manager, go to preferences and enable the inspector tool you will be able to see all of the LDAP records that are internally created and managed as you enable or disable services. You can view the server's GUID in it's computer record beneath the inspector tab in the field "apple-generateduid". You will have to enable the "Show Native Attributes" option to be able to view this data. Click the Options button in the inspector tab (under the computer record):




Select the "dsAttrTypeNative:apple-generateduid" field and click the Edit button to view or copy the whole thing. Do not change this value yourself.



When you create a web site in server admin, that site is also assigned a GUID of it's own. It is the combination of server <=> site GUID's that are used to enable team services for users and groups. The site GUIDs are stored in the computer record as well, in the field "apple-serviceinfo".

There is a "bug" in the script that leopard (up to 10.5.5 at least) uses to generate the squirrelmail configuration file for the built-in webmail service.

The generated configuration results in the following error when attempting to send mail via the webmail service:

ERROR:
ERROR: Could not append message to Sent Messages.
Server responded: Permission denied

ERROR:
ERROR: Bad or malformed request.
Server responded: Unrecognized command

Despite the errors, you will find that message was indeed sent successfully (provided, of course, that you properly configured the outgoing mail server!)


The issue here is that squirrelmail is looking for your "Sent Messages", "Deleted Items", and "Drafts" folders in the wrong location. The solution is to edit squirrelmail's configuration file to include the correct path to these folders.

I love vim, so I access this file with the command "sudo vi /private/etc/squirrelmail/config/config.php".

Once you are in the configuration file, locate the following three lines (a few pages down):

$trash_folder = "Deleted Messages";
$sent_folder = "Sent Messages";
$draft_folder = "Drafts";

When you get into webmail and look at your folder list, you will notice that some of the folders you are not subscribed to are "INBOX/Sent Messages", "INBOX/Deleted Messages", etc... In case you have not guessed the solution already, change the aforementioned lines to:

$trash_folder = 'INBOX/Deleted Messages';
$sent_folder = 'INBOX/Sent Messages';
$draft_folder = 'INBOX/Drafts';

Save, quit, then restart the web server:

sudo serveradmin web stop
sudo serveradmin web start

Log in to web mail and you should now be able to send messages without error and see the contents of the three folders.

I finally have kerberos working properly since upgrading our servers to OS 10.5 last summer. Don't mistake the word "upgrade" for a casual software upgrade -- I would not recommend doing that to a server and that's not what I did. I ran an archive of my open directory and preferences, formatted, installed the new OS, patched it, then reimported the directory. I also ended up resetting all of our account passwords as well (no easy feat, considering the massive 7000 accounts in the system). (see my post on account management on how to go about that)

Anyway, after the upgrade I was never able to get kerberos on other servers to talk to the ODM's kerberos realm. No matter what. Today, after a lot of reading and research, I've got it working.

Here are some of the errors I was getting on the various servers:

Principal or policy already exists

kadmin: No entry for principal afpserver/admin.arts.ohio-state.edu@ODM.ARTS.OHIO-STATE.EDU exists in keytab WRFILE:/etc/krb5.keytab

kadmin: Operation requires ``change-password'' privilege while changing ftp/admin.arts.ohio-state.edu@ODM.ARTS.OHIO-STATE.EDU's key

WARNING: no policy specified for afpserver/admin.arts.ohio-state.edu@ODM.ARTS.OHIO-STATE.EDU; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "afpserver/admin.arts.ohio-state.edu@ODM.ARTS.OHIO-STATE.EDU".


Here's how I fixed it:

1. unbind the client server from the directory

2. remove the clients directory entry

3. make sure the client has a unique LKDC, you may have to reset it

4. make sure the client's "name" is it's short host name, e.g. "print"

5. make sure the client knows it's own FQDN. If you type "hostname" at a prompt it should display it correctly, e.g. "print.arts.ohio-state.edu"

6. remove all references to the client from the directory server (LDAP).

7. remove all references to the client from the kerberos realm - you can retreive a list by typing:

sudo kadmin.local -q "listprincs" | grep servername

To remove items, one a time:

sudo kadmin.local -q "delete_principal item/servername.arts.ohio-state.edu@ODM.ARTS.OHIO-STATE.EDU"

8. using the directory access application on the client, add the directory server using it's FQDN, e.g. "odm.arts.ohio-state.edu"

9. bind to the directory server using directory administrator credentials and USE the client servers FQDN as the computer name. This is tricky since directory access automatically fills in the servers' host name for you -- do not be misled -- fill in the fully qualified domain name, e.g. "print.arts.ohio-state.edu"

10. remove the clients key tab:

ON THE CLIENT:

sudo rm /etc/krb5.keytab

11. restart the client server

12. now join the kerberos realm, again on the client:

sudo sso_util configure -r ODM.ARTS.OHIO-STATE.EDU -a diradmin -p password -v 1 all

13. refresh open directory in the server admin gui to confirm you are bound to the kerberos realm.

As it turns out, the default installation of PHP 5.2.6 in Mac OS 10.5.5 is great for just about everything we do, save one thing -- it does not include the GD graphics library.

Here's what I had to do to make it work:

Install or update the Xcode (Developer) Tools on each server, if they are not already installed.

Download the following libraries:

LibJPEG 6b
LibPNG 1.2.16
PHP 5.2.6
MySQL 5.0.37 (get the package for os x, no need to custom build one)
OpenSSL
MCrypt 2.6.8
mhash 0.9.9.9 http://mhash.sf.net/
zlib 1.2.3
-----------------

install the 32 bit mysql package (not the binary)

-----------------

Run all commands as root:

sudo -s

-----------------

build and install LibJPEG

export MACOSX_DEPLOYMENT_TARGET=10.5
cp /usr/share/libtool/config.sub .
cp /usr/share/libtool/config.guess .
./configure --enable-shared --disable-static
make
mkdir -p /usr/local/include /usr/local/lib /usr/local/man/man1 /usr/local/bin/cjpeg
make install
ranlib /usr/local/lib/libjpeg.a

-----------------

build and install LibPNG

./configure --prefix=/usr/local/png
make && make install

--------------------

openssl

./config --prefix=/usr
make && make install

----------------

use freetype that comes with OS X in /usr/X11R6/

------------------

build and install zlib

./configure
make
make install

------------------

build and install mhash

./configure
make && make install

------------------

build and install libmcrypt

./configure
make && make install

------------------

build and install mcrypt

./configure

vi src/rfc2440.c
replace the line "#include <malloc.h>" with "#include <malloc/malloc.h>"

make && make install

------------------

build and install php

./configure '--prefix=/usr' '--with-apxs2=/usr/sbin/apxs' '--sysconfdir=/etc' '--with-zlib=/usr' '--with-zlib-dir=/usr' '--with-xmlrpc' '--with-xsl=/usr' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-ldap=/usr' '--enable-cli' '--enable-exif' '--enable-ftp' '--enable-mbstring' '--enable-mbregex' '--enable-sockets' '--with-iodbc=/usr' '--with-curl=/usr' '--with-mysql=/usr/local/mysql' '--with-gd' '--with-png-dir=/usr/local/png/' '--enable-fastcgi' '--enable-zip' '--with-kerberos=/usr' '--with-config-file-path=/etc' '--with-mysqli=/usr/bin/mysql_config' '--with-mysql-sock=/var/mysql' '--without-pear' '--with-mcrypt' '--with-freetype-dir=/usr/X11R6/' '--with-openssl=shared,/usr' '--with-jpeg-dir=/usr/local'

make

make install

----------------

now convert 64-bit universal apache to 32-bit for i386:

cp /usr/sbin/httpd /usr/sbin/httpd-fat
lipo /usr/sbin/httpd -thin i386 -output /usr/sbin/httpd

----------------

The PHP session path in this config seems to be /etc/phpsessions/

If that folder does not exist PHP will throw an error and sessions will not work.

vi /etc/php.ini

set phpsessions path to /etc/phpsessions/

sudo mkdir /etc/phpsessions
sudo chown -R www /etc/phpsessions

We set our sights pretty high since this summer upgrade, but gradually all of the pieces are falling into place.

The goal is to maintain maximum availability and performance for all of the services we host. As such I have ranked all of our servers and services in order to deduce what needs to be redundant and how we can perform maintenance with as little affect as possible on our end users. When I first started here we six computer labs which were locally managed and nearly every faculty and staff computer was managed by the end user except when there was a significant problem.

Today my office centrally manages computers, software, and accounts for ten computer labs (all but one are used for instruction as well as open lab hours) and every faculty, staff, and graduate student machine in the college. This has put a significant strain on our central servers despite our aggressive upgrade schedule. As of this moment we are managing twelve public university subnets, ten private internal subnets, forty-one servers, and approximately 1500 active users each quarter. We have implemented load balancers, a SAN, and server virtualization over the last six months in an attempt to separate services to increase availability and use our resources more efficiently.

I won't get in to many more details, but suffice to say that it's been a long road getting to this point - especially considering that the most complicated upgrades I have had to install so far were all over the summer which just so happened to coincide with my wedding.

The subject of this post is "PHP, Apache, and iCal Server with Load Balancing." Now that I've laid out the foundation, here is my goal for our central web services:

• 1+ Apache web servers hosting the sites for the college. Sites and configuration files are stored on the SAN such that I can set up N servers to host all of our web sites, allowing for seamless upgrades and maximum up time.
• 1+ Mail servers to serve as the MTA for all web services and users; mail is stored on the SAN.
• 1 iCal server to host user, group, resource, and location calendars for the college.

To implement this I purchased two Coyote Point E350 load balancers and placed the web servers behind them. The mail server is a seperate Xserve and the iCal server is virtualized on Parallels Server for Mac.

This seemed like a workable set up but I had a lot of trouble getting the wiki, blog, and web calendar services to work. I quickly discovered that Apple has designed Leopard server assuming that most users would server all of these functions on the same machine. There is no way to set these up in my desired configuration with the GUI alone.

Server and SAN Configuration

Since we are running Apple's Xsan, there are some additional components to this configuration that I haven't mentioned. We have two "Metadata Controllers" which are just two PPC G5 Xserves sharing the master/slave configuration to host the SAN volumes. There are also two fibre-channel SAN switches (notice the theme here -- everything in pairs just in case), a managed ethernet switch to handle the metadata network and the data network, and two load balancers.

Here's what we're using:

• Load Balancers - two Coyote Point E350 Series Switches. Coyotepoint.com
• Ethernet Switch - one Cisco Catalyst 3750 Series switch. Cisco.com
• SAN Switches - two Qlogic SANbox 5602Q Switches. Qlogic.com
• Servers - two Apple Xserve G5's with 4 GB memory.

The actual LUNs consist of Apple Xserve RAID devices of various capacities. I have pooled our existing drives together to form six total volumes, three for metadata/journaling and three for the actual data. This seems to be working quite well. One volume is dedicated to network storage, particularly web sites, file sharing, mail data, etc..., one volume is for virtual machines only, and one volume is for home directories. I believe when all is said and done, the capacities will be 13.5 TB, 2.0 TB, and 4.4 TB respectively.

To address the meta data network required for Xsan, I have created a vlan on the cisco switch with an assigned private subnet and connected all of the Xsan server's secondary ethernet ports to that vlan. I then have the other devices connected to either another private vlan or the public network, depending on their use.

The load balancers work by sharing various "cluster" IP addresses between them, such that if one of them fails, the other will take over and keep the services online. The load balancers are then configured to share traffic between any available, assigned servers for each virtual "cluster", which is configured per port and cluster IP.

For example, our web server cluster:

128.146.45.5 on TCP port 80

As far as the web servers are concerned, all of the traffic they receive is simply routed through the load balancers -- the logs still show traffic coming from the original IP addresses and the firewalls and act accordingly.

Now since the servers need to host all of the same web sites and run the same, identical configurations, I have relocated all of the apache configuration files and the web sites onto the SAN volume and mounted it on both servers.

There is no need to edit any conf files here, just

sudo rm -rf /etc/apache2/sites

and then create a symbolic link:

sudo ln -s /etc/apache2/sites /Volumes/sanvolume/newsitespath

restart apache to reload the configurations

sudo apachectl restart

You can even use server admin to maintain the sites, if that's what you prefer, and they will be maintained across all web servers. This configuration has the benefit that you can add as many web servers and you want (within reason) without as much as a hiccup.

Parallels and Server Virtualization for Near-High Availability

I have already noted that there is a single SAN volume for the virtual machines. I am running three brand new intel xserves, each with a copy of Parallels Server for Mac and each connected to the SAN. On the SAN volume I have created three directories, for organizational purposes -- one for each server.

I have a disk image of the Mac OS X and Windows 2008 install disks on the SAN volume so that i can mount it on any virtual machine when I need it. (To date I have been unsuccessful in mounting a bootable leaopard server ISO so I've been using real DVDs in the host machine's dvd drive) I have also created a template of an installed Mac OS X server so that I can quickly deploy a new virtual machine at a moments notice.

Please note -- they key to using a pre-installed unconfigured Leopard Server template is to reset the LKDC each time after deploying the template:

sudo dscl /Local/Default delete /Config/KerberosKDC

sudo rm -rf /var/db/dslocal/nodes/Default/config/KerberosKDC.plist

sudo rm -rf /var/db/krb5kdc

sudo /usr/libexec/configureLocalKDC

Since there are three parallels servers, if one needs to go offline for maintenance I can quickly transfer the virtual machines to another host and start them up again. Though this process is manual, I hope to either automate using the parallels command line interface or see some clustering come out of the parallels development team. This is not nearly as slick as the VMware ESX solution, but again, I'm betting parallels to catch up rather quickly.

I have used this configuration to set up the virtual machine that runs our iCal Server. After that all I had to do was to run updates, bind to the directory, and enable iCal Server.

Directory Structure for Team Services and iCal Server

The most important thing to note with this setup is the directory entries for each server, service, and group that relates to the wiki/blog/calendering. Just enabling apache and php to host web sites in the load balanced configuration works quite well and chugs along happly. As soon as you try to use team services you start to run into problems.

The initial behavior you will notice is that when you use server admin to enable group wiki/blogs/calendaring you will see each server appear seperately in the directory. That means that when you enable the team services for a group, you have to select one and only one of the servers. You will then see that the particular server you selected will be the only one that actually hosts those services -- such that depending on which server the load balancer directs you to, you will either see your group listed or an empty list.

more to come...

I've gone back and forth on releasing this publicly for a while. I think I'm comfortable enough in developing it to just go ahead and post it now, since the next version I'm working on should be much more dynamic, portable, and ... cool.

After six to eight hours of banging my head on my desk, it looks like I FINALLY have the FTP servers working as well as they were before our leopard upgrade this summer.

( more to come )

Halloween party tonight!

I was out to dinner last night at around 6:30 when I got a page from our status monitor machine -- the vmware host was offline. Then, another -- this time the only server I have virtualized on that box now, our DHCP server, was also down. Since our shortest leases are four hours (and most are eight), I figured I did not have to leave right then and there but make a note and fix it when I got home.

When I got home I confirmed that, yes, the DHCP server was off and there was no chance to recover it remotely. Instead of having to come into the office, thanks to our Parallels servers -- I brought up the new dhcp server I've been working on, restored a backup dhcp configuration and voila -- leases abound.

Apparently one can not restore a standard GUI dhcp backup from a windows 2003 server onto a windows 2008 server. I don't know why this would matter, but then again I don't understand why Microsoft does what it does.

This is from an article I found online:

On the windows 2003 server from a command prompt run:

netsh dhcp server export C:\dhcp.txt all

Copy the dhcp.txt (or backup.txt or whatever you want to call it) to a network share, then down to the windows 2008 server.

On the windows 2008 server from a command prompt run:

netsh dhcp server import c:\dhcp.txt all

Oh, and the DHCP service should be installed and running (no scopes need be definied) when you do the import.

Granted I'm going to have to go downstairs now, recover the vmware machine, and get a newer backup, but I think I'm done with vmware server -- I'm sure it's stable elsewhere, but it's caused nothing but problems for us.

It's been about eight months I'm guessing since I upgraded home-2 to Leopard Server. Today I finally got the weblog service working. I've tried off and on since around February to get it enabled.

The solution?

Thanks to my overzealous sense of security I had several optional authentication mechanisms disabled on our directory server -- one of these being "WebDAV-Digest" which oh-so-eloquently says in plain text "web service clients". It wouldn't be a valid solution if it wasn't completely obvious. I probably would have continued to think something was wrong with the weblog service had I not looked in the password server log (another obvious one I know...):

Oct 28 2008 16:58:40 AUTH: {0x44abf4fe42039ffd00000f9a00001023, strucke1} requested mechanism WEBDAV-DIGEST is not available.

So I went ahead and enabled the proper authentication and naturally, I was still unable to log into the weblog service. This time I got a different error on the directory server:

Oct 28 2008 17:00:37 AUTH2: {0x44abf4fe42039ffd00000f9a00001023, strucke1} WEBDAV-DIGEST authentication failed, SASL error -13 (password incorrect).

Now this error is all to familiar -- reminiscent of our NTLM authentication failures. Since I know exactly how to fix that, this one was easly -- reset the users password to add the appropriate password hash.... and BAM here we are.

Not a bad way to end the day.

Oh -- and I don't care how the game turned out -- Penn State still sucks and GO BUCKS!