[THIN] Re: Best Practices on support structure for large farm?XYZ Customer

  • From: "Ron Oglesby" <roglesby@xxxxxxxxxxxx>
  • To: <thin@xxxxxxxxxxxxx>
  • Date: Fri, 17 Oct 2003 23:12:23 -0500

Enterprise Deployment of Server Based Computing Utilizing Citrix
MetaFrame XP Support Model 

 

Prepared by: Ron Oglesby

Table of Contents

Introduction. 3

Support Model Overview.. 4

Engineering Team. 4

Operations Team. 5

Team Structures. 7

Engineering Team Structure. 7

Skill Sets Required in the Engineering Team.. 8

Engineering Team.. 8

Operations Team Structure. 9

Skill Sets Required in the Operations Team.. 9

Operations Team.. 10

Conclusion. 11

 




Introduction


This document is intended to outline an ideal support and engineering
model used to support an Enterprise class Server Based Computing
deployment.   In this document, the resources required to support the
SBC environment on an ongoing basis are defined in the following ways:

*         A listing of teams within the support structure and their
responsibilities/functions within the environment 

*         A high level decomposition of the skills required within in
team

*         A listing of individual team members and their required skill
sets.

These documents along with the initial design document will give XYZ
Customer a good idea of the overall staff requirements to maintain and
scale an enterprise class farm.  

 


Support Model Overview


The support structure is divided into two categories, the SBC or Citrix
Engineering Team and the Operations or Support Team.  Each of these
teams has different functions within in the environment but some
responsibilities will overlap between the teams.   


Engineering Team


The Engineering Team is responsible for providing solutions to end user
requirements and is in charge of the overall SBC architecture.  Designs,
changes, new implementations, application testing and farm change
control are all the responsibility of the Engineering Team.  This team
will also provide third level support to the Operations Team for issues
that can not be resolved through first and second level support.

Some of the functions assigned to the Engineering Team include:

*         Application engineering

*         DR/Business resumption planning

*         Operating System and MetaFrame "base" build development

*         Testing hot-fixes and service packs

*         Server image maintenance

*         Developing standards for the environment and clients

*         Maintaining the development environment

*         Deploying images and applications from
development-test-production 

*         Application update testing with the application owner or
vendor

*         Managing Citrix licensing 

*         Designing IIS/NFuse configuration and maintenance procedures

*         Designing SQL data store configuration and maintenance
procedures

*         Maintaining Citrix administrators group security

*         Design and implementation of the NFuse portal

*         Developing custom deployment and user scripts

*         Developing and maintaining group policy objects

*         Third-level troubleshooting of escalated issues

*         Developing mandatory and hybrid profiles 

*         Testing and certifying print drivers

*         Trend analysis for server performance and capacity

*         Deploying new servers

*         Developing maintenance and reboot schedules and scripts

*         Organizing and documenting changes to production environment

*         Scheduling production updates with the Operations Team 

*         Testing and certifications of new Citrix clients.

For the most part, the engineering functions are "back-end" functions.
This team will provide solutions to the Operations Team and eventually
the end user.  All new changes are tested at this level and deployment
processes are developed here before moving the changes into production. 

Once the solution is deployed to production it becomes the Operations
Team's responsibility to manage, maintain and troubleshoot the
environment.  The Engineering Team is responsible for deploying a solid
solution and providing third level support to the operations team for
changes that need to be implemented. 


Operations Team


The Operations Team is in charge of the day to day maintenance of the
Citrix servers and farm.  The monitoring of server performance,
maintenance of hardware and operating systems and the troubleshooting of
Citrix issues initially all fall within their responsibility.  In simple
terms, the Operations Team maintains what the Engineering Team designs
and deploys.

The Operations Team will have many different skill sets and levels of
expertise.  These are generally broken down into two roles, server
support functions and end user support functions.  The skill sets
required for these two roles will vary greatly and will be discussed
later in the document.

Some of the functions assigned to the Operations Team include:

*         Managing published applications 

*         Managing the IIS NFuse servers 

*         Managing and backing up the SQL data store  

*         Managing user profiles

*         Managing logon scripts 

*         Re-deploying or recovering Citrix servers

*         Scheduling and maintaining server cycling

*         Identifying problematic servers

*         Monitoring server performance

*         Monitoring event and Insight Manager logs

*         Adding new users

*         Analyzing support ticket trends

*         Managing print drivers within the farm

*         Maintaining virus software 

*         Managing published application settings

*         Supporting client/browser 

*         Resetting/Managing user sessions

*         Troubleshooting login and connection issues

*         Handling client requests for profile modifications

*         Supporting client printing 

*         Organizing and documenting changes to production environment

*         Scheduling production updates with the Engineering Team

The list of Operations functions contains the day to day tasks within
the production farm.  The Operations Team will use guidelines and
procedures created by the Engineering Team to maintain the farm and its
infrastructure.  Changes to the farm or servers will be requested by the
Operations Team, approved and tested by the Engineering Team, then
deployed with the cooperation of both teams.


Team Structures


Each of the teams will require a certain level of expertise.  The
Engineering Team will be recognized as the SBC or Citrix experts and are
responsible for the overall stability of the farm and management of the
processes and procedures used to run the farm.  The Operations Team on
the other hand will be required to have a level of expertise that will
allow them to follow these procedures to maintain the environment and
support the servers and users.  Typically the Engineering Team is much
smaller than the Operations Team but will consist of resources with very
strong skill sets.

The number of resources required in each of the Teams will vary
depending on several factors:

*         Size of the farm

*         Number of end users 

*         Geographical locations of servers and users 

*         Number of applications and changes that are required on a
regular basis

*         Experience of team members

*         Stability of existing environment

*         Policies and procedures in the environment

*         Number of on-going projects and deployments.

For the purposes of this document we will assume a fairly stable farm of
200 Citrix servers with a base of 100 supported applications.  This farm
consists of ongoing application changes and new application requests,
along with two data centers housing Citrix servers.


Engineering Team Structure


The Citrix Engineering Team for this farm will contain four or five
members including an Engineering Team Lead.  The members of this team
will have a high level of expertise with Citrix and the Operating System
on which it runs, a general knowledge of networking and the environment
in which the Citrix servers reside, and application and troubleshooting
skills.  The team members should also be able to function in a
structured and controlled environment and have the ability to design
solutions around and within a defined set of parameters. 

The Team Lead will have overall responsibility for technical solutions
coming from the engineering team. He will also participate in, and
supervise or sign off on, all technical projects within the Engineering
Team.  The Team Lead should have a mix of technical abilities and
project management skills along with the ability to view the "big
picture".




Skill Sets Required in the Engineering Team


Ideally the Team Lead will have the strongest Citrix and Terminal
Servers skills in the group and will make decisions on technical
projects based on their knowledge of the servers and their environment.
Other skills will be required within the team and some skills will need
to be found outside the team.  At times the Engineering Team will need
to go outside the group for these specific skills such as a DBA or
Security Analyst. 

Below is a chart that lists the skill sets required by team members.
These skills may be mixed and matched within the team.  

Skills are listed using the following scale:

Familiarity:        Rudimentary knowledge of the process or tool to
perform a predefined task.

Understanding:  General knowledge of a process or tool that allows the
design of a simple solution or successful troubleshooting of a simple
problem.

Knowledge:        Proficiency with a process or tool to design most
solutions or the ability to troubleshoot most problems.

Expertise:           Comprehensive knowledge to design detailed
solutions and troubleshoot complicated problems for a particular process
or tool; "The buck stops here."


Engineering Team


Sill Sets and Experience

Role

Skills Required

Number Required

Citrix Team Lead

*         Knowledge of Applications

*         Expertise of Tools

*         Expertise of Citrix

*         Expertise of Windows 2000

*         Knowledge of scripting

*         Knowledge of Project Mgmt

*         Knowledge of Environment

 

 

1

Sr. Citrix Team Members

*         Knowledge of Applications

*         Knowledge of Tools

*         Expertise of Citrix

*         Knowledge of Windows 2000

*         Knowledge of scripting

*         Understanding of Project Mgmt

*         Knowledge of Environment

2

Jr. Citrix Team Members

*         Knowledge of Applications

*         Knowledge of Tools

*         Knowledge of Citrix

*         Knowledge of Windows 2000

*         Understanding of scripting

*         Familiarity of Project Mgmt

*         Understanding of Environment

0-1

 




Operations Team Structure


The Operations Team for this farm is integrated directly into the normal
support structure for the network.  Members of the Operations Team will
all have a general knowledge of the Citrix environment and be able to
provide basic troubleshooting for problems.  Within the Operations
support structure though a subset will more than likely become the
operational experts and be used for the majority of the troubleshooting
and maintenance.

This team should also have a technical Team Lead that will have overall
responsibility for the management of the environment.  This lead should
work closely with the Engineering Team Lead to help coordinate and
schedule changes within the environment.  The Team Lead for the
Operations Team should be expert enough with the environment that he
could be a senior member of the Engineering Team.  This person's skill
set should include expertise in Citrix and the operating system along
with knowledge of proper project management and change control
procedures. 


Skill Sets Required in the Operations Team


All members of the network support staff will have some general
knowledge of Citrix but will generally not deal with the routine
maintenance. These staff members are generally located in the helpdesk
and are not what we are describing in the following sections. Instead
all helpdesk personnel should have a basic knowledge of Citrix, Printer
creation, sessions, and basic trouble shooting to determine what type of
problem the user is experiencing.  

The staff that maintains the Citrix servers and environment should have
more experience or more training with regards to Citrix. These team
members will maintain the environment based on policies and procedures
developed in the Engineering Team. Troubleshooting and problem isolation
will also be their responsibility. Basic "fixes" and workarounds are
generally developed by this team. If a problem is found that requires a
change in the environment or a problem can not be resolved at this level
it should be escalated to the Engineering team where a resolution can be
found and if necessary a system or design change made.

Below is a chart that lists the skill sets required by team members.
These skills may be mixed and matched within the team and obviously some
team members will be stronger at one skill and weaker at others.

Skills are listed using the following scale:

Familiarity:        Rudimentary knowledge of the process or tool to
perform a predefined task.

Understanding:  General knowledge of a process or tool that allows the
design of a simple solution or successful troubleshooting of a simple
problem.

Knowledge:        Proficiency with a process or tool to design most
solutions or the ability to troubleshoot most problems.

Expertise:           Comprehensive knowledge to design detailed
solutions and troubleshoot complicated problems for a particular process
or tool; "The buck stops here."

 


Operations Team


Sill Sets and Experience

Role

Skills Required

Number Required

Team Lead

*         Understanding of Applications

*         Knowledge of Tools

*         Expertise of Citrix

*         Expertise of Windows 2000

*         Understanding of scripting

*         Knowledge of Project Mgmt

*         Knowledge of Environment

 

 

1

Sr. Team Members

*         Understanding of Applications

*         Understanding of Tools

*         Knowledge of Citrix

*         Knowledge of Windows 2000

*         Understanding of scripting

*         Familiarity of Project Mgmt

*         Understanding of Environment

1-2

Jr. Team Members

*         Understanding of Applications

*         Understanding of Tools

*         Understanding of Citrix

*         Understanding of Windows 2000

*         Familiarity of scripting

*         Familiarity of Project Mgmt

*         Familiarity of Environment

3-4

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

As illustrated in the above skills matrix, these team members need the
basic ability to troubleshoot common problems and situations within the
MetaFrame environment.  Help desk calls that cannot be resolved should
be escalated to this team and troubleshot until a root cause for the
problem is found.  If the cause is found, it can then be fixed or
escalated to the Engineering Team for review. 

These team members can also be multipurpose team members and have duties
outside of Citrix.  It is preferable that their expertise is Citrix and
that they have the time available to support the Citrix environment. 

 


Conclusion


The support requirements for a large server farm are much like any other
software in the enterprise.  For example, if Exchange is used in an
enterprise environment, the enterprise usually has a dedicated Exchange
Team.  The Operations Team will have a general knowledge of the product
and how it functions and can resolve small issues.  The helpdesk will
take calls from users on general connectivity issues and problems with
the client.  Problems that cannot be fixed at the helpdesk or by the
Operations Team are then escalated to the Exchange Team for a
resolution. 

This division between teams allows expertise to be built within a
specific job function.  Focusing on the same types of tasks consistently
increases the team's job proficiency.  The Operations Team will become
familiar with all the maintenance procedures and be able to handle most
small problems that arise.  The Engineering Team becomes very good at
designing solutions, testing applications, working with application
owners to build stable scalable servers for clients.  This highly
developed support structure will result in a stable, easy to manage,
Server Based Computing environment.

 

 

Ron Oglesby

Senior Technical Architect

 

RapidApp

Office 312.372.7188

Mobile 815.325.7618

email roglesby@xxxxxxxxxxxx

 

 

                                                                                                                                                                                               

                                                                                                Kemper Claims Server Based Computing Environment

ABN AMRO                                                                                                                                      Support Model Best Practices

 

                                                                                                                                                                                              2

Prepared by RapidApp                                                                                                                                                                   4

ABN AMRO                                                                                                                                      Support Model Best Practices

Prepared by RapidApp                                                                                                                                                                   ii

                                                                                                Kemper Claims Server Based Computing Environment

CUSTOMER                                                                                                                                    Support Model Best Practices

 

                                                                                                                                                                                              2

Prepared by RapidApp                                                                                                                                                                   4

XYZ Customer                                                                                                                                Support Model Best Practices

Prepared by RapidApp                                                                                                                                                                   ii

Other related posts:

  • » [THIN] Re: Best Practices on support structure for large farm?XYZ Customer