Home
A Partially Complete IT Framework Analysis
Goal: Make a recommendation for a worthwhile and “productive” IT Framework, along with an OS stack offering high availability and full redundancy.
Framework Comparisons
Option 1: The ITIL Framework - http://itsm.certification.info/itilv3quals.html
- Pros:
- Normalizes terminology (although conflicting)
- Provides transparency into IT operations
- Doesn’t advocate reinventing the wheel
- Cons - http://itsm.certification.info/itildisads.html
- Only focuses on a very small area of IT
- Terminology conflicts majority of industry
- Tendencies to over complicate
- Potential for extreme cost and time required, if not implemented properly
Option 2: Val IT Framework v2.0 - http://en.wikipedia.org/wiki/Val_IT
- Pros:
- Combines & extends principles from both ITIL and COBIT
- Introduces some Tangibility (balancing the apples & oranges)
- Introduces full life-cycle management of IT investments.
- Covers a full scope of IT
- Presents value delivery practices
- Categorizes investments
- Assigns accountability
- Largely proactive by nature
- Streamlined approach
- Introduces Portfolio management and presentation
- Is versatile enough to be applied to non-IT governance as well
Option 3: COBIT - http://www.isaca.org/Knowledge-Center/cobit/Pages/FAQ.aspx
Option 4: The Risk IT Framework
Option 5: Utilize a combination of multiple frameworks, (like most companies have ended up doing), to provide support for all areas of IT relations.
# Customized (high-level) suggestion of merged governance
IT Framework Consultants & Training
Archived Webinars - http://www.isaca.org/Education/Online-Learning/Pages/webinars.aspx
On-Site IT Audit Training Courses - http://www.isaca.org/Education/On-Site-Training/Pages/On-site-Training-Course-Descriptions.aspx
IT Infrastructure Requirements

Load Balancing All Resources for high-availability
Web server farm
A Do-It-All Attempt at IT Framework & Governance

Top Framework Requirements:
- Incident Management (Helpdesk Software)
- The objective of Incident Management is to restore normal operations as quickly as possible with the least possible impact on either the business or the user, at a cost-effective price.
- Processes Include:
- Identifying the incident’s root cause
- Develop & document a peer reviewed & approved action plan
- Consult Change Management
- Calculate required implementation time
- Execute action plan & prove it’s completeness via QA
- Catalog documentation for future incident reference
- Support Problem Management
- Provide a helpdesk ticket request system to centralize & track bug reports
- Processes Include:
- The objective of Incident Management is to restore normal operations as quickly as possible with the least possible impact on either the business or the user, at a cost-effective price.
- Release Management
- Implement changes to IT services which consider all aspects of a change including planning, designing, documenting, building, continuous integration, unit testing, training, communications and deployment activities.
- Utilize Continuous Integration and application unit testing
- Identify & resolve potential application bugs prior to QA review
- Establish a proven & stable method of deploying approved application builds for release.
- Document procedures for each release
- Puppet configuration changes
- Required webserver configuration changes
- Application global config changes
- Configuration deployment in sync with application releases
- Document complete Roll-Back procedures
- Support Incident Management
- Perform adequate QA evaluations of application post-release
- Maintain a checklist evaluating all application functionality
- Newly implemented functionality
- Known related legacy functionality
- Review remaining unknown functionality
- Append to a collection of any release related incidents
- Future support of Incident Management
- Maintain a checklist evaluating all application functionality
- Puppet configuration changes
- Utilize Continuous Integration and application unit testing
- Service Desk
- Provide a strategic central point of contact for customers and support the Incident Management process by providing an operational single point of contact to manage incidents to resolution.
- Manage bug reports
- Identify levels of incident criticality supporting Incident Management
- Graph & Report on incident trending
- Support Capacity Management requirements to meet expected service levels
- Provide a strategic central point of contact for customers and support the Incident Management process by providing an operational single point of contact to manage incidents to resolution.
- Service Level Management
- Plan, coordinate, negotiate, report and manage the quality of IT services at an acceptable cost.
- Providing & maintaining a highly-available, high demand infrastructure on the Amazon Cloud supporting
- Combine aspects of Capacity Management & Service Desk to evaluate regular review of IT resource availability.
- Plan, coordinate, negotiate, report and manage the quality of IT services at an acceptable cost.
- IT Financial Management
- Provide budgeting, accounting and charging services to control, manage and recover IT cost and spend.
- Evaluate different operational cost comparisons
- Support cost effective decisions for IT purchases
- Estimate & evaluate application & traffic resource demands
- Establish padding for a potential business growth factor
- Growing storage needs
- Customer traffic demands
- Application dependencies for high-availability
- Legacy hardware end-of-life cycles
- Establish padding for a potential business growth factor
- Evaluate different operational cost comparisons
- Provide budgeting, accounting and charging services to control, manage and recover IT cost and spend.
- Capacity Management
- The discipline that ensures IT infrastructure is provided at the right time in the right volume at the right price, and ensuring that IT is used in the most efficient manner.
- Maintain current IT employee workload padding for unexpected event handling
- Developers - Identifying Incident extent, development and deployment of application hot fixes
- QA Team - Provide and extent of on-demand availability for unexpected incident testing and release.
- Maintain current IT employee workload padding for unexpected event handling
- The discipline that ensures IT infrastructure is provided at the right time in the right volume at the right price, and ensuring that IT is used in the most efficient manner.
- Availability Management
- Optimize the capability of the IT infrastructure, services and supporting organization to deliver a cost effective and sustained level of service availability that meets business requirements.
- IT Service Continuity Management
- Support business continuity management functions by ensuring that IT services can be recovered in the event of a major business disruption within required timescales.
- IT Security Management
- To prevent the occurrence of security-related incidents by managing the confidentiality, integrity and availability of IT services and data, inline with business requirements at an acceptable cost.
- Regularly update server & application software
- Apply software security patches
- Vulnerabilities as discovered & released
- Maintain forward compatible application code
- Apply software security patches
- Restrict SSH access to Public Key authentication only
- Maintain a repo & puppet template for authorized_keys
- Restrict SSH to non-standard port (i.e. 2222)
- To reduce brute SSH probe password attempts
- Maintain STRICT firewall configurations, for only necessary services.
- Utilize centralized authentication management, such as
- LDAP Authentication
- Utilize AWS Route53 for ultimate DNS stability
- Perform regular external vulnerability evaluations
- PCI Compliance scans
- McAfee HackerSafe audits
- XSS Scripting Vulnerabilities
- SQL Injection Vulnerabilities
- Regularly update server & application software
- To prevent the occurrence of security-related incidents by managing the confidentiality, integrity and availability of IT services and data, inline with business requirements at an acceptable cost.
Google Android - AppInventor
Well, we knew this day would come. Our friends at Google have finally developed a WYSIWYG GUI for Android Development that runs in your browser!
I've already built and tested 4 Apps for my phone. I've also registered for the Android Developer Network, and I'm all set to publish meaningless apps to the Android Marketplace.
Check out these screenshots of the interface:

Simple drag n' drop interface for adding controls and elements to the view port.
The Blocks Editor provides an additional drag n' drop, puzzle-like interface for constructing logical flows and event handlers in your app.

Is it me, or does it seem like SkyNet is only a few months away at this point?
Cisco ACE 4710 Load Balancer - CLI Integration
Managing hundreds of websites will stretch almost any piece of hardware to it's limits. Combining that with SSL's and IP address configurations on the load balancer level, makes it an even bigger ballpark. One of the greatest tools I've ever stumbled-upon (Thanks Clark) was Expect.
Expect, is a great tool for compiling action scripts for your server to execute. In most other methods of CGI initiated shell execution, commands are cut and dry. You can send a command and buffer the standard out returned.
This method is very limited, because it doesn't allow you to do anything that involves an interactive shell, such as entering a password to SSH to another server, etc.. Expect allows you to write logic into these "Expect Scripts" that you write.
For example,
spawn ssh # open the new ssh session within the script executeion
send "root\n" # send the username
expect "Password: " # expect the following prompt to be returned from the server
send "mypasswd\n" # send the password
expect "root@web01:/root# " # expect the shell prompt
And you can continue from there.
After inheriting the vast riches of Cisco, our new ACE Balancers required numerous actions to configure new domains and their SSL proxies. I believe we cataloged 35 different actions needed to correctly install a new domain/ssl for our platform. Since we had to do multiple per day, this quickly shaped up to be a fulltime job.
And rightly so, if anyone remembers back to the original Coyote Point balancers? They foolishly did reverse DNS lookups for every cluster configured, on EVERY PAGE LOAD of the gui, because they were all displayed in an html frame on the left hand side. When we finally hit our limit of 500 clusters, and the system started refusing to run, the gui page loads took soo long, the browser window often timed out.
Not that that would be the case with these new Cisco boxes, since we were able to import the original 500+ sites initially, and since added another 250 more, we only needed to worry about the administrative time required for each setup.
Anyone who knows Cisco, knows it always provides a command line interface to configure it's hardware. Basic and Enabled access over SSH. When we were first introduced to the new hardware, and a mass confusion of code developed by the former Soviet Union of Vladulence, we found out just how NOT to do things!
These guys, a team of outsourced cyber-mafia-outcasts, came up with the spectacular idea to use PHP's fsockets() and create Telnet sessions to the balancers! TELNET! Of all things, when SSH was readily available. To make a long story short, the sessions almost ALWAYS dropped or timed out. This usually occurred right in the middle of creating a domain cluster, and as a result would trash the master config file of the balancer, and Down Came The Wall!
We had the great idea to use PHP to generate the Expect script, which was then executed to create the SSH tunnel & perform all of the CLI commands needed to setup each cluster, including generating the SSL Key, CSR, and utilizing the Comodo API to actually submit the CSR, purchase the Certificate, download it, and install it in the balancer.
This was very tricky, thanks to Comodo's validation service. CSR's submitted would often NEVER return the certificate immediately, in time to maintain script execution. To compensate this, we developed a queue of actions pending execution. Things such as generateKey, generateCSR, purchaseSSL, collectSSL, importCert, createVirtualServers, etc..
Each one of these actions had it's own modularized dynamically parsed Expect Script. This allowed each action to be completely independent of one another. All of these were based on a framework controller written in PHP that checked statuses of each domain being configured. Each one was configured with prerequisite actions requiring completion before it's execution was allowed. (Can't buy the cert, if the CSR was never generated, etc..)
A cron service would constantly check the table and query Comodo's API for issuance of each pending Cert purchase. Once the cert was made available, it would be downloaded and instantly configured into the balancers.
After a year passed by, it became apparent that alot of "junk" clusters existed and they were not only slowing down the balancer resources, (regex & cpu threads) but they were also holding our IPv4 addresses for ransom. Think it would be easy to just log in and click delete on a cluster to remove it, and unbind the IP ? Think Again!
Cisco's software required us to perform EVERY SINGLE COMMAND required to create the entire domain configuration, IN REVERSE, in the order that it was entered! Mainly due to the complex nesting that took place with the associations of server farms, policies, proxies, etc..
Needless to say, it took 4 TIMES longer to remove a domain, than it did to add it!
Expect To The Rescue!
All it took was to have php dynamically compile one more Expect action script, and send it off to the balancer for processing. The cron queue action used to identify it was called "bleachDomain". The name says it all.
Sorry, but I no longer have the ability to provide any screen shots of this system, since the hardware has all been decommissioned, after we migrated the entire datacenter to the Rackspace Cloud.
Thanks for reading....
MySQL Based Apache Config Generator
In an industry where websites are built from scratch, pumped into a home-brewed e-commerce/content management system and tore back down 3 months later, an ever increasing burden existed to constantly be modifying Apache configurations. Anytime more than one person is involved in modifying a file, there's potential for integrity corruption.
After scouring the internet for some such existing solution, (and lacking the ability to switch to another web server), I quickly found the need to develop a managed and "idiot proof" method of configuring Apache Virtual Hosts.
The Back End:
A database table of "vhosts" was created to store all the individual host configurations such as ServerName & Alias, DocumentRoot, Logging Levels & Locations, and the ability to perform trackable redirects to other sites/domains. A lot of instances existed where upper management and/or clients would decide to buy 50 different misspelled versions of their product's name, and ask on a Friday afternoon, if they could be ALL setup, tracked independantly, and redirected to the original website, in time for distributed media ads that were scheduled to run the next morning.
A template file was used to set the basic layout of the vHost config block.
Tags such as {DOCUMENTROOT} and the like, were used to define the layout positions in the file. The database was queried for all active records and the config file rebuilt and reloaded each time qualifying changes were made to the database. Servers each ran independently of each other on 15 minute crontab entries. Each server would query the queue for a "RUN NOW" flag identified by their hostname. If found, the script would then begin creating the new config file from the entries returned from the database query.
After the content for the new config file was built, the parent processor would then backup the original file, and perform an "apache2ctl configtest" on the new one. Standard Out was buffered and parsed via string comparison to determine Apache's test results. If any error messages were detected, the new config would be abandoned, the old restored, and the admins notified of the issue and offending vhost record. This allowed a fully automated solution, while still protecting the web server from Apache failing to start due to config file errors.
The Front End:
We also quickly discovered the need for an easy interface to enter all these configurations. Ultimately our goal was to start training other "less technically inclined" people to use it and "share" the programming workload. (Especially since they were the ones who procrastinated and dumped it on the programmer at the last minute anyway!)
I built this visual tool to view all the domains in the database, and added Aplha filtering for easy sorting and view. Over 1000 domains passed through this system over the course of the years in operation.

Modifying the individual Virtual Host config.



