The Garage Datacenter Hardware Stack ...

So lets get down to it, what is the hardware stack we are using for the application ...

Servers

Application Servers

The servers used to house the application in a HA and Load Balanced environment are the Compaq Proliant ML350 servers. They make up the first half of the applcation cluster'd environment

The servers each have the following specifications:

2 x Compaq Proliant ML350's Running
- Dual 1.4 ghz CPU
- 2 Gig of Ram
- 6 x 18.2 SCSI disk

The second half of the of the application cluster are the Compaq Proliant ML370 servers. The specification of the servers are as follows:

2 x Compaq Proliant ML370's Running
- Dual Pentium III 1.4 ghz CPU
- 2 Gig of Ram
- 6 x 18.2 SCSI disk

(pretty much the same as theh ML350's)

Database Servers

The database servers are also going to be configured in a clustered environment. They will be running the mySQL clustering suite over 4 servers. The servers to be used have the following specification:

4 x Compaq Proliant 6400 Running:
- Quad Pentium III 700mhz Xeon
- 3 Gigz of Ram
- 4 x 36.4 Gig hard disk

Storage Servers

The storage servers are used to store various files and videos that are uploaded by the users. The servers used for this purpose have the following specification:

2 x Compaq Proliant ML570 Running
- Quad Pentium III 700mhz Xeon
- 2 Gigs of Ram
- 12 x 36.4 Gig hard disks

Batch Processing Cluster

There are various batch job that need to be run and various user executed processes that can contribute to the workload on the application servers. These processes have been identified and have been migrated to a seperate cluster so that any execution of these processes do no conflict with the operations of the application servers. This is to ensure that the user browsing experience is not hindered or degraded by any backend processing jobs. The following hardware has been used for these servers

10 x Dell GX270 optiplex Running
- Pentium IV 1.8ghz
- 768 megs of Ram
- 1 x 70 Gig hard disk

Racking and Storage

All of the equipment mentioned above along with the switches and networking equipment is housed within 2 x compaq 9000 racks (white in color).

Well there you have it .... the initial datacenter setup. The next post is the more interesting area of hardware, that being ... how we procured all this hardware at reasonable costs.

Procuring hardware ... or not ... decisions decisions ...

To buy the hardware or not to buy the hardware .... that is the question .... hmmm ....

So the goal of the garage datacenter team was to purchase the hardware as cheap as possibe but to ensure that the hardware platform does not jeopordise the application performance. There are many options to go with regard to the hardware platform , there is the hosting of own hardware and there is the VPS route. There are pros and cons to both and its basically about weighing out the best options:

Hosting our own hardware

Pros:

Pros1:

There is full flexibility on our server configuration. We can use as much hardware as we like and there is no additional price for extra hardware or resources other then the cost of the equipment.

Pros2:

We have full control of the parameters that we require and we can do various different clustering configurations with ease

Pros3:

There is no prices for rack space or any additional cost of hosting

Pros4:

Increasing resources like capacity, memory etc, doesnt attract a premium price as you can buy the memory cheap and do it yourself.

Cons:

Cons1:

Electricity and power consumption needs to be considered. It may be insignificant but it depends on how much power we are drawing per day. We also need to consider the amount of watts being used from a saftey point of view and ensure all our power sockets, fuses and breakers can withstand the current being drawn.

Cons2:

Internet costs need to be considered because if the application becomes popular, increasing bandwidth may require clever bonding or aggregation of connectivity to increase the speed of the internet or going with a bigger pipe which may have a much larger cost then currently being used, if available. Worst case scenario is that there is no option to increase bandwidth and it may require that there is a relocation of equipment

Cons3:

You must be full in charge of your own own SLA's and uptimes. If a server fails then you need to make sure you are on hand to ensure it comes back up in an acceptable amount of time.

Virtual Private Server - (VPS)

Pros:

Pros1:
SLA's are taken care of by the datacenter operations team so any downtime will be addressed by a specific datacenter team

Pros2:
Speed of the internet is taken care of by the datacenter operations team and will increase when required

Pros3:
There is no seperate price for elecricity as it is bundled with the whole package

Pros4:

Prices for VPS's are usually very resonable if only a small amount of servers is required, however when requiring more then one or two servers, the price can be quite costly

Cons:

Cons1:

If you require a standard server configuration its fine, but anything outside the standarad requirement will attrack a premium price

Cons2:

Requiring an increase in the amount of servers or specifications can be quite expensive, so requireing 5 servers can attract a price of $250 to $300 a month assuming that a VPS is about $30 - $50

Cons3:

Less ability to tune and tweak the network and hardware setup to meet needs

The Decision

The decision is not an easy one to make cause as you can see there are quite a few for and against. You need to think about growth, ease of maintanence and how you expect to grow. It is also highly dependent on how you want to run your application.

The decision that the garage datacenter team was to host our own equipment. We had alot of equipment already lying around which we could use, and we knew that purchasing other equipment second hand could allow us to get the hardware we wanted at a lower price.

One of our main concerns was the power saftey issue but after discussing with various electricans installing larger power sockets is not an issue. Also with the introduction of Annex M we can get very good upload speeds at a very good rate and if larger bandwidth is required, we can go via the bonding method or increase our internet requirements with the ISP as there is still room to move.

With this we began to build our equipment requirements and began to purchase our servers nd network equipment from all different locations to ensure that we can get the most reasonable price and still run our application correctly. Our next phase is the actual procurement of our hardware, this alone was a pretty interesting journey ....

Operating System Install ... Boring ... How to pass the time ...

OK so we are installing operating systems on the hardware that we have. Its a dead boring but obviously critical activity. You cant leave the machine to install on its own cause you have to press the 'next' button and you cant ask your little sister to do it cause she would either one, make a mistake or two ask for a significant amount of funds to do it, so like everything else in this world its better that you do it yourself. So how do you install the operating system without falling asleep ? There are a multitude of ways to do this so let me list them ...

Argue about politics ... never ending discussions about why we are getting ripped off by the government and why we are paying to much taxes and why we should over throw the government

Argue about why the operating system is taking so long to install.

Argue about why the IBM x340's are not working and why swapping disks around and attempting to solder a larger capacitor on the scsi card is a waste of time

Argue about why t isnt a good idea to paint the roof white even though it might increase the brightness of the garage

Argue why beer *does* make you bloated and fat and the fact that beer doesnt make you bloated is BS

Argue about why my 1.1 litre mazda 121 *definetly* uses less fuel then the 3.6 litre holden commodore even though the driver of the holden commodore may be an *extremely* good driver and can see how much fuel is being used up depending on how hard the throttle is pushed

Argue about why using a particular ISP is crap and that why its better to go with another even after obvious figures

Play music and sing happily ...

So there you have it, a list of things to do whilst installing the operating system. All very productive things to do as you can see. Oh and the last point of 'Play music and sing happily' ... Here is a list of songs we play

- Stand by me - Ben . E . King
- Living on a prayer - Bon Jovi
- To late to apologize - One Nation
- Always - Bon Jovi
- Imagine - Beatles
- Sweet child of mine - Guns & Roses
- summer of 69 - Bryan Adams
- heaven - Bryan Adams
- wonderful tonight - Eric Clapton

And a whole list of other songs i cant remember right now ....

Putting together all the pieces ...

So all the peices are there, well the main components anyway,

The application idea
The working space
The people (Garage BA, Garage CEO and Datacenter Hitman)
(and most importantly) The drive and motivation to finish the project

Once all the main ingredients are there the aim of the game is to combine it all to produce your solution and most importantly to keep the momentum going and drive throughout.

The development of the solution began with discussing the business model and the functionality of the application. The initial framework and functionality of the application was discussed by myself (Garage CEO) and my first partner (Garage BA). We looked at the different functions that could be implemented into the application that would make it unique and different to all other similar applications currently out in the market.

This process took about 3 weeks in total and the output of this was some business process flows along with some initial designs of what the user interface would look like. The detail of the documents was not very deep but as we were going to develop it ourselves we knew what we wanted to do in detail so we used the business process flow documents just as a guideline during the development. I created the user interface documents so that when designing the initial cut of the user interface i had a rough of idea what it would look like to prevent me from designing it as i was developing.

A parallel activity to the development of the business process flows of the application and the user interface design was the procurement of hardware. The idea was that we would have redundancy and decent processing power at a total cost of below $1000 dollars. I along with Datacenter hitman embarked on this journey of purchasing low cost hardware and equipment. Let me outline what we run today as an initial setup. The journey of how we procured this hardware will be done in detail as the story progresses as it is a story on its own however below is the outline of what we are now currently running.

Server equipment
- Compaq proliant G2 ML350 x 2 (dual 1.4 ghz + 2 gig of ram)
- Compaq proliant G2 ML370 x 2 (dual 1.4 ghz + 2 gig of ram)
- Compaq proliant 6400 x 4 (quad 700mhz Xeon + 4 gigs of ram)
- Dell poweredge 2550 x 2 (dual 1.1 ghz + 1 gig ram)
- Dell GX270 desktops x 10 (1.8 ghz + 512 gig ram)
- Compaq proliant ML 570 ( quad 700mhz Xeon + 2 gigs of ram)

Network equipment
- Cisco 5000 series switch
- 3com superstack switch
- baynetworks stackable switch
- Cisco 877 m k9 router

Misc equipment
- Compaq 9000 rack x 2

Equipment thrown out after either getting them for free or purchasing at low cost
- IBM x340 eServer x 2 (didnt work as had faulty mother boards)
- Compaq Alpha upright server ( had no memory etc etc)

Of this equipment the described above the first set of servers that were bought was the ML350's. They are great servers and dont have any problems even until now. Very easy to configure and setup and also to maintain. Datacenter hitman had the role of checking all the equipment and making sure that there were not faulty components and if everything checked out alright, to install the required operating system.

Both the activites progressed in parallel and as Garage CEO i was across both the activities. I would schedule my time as follows to ensure that i could cover both parts.

Weekdays:
During the week days I would come home from work and develop the application, (in the early stages) either on business processes or the user interface design. I would get home at about 6:00pm from work, have a shower, eat then begin coding

Weekends:
I would get up at about 6:00am in the morning, have breakfast, have a shower then begin coding until about 12:00 in the afternoon. Garage hitman would usually arrive at about 12:00 - 1:00 so at that point in time we would begin on the hardware configuration.

It was a great setup and things would get done and progress smoothly. Setting up hardware can be an exciting job but when installing the OS it gets down right boring ... however we are resourceful guys and we would do different things to pass the time ....

The Garage Datacenter Build ...

Lets step back to the post on the birth of the garage datacenter ...

It was by chance that we were able to get this garage. I was contemplating how i could get space to house this project and didnt have space in my current garage as it was packed to the brim full of old fridges, TV's, furniture clothes and other stuff that belongs in the tip.

I was standing infront of the garage and looking at how i could move all this equipment. I called various different storage facilities but i wasn't happy with the prices they were quoting, it just didnt seem like it was worth it.

One day, simply by chance i was walking upstairs and noticed that some new people had moved in downstairs, they were a new young couple and it seemed like they didnt have to many possessions at the time. It seemed like they were recommended to move in to this flat by a next door neighbour of mine. As i was walking upstairs i heard them having a discussion about the space, about the garage and i overheard the new couple saying they didnt have much things and the garage would be empty. That night as i was laying in bed i decided to walk downstairs the next morning and ask if they would rent the garage to me.

The next morning i actually went downstairs and knocked on the door and asked them about paying some rent for their garage space. Without to much hesitation they agreed .... i was shocked cause an initially seemingly difficult problem to solve was solved, i now had a whole garage to use and it only set me back $10 a week ! I thanked them and asked them to give me a reciept everytime i paid, they agreed and its been like so ever since.....

Moving all the bits from my current garage into the new garage wasnt easy, it took three days of back breaking lifting and wheeling. There was rearranging and arguments about what should be thrown out and what shouldnt. Eventually everything migrated from the current garage to the new one i had rented. It was great i now had all the space i needed and i started to rearrange the equipment i already had. I pulled in my development desk, my stack of dell pc's my switch and other cabling that i already had .....

Now that was needed was to spruce up the garage and to start purchasing the required equipment. Purchasing equipment sounds easy right ! Give money get equipment, well this wasnt the case for the garage datacenter team, lots of interesting events along the way but i guess it adds to the experience and the overall journey that we have embarked on.

Beginning the development, past present and future ....

In the past i have done various different types of development ....

As a graduate i worked in the SCADA engineering area and developed modules for realtime systems more specifically control systems for the electrical industry. I used to develop some of the bearer modules that communicated with the remote RTU's and used to develop some of the monitoring processes that would generate alarms when various components out in the field failed.

As a graduate the work in SCADA was great. My experience in this industry taught me alot and opened up alot of doors. You may think that SCADA is a very specialized field (which it is) but if you look at it from an IT perspective it covers alot of key areas.

- The requirement for redundancy
- Mission critical applications
- Required quick response times
- Protocols and communications
- Communications with external devices
- Realtime processing
- etc etc ...

As a graduate usually the menial jobs are passed on and all we get are the crappy repetative development roles but the development that i was doing was exciting. Along with development of code and i also go a view of how infrastructure is built to support realtime mission critical systems, something that i dearly value when looking back as it was experience that would contribute to my future roles.

Most of my development here was in C and C++ with alot of database work and work with different protocols used in the SCADA industry such as DNP and Baker.

I did various other support roles in different industries over time and i had a good technical background because of this experience.

Over time i moved into purely infrastructure work and initially did work as a unix/linux engineer and also started to get into network engineering through my own studies of cisco technology. It was great cause infrastructure work makes way for alot of travelling and as a young dude travelling for work is ideal. I was able to get into infrastructure work (initially) because i had experience when i was a graduate working on the SCADA systems infrastructure doing the basic network setup, configuring and setting up all the linux and sun servers along with the (at the time) windows 2000 servers.

Over time my development work decreased other then perl and shell along with some basic C and my infrastructure work started to increase until eventually i didnt do any more development except for unix shell.

So what does this have to do with the garage datacenter build and the development of the software to be housed by the garage datacenter ...... ?!? Well everything ......

See when you have done development before you think that you will never loose what you know, you developed throughout university, you developed in the early stages of your career, you look at code and you understand it, but for me anyway ... it did start to slip away and when i came to begin developing the application to be housed in the garage datacenter it didnt flow like it used to.

At the time when i was developing web applications they mostly consisted of cgi scripts (very complicated ones) to do alot of what is now support by the various frameworks. Now we have great technology such as ajax, XML and alot of remote communication libraries are built into the standard release of various programming languages, the problem was i had no idea about these and had to begin the learning processes again (dont get me wrong though ... it was fun and i enjoyed it).

After considering various frameworks and building all the sample applications from each of the frameworks such as django, ruby on rails, turbogears, zope and all the others i settled for a slightly different option and im glad i took the alternative route.

The development now is going well (i believe) and i feel i am comfortable with the option that i have chosen. Development happens everynight after dinner which is about 7:30pm - 8:00pm and extends to about 12:00am - 1:00am depending on how sleepy or how close i am to completing a certain module. I enjoy the development because its a great change from the mind numbing conversatios i have at work regarding issues that are really none issues but creates work and makes people look busy.

I can pretty much speak for all the members of the garage datacenter team when i say, i think we all enjoy the garage alot more then our day to day jobs. We thank the man upstairs we have them to support this venture but we all hope to be able to do something more worth while in the future to come ..

Assembeling the team part 1 ...

Lets take a bit of a flash back before the actual birth of the datacenter, some 3 years ago ....

I used to work in south east asia and was building datacenters for telco startups. I had a great experience working in one particular country and met alot of great friends. One person in particular, lets call him the 'datacenter hitman' was someone i worked with closely during the build of one of the bigger datacenters. We did alot of great work and had alot of fun.

After the completion of the datacenter build he returned back and i was asked to stay to do support on the networks and servers as a network/unix engineer. It was a great experience and i ended up staying abroad for just over 2 years. I lost contact with datacenter hitman and thought that i wouldnt get a chance to see him anymore as he also left the company we were initially both working for.

Out of the blue i got an email from datacenter hitman ! some 3 years after we worked together abroad. Datacenter hitman had called all the people in the phone book with my surname and had called the previous company we used to work for and eventually tracked down my email.

To cut a long story short, i gave him a call and we had a great chat, then eventually we caught up and had a beer, talk about old times and the progress we had made over the 3 years. I told him that i had ideas and was working with Garage BA on a project and thought that he would play a great role with his skill sets in assisting with the build up of the garage datacenter. After a few more beers and a couple more chats the idea for the application sounded better and better and datacenter hitman was on board. The team was growing and the idea was being brought to life by initial team. The garage datacenter team now consisted of Garage CEO, Garage BA and Datacenter hitman, the three of us were going to do great work and build something that we could be proud of ....

The Birth of the Garage Datacenter ....

This is a long story so bear with me okie ... yep ... cewl

Continuing on from my previous post, i have 2 jobs, i am Project Manager by Day and CEO by night. I discussed my job as a project manager in my previous post and i think thats all i will write about on the Project Manager side of things. In actual fact this whole blog is put together to discuss my job as CEO and managing director of the garage, i wrote about the Project Manager stuff to lead into why I now have a garage datacenter.

My garage is located 2 stories below where i live and to me its more exciting and alot more fun to work in then my monday to friday - 9:00 - 5:00 office. The only downside right now is even though i am CEO it pays less then what i used to get as a 12 year old boy selling my old stuff that i didnt use anymore on the side of the street. Atleast then i sold a pair of old rollerskates i made 5 bucks out of it .... at the moment as CEO i make diddly squat.

In my previous job i was taking care of environments on IBM infrastructure at a bank. I had a friend who was shadowing me at the time cause i was transitioning my work to him as i was leaving the company to join this new one with the airlines. He lent me a book which was the unoffical google story and this was the catalyst, ths is where it all began. I found the book to be really great and motivational, its the utlimate dream of any computer jocky. To build a technology company and to provide a useful service to many people through the technology you have built. Building a company and succeeding gives satisfaction and fulfilment.

At the same time as i was with the bank i was working on another business venture/idea which was the development of a particular product that is not computer based. I came up with this idea cause i was inconvienced by *something* and decided to buy *something* to rectify this inconvience. To my amazement this *something* that i wanted to buy didnt exist. I thought to my self if this *someting* is useful to me then its possible that this *something* could be useful to other people. On that note i spoke to a patent attorney, she did a patent search then got a provisional patent for this *something* which i am trying to develop right now. The story of this *something* is currently still in progress but its a different story to the Garage datacenter. The only reason why im discussion this *something* and the development of this *something* is because it has some crossroads to the garage datacenter so it makes up the story.

(hmmm... on a side note) I think im structuring this like code, see initially i talked about reading the google unofficial story (thats one class) then i discussed the development of the *something* (thats the second class). I talked about my experiences as a project manager (thats the third class). Now here, i have the main function were i will use both the previous points (classes) to make up the story (program).

OK anyway, so after reading googles unoffical story and considering the *something* i was developing and how much i hate working in big corporations with egos flying around that are just as big, i came up with an idea that could help the development of my *something*, create a business for myself and relinquish the need to work for other people. At the time I told my mate (lets call him the Garage Business Analyst or Garage BA) of my idea to enhance the *something* through building a web application. He liked the idea and we got on with design plans, business processes and interface designs.

I live in an apartment with 3 other people, its difficult to concentrate and focus on the designs cause of the noise so i was thinking of how I could find a place where i could have peace and quiet whilst i began on the design process. The garage at the time was not an option because it was packed to the brim full of junk such as TV's, Fridges, Old Cushions, Boxes of who knows what and just stuff that wasnt being used anymore. During one fateful day i overheard that a new couple had moved in and they didnt use their garage as they didnt have a car or anything to put in there. I decided to try my luck so i asked them if i could rent out the garage. Belive it or not they said yes and for only $10 a week. The minute i got this approval from them along with the garage keys i began moving all the junk from my current garage to this new garage so that i could use my own garage for my datacenter. Thus is the birth and beginning of the garage datacenter....

The Beginning

It all starts somewhere doesnt it. And usually where it starts is the most important and most interesting part of the story. I wish i had put this blog together from the beginning as i could have probably remembered the details better and written better accounts of what happened but im not really a writer i dont think i can even spell properly, im a techo, programmer type guy at heart so what happens is i occasionally forget that im writing in english and i write everything as if its a variable name, with this in mind, if i have spelling mistakes or grammatical mistakes dont be surprised OK.

So let me begin by letting you know how it all began ! At the moment i have 2 jobs one that is interesting and one that is not, one that pays and one that doesnt. One of my jobs causes me (during multiple instances of the day) to considered whether i should prick a pin in my eye to take away the pain, yes really... I think the main reason why i dont prick a pin in my eye is because for one, the pain caused from the pin pricking act wont take away the pain and the reason being is that the pain of being where i am for 8 hours of the day is worse ! For example you may complain to a friend that your leg hurts from last weekends gym session, he may then slap you across the head really hard so that you forget about the pain in your leg cause now your head hurts, it works right, ?!? cause your rubbing your head. In this case however, pin pricking the eye just wont do the trick !

My second reason why pin pricking the eye wont work is i have minus 3 vision and progressivly getting worse which means im blind anyway so really there is no point pricking a pin in my eye as i would achieve nothing on both accounts !

So lets look at my first job (the boring one) ... i am currently working as a Project Manager at a pretty big airlines company deploying a new system. Yes, thats correct, im a techo working as a Project Manager, simply because this is where the quick money is. We have meetings where we talk absolute crap, where simple issues that should be sorted in minutes causes the biggest confusion and where ego's are so big that the owners of those ego's should be driving home in mac trucks so they can carry home those ego's without getting pulled over by the police for over packing a standard class C family car. Its funny to see and i guess you can approach this situation in multiple ways.

Option 1:
You can argue and fight every point you dont agree with and pull out your *stick* PM's love the word *stick*

Option 2:
You can totally not give a rats arse and let things fly as they do

Option 3:
You can attempt to pre-empt alot of the work and do alot of it so you dont have to do the activity crappy activity of following up on people who dont want to be followed up on or worse dont want to do any work

Option 4:
You can take a tazer into meetings and start tazering people who arnt listening and not delivering on time

Option 4 seems best and most practical as it would solve many problems but i guess not exactly legal so i usually go with option 3 cause its easier for me.

Even through this daily grind and paper pushing there is some consolidation. My body might be in the office for 8 hours of the day but in spirit and in mind i am in my garage, yes my garage .... which leads me to my second job. The more interesting one ... CEO of the garage ...