Started using pfSense, Will Never Go Back

I decided it was time to go over my home office network architecture and make some tweaks. The architecture that I started with is Comcast Business Class service for my ISP, and Windows Server 2008 that serves not only as my domain controller, but also as my NAT router and firewall.
I had an older Windows XP computer that I figured I could still use as a firewall. I did some research on firewall software and chose pfSense. pfSense is a free, open source firewall/router that is built upon FreeBSD. I added a 2nd network card to the Windows XP box and installed pfSense, which replaced Windows XP. So now I have a dedicated firewall, with 2GB of memory and an Intel Pentium Dual E2180 @ 2.00 GHz CPU. I added it to my network and proceeded to do some tests.
One other thing that I did was to add Google's public DNS servers. The first test I ran was a DNS lookup from within pfSense itself. I used the DNS Lookup under the Diagnostics menu to look up a domain name, and I chose the Democracy Now! domain name.

Google's DNS servers are at 8.8.8.8 and 8.8.4.4, while Comcast servers are at 75.75.75.75 and 75.75.76.76.  Google's DNS servers perform much better, so that is an improvement already.
Next, I tested the throughput of my 2 routers - the Windows 2008 box vs the pfSense box. To do this, I used ZDNet's Broadband Speed Test from my workstation computer, which I alternatively configured to use my Windows 2008 box as the gateway and then configured to use my pfSense box as the gateway. With my workstation configured to use the Windows 2008 box, I was getting about 12 to 14 Mbps performance. With the pfSense box, my speed went up to 26 to 29 Mbps performance. Wow, what a performance improvement! I am really kicking myself now just thinking of all the bandwidth I was loosing in my network infrastructure, which translates to lost dollars.
With this new configuration, I also get added security. With pfSense, I installed a package called Snort that adds intrusion detection and prevention. Snort provides alerts for the incoming attacks that it detects. It always amazes me the number of continual attacks there are out in the Internet wilderness.

 

The moral of this story is, pay attention to your network infrastructure. You may be able to make large improvements in both performance and security. If you have an old computer that works, don't throw it away. Make it into a high performance router.

Gotta love Microsoft - still dishing out the classics

Haven't seen the old "Divide by Zero" in a while. Thanks Bing!

WTF GoDaddy and Optimizing My Sites on Another ISP - DreamHost

One of the websites I have been working on was hosted at GoDaddy. I had a hosting plan there for years. several weeks ago, I had requested SSH shell access to my account. For some reason, GoDaddy had to move my hosted web account to another server in order to give me SSH access.
Well, the move itself was not smooth. Turns out that one of my MySQL databases was running on an older MySQL server that was not supported on the newer host. So the migration ended up stuck in limbo. I found out that it was the database issue after I contacted GoDaddy support regarding the migration taking such a long time. They told me to upgrade the database. The only problem was, with the migration stuck in limbo, I couldn't access the databases from their control panel. I contacted their support again and explained the situation and told them just to delete the database because I didn't need that one.
Well, that was too complicated for their "level 1" support, so they kicked it up to "level 2" support, where they upgraded the database, completed the migration, and fixed up all my PHP configuration files to point to the shiny new databases on the new database. All seemed to be good.
The first time I tried to visit a site on the new server, all I got was one of those server error messages showing up in my browser where my site was supposed to be. After retrying the various sites that I have hosted their, the websites finally started appearing normally. I figured it must have been some transient temporary issue that occurred after the move to the new server.
However, later I noticed that when I was in meetings with the groups that I created the websites for, that when I tried to pull up a website during the meeting, I was regularly disappointed by how slow the sites would load. They were loading very slowly, which made it difficult to discuss them during the meetings.
Later, I used the free online analyzing tool, WebPageTest. It will give you real-world timings of all the aspects of loading a web page. And Wow. I couldn't believe how poor the performance was on the Godaddy server. It was taking more than 10 seconds just to get the first byte of response from the initial website request! Also, it was taking sometimes over a minute for the site to completely finish loading! There must have been something terribly wrong with the new server. So I contacted Godaddy support again.
I had explained to them in the new ticket I submitted in detail what the issue was, what timings I was getting, and that I would periodically get server error messages. The response was that I should "upgrade" from a shared plan to a "hosted" plan for better performance, a "level 1" response I presume, especially given that I was already on a "hosted" plan.
Then it came to me - I need to move my sites to another ISP. I have sites on BlueHost which are doing well. I also have a site on an Amazon EC2 micro server that screams. However, I noticed that DreamHost has a pretty good deal on virtual private servers (VPS) that have unlimited bandwidth and storage. I also like DreamHost because they are carbon neutral and employee owned. When you sign up, you also get a week for free on a VPS. Their VPS is also configurable on a sliding scale, literally. There is a slider control where you can dynamically change the amount of memory your server has, which also changes the amount that you are billed. So, you only pay for what you use.
The DreamHost VPS is not as fast as an Amazon EC2 instance, but you get easy management through DreamHost's very own custom control panel. This makes it easy to set up multiple websites on your VPS, whereas on Amazon EC2, you have to do all the management yourself through SSH, or whatever applications you install.
So, I chose to go with DreamHost, because the performance is good, and I can perform optimizations and use a cloud service to get even greater performance. I began by copying my site files from my GoDaddy server to the DreamHost server. I have many media files on one site, which would take a long time to download and then re-upload again. To speed the process up, I just connected via SSH to my new DreamHost server and used SCP to directly copy the files from my GoDaddy server. This worked like a charm. Then, I backed up my databases on GoDaddy and imported them into new databases at DreamHost. I got the sites all up and running and already they were performing much much better than they were with GoDaddy.
Now for the optimizing. I ran WebPageTest on one of my sites and this is what I got:

As you can see, it got some F's in some of the ratings. It also has an "X" for CDN which means that I am not using any cloud services for distributing static content. The first thing I did was to optimize some of the JPG images that I have on the site. The second step was to add some code to the ".htaccess" file on the server that is in the root website folder. I added these lines:

<IfModule mod_expires.c>
     # Activate mod_expires for this directory
     ExpiresActive on

     # locally cache common image types for 7 days
     ExpiresByType image/jpg "access plus 7 days"
     ExpiresByType image/jpeg "access plus 7 days"
     ExpiresByType image/gif "access plus 7 days"
     ExpiresByType image/png "access plus 7 days"
     ExpiresByType image/x-icon "access plus 7 days"

     # cache CSS, js, and other files
     ExpiresByType text/css "access plus 7 days"
     ExpiresByType application/x-javascript "access plus 7 days"
     ExpiresByType application/javascript "access plus 7 days"
     ExpiresByType text/javascript "access plus 7 days"
     ExpiresByType application/x-shockwave-flash "access plus 7 days"
</IfModule>

#Gzip
<IfModule mod_deflate.c>
  AddOutputFilterByType DEFLATE text/text text/html text/plain text/xml text/css
  AddOutputFilterByType DEFLATE application/x-javascript application/javascript
  AddOutputFilterByType DEFLATE text/javascript image/x-icon
</IfModule>
#End Gzip

The first set of lines enables caching of much of the static content on the client browsers computer. This speeds up browsing in that the browser only has to download the static content (images, etc.) once, instead of with each page load. It is important to note that if you are working on the site and are editing images, javascript files or other files that are cached, you will want to turn off caching in order to make it easier to test your changes.
The second group of lines (under Gzip) tells the server to compress certain file types, which decreases the time that it takes for the browser to download these files. All of the standard, compressible file types are listed there. I also added the "image/x-icon" types. I wanted to provide a site icon file that supports higher resolution images. These icon files with higher resolutions can become large fast, because they are essentially raw bitmap files. The good news is that they are highly compressible, so by adding it to the "Gzip" list, you don't have to worry so much about the size.
So, now for another test:

The scores are much better here. 5 seconds for the first load is still a bit long. I might be able to shave that a bit more by installing a Joomla extension which combines the javascript and stylesheet files into one javascript file and one css file so that the browser only has to perform one download request for each type. However, I decided for now to move straight to enabling a cloud service. DreamHost supports easy integration with a free (yes, that's right - free) cloud service called CloudFlare. CloudFlare is ridiculously easy to use, especially with DreamHost. To enable it, just "edit" your "domain" for your website and turn it on:

That is all there is too it. You have to wait several hours for it to fully take effect. I waited until the next day to run another test:

Look at that! First view is now down to 2.4 seconds. Note that the "CDN" score now has a check mark in it. I am quite happy now with the performance improvements I was able to obtain.
So, in conclusion, you can dump GoDaddy, move to a smaller ISP, such as DreamHost, and get near enterprise performance for your website for low cost. You don't have to dump more money into GoDaddy in an attempt to get reasonable service.

Kids and Reproducing Errors

After spending over a day setting up more virtual servers in my virtual domain, installing SQL Server, installing Microsoft System Center Operations Manager and installing ACS Services, all on my laptop, I exclaimed out loud, that I had successfully reproduced the customer’s issue.

My older son heard and was confused by this. He asked me why I would ever want to “re-create a problem”. I then explained to him that if you can set up your own system that has the same problem as your customer’s system, you are %90 closer to fixing the customer’s problem. Not only does this give you tremendous insight as to what is causing the problem so that you can track it down and fix it, it gives you a test bed so that you can verify that your fix works.

Of course, that opened the door to my wife’s story about way back when she did a short stint as a support engineer and fixed a bug on the IBM System/36 Operating System where she added an “else” to an “if” statement. It is about the hundredth time that I heard it, but the kids didn’t remember hearing it before and got a kick out of it. Gotta love her. ;)

Hyper-V on Windows 8

I recently got a fairly high end laptop so that I could take my software development work with me wherever I go. My goal is to be able to get work done and score wife/family points by going to those various family functions too. "Go to your family reunion? Sure, just let me pack my laptop here". "Take the kids to their curling match up in Canada? Sure can Dear, I've got everything I need right here in my laptop bag". It has a quad core I7 with 16GB ram and a 1TB hard drive, which I replaced with a 1TB Seagate SSD/HD hybrid drive. It also came with Windows 8 Home, which I upgraded to Windows 8 Pro.
I had started to install Oracle's Virtual Box when I couldn't get Virtual Box to recognize the hardware virtualization that I had enabled in the BIOS configuration. That was when I discovered Windows 8 had Hyper-V. The thing about Hyper-V, is that it takes over the hardware virtualization feature and leaves other VM software (such as Virtual Box) out in the cold. I could have disabled Hyper-V and been on my way, but I chose to check out the new Windows 8 Hyper-V technology. I have had nothing but fun with it. It perfect for some of the work that I do. I really love the "Dynamic Memory" feature that allows you to efficiently use memory amongst your various VMs. Here is a snapshot of my VMs that I currently have on my laptop.

One of my jobs is to maintain and develop software that is built vertically on Microsoft's System Center Operations Manager. This software has some very large customers with large networks that they operate this software on. With Hyper-V, I am able to set up a complete virtual network, albeit on a much smaller scale, all on my laptop. As you can see, I can have several Windows Servers running with various services installed and still have plenty of memory left over for my host system, thanks to Dynamic Memory. You can see the server running SQL Server has allocated more memory because of the way SQL Server pre-allocates memory for fast queries.
I also like the Hyper-V concept of Virtual Switches. This really helps in setting up a virtual domain. As you can see, I have one server working as a Domain Controller. The other servers, except for the NAT server (and of course Kubuntu) are part of the domain. I have two virtual switches set up, one is external and the other is private.

The NAT server connects to both switches and serves as the default gateway for the other Windows VMs. The Domain Controller also serves as DHCP and DNS for the private network on the private switch. And yes, I do know that the name of my private switch has "SCOM 2007" in it. I was going to have two separate switches from my SCOM 2007 and SCOM 2012 setups, but later I discovered that I really didn't need to. So, I'm just living with the name I currently have.
This all seems to work great. There is just a minor issue a run into when, say I need to run out the door, and I close my laptop (where it sleeps or hibernates) and change locations. The servers still seem to run OK, however they seem to sometimes lose connectivity. I find that I may need to run "ipconfig /renew" on the NAT server. One time I had to do this on my other servers as well. One particularly peculiar thing was that a server had the gateway correct, but it's default DNS was pointed at the NAT server, even though the DHCP on the Domain Controller specifies the Domain Controller for the DNS. I think this has to do with a virtual network timing issue when my laptop wakes up again after sleeping. It's just a minor annoyance however.
Why Kubuntu you might ask? Well, I have that for Ruby on Rails development. On a previous contract I was on at Amazon, I was developing Ruby on Rails in the Red Hat Linux environment. For my personal Ruby development, I decided to set up an environment on a Kubuntu VM in Hyper-V. I'll probably write more on that later.

So, just what is a "Capacitator"

One of the many "features" of living in a rural area is the water well, complete with water pump. Our water well happens to be shared among four houses. A couple days ago, the pump in our water well quit working. We eventually had to replace the pump, however, before that, a component in the starter circuit "fried". One of my neighbors replaced the component. However, it was another neighbor, an elderly gentleman, who told me which component was replaced. He said it was the "capacitator". Now, I am familiar with capacitors, but this is the first I have heard of a "capacitator". I tried to find one on the internet, but the search was fruitless. So, I had this mental picture of what a "capacitator" might look like and proceeded to create this image in a much more shareable format.

As you can see, I included capacitors on the right, and added what I suspect is a "capacitator" on the left.  Now, I wonder what the electrical symbol for such a device looks like.