The Beauty of Open Source

The beauty of Open Source is that you can look at the most up-to-date detailed version of the documentation: the source code itself. Whenever you want to know how to do something or how it works, you can take a look at the source code. Yes, this is not always advisable as the first course of action as it can be quite time consuming, especially if it is a large and complex project. However, when you do figure out what you are after by reviewing the source code, it is a much more rewarding process than finding the answers in the docs (plus you may learn a new coding practice / technique).

I wanted to use the Serilog.Extras.Web nuget package to enrich my log entries with the HttpRequestId for log message correlation. I noticed in addition to the HttpRequestId being added to each log message, there were additional log messages being created for "HTTP GET..." or "HTTP POST...". I did not really want these log messages being generated as they were filling the log viewer (Seq) with a lot of noise. A quick Google search lead me to this Github page and the following ApplicationLifecycleModule class:

// Copyright 2014 Serilog Contributors
// 
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
// 
//     http://www.apache.org/licenses/LICENSE-2.0
// 
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

using System;
using System.Linq;
using System.Web;
using Serilog.Events;

namespace Serilog.Extras.Web
{
    /// <summary>
    /// HTTP module that logs application request and error events.
    /// </summary>
    public class ApplicationLifecycleModule : IHttpModule
    {
        static volatile bool _logPostedFormData;
        static volatile bool _isEnabled = true;
        static volatile LogEventLevel _requestLoggingLevel = LogEventLevel.Information;

        /// <summary>
        /// Register the module with the application (called automatically;
        /// do not call this explicitly from your code).
        /// </summary>
        public static void Register()
        {
            HttpApplication.RegisterModule(typeof(ApplicationLifecycleModule));
        }

        /// <summary>
        /// When set to true, form data will be written via a debug-level event.
        /// The default is false. Requires that <see cref="IsEnabled"/> is also
        /// true (which it is, by default).
        /// </summary>
        public static bool DebugLogPostedFormData
        {
            get { return _logPostedFormData; }
            set { _logPostedFormData = value; }
        }

        /// <summary>
        /// When set to true, request details and errors will be logged. The default
        /// is true.
        /// </summary>
        public static bool IsEnabled
        {
            get { return _isEnabled; }
            set { _isEnabled = value; }
        }

        /// <summary>
        /// The level at which to log HTTP requests. The default is Information.
        /// </summary>
        public static LogEventLevel RequestLoggingLevel
        {
            get { return _requestLoggingLevel; }
            set { _requestLoggingLevel = value; }
        }

        /// <summary>
        /// Initializes a module and prepares it to handle requests.
        /// </summary>
        /// <param name="context">An <see cref="T:System.Web.HttpApplication"/> that provides access to the methods, properties, and events common to all application objects within an ASP.NET application </param>
        public void Init(HttpApplication context)
        {
            context.LogRequest +=LogRequest;
            context.Error += Error;
        }

        static void LogRequest(object sender, EventArgs e)
        {
            if (!_isEnabled) return;

            var request = HttpContext.Current.Request;
            Log.Write(_requestLoggingLevel, "HTTP {Method} for {RawUrl}", request.HttpMethod, request.RawUrl);
            if (_logPostedFormData && Log.IsEnabled(LogEventLevel.Debug))
            {
                var form = request.Form;
                if (form.HasKeys())
                {
                    var formData = form.AllKeys.SelectMany(k => (form.GetValues(k) ?? new string[0]).Select(v => new { Name = k, Value = v }));
                    Log.Debug("Client provided {@FormData}", formData);
                }
            }
        }

        static void Error(object sender, EventArgs e)
        {
            if (!_isEnabled) return;

            var ex = ((HttpApplication)sender).Server.GetLastError();
            Log.Error(ex, "Error caught in global handler: {ExceptionMessage}", ex.Message);
        }

        /// <summary>
        /// Disposes of the resources (other than memory) used by the module that implements <see cref="T:System.Web.IHttpModule"/>.
        /// </summary>
        public void Dispose()
        {
        }
    }
}

From this I could see the method generating the unwanted log message (LogRequest) checks _isEnabled before logging the "HTTP..." message. So, to stop these messages all you have to do is set _isEnable to false. The code clearly tells you how to do this too: set the static property IsEnable to false or ApplicationLifecycleModule.IsEnabled = false. Of course, I could have found this out from reading the docs but it would not have been half as much fun or as an insightful learning experience to what was going on inside this small part of Serilog.Extras.Web. 

What else did I learn from this exercise? I realised I could easily enrich the action method execution time telemetry logging I was already collecting with the requests raw URL using HttpContext.Current.Request.RawUrl to facilitate request URL correlation. I noticed that the Error method appears to be a global exception handler which could lead to this being a replacement for Elmah. I also saw the "volatile" C# keyword in use which prompted me to look into this further and discovered it's something not to be used lightly.

Maybe the next time you experience some behavior you were not expecting or want to know how to do something with an open source project, before rushing to the docs for an explanation, take a quick look at the source code and see what you can uncover first. You may just come away with a small sense of achievement and learn something new in the process.

Passing Property Names with C# Lambdas

How many times have you heard that using magic strings is a bad practice? Imagine you are passing a property name to a function that is going to use reflection to read its value. How do you ensure that the string is typo free and accurately represents the property? You could write a unit test to validate the name being passed, but there's nothing stopping you from copying and pasting the typo; you could create a static class and store the names as static, read-only properties; or maybe even retrieve it from another source that could be internal or external to the project. The problem with all these methods is that if there is any change to the property name itself, they also need to be updated.  

var book = new Book()
{
    Author = "George R. R. Martin",
    Title = "A Game Of Thrones",
    Publisher = "Bantam"
};

Console.WriteLine(ReadProperty(book, "Author"));

public object ReadProperty(object source, string propertyName)
{
    var propertyInfo = source.GetType().GetProperty(propertyName);

    return propertyInfo.GetValue(source);
}

If the property's name has been changed, this will not be detected until the function tries to retrieve the PropertyInfo for the property that has been renamed. As it no longer exists, this will result in a NullException being thrown. 

Miguel Castro introduces a simple solution for this in his Pluralsight course that ensures you are only using valid properties. This solution uses a lambda expression to validate that the property identified in the expression actually exists and if it doesn't, a compile error will result. By passing the name as a strongly typed property instead of as a string, you are rewarded with a safety net against typos and changes.

var book = new Book()
{
    Author = "George R. R. Martin",
    Title = "A Game Of Thrones",
    Publisher = "Bantam"
};

Console.WriteLine(ReadProperty(book, () => book.Author));

public object ReadProperty<T>(object source, Expression<Func<T>> propertyExpression)
{
    var memberExpression = propertyExpression.Body as MemberExpression;
    var propertyName = memberExpression.Member.Name;
    var propertyInfo = source.GetType().GetProperty(propertyName);

    return propertyInfo.GetValue(source);
}

The Expression wrapping the Func of T causes the compiler to "emit code to build an expression tree that represents the lambda expression." This provides the functionality to allow us to very easily interrogate the lambda expression and retrieve the name of the property that was identified.

If you are familiar with the ASP.NET MVC framework and HtmlHelpers, you should recognize the application of the above pattern as this is exactly what is being used to identify a property on the view model:

As the in type of the Func is set as the type of the view model, this limits the scope of the properties that can be identified to only those present on the view model; the out type (labeled as TProperty) still remains as the type of the property being identified.

That's it--now you no longer have to worry about passing property names as strings!

Tech Ed - Why a Hacker can own your servers in a day!

I have watched a number of security related sessions but the TWC Why a Hacker Can Own Your Web Servers in a Day! session at Tech Ed Europe 2014 by Marcus Murray and Hasain Alshakarti was one of the best that I have ever seen. They both work at Truesec who conduct Penetration Testing amongst many other security related services. What made this security session stand out is that they went beyond the basic demonstrations of commonly known exploits (SQL Injection, XSS etc). They continued to progress the exploit to show how professional hackers would spread their sphere of influence to ultimately compromising your entire network. What made this even more relevant in the real world was that all the attack vectors they demonstrated had been previously found whilst performing pen testing against some of their customers. This is seriously scary stuff but fascinating at the same time.

In summary, their session consisted of demonstrating three exploits:
1) An image upload exploit which ended up with the web server, SQL server and Domain Controller being compromised.
2) A SQL injection attack which resulted in the SQL server becoming compromised and in turn, only a matter of time before other servers were compromised thanks to what was demonstrated in the first exploit.
3) XSS attack using a live website which results in the attacker being able to do some very scary stuff to your browser session.

I will not go into detail here about the attacks they demonstrated as to do them justice you really need to watch the Tech Ed session. All I will say is the photo upload exploit is pure genius. From uploading a photo to owning the Domain Controller--it's chilling. Granted, it's a very specific exploit but for impact purposes, I have not seen anything else that beats it yet. Kudos to them for finding and being able to exploit this.

They introduced a range of security / hacker tools which I had never heard of before. These tools made the process of "owning" servers through known exploits like "Token Kidnapping" so easy it was scary. 

If I have some spare time on my hands, I would like to attempt to recreate these exploits with the aid of some virtual infrastructure on Azure as there is plenty to be learnt here.

As a developer, if you think security is something that you do not have to be concerned about, watch this session and I hope you will be scared into thinking otherwise by the end of it. This session demonstrates the sophistication and lengths a professional hacker will go to when attempting to compromise your applications security.

A quick look at Marcus's previous Tech Ed sessions shows a string of them running back to 2010. I will be checking these out for sure for some more security goodness.

Introduction to Grunt Presentation

A few weeks ago I gave an hour presentation on using Grunt to automate development and build tasks. I decided to present this as a live demonstration where I would do everything from installing node.js, to installing and configuring the packages and running the grunt tasks. This was a perfect fit for the audience as none of them had any experience with using node.js and by using a clean Windows 8.1 VM, I was able to highlight just how easy it is to get up and running with Grunt.

I encountered one issue with NPM where it didn't create a "npm" folder in the user's roaming profile. I didn't encountered this issue when installing node.js on my laptop or Surface Pro 3, but it was an issue with VMs hosted by Azure. Fortunately as I had done a run through of my demo I was aware of this and how to fix it.

The demonstration covered installing and configuring the following packages:

• grunt-contrib-jshint
• grunt-contrib-csslint
• grunt-htmlhint
• grunt-contrib-copy
• grunt-contrib-clean
• grunt-contrib-concat
• grunt-contrib-uglify
• grunt-processhtml

The feedback I received indicated the audience felt confident that they would be able to start working with Grunt without any difficulties and believed that they would be able to use it for automating their own build tasks. A common suggestion was to include a brief introduction of Grunt before getting into how to use it. I have to agree, that I did gloss over this so this is something that I will incorporate if I get the chance to give this demonstration again.

The website that I used for the demonstration can be downloaded from the following links:

• Starter version
• Complete version 

The starter version is a basic website that contains a single HTML page and a few JavaScript and CSS files that can be processed using Grunt. The HTML page contains two buttons that display alert messages when they are clicked, which provides an easy way to demonstrate that the JavaScript is still working post build. I decided to keep the website as simple as possible to ensure that the focus was on using Grunt and not what the website was doing. The complete version contains the package files (package.json), the grunt file (gruntfile.js), and a modified index.html. I have not included the packages, so you will need to run "npm install" to download them when using the complete version. I have created two tasks that can be executed via the following commands:

• "grunt default" or "grunt": this task will lint your JavaScript, CSS and HTML. I suggest running it with the "--force" option so you end up with a full list of all the linting errors, rather than the task prematurely aborting as soon as it detects the first one.
• "grunt release": this task will copy the website into a release folder, concatenate and minify the JavaScript files and update the index.html file to reference this single JavaScript file.

Working with Age Groups in SQL

Following on from my previous post SQL - Beyond the standard ASC / DESC ORDER BY which introduced the CASE WHEN statement, this post tackles the problem of handling age groups with SQL and introduces a technique I call “query wrapping”.

When writing queries to generate reports, it is a common requirement to aggregate records into groups and a typical grouping is age. The following example outlines the process of aggregating a set of employee records into the following age groups:

Under 18
18 – 25
26 – 35
36 – 55
56+

For this post, the sample employee list I am using looks like this:
 

Before assigning an employee an age group, the employee’s age needs to be calculated. To do this the DATEDIFF T-SQL function can be used. MSDN defines the syntax as:
 
DATEDIFF(datepart, startdate, enddate)
 
To calculate an employee’s age, the number of years in between their date of birth and today’s date is required:
 
DATEDIFF(yy, DateOfBirth, GETDATE())

Note “yy” specifies that the number of years between the two dates is required and GETDATE() will return today's date.
 
The DATEDIFF function can be used in conjunction with a CASE WHEN statement to create a query which will add an age group label to each employee and can in turn be grouped upon:
 

SELECT
    CASE
        WHEN DATEDIFF(yy, DateOfBirth, GETDATE()) <= 17
            THEN 'Under 18'
        WHEN DATEDIFF(yy, DateOfBirth, GETDATE()) BETWEEN 18 AND 25
            THEN '18-25'
        WHEN DATEDIFF(yy, DateOfBirth, GETDATE()) BETWEEN 26 AND 35
            THEN '26-35'
        WHEN DATEDIFF(yy, DateOfBirth, GETDATE()) BETWEEN 36 AND 55
            THEN '36-55'
        WHEN DATEDIFF(yy, DateOfBirth, GETDATE()) > 55
            THEN '56+'
    END AS YearGroup
FROM Employees

This query will generate the following result set:

This result set will be the basis to determining how many employees are in each age group. To create the age group aggregation it is required to group on the result set displayed above. How do you group on the result set of another query? You use that result set as the input to your GROUP BY query. It is possible to dynamically generate a query's FROM data-set from an SQL query instead of referencing an actual table. Take a look at the query below to see this in action:
 

SELECT AgeGroups.YearGroup AS [Year Group]
    , COUNT(AgeGroups.YearGroup) AS [Group Count]
FROM
(
    SELECT
        CASE
            WHEN DATEDIFF(yy, DateOfBirth, GETDATE()) <= 17
                THEN 'Under 18'
            WHEN DATEDIFF(yy, DateOfBirth, GETDATE()) BETWEEN 18 AND 25
                THEN '18-25'
            WHEN DATEDIFF(yy, DateOfBirth, GETDATE()) BETWEEN 26 AND 35
                THEN '26-35'
            WHEN DATEDIFF(yy, DateOfBirth, GETDATE()) BETWEEN 36 AND 55
                THEN '36-55'
            WHEN DATEDIFF(yy, DateOfBirth, GETDATE()) > 55
                THEN '56+'
        END AS YearGroup
    FROM Employees
) AgeGroups
GROUP BY AgeGroups.YearGroup

The query previously written to add an age group label against each employee has been included as the FROM dataset for the GROUP BY query. I think of this as wrapping the group by statement around the result set of the inner age group query. This is an important technique to understand as it allows you to build up very powerful SQL queries by wrapping queries around one another. Data-sets which are being JOINED on to can also be created dynamically using a separate SQL statement opening up a limitless number of querying options.
 
This is where thinking in sets when working in SQL helps. Think of the inner age group labeling query as generating your initial data-set and the outer group by query filtering (or in this case aggregating) down this result set further to something closer to what is required.
 
When executed, the updated query produces the required aggregated result set:
 

When writing an SQL query it is usually habit to start at the top and work down--what you want as the end result. I find it easier to write the inner most queries first where I am pulling in all the data I require to work with in its most primitive form. I then start to wrap these queries in other queries which will better shape the data into the end result that I require and look for ways to JOIN this data on to other data-sets as required.
 
Hopefully this has illustrated the technique of “query wrapping” whilst demonstrating how to work with age groups in SQL and helped open a couple more SQL querying doors for you.

C# Recursion with a Lambda

This is the first in a series about exploring the code patterns that Miguel Castro introduces in his Pluralsight course titled "Building End-to-End Multi-Client Service Oriented Applications". I am creating these blog posts as a learning exercise where I can delve deeper into the pattern and solidify my understanding of it. Hopefully this can be of some help to other programmers who are also taking Miguel's course or anyone trying to improve their C# skills.

Traditional recursion

If you're like me, when it has come to writing a recursive algorithm, you generally follow the pattern of defining a function that contains all the code that needs to be executed repeatedly and then call this from another function. It's simple and it works, but it does require that you create at least two functions. Let's take a look at an example where recursion is used to calculate the factorial for a given number:

public void TraditionalRecursion()
{
    Console.WriteLine(ComputeFactorial(5));
}

private int ComputeFactorial(int n)
{
    if (n == 0) 
    { 
        return 1; 
    }

    return (n * ComputeFactorial(n - 1));
}

The "work" is being done by the ComputeFactorial function, but as it needs to be able to call itself, it has to be created as a separate function. Like with many recursive algorithms, the first function (TraditionalRecursion) doesn't contain any logic or functionality that requires it to be separated--it just has to act as the starting point for the algorithm.

Lambda recursion

Now, wouldn't it be nice if you could do this with just a single function where you no longer need a second function just to start the algorithm. Imagine being able to define the recursive algorithm and start it all within the same function. Well, it turns out that this is quite easily achievable through the use of a lambda expression:

public void LambdaRecursion()
{
    Func<int, int> computeFactorial = (n) =>
    {
        if (n == 0) 
        { 
            return 1; 
        }

        return n * ComputeFactorial(n - 1);
    };

    Console.WriteLine(computeFactorial(5));
}

The inline delegate, a Func in this example, contains exactly the same behavior as the ComputeFactorial function; the only difference is that it has been declared as a variable. An easy way to look at this, is to image that ComputeFactorial has been wrapped up and is now contained within the computeFactorial variable (declaring functions as variables should be very familiar concept to a JavaScript programmer). Once declared, all you have to do it call this variable to start the recursive algorithm--it really is as simple as that!

SPA Silos

I was recently listening to a fascinating .NET Rocks podcast which featured Miguel Castro discussing MVVM on the Web. Miguel talked about his experiences with Knockout.js and AngularJS as client side MVVM frameworks. What I really enjoyed about this podcast was the insight he brought to the table of developing commercial SPA's (Single Page Applications) but using a more hybrid approach. What am I saying here? I think Miguel used the phrase “pages with SPA pockets” but said Brian Noyes coined a much more accurate description—SPA Silos.

SPA Silos are where a single web application is actually made up of many SPA’s. When we envisage creating a SPA, we normally have a vision of a single HTML base page that will initially load all of your SPA code. From there the browser will not actually request any new pages from the server but only swap out views and make web API calls in the background as you navigate around the app. The SPA Silo approach is to break that single SPA into many mini SPA’s each responsible for its own area of functionality. Navigation within the mini SPA is handled by routing in the SPA framework. When moving to different areas of the application, this is handled by requesting a new page from the server which bootstraps all the SPA code for the mini SPA that will provide the functionality for the area of the application you are now navigating to.

Miguel gave a nice example which helped illustrate this practice very well:
Let’s say you are building an e-commerce management web application. The application would have areas such as Products, Customers, Orders, Admin etc. Instead of having a single SPA which contained the functionality for all of these areas, you would actually have 4 mini SPA’s which are each loaded when navigating to the required area instead of upfront when first loading the application.

What are the benefits from using this hybrid SPA Silos approach I hear you asking? There is one very big benefit which immediately springs to mind--Separation of Concerns. I cannot stress the importance of this principle enough. If this hybrid approach was not used you would have one very big SPA application with potentially a very complex and large routing table as well as a very large code base. By breaking the application out into many smaller applications you now only need a routing table for the area your application is concerned with (products or customers for example). Another benefit is that the code base for each area of the e-commerce management application is modularised due to it being implemented in a self-contained SPA. This will help make finding the required code easier assisting with maintaining and enhancing the web application in the long run. SPA applications force the browser to work in a way it was never originally designed to. Browsers work best when they can navigate from page to page and forget everything that happen on the previous page. They like to "wipe the slate clean". SPA’s force the browser to stay on a single page but update the current page's content dynamically as the user navigates around the application. This behaviour could lead to memory leaks as a browser could go for hours, days or even weeks without the page actually being refreshed. This hybrid model helps encourage page refreshing as each time the user moves from one area of the application to another, the application will make a page request and the browser will be breathing a sigh of relief. Of course, if the user were to spend all their time in a single area of the application the original problem will still be present. However, with the hybrid approach the functional surface area which a user can operate in without triggering a page refresh is greatly limited, increasing the likelihood of a page refresh occurring on a more regular basis.

On the downside there is the added complexity of handling a combination of client-side and server-side routing within the web application but I feel this is outweighed by the benefits of this approach.

Miguel will be presenting a session on this approach at NDC London in December so I will be watching this session as soon as it’s posted on their Vimo channel. If you cannot wait until then, Miguel's Building End-to-End Multi-Client Service Oriented Applications - Angular Edition course on Pluralsight also provides an insight into this hybrid approach.

Quoting Miguel, "It’s the best of both worlds".

EDIT:
You can find Miguel's talk at NDC London here: AngularJS for ASP.NET MVC Developers by Miguel A Castro.