Ad hoc Geek

Investigating the Visual Cron API

admin — Fri, 30 Sep 2011 09:42:03 +0000

We recently moved a lot of our processes to Visual Cron. This has given us a lot more control over how things run and how we’re notified when things go wrong. It’s also given us a very straightforward .NET API to query Visual Cron itself.

Here’s a simple starter script that enumerates all the active jobs and active tasks and appends the most recent exit code. We use something like this to monitor the tasks themselves and figure out whether any of them are consistently failing.

using System;
using System.Linq;
 
using VisualCron;
using VisualCronAPI;
 
namespace VisualCronTest
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                ExitCodes.Load();
 
                OutputJobsAndTasks(new Connection()
                {
                    Address = "VCSERVER01",
                    UserName = "admin",
                    PassWord = "",
                    Port = 16444,
                    ConnectionType = Connection.ConnectionT.Remote
                });
 
                //Add further Visual Cron server connections here
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
            }
        }
 
        public static void OutputJobsAndTasks(Connection conn)
        {
            var c = new Client() { LogToFile = false };
            var s = c.Connect(conn, true);
            var jobs = s.Jobs.GetAll();
 
            foreach (var j in jobs.Where(j => j.Stats.Active))
            {
                Console.WriteLine("JOB : " + j.Name);
                foreach (var t in j.Tasks.Where(t => t.Stats.Active))
                    Console.WriteLine("    TASK : " + t.Name + " - " + t.Stats.ExitCode == 0? "OK" : "ERROR");
            }
        }
    }
}

The only thing I’m personally hating about Visual Cron at the moment is that every time they make a release they break the remote connection API. This is a real pain because you have to recompile everything that was compiled against the previous version. I don’t believe this is down to fundamental API changes, I think it’s something deliberate, but I haven’t had time to investigate exactly what’s going on yet.

CSV Mangling in VB.NET

admin — Mon, 07 Feb 2011 16:03:35 +0000

I’ll probably go to hell for this one since I imagine there are much better libraries out there to do the right thing, but I needed to write a quick and dirty function in VB.NET to split some quoted CSV into it’s constituent fields.

This is horrible, but it works for the moment.

    'A function to split a quoted comma delimited string into separate fields
    Public Function GetFields(ByVal row As String) As String()
        Dim fields As New List(Of String)
        If row.Length = 0 Then
            Return fields.ToArray()
        End If
 
        Dim current As Integer = 0
        Dim buffer As New StringBuilder()
        Dim betweenQuotes As Boolean = False
        Dim eol As Boolean = False
        While Not eol
 
            Select Case row(current)
                Case CChar("""")
                    betweenQuotes = Not betweenQuotes
 
                Case CChar(",")
                    If betweenQuotes Then
                        buffer.Append(row(current))
                    Else
                        fields.Add(buffer.ToString())
                        buffer = New StringBuilder()
                    End If
 
                Case Else
                    buffer.Append(row(current))
 
            End Select
 
            current += 1
            eol = (current = row.Length)
        End While
 
        'Add the last field
        fields.Add(buffer.ToString())
 
        Return fields.ToArray()
    End Function

NB, you’ll need to add these lines to the top of the file :

Imports System.Text
Imports System.Collections.Generic

dtutil : Importing SSIS packages via batch scripting

admin — Fri, 28 Jan 2011 17:09:55 +0000

Just a quick and simple task here. I got fed up of manually importing dtsx files into our SSIS file store so I looked into a scripting solution. This relies entirely on dtutil which is installed with SSIS.

@echo off
 
set root=My SSIS Packages
set server=SSIS01
 
echo Importing Some Foo Packages
 
set tmpFolder=Foo
dtutil /SourceServer %server%  /FExists SQL;"\%root%\%tmpFolder%"
if errorlevel=1 dtutil /SourceServer %server% /FCreate SQL;"\%root_folder%";"%tmpFolder%"
 
dtutil /File "Foo1.dtsx" /Encrypt SQL;"\%root%\%tmpFolder%\Foo1";5 /DestServer %server% /Q
...

This is pretty simple – it checks whether there’s a “Foo” folder under the root path “My SSIS Packages” on the destination SSIS instance, if not it creates the folder and then uploads the package. The only thing to note here is that I’m using /Encrypt rather than /Copy, this is the same as manually uploading and setting the protection level to “Rely on server storage and roles for access control” (essential if you’re uploading packages which will be scheduled to run under a different account to your development account).

Playing with Powershell

admin — Thu, 06 Jan 2011 15:53:34 +0000

I’ve not actually used powershell before, but we had a requirement to store a list of security camera files and dates in a database so that we could track historical camera activity and this seemed like a good excuse to use it rather than writing a short C# executable.

What I ended up with was this :

# Configuration
$camera_home = "C:\Recordings";
$connection_string = "Data Source=localhost;Initial Catalog=MyDatabase;Trusted_Connection=True";
 
# Sanity Check
if (!(Test-Path $camera_home)) 
{
    [Console]::WriteLine($camera_home + " not found!"); 
    return;
};
 
# Set up the SQL Connection
$conn = new-object System.Data.SqlClient.SqlConnection($connection_string);
$conn.Open();
 
# Get a list of all cameras
$cameras = Get-ChildItem $camera_home;
 
# Iterate over the list
foreach($camera in $cameras)
{
    [Console]::WriteLine($camera);
    $guid = $camera.Name;
 
    # Check whether the current camera exists in the database.
    $camera_getid = new-object System.Data.SqlClient.SqlCommand("select ID from dbo.Camera where CameraGUID = @CameraGUID", $conn);
    $camera_getid.Parameters.Add("@CameraGUID", [System.Data.SqlDbType]"NVarChar").Value = $guid;
    $id = $camera_getid.ExecuteScalar();
    if ($id -eq $Null)
    {
        # Create a new camera record
        $camera_create = new-object System.Data.SqlClient.SqlCommand("insert into dbo.Camera(CameraGUID, CameraName) values(@CameraGUID, 'New Camera')", $conn);
        $camera_create.Parameters.Add("@CameraGUID", [System.Data.SqlDbType]"NVarChar").Value = $guid;
        $camera_create.ExecuteNonQuery();
        $id = $camera_getid.ExecuteScalar();        
    }
 
    # Get the date of the last recording we have a record for
    $camera_getlastrecording = new-object System.Data.SqlClient.SqlCommand("select case when Max(RecordingDate) is null then convert(datetime, '1/1/2001', 103) else MAX(RecordingDate) end from dbo.CameraData where CameraID = @CameraID", $conn);
    $camera_getlastrecording.Parameters.Add("@CameraID", [System.Data.SqlDbType]"Int").Value = $id;
    $lastrecording = $camera_getlastrecording.ExecuteScalar();
    [Console]::WriteLine([String]::Concat("    Last recording date : ", $lastrecording ));
 
    # Get a list of new recordings
    $recordings = Get-ChildItem -recurse -include "*.acsm" $camera_home\$camera | Where-Object {$_.CreationTime -gt $lastrecording};
    if ($recordings -eq $Null) 
    { 
        [Console]::WriteLine([String]::Concat("    No new recordings found."));
        continue 
    }
    [Console]::WriteLine([String]::Concat("    ", $recordings.Count, " new recordings found."));
 
    # Insert new recordings into the database
    foreach($recording in $recordings)
    {
        [Console]::WriteLine([String]::Concat($recording, ",", $recording.CreationTime.ToString("dd/MM/yyyy HH:mm:ss.FF")));
        $recording_create = new-object System.Data.SqlClient.SqlCommand("insert into dbo.CameraData(CameraID, RecordingName, RecordingDate) values(@CameraID, @RecordingName, @RecordingDate)", $conn);
        $recording_create.Parameters.Add("@CameraID", [System.Data.SqlDbType]"Int").Value = $id;
        $recording_create.Parameters.Add("@RecordingName", [System.Data.SqlDbType]"NVarChar").Value = $recording.FullName;
        $recording_create.Parameters.Add("@RecordingDate", [System.Data.SqlDbType]"DateTime").Value = $recording.CreationTime;
        $recording_create.ExecuteNonQuery();
    }
}

Probably the most interesting thing about this post was how long it took to actually make it because of this :

http://wordpress.stackexchange.com/questions/6421/how-can-a-single-line-in-a-blog-post-take-down-my-server/6428#6428

I suspect this kind of thing is going to reoccur if I post anything else with SQL in it, but unfortunately, I don’t have the ability to switch off the ModSecurity rules…

An animated, Reuters-style ticker in WPF

admin — Thu, 19 Aug 2010 15:26:56 +0000

I just need to get this out onto the blog for posterity, as it were.

This is a class that derives from StackPanel and Implements a Reuters style ticker behaviour with its children. Anything you add to the Children collection will be ticked from left to right.

The only issue with it so far is that adding something to the Children collection makes it initially show up on the left (intialised to coordinates (0,0)). This could probably be solved by adding something like an AnimatedChildren collection, and only adding to the Children collection when the animation is initiated?

This is hugely untidy at the moment, but it does work after a fashion. I don’t in any way claim this is the right way to do this or that any of this is best practice

Update

Ok, so I only realised a short while ago that you don’t strictly need the Storyboard class to initiate animations when you’re not using XAML. I haven’t yet investigated removing them from this code, but I’ll probably get round to that when I have a Windows VM finally set up.

public class MyPanel : Canvas
{
    private Timer _timer = new Timer(500);
    public MyPanel()
    {
        LayoutUpdated += new EventHandler(MyPanel_LayoutUpdated);
        _timer.Elapsed += new ElapsedEventHandler(_timer_Elapsed);
        _timer.Start();
    }
 
    void  _timer_Elapsed(object sender, ElapsedEventArgs e)
    {
        Dispatcher.BeginInvoke(new Func<object>(RunNextAnimation));
    }
 
    object RunNextAnimation()
    {
        if (storyBoardQueue.Count == 0) return null;
 
        //check the last animated child
        //if there is space, begin the animation of the next child.
        if (lastAnimated == null)
        {
            var sb = storyBoardQueue.Dequeue();
            var child = childQueue.Dequeue();
            animatingChildQueue.Enqueue(child);
            animatingStoryBoardQueue.Enqueue(sb);
            sb.Begin(child, true);
            lastAnimated = child;
        }
        else
        {
            var x = lastAnimated.RenderTransform.Value.OffsetX;
            if (x + lastAnimated.ActualWidth < ActualWidth - 10)
            {
                var sb = storyBoardQueue.Dequeue();
                var child = childQueue.Dequeue();
                animatingChildQueue.Enqueue(child);
                animatingStoryBoardQueue.Enqueue(sb);
                sb.Begin(child, true);
                lastAnimated = child;
            }
        }
        return null;
    }
 
    void MyPanel_LayoutUpdated(object sender, EventArgs e)
    {
        foreach(var child in Children)
        {
            if (animatedChildren.Contains(child as FrameworkElement)) continue;
            AnimateChild(child as FrameworkElement);
        }
    }
 
    //just to keep track of things we've created storyboards/translations for
    private List<FrameworkElement> animatedChildren = new List<FrameworkElement>();
 
    private Queue<Storyboard> storyBoardQueue = new Queue<Storyboard>();
    private Queue<FrameworkElement> childQueue = new Queue<FrameworkElement>();
 
    private FrameworkElement lastAnimated;
 
    private Queue<Storyboard> animatingStoryBoardQueue = new Queue<Storyboard>();
    private Queue<FrameworkElement> animatingChildQueue = new Queue<FrameworkElement>();
 
    private void AnimateChild(FrameworkElement child)
    {
        if (child == null) return;
 
        NameScope.SetNameScope(child, new NameScope());
 
        var anim = new DoubleAnimation
                   {
                       From = ActualWidth,
                       To = -100,
                       By = -5,
                       Duration = new Duration(TimeSpan.FromSeconds(ActualWidth/40)),
                   };
 
        var transform = new TranslateTransform();
        child.RegisterName("translate", transform);
 
        child.RenderTransform = transform;
        Storyboard.SetTargetName(anim, "translate");
        Storyboard.SetTargetProperty(anim, new PropertyPath(TranslateTransform.XProperty));
 
        var myStoryBoard = new Storyboard();
        myStoryBoard.Children.Add(anim);
        myStoryBoard.Completed += myStoryBoard_Completed;
        storyBoardQueue.Enqueue(myStoryBoard);
        childQueue.Enqueue(child);
 
        animatedChildren.Add(child);
    }
 
    void  myStoryBoard_Completed(object sender, EventArgs e)
    {
        storyBoardQueue.Enqueue(animatingStoryBoardQueue.Dequeue());
        childQueue.Enqueue(animatingChildQueue.Dequeue());
    }
}

Uploading large files with ASP.NET

admin — Tue, 20 Oct 2009 11:29:49 +0000

I’ve recently had a need for users of an intranet application to upload comparatively large text files (1-200MB) to a web server. There are only a couple of ways I can think of to get around the limits imposed by IIS and ASP.NET without writing code : train the users to upload smaller files which could be concatenated at the server, or allow the users to “upload” from a network share which the server has access to. These are obviously inelegant solutions, and after a little research I’ve found that the necessary code to enable large uploads yourself is surprisingly easy to write.

Jon Galloway has written a useful article which gives a more complete and eloquent view of the ASP.NET large file upload problem than I have time for here. Particularly of interest is the discussion he links to, titled “HttpHandler or HttpModule for file upload, large files, progress indicator?”. I’ve adapted or rewritten some of the suggested code from there for my solution. I’m particularly indebted to Travis Whidden whose code is much more complete than mine (it handles multiple files, for a start). Part of the reason I ended up rewriting it is because I didn’t think this solution handled a particular edge case, but I also needed to understand what it was doing, and the best way for me to do that was rewrite from scratch (this took me about a day of thinking, poking around and testing – I don’t claim to be a fast learner .

Essentially the code consists of two classes, UploadModule and RequestProcessor, along with some minor changes in the web.config file :

UploadModule

In order to intercept the HttpRequest and deal with it in a stream-wise fashion, we have to implement an IHttpModule :

using System;
using System.Diagnostics;
using System.Text;
using System.Web;
using System.Reflection;
 
namespace MyIntranetSite
{
    public class UploadModule : IHttpModule
    {
        #region IHttpModule Members
 
        void IHttpModule.Dispose()
        {
        }
 
        void IHttpModule.Init(HttpApplication context)
        {
            context.BeginRequest += new EventHandler(context_BeginRequest);
        }
 
        void  context_BeginRequest(object sender, EventArgs e)
        {
            HttpApplication application = (HttpApplication) sender;
            HttpContext context = application.Context;
 
            if (context.Request.ContentType.IndexOf("multipart/form-data") == -1)
            {
                //Not our bag, baby.
                return;
            }
 
            try
            {
                HttpWorkerRequest workerRequest = (HttpWorkerRequest) context.GetType().GetProperty("WorkerRequest", BindingFlags.Instance | BindingFlags.NonPublic).GetValue(context, null);
                if (workerRequest.HasEntityBody())
                {
                    long defaultBuffer = 500000; 
                    long contentLength = long.Parse((workerRequest.GetKnownRequestHeader(HttpWorkerRequest.HeaderContentLength)));
 
                    byte[] preloadedBufferData = workerRequest.GetPreloadedEntityBody();
                    RequestProcessor rp = new RequestProcessor(contentLength);
                    rp.ReadBuffer(ref preloadedBufferData);
 
                    long remaining = contentLength - preloadedBufferData.Length;
                    byte[] bufferData;
                    while (remaining > 0)
                    {
                        bufferData = new byte[(remaining > defaultBuffer)? defaultBuffer : remaining];
                        remaining -= bufferData.Length;
                        workerRequest.ReadEntityBody(bufferData, bufferData.Length);
                        rp.ReadBuffer(ref bufferData);
                    }
                }
            }
            catch(Exception ex)
            {
                EventLog.WriteEntry("Custom ASP.NET Upload Module", ex.Message);
            }
 
            context.Response.Redirect(context.Request.RawUrl);
        }
 
        #endregion
    }
}

This handles ALL “multipart/form-data” requests at the moment. You would probably want to check the url of the request and match it against a list of expected pages, otherwise all of your web requests (that is, every postback) would get processed by this module, and much of your code would be bypassed!

RequestProcessor

This class takes a series of byte array buffers and parses them for a start and end pattern. Hopefully, I’ve commented my code well enough for it to be read and understood. However, I should say that this only deals with single file uploads at the moment and expects to be able to write to “C:\temp\”. It would be possible to improve the code to handle multiple files and making the upload directory configurable would be fairly trivial, but I think it’s more useful as a learning tool if I keep it simple for now.

using System;
using System.Diagnostics;
using System.Collections.Generic;
using System.IO;
using System.Text;
 
namespace MyIntranetSite
{
 
    //
    /// 
    /// Takes byte[] chunks from an HTTP request and processes them looking for (currently) the first file.
    /// Each file will be wrapped by the lines :
    /// (start) "Content-Type: [some content type]\r\n"
    /// (end) "-----------------------------[a form post ID]\r\n\r\n" 
    ///       (that's 29 "-"s followed by a number, followed by 2 * carriage return + newline
    /// 
    /// Problems arise because the start and end patterns could span two buffers.
    /// This means we can't write from the latest buffer - we have to always be writing from the previous buffer,
    /// since we can never know if the latest buffer (assuming there are more bytes to read) contains the start of the
    /// end pattern, but not all of it.
    /// 
    public class RequestProcessor : IDisposable 
    {
        public long Length { get; private set; }
        public long BytesRead { get; private set; }
        public List<string> FinishedFiles = new List<string>();
 
        private BufferChunk previous;
        private bool _startFound = false;
        private bool _endFound = false;
        private List<byte> startPatternBegin;
        private List<byte> startPatternEnd;
        private List<byte> endPattern;
        private FileStream currentFileStream;
        private string currentFileName = Guid.NewGuid() + ".bin";
 
        public RequestProcessor(long length)
        {
            Length = length;
            BytesRead = 0;
            startPatternBegin = new List<byte>(Encoding.UTF8.GetBytes("Content-Type: "));
            startPatternEnd = new List<byte>(Encoding.UTF8.GetBytes("\r\n\r\n"));
        }
 
        public void ReadBuffer(ref byte[] buffer)
        {
            if (_endFound) return;
 
            BufferChunk current = new BufferChunk(ref buffer);
            if (previous == null)
            {
                //first buffer chunk
                //the first line of this will give the form content separator, which is also the endPattern
                int i = 0;
                endPattern = new List<byte>();
                while (current.Data[i] != Encoding.UTF8.GetBytes("\r")[0])
                {
                    endPattern.Add(current.Data[i]);
                    i++;
                }
            }
 
            //Merge the previous and current buffers
            List<byte> mergedBuffers = new List<byte>();
            if (previous != null) mergedBuffers.AddRange(previous.Data);
            mergedBuffers.AddRange(current.Data);
 
            if (!_startFound)
            {
                //Look for start pattern in the current buffer.
                //It could span this buffer and the one before (in which case the start point is in THIS buffer)
                //or it could span this buffer and the next (in which case the start point is in the NEXT buffer)
                //The latter case has to be checked when the next buffer comes in.
 
                int startBegin;
                if ((startBegin = FindBytePattern(mergedBuffers, startPatternBegin, 0)) != -1)
                {
                    //found a content-type declaration, look for the end of that line :
                    int startEnd;
                    if ((startEnd = FindBytePattern(mergedBuffers, startPatternEnd, startBegin + startPatternBegin.Count)) != -1)
                    {
                        //found the end of the line
                        if (startEnd + startPatternEnd.Count < mergedBuffers.Count - 1)
                        {
                            int startByte = startEnd + startPatternEnd.Count;
                            if (previous != null)
                            {
                                current.Start = startByte - previous.Data.Count;
                            }
                            else
                            {
                                current.Start = startByte;
                            }
                            _startFound = true;
                        }
                        // else the start byte is in the next buffer.
                    }
                }
 
                if (!_startFound) current.Start = current.Data.Count;
            }
 
            if (_startFound && !_endFound)
            {
                //Look for the end pattern in the current buffer
                //As with the start it could span beginning (in which case the last byte is in the PREVIOUS buffer)
                //Or it could span the end (in which case the last byte is in THIS buffer)
                //The latter case has to be checked when the next buffer comes in.
 
                int endBegin;
                int searchStart = previous != null? previous.Start : current.Start;
                if ((endBegin = FindBytePattern(mergedBuffers, endPattern, searchStart)) != -1)
                {
                    int endByte = endBegin - 1;
                    if (previous != null)
                    {
                        if (endByte < previous.Data.Count)
                            previous.End = endByte;
                        else
                            current.End = endByte - previous.Data.Count;
                    }
                    else
                    {
                        current.End = endByte;
                    }
                    _endFound = true;
                }
                // else the end byte is in the next buffer.
 
                if (!_endFound && previous != null) previous.End = previous.Data.Count;
            }
            BytesRead += current.Data.Count;
 
            //FILE CREATION
            if (previous != null && _startFound && previous.WriteBytes > 0)
            {
                //Write out the previous buffer from Start to End
                if (currentFileStream == null)
                {
                    currentFileStream = File.OpenWrite(@"C:\temp\" + currentFileName);
                }
                currentFileStream.Write(previous.Data.ToArray(), previous.Start, previous.WriteBytes);
            }
 
            if (_startFound && _endFound || BytesRead == Length)
            {
                //Write out the current buffer from Start to End
                if (currentFileStream == null)
                {
                    currentFileStream = File.OpenWrite(@"C:\temp\" + currentFileName);
                }
                currentFileStream.Write(current.Data.ToArray(), current.Start, current.WriteBytes);
                currentFileStream.Close();
                currentFileStream.Dispose();
            }
 
            previous = current;
        }
 
        private static int FindBytePattern(List<byte> container, List<byte> pattern, int startIndex)
        {
            int i, position;
            if (pattern.Count > container.Count - startIndex) return -1;
 
            for (position = startIndex; position < container.Count; position++)
            {
                if (container[position] == pattern[0])
                {
                    for(i = 1; i < pattern.Count; i++)
                    {
                        if (position + i == container.Count || pattern[i] != container[position + i]) break;
                    }
                    if (i == pattern.Count) return position;
                }
            }
 
            return -1;
        }
 
        #region IDisposable Members
 
        void IDisposable.Dispose()
        {
            if (currentFileStream != null)
            {
                currentFileStream.Close();
                currentFileStream.Dispose();
            }
        }
 
        #endregion
    }
 
    public class BufferChunk
    {
        public List<byte> Data;
        public int Start;
        public int End;
        public int WriteBytes { get { return End - Start; } }
 
        public BufferChunk(ref byte[] buffer)
        {
            Data = new List<byte>(buffer);
            Start = 0;
            End = Data.Count;
        }
    }
}

These files can sit within your web project or in a separate assembly if you want. Personally I’d rather have them sitting with the web project since that makes them more straightforward to debug and more obvious as to where they belong.

Web.Config changes

This is almost laughably trivial :

>
     name="UploadModule" type="MyIntranetSite.UploadModule"/>
>

And that’s it! There’s obviously a lot more that could be done (such as the progress indicator Travis incorporated), but this seems like a decent start to me.

Ideally, ASP.NET 3.0(?) and IIS 7.0 would address this kind of problem once and for all, but I’m not holding my breath. I also suspect a lot of businesses will remain on IE6.0, IIS 5/6 and ASP.NET 2.0 for another few years, so this approach will remain relevant a little while longer.

Update (a warning)

It’s entirely possible that the parser will bomb out with some kind of error on occasion. If this happens when the number of bytes left to process is greater than the ASP.NET maxRequestLength (or the IIS request length) then the site will (seemingly) silently fail and you’ll get the dreaded “Connection was reset” error page!

Enumerating Users with LDAP

admin — Fri, 09 Oct 2009 10:25:20 +0000

I’m sure there are lots of other tutorials, blog posts and actual documentation telling you how to do this, and this probably goes for a lot of the things I’ll post here, but this blog isn’t necessarily for other people. I’m beginning to realise that a lot of the code I post here is simply for my own benefit, and if it happens to coincide with the needs of someone else, that’s fine.

Anyway, I’m currently on-site again so I don’t have a lot of time to play and post snippets, but the LDAP code I was toying with this morning struck me as something I might want to revisit at some point in the future. It’s not exactly rocket science, but I often find that a short piece of working code can be an essential starting point when embarking on the understanding of a complex new library (for example, knowing that the technology is called LDAP and even understanding what that means, doesn’t tell you which .NET assemblies you should be investigating).

The following code is a C# console application which simply enumerates the users that the current account has access to know about in the given domain. An administrator running this should see everyone, but a lowly user with minimal rights may see only themselves (or possibly nothing at all!).

using System;
using System.Collections.Generic;
using System.DirectoryServices;
 
namespace ADTest
{
  class Program
  {
    static void Main(string[] args)
    {
      DirectoryEntry entry = new DirectoryEntry("LDAP://FM");
      DirectorySearcher search = new DirectorySearcher(entry); 
      search.Filter = "(&(objectClass=user)(objectCategory=person))";
      SearchResultCollection src = search.FindAll();
 
      Console.WriteLine(src.Count);
      foreach (SearchResult sr in src)
      {
        List<string> props = new List<string>();
        foreach (string propName in sr.Properties.PropertyNames)
        {
          props.Add(propName + "=[" + sr.Properties[propName] + "]");
        }
        Console.WriteLine(String.Join(", ", props.ToArray()) + "\n");
      }
    }
  }
}

This immediately gives me something to play with, it tells me I should investigate System.DirectoryServices, it tells me something about the way active directory search results are organised, and it tells me there is a curious filter syntax to investigate…

Binomial Option Pricing with Alchemy

admin — Thu, 17 Sep 2009 14:30:47 +0000

Yet another version of the binomial option pricing algorithm (sorry), but this time I’m using Alchemy!

I have to admit to not entirely understanding how Alchemy works, but I understand enough C to write a simple pricer in it. Here’s the relevant C-code (which is, in part, copied from Mike Chambers’ stringecho.c sample) :

#include 
#include 
#include 
 
//Header file for AS3 interop APIs
#include "AS3.h"
 
static AS3_Val getPrice(void* self, AS3_Val args)
{
    double N, T, S, K, v, r, price;
    AS3_ArrayValue( args, "DoubleType, DoubleType, DoubleType, DoubleType, DoubleType, DoubleType", &N, &T, &S, &K, &v, &r);
 
    double dt = T/N; //one time step
 
    double u = 1 + v*sqrt(dt); //up-tick
    double d = 1 - v*sqrt(dt); //down-tick
    double p = 0.5 + r*sqrt(dt)/(2*v); //risk-neutral probability of up-tick
    double df = 1/(1+r*dt); //discount factor over 1 time step, dt
 
    double* optionValues = malloc((N+1) * sizeof(double));
 
    //populate the tree (for N-steps there will be N+1 values)
    int i, j;
    double ST;
    for (i=0; i < N+1; i++) 
    {
        ST = S * pow(u, i) * pow(d, N-i);
        optionValues[i] = (ST > K)? ST - K : 0;
    }
 
    //now work backwards to get expected 
    //option value at each previous stage
    for (i=N; i >= 0; i--)
    {
        for(j=0; j<i; j++)
        {
            optionValues[j] = (p*optionValues[j+1] + (1-p)*optionValues[j])*df;
        }
    }
 
    price = optionValues[0];
    free(optionValues);
 
    return AS3_Number(price);
}
 
int main()
{
    //define the methods exposed to ActionScript
    //typed as an ActionScript Function instance
    AS3_Val getPriceMethod = AS3_Function( NULL, getPrice );
 
    // construct an object that holds references to the functions
    AS3_Val result = AS3_Object( "getPrice: AS3ValType", getPriceMethod );
 
    // Release
    AS3_Release( getPriceMethod );
 
    // notify that we initialized -- THIS DOES NOT RETURN!
    AS3_LibInit( result );
 
    // should never get here!
    return 0;
}

It’s been a long time since I used C in anger though, so it’s entirely possible I’m using malloc and free badly here! If so, please let me know.

Using the compile SWC in a flex project is incredibly simple, here’s the relevant exerpt :

import cmodule.binomial_pricer.CLibInit;
 
private function calc_Click():void
{
    var start:Number = new Date().time;
    var result:Number;
    var loader:CLibInit = new CLibInit();
    var lib:Object = loader.init();
    for (var i:int=0; i < repetitions.value; i++)
    {
        result = lib.getPrice(    steps.value, 
                            timeToExpiry.value, 
                            spot.value,
                            strike.value,
                            volatility.value,
                            rate.value);
    }                                
    price.text = result.toString();            
    var end:Number = new Date().time;
    averageTime.text = ((end-start)/repetitions.value).toString() + "ms";
}

And the finished result, which is about 8x (!) faster than the pixel bender version (on my machine at least) :

Binomial Option Pricing with Pixel Bender

admin — Tue, 15 Sep 2009 11:13:03 +0000

Pixel Bender has been around for a little while now and it’s a very cool idea, promising incredible speed improvements by allowing developers to make the most of multi-core processors and GPUs. It seems as though it was designed primarily for Adobe image manipulation software, but, since images are just arrays of bytes it can also be used to speed up any parallelisable code you might have within Flash.

A fast intro to Pixel Bender

The main unit of operation in pixel bender is the kernel. Generally, each kernel will take a source image and calculate the value of a single pixel in the output image. i.e. for every pixel in your source image, a single kernel instance will be run which can use that whole image to calculate the value of a single pixel (although you won’t usually want it to look at the whole image – the less dependent it is on other pixels, the faster it can run).

The kernel language is intended to be fast and as such is fairly simplistic. A simple kernel which converts every pixel in the source image to black would be written as follows :

<languageVersion : 1.0;>
 
kernel Gothify
<   namespace : "com.example";
    vendor : "Example.com";
    version : 1;
    description : "Fills every pixel in the source image with black";
>
{        
    input image4 source;
    output pixel4 result;
 
    void evaluatePixel()
    {
        result = pixel4(0,0,0,1.0);
    }
}

The developer’s guide says to use pixel4(0,0,0,0), but since the last value is the alpha channel, that just ends up with a transparent black image. It also doesn’t include the input image4, and while this may do something in photoshop, it’s not going to give you any output in the pixel bender IDE.

Now, an image is basically just an array of pixels, and, within pixel bender, a 4-channel pixel (a pixel4 type) is just a vector of floats. We can use any values within a pixel4, limited only by the bounds of a standard float size. So, we could change the evaluatePixel function above to the following and it wouldn’t make any difference to the output image :

void evaluatePixel()
{
    result = pixel4(-1500.67,-45.0,-32.333,515.47);
}

It’s only the values between 0.0 and 1.0 that can change the output colour, anything above 1.0 is at maximum and anything below 0.0 is at minimum. This means we can use pixel bender to operate on any array of floating point values as long as we’re happy for it to operate on pixel sized chunks of data in isolation (the pixel bender graph language does allow chaining of kernels, but this isn’t available to us in the flash player, so I’m going to ignore it for now).

Parallelising the Binomial Option Pricing Algorithm

There’s a short, fairly readable paper here [Ganesan, Chamberlain, Buhler] which gives some ideas for how to parallelise the algorithm. I’m only going to implement the scheme which is probably best illustrated by their fig. 2. This requires me to pass to each kernel the input image (array of option values at time T), the risk-neutral probability of an up-tick and the discount factor over one time step.
I’ll then have to call the resulting kernel (or array of kernels, depending on how you think of it), N-1 times to get the final option price.

The kernel I wrote ended up like this :

<languageVersion : 1.0;>
 
kernel BinomialOptionPricer
<   namespace : "com.adhocgeek";
    vendor : "Ad hoc Geek";
    version : 1;
    description : "Calculates the one step binomial option price";
>
{
    input image3 src;
    output pixel3 dst;
 
    parameter float p; //risk-neutral probability of an up-tick
    parameter float df; //discount factor over one time step
 
    void evaluatePixel()
    {
        pixel3 down = sampleNearest(src, outCoord());
        pixel3 up = sampleNearest(src, outCoord() + float2(1.0, 0));
        float V = (p*up.x + (1.0-p)*down.x)*df;
 
        dst = pixel3(V, 0, 0);
    }
}

Using the Pixel Bender Filter in Actionscript

This is actually incredibly easy. Flash 10 gives us a new class called the Shader which represents a Pixel Bender kernel. You can use this with the new ShaderJob class to trigger a set of kernel operations on any vector or byte array that you’ve created. I’m not going to try and write a tutorial, so here’s the final actionscript code I wrote (you should notice it’s quite similar to the previous version):

[Embed("BinomialOptionPricer.pbj", mimeType="application/octet-stream")]
private var BinomialOptionPricer:Class;
 
private function getPrice(  N:Number, //Number of steps
                                    T:Number, //Time to expiry (in years)
                                    S:Number, //Spot price of underlying
                                    K:Number, //Option strike
                                    v:Number, //Volatility of underlying
                                    r:Number, //risk-free rate
                                    NUMOPTIONS:Number 
                                 ):Number
{
    var dt:Number = T/N; //one time step
 
    var u:Number = 1 + v*Math.sqrt(dt); //up-tick
    var d:Number = 1 - v*Math.sqrt(dt); //down-tick
    var p:Number = 0.5 + r*Math.sqrt(dt)/(2*v); //risk-neutral prob. of up-tick
    var df:Number = 1/(1+r*dt); //discount factor over 1 time step, dt
 
    var optionValues:Vector.<Number> = new Vector.<Number>();
 
    //Populate the fake pixels array 
    var i:int, j:int, ST:Number;
    for (j=0; j < NUMOPTIONS; j++)
    {
        for (i=0; i < N+1; i++) 
        {
            ST = spot.value * Math.pow(u, i) * Math.pow(d, N-i);
            optionValues.push((ST > K)? ST - K : 0);
            optionValues.push(0);
            optionValues.push(0);
        }
    }
 
 
    //work backwards to get expected option value at each previous stage
    var sj:ShaderJob;
    var shader:Shader = new Shader(new BinomialOptionPricer() as ByteArray);
    shader.data.src.input = optionValues;
    shader.data.src.height = NUMOPTIONS;
    shader.data.p.value = [p];
    shader.data.df.value = [df];
    for (i=N; i > 0; i--)
    {
        shader.data.src.width = i+1;
        sj = new ShaderJob(shader, optionValues, i+1, NUMOPTIONS);
        sj.start(true);
    }
 
    return optionValues[0]; 
}

Points to note are where I’ve embedded the kernel in the actionscript itself and where I’ve added the ability to run the calculation multiple times in parallel (simulating running the calculation for many different options at once). Each line in the source “image” represents a different option (although they’re all using the same source data at the moment).

Here (if you have Flash Player 10 installed) is the resulting swf (view source enabled, of course) :

It’s slightly faster than the straight actionscript version, but not by a great deal, where it really shines is in calculating multiple options at once. There’s an interesting non-linear relationship between the number of lines in the source “image” and the speed of this particular algorithm. I might graph it at some point…

Binomial Option Pricing in Actionscript

admin — Wed, 09 Sep 2009 12:20:33 +0000

The binomial option pricing model is one of the simplest pricing models to understand and code. I’m not going to go into an explanation of the model itself, as that would take more time than I’m willing to expend, however I think it’s always useful to have example code lying around and I’m quite happy for some of it to lie around on my website.

private function getPrice(  N:Number, //Number of steps
                            T:Number, //Time to expiry (in years)
                            S:Number, //Spot price of underlying
                            K:Number, //Option strike
                            v:Number, //Volatility of underlying
                            r:Number //risk-free rate 
                         ):Number
{
 
    var dt:Number = T/N; //one time step
 
    var u:Number = 1 + v*Math.sqrt(dt); //up-tick
    var d:Number = 1 - v*Math.sqrt(dt); //down-tick
    var p:Number = 0.5 + r*Math.sqrt(dt)/(2*v); //risk-neutral prob. of up-tick
    var df:Number = 1/(1+r*dt); //discount factor over 1 time step, dt
 
    var optionValues:Array = new Array(N+1);
 
    //populate the tree (for N-steps there will be N+1 values)
    var i:int, j:int, ST:Number;
    for (i=0; i < N+1; i++) 
    {
        ST = spot.value * Math.pow(u, i) * Math.pow(d, N-i);
        optionValues[i] = (ST > K)? ST - K : 0;
    }
 
    //now work backwards to get expected option value at each previous stage
    for (i=N; i >= 0; i--)
    {
        for(j=0; j<i; j++)
        {
            optionValues[j] = (p*optionValues[j+1] + (1-p)*optionValues[j])*df;
        }
    }
 
    return optionValues[0]; 
}

What I’ve written is slightly different from some of the examples you might see (though exactly what Wikipedia suggests) – I don’t bother building up an array of stock prices since each value in the final array can be calculated with a simple formula. This means that I’ve removed a couple of loops from the calculation and (roughly) halved the execution time. I’m also assuming that we’re pricing a vanilla call option and that the risk-free rate is discretely compounded, though it would be fairly trivial to replace the call payoff function ((ST > K)? ST – K : 0) with any kind of external payoff function and to replace the discrete discount factor with a continuous one (1+rt becomes e^rT, essentially). In the end, with enough time steps, these kinds of changes make only a little difference though.

Have a play :

And for comparison, here’s a Black-Scholes pricer :