Ad hoc Geek

An animated, Reuters-style ticker in WPF

admin — Thu, 19 Aug 2010 15:26:56 +0000

I just need to get this out onto the blog for posterity, as it were.

This is a class that derives from StackPanel and Implements a Reuters style ticker behaviour with its children. Anything you add to the Children collection will be ticked from left to right.

The only issue with it so far is that adding something to the Children collection makes it initially show up on the left (intialised to coordinates (0,0)). This could probably be solved by adding something like an AnimatedChildren collection, and only adding to the Children collection when the animation is initiated?

This is hugely untidy at the moment, but it does work after a fashion. I don’t in any way claim this is the right way to do this or that any of this is best practice

public class MyPanel : Canvas
{
    private Timer _timer = new Timer(500);
    public MyPanel()
    {
        LayoutUpdated += new EventHandler(MyPanel_LayoutUpdated);
        _timer.Elapsed += new ElapsedEventHandler(_timer_Elapsed);
        _timer.Start();
    }
 
    void  _timer_Elapsed(object sender, ElapsedEventArgs e)
    {
        Dispatcher.BeginInvoke(new Func<object>(RunNextAnimation));
    }
 
    object RunNextAnimation()
    {
        if (storyBoardQueue.Count == 0) return null;
 
        //check the last animated child
        //if there is space, begin the animation of the next child.
        if (lastAnimated == null)
        {
            var sb = storyBoardQueue.Dequeue();
            var child = childQueue.Dequeue();
            animatingChildQueue.Enqueue(child);
            animatingStoryBoardQueue.Enqueue(sb);
            sb.Begin(child, true);
            lastAnimated = child;
        }
        else
        {
            var x = lastAnimated.RenderTransform.Value.OffsetX;
            if (x + lastAnimated.ActualWidth < ActualWidth - 10)
            {
                var sb = storyBoardQueue.Dequeue();
                var child = childQueue.Dequeue();
                animatingChildQueue.Enqueue(child);
                animatingStoryBoardQueue.Enqueue(sb);
                sb.Begin(child, true);
                lastAnimated = child;
            }
        }
        return null;
    }
 
    void MyPanel_LayoutUpdated(object sender, EventArgs e)
    {
        foreach(var child in Children)
        {
            if (animatedChildren.Contains(child as FrameworkElement)) continue;
            AnimateChild(child as FrameworkElement);
        }
    }
 
    //just to keep track of things we've created storyboards/translations for
    private List<FrameworkElement> animatedChildren = new List<FrameworkElement>();
 
    private Queue<Storyboard> storyBoardQueue = new Queue<Storyboard>();
    private Queue<FrameworkElement> childQueue = new Queue<FrameworkElement>();
 
    private FrameworkElement lastAnimated;
 
    private Queue<Storyboard> animatingStoryBoardQueue = new Queue<Storyboard>();
    private Queue<FrameworkElement> animatingChildQueue = new Queue<FrameworkElement>();
 
    private void AnimateChild(FrameworkElement child)
    {
        if (child == null) return;
 
        NameScope.SetNameScope(child, new NameScope());
 
        var anim = new DoubleAnimation
                   {
                       From = ActualWidth,
                       To = -100,
                       By = -5,
                       Duration = new Duration(TimeSpan.FromSeconds(ActualWidth/40)),
                   };
 
        var transform = new TranslateTransform();
        child.RegisterName("translate", transform);
 
        child.RenderTransform = transform;
        Storyboard.SetTargetName(anim, "translate");
        Storyboard.SetTargetProperty(anim, new PropertyPath(TranslateTransform.XProperty));
 
        var myStoryBoard = new Storyboard();
        myStoryBoard.Children.Add(anim);
        myStoryBoard.Completed += myStoryBoard_Completed;
        storyBoardQueue.Enqueue(myStoryBoard);
        childQueue.Enqueue(child);
 
        animatedChildren.Add(child);
    }
 
    void  myStoryBoard_Completed(object sender, EventArgs e)
    {
        storyBoardQueue.Enqueue(animatingStoryBoardQueue.Dequeue());
        childQueue.Enqueue(animatingChildQueue.Dequeue());
    }
}

Uploading large files with ASP.NET

admin — Tue, 20 Oct 2009 11:29:49 +0000

I’ve recently had a need for users of an intranet application to upload comparatively large text files (1-200MB) to a web server. There are only a couple of ways I can think of to get around the limits imposed by IIS and ASP.NET without writing code : train the users to upload smaller files which could be concatenated at the server, or allow the users to “upload” from a network share which the server has access to. These are obviously inelegant solutions, and after a little research I’ve found that the necessary code to enable large uploads yourself is surprisingly easy to write.

Jon Galloway has written a useful article which gives a more complete and eloquent view of the ASP.NET large file upload problem than I have time for here. Particularly of interest is the discussion he links to, titled “HttpHandler or HttpModule for file upload, large files, progress indicator?”. I’ve adapted or rewritten some of the suggested code from there for my solution. I’m particularly indebted to Travis Whidden whose code is much more complete than mine (it handles multiple files, for a start). Part of the reason I ended up rewriting it is because I didn’t think this solution handled a particular edge case, but I also needed to understand what it was doing, and the best way for me to do that was rewrite from scratch (this took me about a day of thinking, poking around and testing – I don’t claim to be a fast learner .

Essentially the code consists of two classes, UploadModule and RequestProcessor, along with some minor changes in the web.config file :

UploadModule

In order to intercept the HttpRequest and deal with it in a stream-wise fashion, we have to implement an IHttpModule :

using System;
using System.Diagnostics;
using System.Text;
using System.Web;
using System.Reflection;
 
namespace MyIntranetSite
{
    public class UploadModule : IHttpModule
    {
        #region IHttpModule Members
 
        void IHttpModule.Dispose()
        {
        }
 
        void IHttpModule.Init(HttpApplication context)
        {
            context.BeginRequest += new EventHandler(context_BeginRequest);
        }
 
        void  context_BeginRequest(object sender, EventArgs e)
        {
            HttpApplication application = (HttpApplication) sender;
            HttpContext context = application.Context;
 
            if (context.Request.ContentType.IndexOf("multipart/form-data") == -1)
            {
                //Not our bag, baby.
                return;
            }
 
            try
            {
                HttpWorkerRequest workerRequest = (HttpWorkerRequest) context.GetType().GetProperty("WorkerRequest", BindingFlags.Instance | BindingFlags.NonPublic).GetValue(context, null);
                if (workerRequest.HasEntityBody())
                {
                    long defaultBuffer = 500000; 
                    long contentLength = long.Parse((workerRequest.GetKnownRequestHeader(HttpWorkerRequest.HeaderContentLength)));
 
                    byte[] preloadedBufferData = workerRequest.GetPreloadedEntityBody();
                    RequestProcessor rp = new RequestProcessor(contentLength);
                    rp.ReadBuffer(ref preloadedBufferData);
 
                    long remaining = contentLength - preloadedBufferData.Length;
                    byte[] bufferData;
                    while (remaining > 0)
                    {
                        bufferData = new byte[(remaining > defaultBuffer)? defaultBuffer : remaining];
                        remaining -= bufferData.Length;
                        workerRequest.ReadEntityBody(bufferData, bufferData.Length);
                        rp.ReadBuffer(ref bufferData);
                    }
                }
            }
            catch(Exception ex)
            {
                EventLog.WriteEntry("Custom ASP.NET Upload Module", ex.Message);
            }
 
            context.Response.Redirect(context.Request.RawUrl);
        }
 
        #endregion
    }
}

This handles ALL “multipart/form-data” requests at the moment. You would probably want to check the url of the request and match it against a list of expected pages, otherwise all of your web requests (that is, every postback) would get processed by this module, and much of your code would be bypassed!

RequestProcessor

This class takes a series of byte array buffers and parses them for a start and end pattern. Hopefully, I’ve commented my code well enough for it to be read and understood. However, I should say that this only deals with single file uploads at the moment and expects to be able to write to “C:\temp\”. It would be possible to improve the code to handle multiple files and making the upload directory configurable would be fairly trivial, but I think it’s more useful as a learning tool if I keep it simple for now.

using System;
using System.Diagnostics;
using System.Collections.Generic;
using System.IO;
using System.Text;
 
namespace MyIntranetSite
{
 
    //
    /// 
    /// Takes byte[] chunks from an HTTP request and processes them looking for (currently) the first file.
    /// Each file will be wrapped by the lines :
    /// (start) "Content-Type: [some content type]\r\n"
    /// (end) "-----------------------------[a form post ID]\r\n\r\n" 
    ///       (that's 29 "-"s followed by a number, followed by 2 * carriage return + newline
    /// 
    /// Problems arise because the start and end patterns could span two buffers.
    /// This means we can't write from the latest buffer - we have to always be writing from the previous buffer,
    /// since we can never know if the latest buffer (assuming there are more bytes to read) contains the start of the
    /// end pattern, but not all of it.
    /// 
    public class RequestProcessor : IDisposable 
    {
        public long Length { get; private set; }
        public long BytesRead { get; private set; }
        public List<string> FinishedFiles = new List<string>();
 
        private BufferChunk previous;
        private bool _startFound = false;
        private bool _endFound = false;
        private List<byte> startPatternBegin;
        private List<byte> startPatternEnd;
        private List<byte> endPattern;
        private FileStream currentFileStream;
        private string currentFileName = Guid.NewGuid() + ".bin";
 
        public RequestProcessor(long length)
        {
            Length = length;
            BytesRead = 0;
            startPatternBegin = new List<byte>(Encoding.UTF8.GetBytes("Content-Type: "));
            startPatternEnd = new List<byte>(Encoding.UTF8.GetBytes("\r\n\r\n"));
        }
 
        public void ReadBuffer(ref byte[] buffer)
        {
            if (_endFound) return;
 
            BufferChunk current = new BufferChunk(ref buffer);
            if (previous == null)
            {
                //first buffer chunk
                //the first line of this will give the form content separator, which is also the endPattern
                int i = 0;
                endPattern = new List<byte>();
                while (current.Data[i] != Encoding.UTF8.GetBytes("\r")[0])
                {
                    endPattern.Add(current.Data[i]);
                    i++;
                }
            }
 
            //Merge the previous and current buffers
            List<byte> mergedBuffers = new List<byte>();
            if (previous != null) mergedBuffers.AddRange(previous.Data);
            mergedBuffers.AddRange(current.Data);
 
            if (!_startFound)
            {
                //Look for start pattern in the current buffer.
                //It could span this buffer and the one before (in which case the start point is in THIS buffer)
                //or it could span this buffer and the next (in which case the start point is in the NEXT buffer)
                //The latter case has to be checked when the next buffer comes in.
 
                int startBegin;
                if ((startBegin = FindBytePattern(mergedBuffers, startPatternBegin, 0)) != -1)
                {
                    //found a content-type declaration, look for the end of that line :
                    int startEnd;
                    if ((startEnd = FindBytePattern(mergedBuffers, startPatternEnd, startBegin + startPatternBegin.Count)) != -1)
                    {
                        //found the end of the line
                        if (startEnd + startPatternEnd.Count < mergedBuffers.Count - 1)
                        {
                            int startByte = startEnd + startPatternEnd.Count;
                            if (previous != null)
                            {
                                current.Start = startByte - previous.Data.Count;
                            }
                            else
                            {
                                current.Start = startByte;
                            }
                            _startFound = true;
                        }
                        // else the start byte is in the next buffer.
                    }
                }
 
                if (!_startFound) current.Start = current.Data.Count;
            }
 
            if (_startFound && !_endFound)
            {
                //Look for the end pattern in the current buffer
                //As with the start it could span beginning (in which case the last byte is in the PREVIOUS buffer)
                //Or it could span the end (in which case the last byte is in THIS buffer)
                //The latter case has to be checked when the next buffer comes in.
 
                int endBegin;
                int searchStart = previous != null? previous.Start : current.Start;
                if ((endBegin = FindBytePattern(mergedBuffers, endPattern, searchStart)) != -1)
                {
                    int endByte = endBegin - 1;
                    if (previous != null)
                    {
                        if (endByte < previous.Data.Count)
                            previous.End = endByte;
                        else
                            current.End = endByte - previous.Data.Count;
                    }
                    else
                    {
                        current.End = endByte;
                    }
                    _endFound = true;
                }
                // else the end byte is in the next buffer.
 
                if (!_endFound && previous != null) previous.End = previous.Data.Count;
            }
            BytesRead += current.Data.Count;
 
            //FILE CREATION
            if (previous != null && _startFound && previous.WriteBytes > 0)
            {
                //Write out the previous buffer from Start to End
                if (currentFileStream == null)
                {
                    currentFileStream = File.OpenWrite(@"C:\temp\" + currentFileName);
                }
                currentFileStream.Write(previous.Data.ToArray(), previous.Start, previous.WriteBytes);
            }
 
            if (_startFound && _endFound || BytesRead == Length)
            {
                //Write out the current buffer from Start to End
                if (currentFileStream == null)
                {
                    currentFileStream = File.OpenWrite(@"C:\temp\" + currentFileName);
                }
                currentFileStream.Write(current.Data.ToArray(), current.Start, current.WriteBytes);
                currentFileStream.Close();
                currentFileStream.Dispose();
            }
 
            previous = current;
        }
 
        private static int FindBytePattern(List<byte> container, List<byte> pattern, int startIndex)
        {
            int i, position;
            if (pattern.Count > container.Count - startIndex) return -1;
 
            for (position = startIndex; position < container.Count; position++)
            {
                if (container[position] == pattern[0])
                {
                    for(i = 1; i < pattern.Count; i++)
                    {
                        if (position + i == container.Count || pattern[i] != container[position + i]) break;
                    }
                    if (i == pattern.Count) return position;
                }
            }
 
            return -1;
        }
 
        #region IDisposable Members
 
        void IDisposable.Dispose()
        {
            if (currentFileStream != null)
            {
                currentFileStream.Close();
                currentFileStream.Dispose();
            }
        }
 
        #endregion
    }
 
    public class BufferChunk
    {
        public List<byte> Data;
        public int Start;
        public int End;
        public int WriteBytes { get { return End - Start; } }
 
        public BufferChunk(ref byte[] buffer)
        {
            Data = new List<byte>(buffer);
            Start = 0;
            End = Data.Count;
        }
    }
}

These files can sit within your web project or in a separate assembly if you want. Personally I’d rather have them sitting with the web project since that makes them more straightforward to debug and more obvious as to where they belong.

Web.Config changes

This is almost laughably trivial :

>
     name="UploadModule" type="MyIntranetSite.UploadModule"/>
>

And that’s it! There’s obviously a lot more that could be done (such as the progress indicator Travis incorporated), but this seems like a decent start to me.

Ideally, ASP.NET 3.0(?) and IIS 7.0 would address this kind of problem once and for all, but I’m not holding my breath. I also suspect a lot of businesses will remain on IE6.0, IIS 5/6 and ASP.NET 2.0 for another few years, so this approach will remain relevant a little while longer.

Update (a warning)

It’s entirely possible that the parser will bomb out with some kind of error on occasion. If this happens when the number of bytes left to process is greater than the ASP.NET maxRequestLength (or the IIS request length) then the site will (seemingly) silently fail and you’ll get the dreaded “Connection was reset” error page!

Enumerating Users with LDAP

admin — Fri, 09 Oct 2009 10:25:20 +0000

I’m sure there are lots of other tutorials, blog posts and actual documentation telling you how to do this, and this probably goes for a lot of the things I’ll post here, but this blog isn’t necessarily for other people. I’m beginning to realise that a lot of the code I post here is simply for my own benefit, and if it happens to coincide with the needs of someone else, that’s fine.

Anyway, I’m currently on-site again so I don’t have a lot of time to play and post snippets, but the LDAP code I was toying with this morning struck me as something I might want to revisit at some point in the future. It’s not exactly rocket science, but I often find that a short piece of working code can be an essential starting point when embarking on the understanding of a complex new library (for example, knowing that the technology is called LDAP and even understanding what that means, doesn’t tell you which .NET assemblies you should be investigating).

The following code is a C# console application which simply enumerates the users that the current account has access to know about in the given domain. An administrator running this should see everyone, but a lowly user with minimal rights may see only themselves (or possibly nothing at all!).

using System;
using System.Collections.Generic;
using System.DirectoryServices;
 
namespace ADTest
{
  class Program
  {
    static void Main(string[] args)
    {
      DirectoryEntry entry = new DirectoryEntry("LDAP://FM");
      DirectorySearcher search = new DirectorySearcher(entry); 
      search.Filter = "(&(objectClass=user)(objectCategory=person))";
      SearchResultCollection src = search.FindAll();
 
      Console.WriteLine(src.Count);
      foreach (SearchResult sr in src)
      {
        List<string> props = new List<string>();
        foreach (string propName in sr.Properties.PropertyNames)
        {
          props.Add(propName + "=[" + sr.Properties[propName] + "]");
        }
        Console.WriteLine(String.Join(", ", props.ToArray()) + "\n");
      }
    }
  }
}

This immediately gives me something to play with, it tells me I should investigate System.DirectoryServices, it tells me something about the way active directory search results are organised, and it tells me there is a curious filter syntax to investigate…

Binomial Option Pricing with Alchemy

admin — Thu, 17 Sep 2009 14:30:47 +0000

Yet another version of the binomial option pricing algorithm (sorry), but this time I’m using Alchemy!

I have to admit to not entirely understanding how Alchemy works, but I understand enough C to write a simple pricer in it. Here’s the relevant C-code (which is, in part, copied from Mike Chambers’ stringecho.c sample) :

#include 
#include 
#include 
 
//Header file for AS3 interop APIs
#include "AS3.h"
 
static AS3_Val getPrice(void* self, AS3_Val args)
{
    double N, T, S, K, v, r, price;
    AS3_ArrayValue( args, "DoubleType, DoubleType, DoubleType, DoubleType, DoubleType, DoubleType", &N, &T, &S, &K, &v, &r);
 
    double dt = T/N; //one time step
 
    double u = 1 + v*sqrt(dt); //up-tick
    double d = 1 - v*sqrt(dt); //down-tick
    double p = 0.5 + r*sqrt(dt)/(2*v); //risk-neutral probability of up-tick
    double df = 1/(1+r*dt); //discount factor over 1 time step, dt
 
    double* optionValues = malloc((N+1) * sizeof(double));
 
    //populate the tree (for N-steps there will be N+1 values)
    int i, j;
    double ST;
    for (i=0; i < N+1; i++) 
    {
        ST = S * pow(u, i) * pow(d, N-i);
        optionValues[i] = (ST > K)? ST - K : 0;
    }
 
    //now work backwards to get expected 
    //option value at each previous stage
    for (i=N; i >= 0; i--)
    {
        for(j=0; j<i; j++)
        {
            optionValues[j] = (p*optionValues[j+1] + (1-p)*optionValues[j])*df;
        }
    }
 
    price = optionValues[0];
    free(optionValues);
 
    return AS3_Number(price);
}
 
int main()
{
    //define the methods exposed to ActionScript
    //typed as an ActionScript Function instance
    AS3_Val getPriceMethod = AS3_Function( NULL, getPrice );
 
    // construct an object that holds references to the functions
    AS3_Val result = AS3_Object( "getPrice: AS3ValType", getPriceMethod );
 
    // Release
    AS3_Release( getPriceMethod );
 
    // notify that we initialized -- THIS DOES NOT RETURN!
    AS3_LibInit( result );
 
    // should never get here!
    return 0;
}

It’s been a long time since I used C in anger though, so it’s entirely possible I’m using malloc and free badly here! If so, please let me know.

Using the compile SWC in a flex project is incredibly simple, here’s the relevant exerpt :

import cmodule.binomial_pricer.CLibInit;
 
private function calc_Click():void
{
    var start:Number = new Date().time;
    var result:Number;
    var loader:CLibInit = new CLibInit();
    var lib:Object = loader.init();
    for (var i:int=0; i < repetitions.value; i++)
    {
        result = lib.getPrice(    steps.value, 
                            timeToExpiry.value, 
                            spot.value,
                            strike.value,
                            volatility.value,
                            rate.value);
    }                                
    price.text = result.toString();            
    var end:Number = new Date().time;
    averageTime.text = ((end-start)/repetitions.value).toString() + "ms";
}

And the finished result, which is about 8x (!) faster than the pixel bender version (on my machine at least) :

Binomial Option Pricing with Pixel Bender

admin — Tue, 15 Sep 2009 11:13:03 +0000

Pixel Bender has been around for a little while now and it’s a very cool idea, promising incredible speed improvements by allowing developers to make the most of multi-core processors and GPUs. It seems as though it was designed primarily for Adobe image manipulation software, but, since images are just arrays of bytes it can also be used to speed up any parallelisable code you might have within Flash.

A fast intro to Pixel Bender

The main unit of operation in pixel bender is the kernel. Generally, each kernel will take a source image and calculate the value of a single pixel in the output image. i.e. for every pixel in your source image, a single kernel instance will be run which can use that whole image to calculate the value of a single pixel (although you won’t usually want it to look at the whole image – the less dependent it is on other pixels, the faster it can run).

The kernel language is intended to be fast and as such is fairly simplistic. A simple kernel which converts every pixel in the source image to black would be written as follows :

<languageVersion : 1.0;>
 
kernel Gothify
<   namespace : "com.example";
    vendor : "Example.com";
    version : 1;
    description : "Fills every pixel in the source image with black";
>
{        
    input image4 source;
    output pixel4 result;
 
    void evaluatePixel()
    {
        result = pixel4(0,0,0,1.0);
    }
}

The developer’s guide says to use pixel4(0,0,0,0), but since the last value is the alpha channel, that just ends up with a transparent black image. It also doesn’t include the input image4, and while this may do something in photoshop, it’s not going to give you any output in the pixel bender IDE.

Now, an image is basically just an array of pixels, and, within pixel bender, a 4-channel pixel (a pixel4 type) is just a vector of floats. We can use any values within a pixel4, limited only by the bounds of a standard float size. So, we could change the evaluatePixel function above to the following and it wouldn’t make any difference to the output image :

void evaluatePixel()
{
    result = pixel4(-1500.67,-45.0,-32.333,515.47);
}

It’s only the values between 0.0 and 1.0 that can change the output colour, anything above 1.0 is at maximum and anything below 0.0 is at minimum. This means we can use pixel bender to operate on any array of floating point values as long as we’re happy for it to operate on pixel sized chunks of data in isolation (the pixel bender graph language does allow chaining of kernels, but this isn’t available to us in the flash player, so I’m going to ignore it for now).

Parallelising the Binomial Option Pricing Algorithm

There’s a short, fairly readable paper here [Ganesan, Chamberlain, Buhler] which gives some ideas for how to parallelise the algorithm. I’m only going to implement the scheme which is probably best illustrated by their fig. 2. This requires me to pass to each kernel the input image (array of option values at time T), the risk-neutral probability of an up-tick and the discount factor over one time step.
I’ll then have to call the resulting kernel (or array of kernels, depending on how you think of it), N-1 times to get the final option price.

The kernel I wrote ended up like this :

<languageVersion : 1.0;>
 
kernel BinomialOptionPricer
<   namespace : "com.adhocgeek";
    vendor : "Ad hoc Geek";
    version : 1;
    description : "Calculates the one step binomial option price";
>
{
    input image3 src;
    output pixel3 dst;
 
    parameter float p; //risk-neutral probability of an up-tick
    parameter float df; //discount factor over one time step
 
    void evaluatePixel()
    {
        pixel3 down = sampleNearest(src, outCoord());
        pixel3 up = sampleNearest(src, outCoord() + float2(1.0, 0));
        float V = (p*up.x + (1.0-p)*down.x)*df;
 
        dst = pixel3(V, 0, 0);
    }
}

Using the Pixel Bender Filter in Actionscript

This is actually incredibly easy. Flash 10 gives us a new class called the Shader which represents a Pixel Bender kernel. You can use this with the new ShaderJob class to trigger a set of kernel operations on any vector or byte array that you’ve created. I’m not going to try and write a tutorial, so here’s the final actionscript code I wrote (you should notice it’s quite similar to the previous version):

[Embed("BinomialOptionPricer.pbj", mimeType="application/octet-stream")]
private var BinomialOptionPricer:Class;
 
private function getPrice(  N:Number, //Number of steps
                                    T:Number, //Time to expiry (in years)
                                    S:Number, //Spot price of underlying
                                    K:Number, //Option strike
                                    v:Number, //Volatility of underlying
                                    r:Number, //risk-free rate
                                    NUMOPTIONS:Number 
                                 ):Number
{
    var dt:Number = T/N; //one time step
 
    var u:Number = 1 + v*Math.sqrt(dt); //up-tick
    var d:Number = 1 - v*Math.sqrt(dt); //down-tick
    var p:Number = 0.5 + r*Math.sqrt(dt)/(2*v); //risk-neutral prob. of up-tick
    var df:Number = 1/(1+r*dt); //discount factor over 1 time step, dt
 
    var optionValues:Vector.<Number> = new Vector.<Number>();
 
    //Populate the fake pixels array 
    var i:int, j:int, ST:Number;
    for (j=0; j < NUMOPTIONS; j++)
    {
        for (i=0; i < N+1; i++) 
        {
            ST = spot.value * Math.pow(u, i) * Math.pow(d, N-i);
            optionValues.push((ST > K)? ST - K : 0);
            optionValues.push(0);
            optionValues.push(0);
        }
    }
 
 
    //work backwards to get expected option value at each previous stage
    var sj:ShaderJob;
    var shader:Shader = new Shader(new BinomialOptionPricer() as ByteArray);
    shader.data.src.input = optionValues;
    shader.data.src.height = NUMOPTIONS;
    shader.data.p.value = [p];
    shader.data.df.value = [df];
    for (i=N; i > 0; i--)
    {
        shader.data.src.width = i+1;
        sj = new ShaderJob(shader, optionValues, i+1, NUMOPTIONS);
        sj.start(true);
    }
 
    return optionValues[0]; 
}

Points to note are where I’ve embedded the kernel in the actionscript itself and where I’ve added the ability to run the calculation multiple times in parallel (simulating running the calculation for many different options at once). Each line in the source “image” represents a different option (although they’re all using the same source data at the moment).

Here (if you have Flash Player 10 installed) is the resulting swf (view source enabled, of course) :

It’s slightly faster than the straight actionscript version, but not by a great deal, where it really shines is in calculating multiple options at once. There’s an interesting non-linear relationship between the number of lines in the source “image” and the speed of this particular algorithm. I might graph it at some point…

Binomial Option Pricing in Actionscript

admin — Wed, 09 Sep 2009 12:20:33 +0000

The binomial option pricing model is one of the simplest pricing models to understand and code. I’m not going to go into an explanation of the model itself, as that would take more time than I’m willing to expend, however I think it’s always useful to have example code lying around and I’m quite happy for some of it to lie around on my website.

private function getPrice(  N:Number, //Number of steps
                            T:Number, //Time to expiry (in years)
                            S:Number, //Spot price of underlying
                            K:Number, //Option strike
                            v:Number, //Volatility of underlying
                            r:Number //risk-free rate 
                         ):Number
{
 
    var dt:Number = T/N; //one time step
 
    var u:Number = 1 + v*Math.sqrt(dt); //up-tick
    var d:Number = 1 - v*Math.sqrt(dt); //down-tick
    var p:Number = 0.5 + r*Math.sqrt(dt)/(2*v); //risk-neutral prob. of up-tick
    var df:Number = 1/(1+r*dt); //discount factor over 1 time step, dt
 
    var optionValues:Array = new Array(N+1);
 
    //populate the tree (for N-steps there will be N+1 values)
    var i:int, j:int, ST:Number;
    for (i=0; i < N+1; i++) 
    {
        ST = spot.value * Math.pow(u, i) * Math.pow(d, N-i);
        optionValues[i] = (ST > K)? ST - K : 0;
    }
 
    //now work backwards to get expected option value at each previous stage
    for (i=N; i >= 0; i--)
    {
        for(j=0; j<i; j++)
        {
            optionValues[j] = (p*optionValues[j+1] + (1-p)*optionValues[j])*df;
        }
    }
 
    return optionValues[0]; 
}

What I’ve written is slightly different from some of the examples you might see (though exactly what Wikipedia suggests) – I don’t bother building up an array of stock prices since each value in the final array can be calculated with a simple formula. This means that I’ve removed a couple of loops from the calculation and (roughly) halved the execution time. I’m also assuming that we’re pricing a vanilla call option and that the risk-free rate is discretely compounded, though it would be fairly trivial to replace the call payoff function ((ST > K)? ST – K : 0) with any kind of external payoff function and to replace the discrete discount factor with a continuous one (1+rt becomes e^rT, essentially). In the end, with enough time steps, these kinds of changes make only a little difference though.

Have a play :

And for comparison, here’s a Black-Scholes pricer :

VAR, COVAR and COREL in Actionscript

admin — Fri, 28 Aug 2009 11:07:13 +0000

Someone was asking whether or not it’s possible to call Excel functions from a Flex project because they needed to use VAR, COVAR and COREL. The short answer is, of course, no (at least not to my knowledge, and, even if it were possible, I don’t think it’s something you really want to encourage). A better answer would explain that these functions aren’t especially complex and that an Actionscript implementation is fairly straightforward. The wikipedia page on covariance is a little daunting if you’ve never really thought about what these functions entail (or have forgotten), but essentially it boils down to this :

private function corel(X:Array, Y:Array):Number
{
    //correlation coeff between two random variables X and Y is defined as :
    //correlation(X,Y) = covar(X,Y)/(sqrt(Var(X)) * sqrt(Var(Y)))
    //
    //var(X) = covar(X,X);
    //covar(X,Y) = E((X-xm)(Y-ym)); where xm, ym are the population means.
    return covar(X, Y)/Math.sqrt(covar(X, X) * covar(Y, Y));
}
 
private function covar(X:Array, Y:Array):Number
{
    //Sample covariance is Sum((X-xm)(Y-ym))/n-1
    //where n is the sample size and xm & ym are sample means.
    //I'll assume that X and Y are the same size...
    var total:Number = 0;
    var xm:Number = average(X);
    var ym:Number = (X == Y)? xm : average(Y);
    for (var i:int=0; i<X.length; i++)
        total += (X[i]-xm)*(Y[i]-ym);
    return total/(X.length - 1); 
}
 
private function average(X:Array):Number
{
    //Sample mean or average = Sum(X)/(sample size)
    var total:Number = 0;
    for each(var x:Number in X)
        total += x;
    return total/X.length;
}

Of course, it’s not quite that simple – my covar won’t return the same value as COVAR in Excel, because Excel uses the biased estimator (i.e. it divides by n rather than (n-1)), but this cancels out when calculating the correlation coefficient, and correcting covar to the biased estimator is trivial (multply by (n-1)/n). Personally, I think Excel is wrong for defaulting to the biased estimator for COVAR, especially since VAR uses the unbiased one (and, as I’ve written in the comments, VAR(X)=COVAR(X,X)).

Playing with IDataReader and SqlBulkCopy

admin — Tue, 25 Aug 2009 13:57:14 +0000

For importing huge amounts of data into SQL Server, there’s really nothing quite like SqlBulkCopy. I’ve recently had a need to manipulate a (roughly) 330,000 line CSV file and import the results of that manipulation into a single table. Doing this record by record can take minutes, but with SqlBulkCopy, importing that many records can be done in about 4 seconds on my development machine (and it’s definitely not the fastest PC in the world).

Out of Memory

Originally I was reading in the file, manipulating the data and writing out another CSV file I could use with DTS. However, SqlBulkCopy.WriteToServer doesn’t take a CSV file directly, it only takes either a DataTable, DataRow[] or IDataReader, so at first, while writing out the CSV file I was also building up a DataTable to pass to it. For a file like mine of only a few hundred thousand records, it wasn’t a big problem to build that DataTable in memory – it was only taking a few hundred MB – but it occurred to me that there could be a problem if the number of records increased modestly to a million or so. In fact, with a file of only 4 million records, I’d probably be looking at a System.OutOfMemoryException.

IDataReader

The solution to this problem is to write a class which implements IDataReader and pass this to SqlBulkCopy. There are a few implementations out there already, but I couldn’t find anything both free and in C#. I didn’t look terribly hard though, and I was curious to try writing a basic implementation myself just to see how difficult it would be.
It turns out it’s not very difficult at all, it depends on how much effort you want to put in. For a simple spike like this I just wanted to see how long it took to implement enough of IDataReader for SqlBulkCopy to work so I could then see how much memory was being used. This is (part of) what I ended up with :

public class CSVDataReader : IDataReader
{
    private StreamReader stream;
    private Dictionary<string, int> columnsByName = new Dictionary<string,int>();
    private Dictionary<int, string> columnsByOrdinal = new Dictionary<int,string>();
    private string[] currentRow;
    private bool _isClosed = true;
 
    public CSVDataReader(string fileName)
    {
        if (!File.Exists(fileName))
            throw new Exception("File [" + fileName + "] does not exist.");
 
        this.stream = new StreamReader(fileName);
 
        string[] headers = stream.ReadLine().Split(',');
        for (int i=0; i < headers.Length; i++)
        {
            columnsByName.Add(headers[i], i);
            columnsByOrdinal.Add(i, headers[i]);
        }
 
        _isClosed = false;
    }
 
    public void Close()
    {
        if (stream != null) stream.Close();
        _isClosed = true;
    }
 
    public int FieldCount
    {
        get { return columnsByName.Count; }
    }
 
    /// 
    /// This is the main function that does the work - it reads in the next line of data and parses the values into ordinals.
    /// 
    /// A value indicating whether the EOF was reached or not.
    public bool Read()
    {
        if (stream == null) return false;
        if (stream.EndOfStream) return false;
 
        currentRow = stream.ReadLine().Split(',');
        return true;
    }
 
    public object GetValue(int i)
    {
        return currentRow[i];
    }
 
    public string GetName(int i)
    {
        return columnsByOrdinal[i];
    }
 
    public int GetOrdinal(string name)
    {
        return columnsByName[name];
    }
 
    //Other IDataReader methods/properties here, but all throwing not implemented exceptions.
}

It turns out you only need to implement these few properties and methods for SqlBulkCopy (I’m not even sure you need implement this much). Once I had this, it was a mere four lines to import the CSV file into SQL Server :

SqlBulkCopy sbc = new SqlBulkCopy(mySqlConnection);
sbc.DestinationTableName = "MyTable";
sbc.BulkCopyTimeout = 6000; //10 Minutes
sbc.WriteToServer(new CSVDataReader(myFileName));

Of course, this relies on all of MyTable’s columns being of type varchar and the column headers in the CSV file need to match up with the column headers in the table, but this is supposed to be a simple spike.
The first time I ran this, it was using about 12MB of memory for a 12MB file (my original 330K line file), and while this was an improvement over the 100s of MB for building the DataTable, it didn’t really tell me anything about how it might scale. So, I generated a file with about 35 million rows in it just to see what would happen. I was pleasantly surprised to find that it only used about 12MB from start to finish – this is clearly the benefit of using this DataReader model, the whole file/data structure is never in memory so we’re not generating enormous data structures to pass around.

If I have to do something similar to this in the future, I’ll probably tidy up this CSVDataReader and use it again. I may even implement the rest of it…

Actionscript Date Lunacy

admin — Thu, 20 Aug 2009 14:41:31 +0000

I really hate the Actionscript Date class. Mostly because of the way it deals with local time offsets. When I create a new date instance, I generally don’t care about the local time zone settings. I’ll care about them when I have to combine it with other dates, sure, but at the time I set that value I want it to stay that value. The AS date class thinks otherwise.

Say I have a number of milliseconds from epoch (1 Jan 1970) which I want to use to create a specific date so I can use a DateFormatter to output the result, then you’d think that the following would be fine :

var myDate:Date = new Date(1250766979000);
var df:DateFormatter = new DateFormatter();
df.formatString = "YYYY-MM-DD JJ:NN:SS";
var stringDate:String = df.format(myDate);

(Let’s gloss over the batshit-insane need for a DateFormatter class and the actual formatString itself – JJ for 0-23 hour time, really?)

But no. In my area, we’re currently in daylight savings mode, so the Flash player decides that this date must be modified accordingly, and if you format it with the code above you’ll get “2009-08-20 12:16:19″, whereas it should be “2009-08-20 11:16:19″ (you can try and calculate this yourself if you’re feeling particularly bored). This sounds fairly trivial to deal with until you realise that, short of writing your own DateFormatter, there’s no way to output the actual, formatted UTC date and time (ignoring the unformattable toUTCString() method).

To do that, you have to take this approach :

var myDate:Date = new Date(1250766979000);
myDate.setTime(myDate.getTime() + myDate.timezoneOffset * 60 * 1000);
var df:DateFormatter = new DateFormatter();
df.formatString = "YYYY-MM-DD JJ:NN:SS";
var stringDate:String = df.format(myDate);

Which, to me at least, seems mental, because we haven’t really created a UTC date there – we’ve actually had to modify the milliseconds since epoch due to the local timezone settings. If I change the time settings on my computer, the output of this will still change! It’s possible I’ve lived too long in Microsoft-land, but the equivalent code in C# seems a million times better and far more predictable :

DateTime dat = new DateTime(1970, 1, 1).Add(new TimeSpan(12507669790000000L));
string stringDate = dat.ToString("yyyy-MM-dd HH:mm:ss");

Yes, miraculously, dates can format themselves in C#! There’s no need for a superfluous DateFormatter class. The only reason this code might look a litte overinflated is that dates in C# have a wider range, so I’ve had to convert the milliseconds since Unix epoch to 10’s of nanoseconds since 0001-1-1 00:00:00.

Simple IIS Scripting

admin — Tue, 18 Aug 2009 13:27:36 +0000

Using the Windows Scripting Host, it’s possible to write small JScript files which perform administrative tasks. I had a need for a small script to automatically create a virtual directory in IIS and link it with a particular path as part of a (very basic) deployment procedure. After a little bit (maybe 15 minutes?) of searching and reading, this is what I ended up with :

function install()
{
    var hostname	= "localhost";
    var projectName	= "MyProject";
    var projectDir	= "C:\\Projects\\MyProject";
 
    try
    {
        var existing = GetObject("IIS://" + hostname + "/W3SVC/1/Root/" + projectName);
        WScript.Echo("ERROR : An application called " + projectName + " already exists.");
        return;
    } catch(e) {}
 
    try
    {
        //Get the root IIS application
        var objIIS = GetObject("IIS://" + hostname + "/W3SVC/1/Root");
 
        var virtualDir = objIIS.Create("IISWebVirtualDir", projectName);
        virtualDir.AppIsolated		= 2; //pooled process
        virtualDir.AccessScript 	= true;
        virtualDir.AppFriendlyName	= projectName;
        virtualDir.Path 		= projectDir;
        virtualDir.AccessWrite		= true;
        virtualDir.AccessRead		= true;
        virtualDir.AccessExecute	= true;
        virtualDir.AppCreate(true);
        virtualDir.SetInfo();
 
        WScript.Echo("IIS application \"" + projectName + "\" created successfully.");
    }
    catch(e)
    {
        var msg = "";
        for (var prop in e)
        {
            msg += prop + ":" + e[prop] + "\n";
        }
 
        WScript.Echo(msg);
    }
}
 
install();

Pretty simple really. The WScript object is the root of the WSH hierarchy and is always available, but the important type here is IISWebVirtualDir.

It’s possible to do all kinds of things in IIS from a script, but this was all I needed. I might investigate permissions and DNS entries if the deployment gets more complicated (or more frequent).