The Screwball Division

Finally the last post in this serie is done.

In the previous post we covered how we did to store the information we needed in the Image field in Sitecore.

In this post I will show you what we did to get the information stored in the Image field out on the website. I’ll start with a short warning, this will get complicated, so I will not go into details on all parts. At some point in this project we took some shortcuts which will explain some of the solutions we did that could have been better. This post does not contain much code examples since this post would be way to long, so at the end is a zipped file with all classes used in this solution.

When requesting any media from Sitecore, all file ends with .ashx, this ending is mapped to a requesthandler that steams the media to your browser.
Our plan was to use this in such a way that we request an predefined media-item inside Sitecore based on one of our extended media templates with the external media id and media extension as querystrings.
So we created an item /sitecore/media library/EmptyMedia/EmptyImage based on the template Externa Image that we created in one of the earlier posts without any data.
This media-item would get the url:
~/media/EmptyMedia/EmptyImage.ashx
We were going to request this item with both the id from the external media library and the extension of the image like this:
~/media/EmptyMedia/EmptyImage.ashx?extid=30779&extension=jpg

This was going to be our base for requests, now we needed a requesthandler that would take care of our media items and not mess up the existing Sitecore media functionality.
We started by creating a complete copy of the existing MediaRequestHandler that Sitecore uses for handling all .ashx requests. And started modifying it.

Our custom requesthandler would have to check if the request was done with extid as querystring and based on this either handle it as an external media item or as a Sitecore media item.

In Sitecores MediaRequestHandler, the ProcessRequest method wich is triggered on request, runs a method bool DoProcessRequest(HttpContext context)  which validates the request and creates an important object Sitecore.Resources.Media.Media and passes this to the method bool DoProcessRequest(HttpContext context, MediaRequest request, Sitecore.Resources.Media.Media media) which then loads the stream and passes it to the response output.

Simple right? Well we thought so, and we wanted to use these methods since they were already there. We started by checking if the request was done with extid as querystring and if that was the case a ran our custom method:

bool DoProcessExternalRequest(HttpContext context, MediaRequest request, string id, string extension)

This method was responsible for creating an object of the Sitecore.Resources.Media.Media. So we did some research on the Media class and found out that to create this we needed to create a MediaData object and pass it to the constructor of Media. And to create the MediaData object we needed a MediaItem that was based on the EmptyImage item inside Sitecore.
We now knew which objects we needed, so it was time to find out where the stream of data was loaded. After some poking around we found that the MediaRequestHandler calls the method GetStream in Sitecore.Resources.Media.Media wich in turn runs the pipeline getMediaStream with the MediaData object as parameter. All of the processors in the getMediaStream pipeline triggers a chain of methods that at one point ends up running the method GetBlobStream of MediaData which would retrieve the data from the Field by calling its GetBlobStream method. This was our entry point. We were going to check if the stream delivered from the Field objects was null and if so check if so validate if the request was for an external media file and deliver its stream instead.

So we had to be able to make the external id and extension available for this method. Since we were going to use this in some different cases to we decided to make an extension of the complete chain of classes and and mimic the behavior. We already had an extension of the MediaItem called ExtMediaItem created in the previous post which could hold the external id, extension and the width and height. So over to the MediaData class, we needed it to be able to hold and pass the external id and extension and to handle the ExtMediaItem. So the class ExtendedMediaData was created and we did the following alterations in this class:

• Added attribute ExtID to hold and pass the external media id
• Altered MediaId to deliver a unique id for external media. The MediaId is, what I think, used for caching.
• GetBlobStream – this method was altered to try to load the data from the external media library if the stream that was delivered by the field is null.

With that altered we tried to get this working, so we created a ExtMediaItem based on the EmptyImage in Sitecore.
We created a ExtendedMediaData based on the ExtMediaItem and provided it with the  the values from the querystring for extid and extension.
This was then fead into a Media object and sent to
bool DoProcessRequest(HttpContext context, MediaRequest request, Sitecore.Resources.Media.Media media)
in the RequestHandler.
Unfortunately this didn’t, we got an exception telling us that the object provided wasn’t a MediaData object, even though ExtendedMediaData inherits from MediaData.
Because the lack of time we decided to create an extension of the Media class aswell where we altered all MediaData objects handled in the class to ExtendedMediaData, which worked.

So now when I make the following request:
~/media/EmptyMedia/EmptyImage.ashx?extid=30779&extension=jpg
the following steps are made:

1. We check if the url contains extid and calls a custom method to build a Media object.
2. An ExtMediaItem based on the item /sitecore/media library/EmptyMedia/EmptyImage is created
3. An ExtendedMediaData is created with the ExtMediaItem and also provided with the ExtId and Extension values.
4. A custom Media class wich intherits Sitecores Media class i created with the ExtendedMediaData.
5. This Media object is provided to the DoProcessRequest method in the requesthandler.
6. The DoProcessRequest calls the method GetStream on the Media object which triggers the getMediaStream pipeline.
7. At some point one of the processors in the getMediaStream pipeline triggers a chain of methods that calls the GetBlobStream in the ExtendedMediaData.
8. If the stream delivered by the Field object in ExtendedMediaData, it tries to get a stream from the web service of the external media library based on the extid value.
9. Once this stream has been fetched, it’s returned and handled by the processors in the getMediaStream pipeline, like being resized. This is where the extension is needed, if the ExtendedMediaData doen’t have an extension it won’t be handled as a image.
10. When the getMediaStream pipeline is done processing the stream it’s return to the Media object which will add the stream to Sitecores media cache and then return it to the RequestHandler.
11. This is where it all ends, when the stream has been delivered to the RequestHandler, it is written to the output stream of the HTTP response object.

So thats the end of the series on how we integrated an external media library. Below is a link to all files used, they are not in any order what so ever and there might be some or alot of code that isn’t in use

 Here’s the zip-file

In the previous part we looked at how to get external information into the Sitecore media library tree using a DataProvider. Usually we use DataProviders to integrate content into the master database and then publish it to the web database for faster handling of that information.

In this case we don’t want to do that because the external media library contains alot of information and we won’t be using everything on the website. So we had to find another way of storing the information and only the used information and nothing more.

We started looking at how Sitecore stores information in the Image field. Many of Sitecores fields stores its information as xml and our idea was to use this and extend the Image fields to store our extra information about the external media. What we did was create a CustomImage class that inherits the Sitecore.Shell.Applications.ContentEditor.Image and “overrided” the BrowseImage method and added a block of code that checks if the selected image is based on any of our external media library templates. If they are we add our extra attributes.

MediaItem item = new MediaItem(innerItem);
TemplateItem template = item.InnerItem.Template;
if ((template != null) && !(this.IsImageMedia(template) || MediaHelper.IsExternalImageMedia(template.ID)))
{
	SheerResponse.Alert("The selected item does not contain an image.", new string[0]);
}
else
{
	MediaUrlOptions options = new MediaUrlOptions();
	string mediaUrl = MediaManager.GetMediaUrl(item, options);

	this.XmlValue.SetAttribute("mediapath", item.MediaPath);
	this.XmlValue.SetAttribute("src", mediaUrl);
	this.XmlValue.SetAttribute("height", item.InnerItem.Fields["height"].Value);
	this.XmlValue.SetAttribute("width", item.InnerItem.Fields["width"].Value);
	if ((template != null) && MediaHelper.IsExternalImageMedia(template.ID))
	{
		this.XmlValue.SetAttribute("extmediaid", item.InnerItem.Fields["ExtMediaId"].Value);
		this.XmlValue.SetAttribute("mediaid", MediaHelper.EmptyImage.ToString());
		this.XmlValue.SetAttribute("extension", item.InnerItem.Fields["Extension"].Value);
	}
	else
		this.XmlValue.SetAttribute("mediaid", item.ID.ToString());

	this.Value = item.MediaPath;
	this.Update();
	this.SetModified();
}

In the code you can see that when it’s an external media item we set the mediaid to MediaHelper.EmptyImage.ToString() which returns an id. The reason for this is that since we don’t publish the external media items to the web database we don’t have any media item to request. Our idea here was to create an empty media node in the media library that we can refer to and request with our information as querystrings when displaying the image.

After creating this class we reconfigured the Sitecore Image Field to use this class instead of the default Image in the Core database. Now we are able to store information about the selected external image. The next step is to use the information.

We tried to extend the Sitecore.Data.Field.ImageField class but found it easier to create a completely new class since there were so many changes to do. What we mainly did was adding the custom attributes. As we went on we found it necessary to extend the Sitecore.Data.Items.MediaItem to add our attributes and to store the Width and Height. I will link these classes at the end of this post.

Now we are able to use the information stored in the field. Next post I will cover the part where we create a new mediahandler that can deliver the image.

Below is a rar-archive with all the classes created.
classes.rar

In the end of the spring we started a new website project for one of our customers where one of the criterias was to integrate an external media library. The media library had a set of webservice-methods which provided us with the structure and files of the media library and the data.

Our goal was to implement the media library into Sitecores own media library without accualy storing it there, and to use Sitecores media-cache and resize engine.

This posed a series of questions

1. How would we integrate the media library?
2. How would we store the used media-items without having to store everything from the external media library inside sitecore and thus publishing it to the web-database?
3. How would we be able to use Sitecore media-cache and resize engine?
4. And finally, what impact to the systems preformance would all this do?

This part will cover the first questions namely how to integrate the media library.

When integrating the external media library, we constructed a DataProvider that would call the external media librarys webservices and add them as items in the media library tree. For this we used and modified the dataprovider from the Share resource youtube module.

What we did first was creating templates for each of the object-types we were going to collect.

The basic idea here was to use alot of sitecores functionality, so these templates interited from each of the sitecore media templates. And since all of the would have some information in common, we created a base templates which containd the external media item id and the path to the item.

When this was done we created what would become the root of the external media library inside sitecore media library by simply create an item based on the Media Library Folder.

Now we were set to continue with the dataprovider, so now lets go on to the coding part:)

Since we had our set of templates and one of the templates would act as a folder, we only wanted the dataprovider to trigger when the folder template was expanded. To achive this we would have to override the method GetChildIDs.

public override IDList GetChildIDs(ItemDefinition itemDefinition, CallContext context)
{
    if (itemDefinition.TemplateID.Equals(new ID(_folderTemplateID)))
    {
        ID parentId = itemDefinition.ID;
        IDictionary<string, string> ids = LoadDataID(parentId);
        IDList idList = new IDList();
        foreach (var itemId in ids)
        {
            IDTableEntry idEntry = IDTable.GetID(prefix, itemId.Key);
            ID newId;
            if (idEntry == null)
            {
                newId = ID.NewID;
                IDTable.Add(prefix, itemId.Key, newId, parentId, itemId.Value);
            }
            else
            {
                newId = idEntry.ID;
            }
            idList.Add(newId);
        }
        return idList;
    }
    return null;
}

We started of with this. Here we have the folder template id stored in a private variable _folderTemplateID and checks if the current item is based on that template. If the current item is a our folder item we call the LoadDataID which in turn calls the webservices and provides a dictionary with name and id. These are then, based on wether they exist or not in the Sitecore IDTable, either added or fetched from the IDTable.
This worked fine until we started to modify inside the external media library. When we started to move items inside the media library we started to get some NullReferenceException. After some troubleshooting we finally understood what was going wrong. The Sitecore IDTable acctually caches the ID added to it for some time. So if an external media item where to change the its parent, we would stil get the old parent id reference from the Sitecore IDTabel which in turn would spell dissaster for us.

What we did was going through all keys in the IDTable stored for our prefix that had the current parentId and did not exist in the Dictinary of IDs provided by the webservice.

public override IDList GetChildIDs(ItemDefinition itemDefinition, CallContext context)
{
    if (itemDefinition.TemplateID.Equals(new ID(_folderTemplateID)))
    {

        ID parentId = itemDefinition.ID;
        IDictionary<string, string> ids = LoadDataID(parentId);
        IDList idList = new IDList();

        foreach (var itemId in ids)
        {
            IDTableEntry idEntry = IDTable.GetID(prefix, itemId.Key);
            ID newId;
            if (idEntry == null)
            {
                newId = ID.NewID;
                IDTable.Add(prefix, itemId.Key, newId, parentId, itemId.Value);
            }
            else
            {
                newId = idEntry.ID;
            }
            if (!_handledItemIds.Contains(newId))
                _handledItemIds.Add(newId);
            idList.Add(newId);
        }
        foreach (var entry in IDTable.GetKeys(prefix).Where(k => k.ParentID.Equals(parentId) && !ids.Keys.Contains(k.Key)))
        {
            IDTable.RemoveID(prefix, entry.ID);
            if (_handledItemIds.Contains(entry.ID))
                _handledItemIds.Remove(entry.ID);
        }
        context.DataManager.Database.Caches.DataCache.Clear();
        return idList;
    }
    return null;
}

The _handledItemIds is a list that is used to check the current item is something we can process in the GetItemDefinition function which is the next step in our code.
The GetItemDefinition is a method in the DataProvider class that creates a ItemDefinition that tells the system what kind of item it is (Template), what ID it has and its name. Here we only want to handle the items we got from the webservice and to avoid having to look in the IDTable if our ID exist there, which takes a loooong time. So we stored all handled item ids in the _handledItemIds and basically created a method that checks if a certain id exists.
So what we did was override the GetItemDefinition which is called everytime Sitecore displays an item and checked it the current request was for a external media item.

public override ItemDefinition GetItemDefinition(ID itemId, CallContext context)
{
    ItemDefinition itemDef = null;
    string itemName = string.Empty;
    if (CanProcessMediaItem(itemId, context))
    {
        string originalID = GetOriginalRecordID(itemId);
        ExtMediaData itemInfo = GetMediaDataInfo(itemId);
        if (itemInfo == null)
        {
            var n = GetCachedExternalNode(originalID);
            itemName = ItemTools.GetValidNodeName(n.Name);
            var resourceTid = GetResourceID(n);
            itemDef = new ItemDefinition(itemId, itemName, resourceTid, ID.Null);
            _items.Add(itemId, new ExtMediaData(itemId, resourceTid, itemName, n));

        }
        else
        {
            itemDef = new ItemDefinition(itemId, itemInfo.Name, itemInfo.TemplateID, ID.Null);
        }
        try
        {
            ItemCache itemCache = CacheManager.GetItemCache(context.DataManager.Database);
            if ((itemCache != null) && (itemDef != null))
            {
                ReflectionUtil.CallMethod(itemCache, "RemoveItem", true, false, false, new object[] { itemDef.ID });
            }
        }
        catch (Exception exception)
        {
            Log.Error("Can't clear cache for the External Media item", exception, this);
        }
        // Do not cache items for this data provider
        if (itemDef != null)
            ((ICacheable)itemDef).Cacheable = false;
    }
    return itemDef;
}

This method can be a little confusing, but what we do here is first checking if the current request is for a external media item by running “if (CanProcessMediaItem(itemId, context))”, this method checks in the _handledItemIds if the id exist and returns true or false. What we check next is if we have done this procedure before and cached the information. We do this by calling ExtMediaData itemInfo = GetMediaDataInfo(itemId); which returns an ExtMediaData from a Hashtable of it exist. If it exist we create an ItemDefinition based on the info, however if it doesn’t exist we will have to create it request all information. To do this we need the id of the external media item which this request does “string originalID = GetOriginalRecordID(itemId);”. Since we store the original id in the sitecore IDTable this method just checks in the IDTable and returns the referenced id.

string GetOriginalRecordID(ID id)
{
    IDTableEntry[] idEntries = IDTable.GetKeys(prefix, id);
    if (idEntries != null && idEntries.Length > 0)
    {
        return idEntries[0].Key;
    }
    return null;
}

Now we have the id, and early in the project we did another webservice request to get the media from the external media library. However we soon found out that this took to long. What we did was to modify LoadDataID called in GetChildIDs to store all collected media items in a collection that we lated use to fetch the item again from within GetItemDefinition by calling “var n = GetCachedExternalNode(originalID);”. Now we only need the Template id for the item. From the webservices provided from the external media library we only got the file extension, so we created a function that based on an extension return the correct template id. Nothing fancy. Basically a switch.
We now have all the information we need to create an ItemDefinition. This is then cached in the same HashTable that is queried earlier in GetItemDefinition so that we don’t have to request the webservice again for this item during this session.

The last thing we do is override the GetItemFields where we, based on “hard coded” fields, return the values from the external media item.

public override FieldList GetItemFields(ItemDefinition itemDefinition, VersionUri versionUri, CallContext context)
{
    FieldList fields = new FieldList();
    if (CanProcessMediaItem(itemDefinition.ID, context))
    {

        string originalID = GetOriginalRecordID(itemDefinition.ID);
        TemplateItem templateItem = ContentDB.Templates[itemDefinition.TemplateID];
        if (templateItem != null)
        {
            ExtMediaData ytItemInfo = GetMediaDataInfo(itemDefinition.ID);
            if (ytItemInfo != null)
            {
                foreach (var field in templateItem.Fields)
                {
                    var value = GetFieldValue(field, ytItemInfo);
                    if (!string.IsNullOrEmpty(value))
                        fields.Add(field.ID, value);
                }
            }
        }
    }
    return fields;
}

Ofcourse we check if this is an item that we want to handle by calling CanProcessMediaItem. Then we get the cached media item, and the corresponding sitecore template. What we want to do here is to go through all fields, and collect the data from the media item into the corresponding field. We had some trouble here because we allways added the result into the field, even if it was empty, which was a misstake, since we wanted to use the templates standard values aswell. But by checking if the returned value was empty or not, we solved that problem.
The GetFieldValue isn’t realy that interesting, but I’ll show it anyway:)

private string GetFieldValue(TemplateFieldItem field, ExtMediaData itemInfo)
{
    string val = string.Empty;
    switch (field.Name)
    {
        case "External Path":
            val = itemInfo.Data.Adress;
            break;
        case "Width":
            val = itemInfo.Data.Width.ToString();
            break;
        case "Height":
            val = itemInfo.Data.Height.ToString();
            break;
        case "Title":
            val = itemInfo.Data.Name;
            break;
        case "Extension":
            val = itemInfo.Data.Extension.Replace(".", "");
            break;
        case "Mime Type":
            val = itemInfo.Data.MimeType;
            break;
        case "ExtMediaId":
            val = itemInfo.Data.Id.ToString();
            break;
        case "Size":
            val = itemInfo.Data.Size.ToString();
            break;
        case "Format":
            break;
        case "Dimensions":
            break;
    }
    if (val == null) { val = string.Empty; }
    return val;
}

As you can see it only checks if the current field is one of those i want to add information to, and returns the value.

With this done, we had to configure the dataprovider to work in the master database. We did this by creating an include file that looked like this:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:x="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <dataProviders>
            <externalmediadp type="Pyramid.DataProviders.ExternalMediaDataProvider, Pyramid.DataProviders">
                <param desc="folderTemplateID">{416F14A1-0D15-422D-991E-C8039B074F05}</param>
                <param desc="resourceTemplateFolder">{369D90E5-2394-4E6C-8BF1-2BCDE4EDDA28}</param>
                <param desc="externalMediaField">ExtMediaId</param>
                <param desc="contentDatabase">master</param>
            </externalmediadp>
        </dataProviders>
        <databases>
            <database id="master" singleInstance="true" type="Sitecore.Data.Database, Sitecore.Kernel">
                <dataProviders hint="list:AddDataProvider">
                    <dataProvider ref="dataProviders/externalmediadp">
                        <disableGroup>delete</disableGroup>
                    </dataProvider>
                </dataProviders>
            </database>
        </databases>
    </sitecore>
</configuration>

Here we define a dataprovider and provide the constructor with some attributes, “folderTemplateID”, “resourceTemplateFolder”, “externalMediaField”, “contentDatabase”. To get this attributes, the will have to be added to the dataproviders contructor and then store them in private variables. Then we add the dataprovider to the master database.

Now the dataprovider will call the webservice, get all items and folders from the external media library to the sitecore media library. Or will it?
Theoretically yes, however our provider is built in such a way that it needs an external parent id to get the external children… yes?… Well what about the root folder? yeah we will have to configure it with an external id aswell. And as you saw in the GetFieldValue we acctually have a field called “ExtMediaId” where we store all the collected items. So after configuring our root folder, that we manually created earlier, with the media librarys root item id it now works.

To summarize this part, well something we have learned is to never request an item from sitecore using the the database. By doing this you might end up in an infinit loop. For example if we would request a external folder item using for exampel Sitecore.Context.Database.Item[theId], this would trigger all configured dataproviders for that database to call GetItemDefinition for that item, among them your dataprovider will be called. And depending on where in the code you call did the call, you might end up doing the same request again.
Our solution to this was to create an instance of the ItemProvider and do our request using it, like this:

var provider = new ItemProvider();
Item item = provider.GetItem(theIdTotheItemYouWant, ContentDB.Languages[0], Sitecore.Data.Version.Latest, ContentDB, SecurityCheck.Enable);

This specifies that its an item from sitecore you want that exist in the database and nothing more.

Thats all for now… stay tuned for the next part where we will look in to how we did to store the used media without storing the entire media library in sitecore. And we will also look at how we did to use sitecore resize and caching for media.

Best regards