CAML can take a long walk across the desert…

Robert Bogue November 8, 2005 No Comments

Back in March I attended a MS Book Publishers Summit in Redmond. I had the pleasure to sit down with Mike Fitzmaurice. What came out of his blog entry “CAML is Here to Stay“. I was the mysterious “guy” mentioned in the post.

Fitz’s post was reassuring – sort of . It told me that there would be some support for CAML and that there was hope for all of us struggling with the poor (ok, missing) documentation on CAML and how it works. I was encouraged because I knew how much of SharePoint requires CAML and how not knowing the rules for CAML hampers your ability to do anything in SharePoint – from queries to site definitions you’ll find so much CAML you’ll swear you’re in a desert. (Sorry, I had to get that one in.)

However, despite Fitz’s claims we still struggle to find good documentation on CAML. Every new bit of information on the topic seems to lead to more that remains undocumented.

The reason for this post is to make public one of the recent interactions that I had with the product support group – and the response I got. I think it illustrates the problems that I have with CAML – the challenges that I think all of us face.

I’ve got a set of utilities that allow you to migrate configuration data (web part placement, lists, etc.) from one site to another. (The utilities are not unlike the Echo utility from WinAppTechnology.) They rely upon the SharePoint object model to function because frankly they have to – not everything is exposed through the web services (Please make everything available via web services.) or the FrontPage RPCs (Please kill these off.) However, the problem with this is that it limits their ability to be used to a single configuration database. In other words, I can’t push configuration from development to staging. I’ve got a work around in creating a backup of a site in development, restoring it in staging, and using that as my template for changes.

This is difficult to do and requires manual intervention. The possibility for errors is fairly large. This approach doesn’t fly very well when you have a tightly controlled production environment – which most customers have. It ends up being a big issue.

So my current client and I found out that the template file (user site template – STP) is really a renamed cab file with a manifest.xml file in it. Manifest.Xml is CAML. So the thinking was that this would have what was necessary. To replicate the site. If the template can create a site surely it has all of the data about the site and should therefore be possible to use as a starting point to make updates.

Having this XML based serialization of the site means that we can pack up the file and use it as a template on another server. This means that we could have a very clean implementation for deploying from environment to environment. We started down the process of implementing this mechanism believing that we were well within guidelines. We weren’t touching the database – we were using a file that is in a supported format. Everything should be fine.

Well, that’s a good idea, but unfortunately, the word back from the development team is that they don’t want to support people using the manifest.xml file in an STP. At least they don’t want to support the areas of that file which are not documented CAML. They have a binary serialization of web parts in the file which they’re not planning on sharing the details to. The net effect is that it’s not possible to get web parts from an STP files Manifest.xml

That leaves me with very few options. Officially, I can’t use the data in a solution so I’d have to come up with my own mechanism for serializing this data – integrating that into the manifest.xml file and shipping that around. Of course, if I modify the schema to ship the web parts around I’m no longer moving CAML so there’s no telling if I’ll be able to use that data anyplace except in my application.

I’ve not yet figured out how to resolve the issue – however, I’m looking for input here. Am I the only one that thinks that if Microsoft says they’re going to support CAML that they should support it every time they write it out? Am I the only one who sees that this kind of “We’re supporting it” and then when we go to actually use it we get “That’s not supported” is making life really difficult?

Thoughts?

Creating a new set of list items from one list into another

Robert Bogue November 6, 2005 4 Comments

Here’s the other bit of code having to do with the pesky created/modified dates. This is just a function and not a whole program. It deletes all of the items in a destination list and then copies all of the items from the source list. The interesting bit is that the created/modified fields hold up to this mechanism. You can create a new list item that maintains the created/modified dates of the original item. As with the previous code — this is not intended for direct use. It’s designed to show you how you might copy the created/modified fields from one list to another.

/// <summary>
/// Copy list data from one list to another
/// </summary>
/// <param name=”spListRef”>The source of the list data</param>
/// <param name=”spListTarget”>The target (destination) of the list data</param>
public static void CopyListData(SPList spListRef, SPList spListTarget)
{
if (spListRef.BaseType == SPBaseType.DocumentLibrary)
{
CopyDocumentData(spListRef, spListTarget);
return;
}
SPListItem spListItemTarget;

// Delete all Items
while (spListTarget.Items.Count > 0)
{
spListTarget.Items.Delete(0);
}

foreach(SPListItem spListItemRef in spListRef.Items)
{
spListItemTarget = spListTarget.Items.Add();
foreach(SPField spFieldRef in spListItemRef.Fields)
{
if (spFieldRef.Type != SPFieldType.Attachments &&
spFieldRef.Type != SPFieldType.Calculated &&
spFieldRef.Type != SPFieldType.Computed &&
spFieldRef.Type != SPFieldType.Counter &&
spFieldRef.Type != SPFieldType.CrossProjectLink &&
spFieldRef.Type != SPFieldType.Error &&
spFieldRef.Type != SPFieldType.File &&
spFieldRef.Type != SPFieldType.Invalid &&
spFieldRef.InternalName != “owshiddenversion” &&
spFieldRef.InternalName != “InstanceID” &&
spFieldRef.InternalName != “Order” &&
spFieldRef.InternalName != “GUID”
)
{
spListItemTarget[spFieldRef.InternalName] = spListItemRef[spFieldRef.InternalName];
}
}

foreach(string attachmentUrl in spListItemRef.Attachments)
{
string fullUrl = string.Format(“Lists/{0}/Attachments/{1}/{2}”, spListRef.Title, spListItemRef.ID,
attachmentUrl);
SPFile srcAttach = spListRef.ParentWeb.GetFile(fullUrl);
spListItemTarget.Attachments.Add(attachmentUrl, srcAttach.OpenBinary());
}
spListItemTarget.Update();
}
}

Creating a file on WSS with the same created/modified date as a file on the file system.

Robert Bogue November 6, 2005 No Comments

Apparently, I hit a sore spot for the community with my discussion about created and modified fields. The code that follows is a very rough hack at how you can get a file into WSS while setting the created and modified dates. I didn’t bother making the code lookup the users’ ID for the site from the name on the file — but that could be done with another half a dozen lines of code or so. So here’s a command line utility to upload a file to WSS with the same created/modified dates as the original file… Remember this is a hack… it’s definitely not designed to be used for anything more than realizing how it can be done…

using System;
using System.IO;
using Microsoft.SharePoint;

namespace TESTCREATED
{
/// <summary>
/// Summary description for Class1.
/// </summary>
class Class1
{
/// <summary>
/// The main entry point for the application.
/// </summary>
[STAThread]
static void Main(string[] args)
{
if (args.Length < 3)
{
Console.WriteLine(“TESTCREATED siteUrl ListName FilePath FileName”);
}
string paramSite = args[0];
string paramList = args[1];
string paramPath = args[2];
string paramName = args[3];

SPSite spSite = new SPSite(paramSite);
SPWeb spWeb = spSite.OpenWeb();

SPListCollection spLists = spWeb.Lists;
spLists.IncludeRootFolder = true;
SPList spList = spLists[paramList];

SPFolder rf = spList.RootFolder;

FileInfo fi = new FileInfo(paramPath);
FileStream fs = File.OpenRead(paramPath);
byte[] bytes = new byte[fs.Length];
fs.Read(bytes, 0, (int) fs.Length);
SPUser createdBy = spWeb.Users[0];
SPUser modifiedBy = spWeb.Users[0];
SPFile file = rf.Files.Add(paramName, bytes, createdBy, modifiedBy, fi.CreationTimeUtc, fi.LastWriteTimeUtc);

}
}
}

An update on those pesky fields

Robert Bogue November 6, 2005 No Comments

Through some email conversations I’ve got a bit more detail on those pesky created and modified fields. You can set them when you’re adding a new record — and the values hold. However, it doesn’t appear that you can set them on an update… I love wandering around the depths of uncharted (undocumented) SharePoint.

SharePoint: Created By, Created (Date), Modified By, and Modified (Date)

Robert Bogue November 6, 2005 2 Comments

One of the frustrating things about SharePoint 2003 is that the developer documentation is still a bit weak. The way that we see the impact of this is that well meaning folks end up accidentally saying that something is or isn’t supported. I’m quite guilty of it myself — although thankfully I’ve not written down any of these errors that I know of yet.

Bil Simser was noodling on the idea of updating the created and modified fields in a list and mentioned that they’re read only. (ListFormWebPart and Tzuanmi SharePoint Designer). The interesting thing is that I’ve updated these fields in a list. I did it for some technology POC code I was generating around copying list data from one list to another. I didn’t even pay attention to the read only field and I got the results I wanted. I copied the Created By, Created (date), Modified By, and Modified (date) fields.

This wouldn’t be so interesting except Adam Macaulay from CorasWorks and I had a similar discussion on the fields not two weeks ago. Their Data Migrator tool (a great tool by the way) didn’t preserve Created/Modified dates. So, I forwarded him along my POC code so hopefully he’ll be able to figure out why they initially had problems with getting the Created/Modified dates to copy from one list to another.

I wonder how many other things about SharePoint that we collectively have wrong.

SPSiteDefUpd – SharePoint Site Definition Update

Robert Bogue November 5, 2005 1 Comment

I’ve mentioned through the last few posts that I’m working on another custom site definition and that I’m wandering through SCHEMA.XML definitions. Well, one of the things that I decided to do since making changes to Schema definition file is so monotonous is to automate the addition or removal of web parts in all of the appropriate schema.xml files within a site definition.

So I have yet another tool… SPSiteDefUpd — it updates site definitions to add, remove, or enumerate web parts in the various schema.xml files for the list definitions under a site definition. It has a few limitations …

Right now it only processes the List defintions, it won’t automatically update your ONET.XML file to — that would be handy if you’re adding something like a global header or footer. (but modifing the ONET is substantially quicker than hitting all of the schema.xml files.
It does removal by title so if you have two web parts with the same name .. it will remove one of them but which one it will remove is a mystery. (Actually, it’s not, it will remove the first one listed in the Schema.xml file.)
It won’t touch the ListForm or ListView web parts — this is in no small part due to the fact that they don’t actually show up in the web parts section.

OK, now that you know what it won’t do, here’s what it does do:

Processes every file for the list definition. This includes both views and forms.
Processes every list definition — you tell it where the site definition is and it will, if you want, go through and do replacements for every list definition — or for only one if you prefer.
Adds new web parts — it reads the web part zone and web part order to put in the AllUsersWebPart attributes.
Adds the WebParts node to the XML if it doesn’t exist.
Enumerates all web parts on every page in every list definition or any subset that you prefer.
Removes web parts from the site definition by title.
Consolidates write operations if multiple sections of the file are changed to improve performance.

I think it will be very handy as I work on new site definitions and want to do a quick add of my global header, footer, and navigation. It will be trivial to get these pushed through all of the pages now. It will also be good since I will be able to allow clients to create their own new site definitions based on a site definition that I leave with them.

If anyone else is interested in trying the tool out let me know via feedback or comment, I’ll be looking for a few beta testers by the end of the week. (That will give me time to write the documentation and do a bit more testing.)

SCHEMA.XML — Continuation Character?

Robert Bogue November 4, 2005 No Comments

At the client I’m at we’re doing some customization of the SCHEMA.XML files for the list templates. Basically, we’re adding some web parts into the pages since our header, footer, and navigation is driven from web parts. I was over with one of the developers and noticed he was looking at the SCHEMA.XML as a text file. When I asked why he showed me, VS.2003 wasn’t able to show a XML view.

So I fired up my copy of Stylus Studio (which tells me why something isn’t valid XML in addition to just blowing up.)… I found that there were a few lines in the schema.xml which look like they were “continued.” They had an exclamation point and a carriage return at the 990th position on a line. I’ve checked through the other SCHEMA.XML files and can’t find the character … so I don’t know how it got there, but if you run into a schema.xml that won’t load, perhaps some utility has decided to add a line continuation character…

SharePoint Web Part Administrator (SPWPADM)

Robert Bogue November 2, 2005 2 Comments

STSADM in addition to being a swiss army knife of useful little functions is also one of the most frequently used tools in SharePoint deployments. Being the gatekeeper to easy web part deployment has it’s advantages. However, there are limitations that it has and features that it hasn’t implemented. That’s where one of my latest projects comes in.

First, STSADM is very dependent upon access to the SharePoint content database. If you don’t have enough access the tool doesn’t work. This can be worked around, however, in many environments it represents a barrier to easily using STSADM.

Second, although STSADM allows you to add and remove web parts as trusted assemblies on the server — and deploy default property sets (read as “DWP files“) it does not allow you to add or remove web parts from a web part page from the command line.

However, I’ve been working on a tool the “SharePoint Web Part Administrator” (SPWPADM) that solves both of these two limitations. First, it is entirely based on the web service interface to SharePoint. You can run it from your desk, from the server, or anywhere that you can access the SharePoint built-in web services. Second, it’s exclusive function is to manage web parts on web part pages.

It allows you to enumerate the web parts on a page, add new web parts to a page, delete web parts from a page, export web parts from a page, and even to delete closed web parts on a page. Of all of those, perhaps only the last function needs any explanation…

When people “delete” a web part from SharePoint they’re often times just closing it — suppressing the display. The code is actually still running it’s just that people don’t see the results on the page. That’s really not all that great if you’re doing extensive computations to come up with content that will never be displayed. However, the only way to delete a web part is to enter design mode to delete it. The result is most people close web parts without realizing the impact on their server.

The SPWPADM tool will hunt down these closed web parts and actually delete them for you so they’re no longer running.

So, there are two questions you might ask… 1) “Why did you write the tool?” — The primary reason is because I’m working on an article for DevX.com on how to replace the edit control (ListFormWebPart) for a list. To do that you have to add a new web part to the list pages and remove the existing web part. It’s possible to do but certainly kludgy. So I thought the tool would be a nice thing to offer folks as a trial with the article. It’s turned into slightly more than that as I realized other purposes for it, but at heart it’s designed to facilitate management of a web part page from the command line.

2) “So what is it useful for?” — Thus far I’ve used it for two things… First, deleting closed web parts. You would be amazed at how many of them there are in a typical SharePoint implementation. Second, helping to configure site definitions while building/testing them. When I build a site definition I typically have a page that is composed entirely of web part zones. I don’t have a header, I don’t have a footer, nor menus, nor other parts of the page, those functions are represented by web part zones and by web parts. (Yes, I know this is slightly wasteful from a configuration storage perspective — it does, however, make staying supported while making changes much easier.) So I typically make 20 copies of my default page so I can drop in other content as needed. That means adding half a dozen web parts to each page (header, footer, search, global navigation, site navigation, etc.) So I make a batch file that calls the add web part command for each web part. I nest this in a batch file in one that calls it once for each page. It takes a while but eventually the controls are all placed on every page.

So why am I bringing this up? Well, two reasons, I’m ready to let a few people try it out (send me an email and I’ll send you a copy of it along with a license.) Second, I got someone asking about how to lock web parts in place for 50 or so sales guys on a discussion list I watch. It occurred to me it’s possible to put this in a script that runs every night and deletes and then readds the web parts — the net effect of which is forcing them to be on a page since changes, or removal will be removed each day. Not pretty but an idea that would be easy and effective.

So if you’re interested, drop me a line and I’ll get you a beta/trial copy.

CorasWorks: Growing with the deployment

Robert Bogue October 31, 2005 No Comments

I typically don’t make a big deal out of the software I use on a daily basis because I figure that people have their own unique needs and we all get bludgeoned nearly to death by constant bombardment with new tools, techniques, and processes that are going to save the world.

However, I’ve been spending a fair amount of time with CorasWorks lately and there were a few observations I wanted to make about the product suite that I don’t think is being well portrayed in the market. To be clear, I’ve already shared the essence of this post with them in a candid conversation at CorasWorks University last week.

What I’ve found with SharePoint is that it’s really about getting together a set of very flexible and very fundamental tools and building creative things with them. So much has been made of the content editor web part – and what it does is so simple … it emits HTML code. How sophisticated do you have to be to write a web part that emits the HTML that is provided to it.

Some of my clients view CorasWorks in a similar way … how hard is it to roll up a few lists? Frankly, not that hard. However, that’s focusing on the tactical problem without taking a step back. Rolling up a few lists is trivially simple. Creating a framework for managing that rolled up data, providing a complete set of views, providing support, etc. is where it gets hard.

The navigational components that are provided with CorasWork’s suite are fundamental and perhaps even some what trivial – however, they are – for the most part – a good set of tools that do provide great flexibility and most of the fundamentals that you need.

In fact, the biggest complaint I have about the navigational components is that the current released version doesn’t support relative paths. This is eating me alive at my current client as we try desperately not to bind in any environment specific configuration into out site definitions and other files. Although it’s an up-hill challenge, I’m hoping that they will add relative path support to their Special Site Navigation component. (I’d love to hear your comments if you’re struggling with the same issue.)

Their other navigation tools, Workplace View Advanced, and the SPS cousin are more sophisticated but are limited to the two dimensional movement through a site or area hierarchy. It’s useful for site-to-site navigation but doesn’t help when you need to link applications within a portal together.

I mentioned that they are fundamental building blocks. What I have been most impressed about as I’m installing CorasWorks into my third large scale implementation is it’s ability to grow with the environment. Through the use of some undocumented properties (that I’ll likely blog about soon) it’s possible to gain very fine-level control of how things appear. This means that tools that may originally fill a specific niche around roll ups may grow up into a set of tools that continue to drive the core of the portal for years to come.

So if you’re researching CorasWorks and wonder if it will get a few users or many users – think along the side of many users. When you learn all of the advanced tricks it’s really amazing what you can do with the tool – including very customized menus and spectacular roll ups. I’ll start posting on how to do a few of these things in the coming weeks. For now, just know that there are plenty of undocumented or under-documented features just waiting to be unleashed. Know that these features aren’t being discussed much in the market — they’re seen as an entry level tool which is useful to get started, however, the amount of “runway” that you have with CorasWorks is pretty substantial.

A few quick how-to documents

Robert Bogue October 31, 2005 No Comments

I needed to do some simple how-to documents for some developers I’m working with so I thought I would share with everyone these step-by-step guides for doing some SharePoint (and .NET) fundamentals.

I hope they’re helpful.

Public Speaking

Recent Posts

Public Speaking