The most important patent I helped develop was Dynamically mediating multimedia content and devices . It is fundamental for realizing artificial intelligence (AI) and natural user interfaces (NUI).  The base concept is facilitating experiences based on software discovering and making connections with all available resources including content, devices, sensors, services and other technologies, then surfacing them through an interface that provides the best potential experience given those resources. It was in development for years, but became realized in the Microsoft Home in 2004.

In the pre-disclosure we gave examples of how this might manifest. In retrospect, we should have used the term facilitation instead of mediation…

“We have 3 examples reduced to practice in prototypes within the Microsoft Home that use the mediation.
The first and simplest example is with playing media. This addresses mediating the best experience given the content and devices available.
On the TV using a remote control, we choose a piece of media from our media guide. While the TV has speakers, there are also separate speakers available nearby (geographically) on the network. The mediator notes that the quality of the content warrants high fidelity and so chooses the best speakers for the experience, those on the network and not part of the TV.

The second example is in our dining room with our homework scenario.

This addresses not only a mediator for the best experiences but also builds relationships with the variety of content being displayed on the variety of devices and offers new experiences based on correlations it makes between the content and the devices.
On the tablet we see content that is appropriate for a variety of devices within the room. The mediator may be responsible for identifying these or it could be a location based service that shows devices within proximity (we’ve already got a patent in motion for proximity based devices File #: 304166.01).
We push different pieces of content to the devices, in this case content associated with astronomy. A 3D solar system model is pushed to the plasma TV, and a night sky goes to the intelligent lights (or projectors).
In a 2 way communication, the mediator/relationship builder notes the different kinds of content and devices involved and builds relationships.
The first thing that happens is that the nearby room lights dim to accommodate the best scenario/experience for the night sky.  The mediator knows that the content is interactive and that the content on the devices share some common elements. In this case it’s 3 planets, Jupiter, Mars and Saturn. The mediator also knows that the tablet is perfect as a screen based remote control as sends it a message to display links to the 3 planets, building a UI on the fly based on the information gathered. As we choose a planet, both the night sky and solar system model respond to highlight these planets based on the content and devices capabilities. As a result, our remote control is now controlling 2 devices.”

We definitely earned our patent award for this one.