The GAJ & Zeitgeist teams are proud to announce the 0.5.0 development release of the GNOME Activity Journal, codenamed “Defying Gravity”.

Special kudos for major contributions to Hylke, Randy, Seif, and Siegfried!

What is GNOME Activity Journal?

GNOME Activity Journal is an Activity Browser, which lets you quickly find what you did – with files, emails, and on the web. As opposed to File Browsers, it mainly focuses on when you did something rather than where the items you worked with are located. GAJ is powered by the awesome Zeitgeist Framework.

Screenshots

The Columns View of the GNOME Activity Journal with a Preview Tooltip (top left), the Search Tool (top right), a Used-with collection (bottom left), and the Preferences dialog showing Plugins (bottom right):

The Intervals & Thumbs Views of GNOME Activity Journal:

Where?

• Download: http://edge.launchpad.net/…+download/…0.5.0.tar.gz
• About Zeitgeist: http://zeitgeist-project.com
• Wiki: http://live.gnome.org/GnomeActivityJournal

This Release

Since the last release, we included some new features (but postponed others). Alongside multiple bug fixes, major changes and features include:

• Improved start-up time
• Better support for Tomboy and websites
• An experimental toolbar (screenshots below) providing
  • Access to “MultiView”, “TimelineView”, and “ThumbView”
  • Improved search tool (please let us know, what you think of it)
  • Preferences (experimental Blacklists extension)
• Makeover of the visual appearance

Next Releases

What to expect with the next releases:

• A highlighting of categories containing search results
• Category tooltip, so that users can peek into collapsed categories
• More tooltip previews
• Improved widget caching
• Search on all views
• Improved search experience
• More keyboard shortcuts
• Accessibility

Improving Re-Finding upon Resumption of Computer-Work

The paper presents an approach to reducing user efforts when reorienting to interrupted work. This features a time-centric Journal view of a user’s information-work activities, which allows going back to ‘task states’ for activity resumption. We further compute an activity-induced correlation function to reflect information coherence, as emerging and developing on the user’s side when ‘using information items together’. We thereby investigate the expressiveness of easy to understand correlation indicators, such as temporal proximity, window switching, and clipboard use. Finally, we present a list of most-used items, which is considered to provide suitable ‘entry points’ for work resumption.

ReflAction Journal

One approach for determining the information items needed to continue an interrupted activity, is re-finding those items, which have been the object/s of a user’s activities by the time when the interruption occurred (activity-phase specific approach). The ReflAction Journal supports this approach by presenting so-called activity objects to the user.

The ReflAction-Journal visualization is based on computer-observable fractions of information work. In the figure below, the doing pane shows item-related user activities (uA), which mirror that a user’s activities leave traces on those items. Accordingly, an activity object is composed of an information item (defining its name), and the traces related to the item, where the latter are sets of {focus, edit/view, visibility} intervals, representing, if so, interrupted item-related user activities.

The re-finding means enabled by the ReflAction Journal improve today’s common user experience that computers do not well support the “leaving the task (state) as it is” strategy, as they are not hampered by closed windows.

ReflAction Correlated & MostUsed

Imagine that a few days have passed since a user had extensively used some important information sources of a presentation (or any other) document. Even though she meanwhile proceeded with editing the presentation, she still perceives those sources as related to her presentation document a. Therefore, we provide a function called uA-related, which is used to determine correlated items within a time frame (across activity phases). Its user interface (upper part of the below figure) is an early prototype and doesn’t allow convenient control over time scoping yet.

The intuition behind the uA-related function is that information items ‘used together’ implicitly gain activity-induced coherence. Let uA-related(a, b) denote the function mapping the tuple (a, b) to the correlation strength of b to a. The construction of this function is illustrated in the paper. It is called for all b’s within the provided time frame. So, uA-related(Vortrag) for ‘today’ yields the list of items presented in the figure above.

The most-used list shown in the lower part of the figure above provides the main working documents to a user, which are considered suitable ‘entry points’ to resuming one’s work. Each entry of the most-used list is computed by evaluating the aggregated lengths of activity intervals per item within a time frame, where, in general, editing has more weight than viewing.

Open-Source Sister Projects

Together with Federico Mena’s  journal prototype and mayanna, the ReflAction Journal and the ContextDrive system have been influential forerunners and, by now, are  sister projects of the (GNOME) Activity Journal (GAJ) and Zeitgeist. As of today, the latter is one of less than 30  featured projects on launchpad (of approx. 19000 hosted projects) and was recently announced to be a key technology embraced by Unity.

The function behind the ReflAction correlatedItems-list is similar in aim to the experimental ‘findRelatedUris’ Zeitgeist-function by Seif Lotfy. Analogous for the function behind the ReflAction mostUsed-items list and the Zeitgeist-API function ‘findEvents((begin,end),…, mostPopularSubjects,…)’. For more on this, see the Zeitgeist DBus-API documentation and my earlier posts on ‘implicit relating’ (1,2) and ‘personal semantic technology‘.

The Zeitgeist and Elementary teams are currently hacking on Sezen. It is the first user interface built on top of  Zeitgeist, which has the new Zeitgeist ‘full-text search’ extension, provided by Mikkel Kamstrup and  Canonical. Big thanks for providing this awesome extension!!  … GAJ will get it too …  So stay tuned!

FYI, here is what we presented at GCDS/GUADEC 2009 (slides) and at a CHI 2010 workshop (slides, paper).

Topics of this year’s presentation will include the current developments of Zeitgeist and of the GNOME Activity Journal, how to work with and extend them, integration with the GNOME desktop, and cross-desktop development issues. If you want to know more, check out http://zeitgeist-project.com/.

We look forward to meeting everyone at GUADEC 2010.

Seif: “Hey computer, I want to see what I was doing when Ryan Paul wrote his tweet about how to turn on Gwibber’s multicolumn view!”

Computer: “Well, tell me what files you were using then or where they are located!”

Seif: “Mmh, I was programming and listening to music.”

Computer: “That can be a lot of files, man. Tell me more!”

Seif: “OK, I just remembered that python file I was working on and I now also remember one artist and one song title of the music I was listening to at that time.”

Computer: “Great! Now tell this information to me, one at a time, and I’ll provide you the files.”

Seif: “Urgs!”

Ryan Paul and Seif Lotfy having Fun with Gwibber and Zeitgeist

The below video demonstrates that Seif’s computer can do better, if it has Zeitgeist running.

Zeitgeist in Gwibber from Seif Lotfy on Vimeo.

So, the above dialogue becomes considerably shorter to look something like this:

Seif: “Hey computer, I want to see … !”

Computer: “Here you are!”

Seif: “Thanks!”

Implicit Relating via the WHILE-Operator in a User’s Query

Today, a user who wants to get back to what s/he was doing while perceiving or doing something else is given a hard time. Why? Well, because this user is not supplied with a direct link to the information X, which was used while Y happened, upon provision of that Y.

Here is how Zeitgeist solves the riddle: In the above video, Y is a tweet, which can be interpreted as an event because it has a timestamp. Zeitgeist takes the tweet, looks up the contemporaries, and provides them. Thereby, Zeitgeist enables the user to re-find X within a click from Y.

In the shown prototype implementation, Zeitgeist searches for the user’s activities within a five minute radius around the tweet’s timestamp and delivers those files, which have been involved in the activities. So Zeitgeist’s answer can be read like this: “While Ryan sent out his tweet, you were working on actions.py and listening to these two songs.”

Stay tuned!

1. Working Sets – A Case of Implicit Relating

Albrecht Schmidt presented our work [4] at the Personal Informatics Workshop at CHI 2010 and blogged about it. It was him, who coined the distinction of implicit and explicit human-computer interaction (HCI) [5], which is now commonly used in HCI: “In contrast to explicit interaction, implicit interactions are based not on explicit actions by the user (directed at a computer), but more commonly on users’ existing patterns of behavior.” [6, p.191], referring to [5]. Differentiating between implicit and explicit HCI proves productive as an analogy for investigating different forms of relating a user’s information items, as enabled by experience-trace infrastructures, such as Zeitgeist and ContextDrive.

Implicit and Explicit HCI

“Implicit human-computer interaction is an action, performed by the user that is not primarily aimed to interact with a computerized system but which such a system understands as input.” [5]. This is opposed to explicit HCI, where “the user tells the computer at a certain level of abstraction (e.g., by command-line, direct manipulation using a GUI, gesture, or speech input) what she expects the computer to do.” [5]. An example of explicit HCI is using a direct-manipulation graphical user interface (GUI) to drag a file from one folder to another, in order to have the file become located in the target folder. Implicit HCI, on the other hand, requires computers to observe and interpret what the user is doing, via sensors, which may be implemented as physical sensors (e.g., GPS) or digital sensors (e.g., file usage monitor). Thereupon, a computer can offer information such as “you presented conceptReport-v1.1, while being at this customer’s site last time on april 12”. Consequently, monitoring user behavior is a basic prerequisite for implementing implicit HCI.

Implicit Creation of Working Sets

I’ll introduce implicit relating as a specialization of implicit HCI, which means it inherits the goals of implicit HCI. The purpose of implicit HCI is to reduce mental load on the user’s side by reducing attention-intensive efforts directed at explicitly handling a computer’s UI, in order to get something done. Similarly, implicit relating of a user’s information items aims at reducing explicit user actions towards grouping her information items, for example, in order to represent “the stuff she is currently working on” as working sets. Working sets, as an example of information collections of implicitly related items may be related to, but are still different from to be archived information collections [1]. This can be illustrated by comparing (a) paper piles on your desk with (b) folders in your rack: Grouping criteria and purpose of (a) and (b) are simply different.

Working Sets

In my previous post (final section, 4.), I referred to re-finding working sets as one of the new user experiences enabled by experience-trace infrastructures. The notion of a working set is borrowed from Federico’s GNOME 2008 UX hackfest post and his October 2009 post on this topic. The motivation behind this notion is that working on rather complex topics usually involves multiple information items, such as code, emails, design documents, videos, and web resources. Plus, completing complex tasks is seldom achieved within a single uninterrupted time period of activity dedicated to that task. Hence, the need to again and again resume work, which often implies re-using at least some of the involved information items. Beyond, working sets commonly evolve over time, be it shrinking or expanding. Nowadays, it is left to the user to explicitly create and maintain folders to group working-set items, for the purpose of efficiently re-finding them, when resuming an activity.

Scenario

Consider the following common situation: When continuing work with others, one often needs to refer back to items used earlier in the common activity thread. This could be the UI mockup discussed two days ago, along with the IRC discussion related to it, and the existing code you looked at, which would need to be changed, in order to implement the discussed mockup.

As already said above, today, in order to conveniently go back to these items for task resumption, you would have needed to group them together before. So what? Well, first, this grouping is all too often “forgotten”, for various reasons. One of them is that it is perceived secondary to one’s actual task work, i.e. on top of “getting things done”. Let alone the fact that today’s means turn linking together an image, a chat discussion, and code pointers into a rather clumsy process. Additionally, anticipating what one might need in the future is hard in principle.

If not having grouped the needed working-set items beforehand, one is forced to hunt each of them down again, by means of explicit HCI, i.e. to re-find them by browsing and searching. Even if re-finding each relevant item might not be a big problem on its own, re-finding all of them consumes time and potentially interrupts your work flow. Plus, recalling all of the relevant items, if you achieve to do so, implies considerable effort, i.e. memory load.

Presentation of Working Sets

Instead, in the above scenario, your computer could unobtrusively present the relevant working-set items as soon as you are chatting with your collaborator again. Zeitgeist enables this by interpreting the collaborator match, and possibly more matches, between the before and now situations, as an indicator that you might want to continue working on one of the topics you had recently treated together. The presentation of an implicitly created working set, i.e. a set of implicitly related items, necessitates monitoring user behavior, just as is required to implement implicit HCI. In the short scenario above, this is monitoring of using images, of chat conversations, and of programming.

All of this serves the general goal of having information bits and pieces “at hand” when needed. That is the vision behind the early-prototype video by Seif Lotfy below, which implements the saying “stuff you use together belongs together”, which we coined at GUADEC 2009. The corresponding calculations in the Zeitgeist engine are for now performed ad-hoc in nearly real-time. One of the remaining challenges is to suitably draw borders among different working sets, accounting for their dynamic evolvement over time, and distinguishing working sets from what the user does “on the side”.

Implicitly relating your files and contacts with Zeitgeist from Seif Lotfy on Vimeo.

The more we achieve to solve these issues, the better the employment of such implicitly created working sets will work towards faster, less effortful, and so more convenient re-finding and access of previously used information items relevant to the user’s respective activity. Referring to paper-based work at your physical desk again, the above described implicit information organization can be understood as automatically putting paper documents into piles, informed by how each of them is used, by their different arrangements on your desk over time, and by the, possibly multiple, occurrences of the documents being involved in your workflow.

Design Issues

Automatically presenting a working set upon recognizing similarities between earlier situations and the current one, which, by the way, is a case of implicit output, releases the user from the need to explicitly browse or search for the items without requiring him to explicitly create and maintain working sets. Thereby, in the ideal case, the user doesn’t need to recall the relevant information items, but instead just needs to recognize them and choose accordingly from the presented working set. This needs to be complemented by means of querying for working sets, for example by defining a time scope.

Even though Zeitgeist provides support for the above discussed means for implicit creation of working sets, these dynamic collections might, in some cases, still become too large to be dealt with easily by the user. Therefore, our visualization of a working set will reflect a gradual and dynamic relevance notion, i.e. show how much a certain item belongs to a working set and/or to the currently recognized situation. In addition, we are experimenting with the concept of a life time of items with respect to inclusion into a working set, in combination with the relevance concept. This could be visually mirrored as having items fade out over time, if not being used, and fade in again, if being re-used. If users want more control, pinning items onto the working-set canvas will prevent fading behavior.

Beyond the implicit means of a gradual relevance notion and an item life-time, there are also explicit means of letting a user manipulate a working set and its visualization, in order to keep them treatable. These are blacklisting, filtering, and maybe defining size limits. Furthermore, we will investigate letting users partition a working set by arranging its items on a canvas or by employing tags or folders. The two latter naming means would also allow several working sets to be presented at once, reflecting, for example, the case of the same group of people working on different tasks “in parallel”.

Users often experience a loss of control when confronted with dynamic structures, such as dynamic menus. This needs to be avoided with respect to the rather dynamic concept of working sets. Users should not fear “losing” working-set elements or not being able to later re-find and access items they had earlier seen in working sets. This goal could be achieved by automatically turning working sets into persistent collections, for example, daily. These periodic backups would be “lossless” with respect to once included items. Working sets can even be re-computed at any time, as long as no events are deleted from the Zeitgeist log. By default, deletion of essential events therefore only happens upon user request. Consequently, users will not be required to explicitly keep working sets.

Additionally, we will try designing the computation and visualization of working sets such that users will not be forced to organize or re-organize these sets at specific occasions or times. Instead, users should feel and be in control over when to further classify information from working sets. This is because of the known negative effects of imposing interrupts on a user’s work flow, which are perceived as unnecessary overhead on the user’s part. Another rationale for the above design goal is the often reproduced finding of [2], that untitled “piles” are helpful means to represent working-set like information.

As opposed to the world of our physical desktops, the computer world enables us to have the same information item in multiple “piles”, so working sets, at the same time. Beyond, Zeitgeist can also keep track of the evolvement of working sets and the information they “contain”, in order to enable a user to refer back to different versions of them over time. A related challenge of making working sets usable is to design their behavior and user interface in a way, which lets the user distinguish among copy and reference semantics for the “contained” information items.

References

[1] William Jones, Jaime Teevan (2007). Personal Information Management. University of Washington Press, Seattle and London.

[2] Thomas W. Malone (1983). How do people organize their desks? – Implications for the Design of Office Information Systems. ACM TOIS, Vol. 1, No. 1 (January 1983), pp. 99-112.

[3] Thorsten Prante, Richard Stenzel, Kostanija Petrovic, Victor Bayon (2004). Exploiting Context Histories: A Cross-Tool and Cross-Device Approach to Reduce Compartmentalization when Going Back. Proceedings of Informatik 2004, Ulm, September 20-24, pp. 314-318.

[4] Thorsten Prante, Jens Sauer, Seif Lotfy, Albrecht Schmidt (2010). Personal Experience Trace: Orienting Oneself in One’s Activities and Experiences. CHI 2010 Workshop on Personal Informatics, April 10, Atlanta, Georgia, USA.

[5] Albrecht Schmidt (2000). Implicit Human Computer Interaction through Context. Personal and Ubiquitous Computing, Vol. 4, No. 2&3 (June 2000), pp. 191-199.

[6] Andrew D. Wilson (2008). Sensor- and Recognition-Based Input for Interaction. Chapter 10 of: Andrew Sears, Julie A. Jacko (2008). The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications (2nd Ed.), CRC Press, New York, USA.

Here is the abstract: In this paper, we present steps towards adequate support for answering the central questions of orienting oneself within one’s past, current, and anticipated future activities and experiences, along with application areas, which will benefit from such support. The experience-trace concept is introduced and it is shown how support for mental time-travelling within and via elements of one’s activities and experiences can be implemented as re-finding elements within one’s experience trace. To this end, two interactive journal user interfaces are proposed: ReflAction Journal and (GNOME) Activity Journal. They are enabled by the ContextDrive and Zeitgeist frameworks and their respective experience-trace implementations.

Personal Semantic Technology

Trends in technology development indicate that personal computers could develop into always-on companions, i.e. possibly wearable, personal helpmates. Anticipating this sketch of the future, we aim at designing what we call personal semantic technology. This technology will be capable of reflecting personal meaning of computer-represented objects, as emerging and developing from them being involved in a user’s activities and experiences. We investigate and test our approach with today’s mobile computers, which can be seen as forerunners of the above envisioned machinery. Digital technology is accompanying early adopters in an ever increasing amount of situations. On the one hand, this personal technology is tool and medium for a multitude of its users’ activities and experiences. On the other hand, in the future, it might serve as a personal witness of its users’ activities and experiences, possessing precise and reliable memory.

Personal Experience Trace

Personal experience traces are one of the core concepts enabling personal semantic technology: A user’s experience trace represents a consolidation of “computer-experienced” events related to this user’s activities and experiences. The experience-trace notion reflects that a user’s behavior leaves traces within his/her personal computing machinery, which can be exploited towards this user’s benefit. This is facilitated via logging events and marking them with a variety of labels, some computed and others user-indicated.
The purpose of a personal experience trace is to mirror what a person actually does and experiences by capturing a continuous, comprehensive, and coherent picture of it. This picture encompasses traces referring to objects of retrospective, current, and prospective character. It is continuously evolving as the user acts and experiences. This said, we see and accept the principal gap between what humans do/experience and what computers can sense and represent of it. We respectfully understand this gap as a design challenge and consider it mandatory that a user is and feels in absolute control of this representation.

Supporting Mental Time-Travel

Mental time-travelling is what people do when they use their “mental eye” to situate themselves in what happened in the past or in what they imagine happening in their personal future. Personal calendars are one example of a tool facilitating backward and forward mental time-travel. Additionally, they serve to schedule future “events”. Please note, that the “events” represented by calendar items do not denote the same thing as the events capturing a user’s doing and experiencing into the personal experience trace.
The experience-trace implementations ContextDrive and Zeitgeist can not only capture when a calendar item has been created or modified, but, more importantly, what time span it refers to. Thereby, calendar items are turned into labels and container items for what actually happens during the time span in question. Plus, calendar items can be appropriated, for example, to serve users in relating their preparatory and follow-up activities to the represented “events”. The ReflAction Journal facilitates this via drag’n’drop. Further, the analysis engines of both ContextDrive and Zeitgeist are currently expanded, in order to automatically achieve this relating of events to labels and other higher-level entities. This is brought forward by investigating experience-trace induced notions of relatedness, according to the above introduced personal-semantic-technology approach.
To conclude, experience-trace implementations add an application-independent time-bound layer to the experience of using computers. Users are thereby enabled to relate to and situate themselves within representations of their past, current, and future activities and experiences.

New User Experience: Re-Finding Documents, Re-Situating, and Beyond

Employing the above introduced concepts and means opens up new ways of going back to what one did and experienced.

1. In order to re-find a document D, users can employ their memories of situations in which they created, viewed, modified, or otherwise used this document.
2. Vice versa, users can re-find all situations in which they interacted with a certain document.
3. Users can re-find other items interacted with while, before, or after working with D.
4. Employing “user-language labels”, such as calendar items or task representations, users can re-find “collections of use”. Such a collection of use could, for example, be the working set of documents last used for working on a task, before having switched to another activity.
5. Experience-trace implementations support users in bringing to their minds what they did during the last month, which helps them with reporting tasks.

In referring to the ReflAction Journal as a visualization of their personal experience trace, users see their item-related interaction intervals composed from events. They also have available labels denoting, for example, their scheduled “events” and tasks. Furthermore, properties of these labels and of the item-related events can be employed to search for items and events as well as to filter and scope the result set displayed, according to the users’ needs.