On behalf of the Zeitgeist team, I am pleased to announce that we will present the latest cooking from the Zeitgeist kitchen at GUADEC 2010.
FYI, here is what we presented at GCDS/GUADEC 2009 (slides) and at a CHI 2010 workshop (slides, paper).
Topics of this year’s presentation will include the current developments of Zeitgeist and of the GNOME Activity Journal, how to work with and extend them, integration with the GNOME desktop, and cross-desktop development issues. If you want to know more, check out http://zeitgeist-project.com/.
We look forward to meeting everyone at GUADEC 2010.
Gwibber and Zeitgeist – Another Case of Implicit Relating
Seif: “Hey computer, I want to see what I was doing when Ryan Paul wrote his tweet about how to turn on Gwibber’s multicolumn view!”
Computer: “Well, tell me what files you were using then or where they are located!”
Seif: “Mmh, I was programming and listening to music.”
Computer: “That can be a lot of files, man. Tell me more!”
Seif: “OK, I just remembered that python file I was working on and I now also remember one artist and one song title of the music I was listening to at that time.”
Computer: “Great! Now tell this information to me, one at a time, and I’ll provide you the files.”
Seif: “Urgs!”
Ryan Paul and Seif Lotfy having Fun with Gwibber and Zeitgeist
The below video demonstrates that Seif’s computer can do better, if it has Zeitgeist running.
So, the above dialogue becomes considerably shorter to look something like this:
Seif: “Hey computer, I want to see what I was doing when Ryan Paul wrote his tweet about how to turn on Gwibber’s multicolumn view!”
Computer: “Here you are!”
Seif: “Thanks!”
Implicit Relating via the WHILE-Operator in a User’s Query
Today, a user who wants to get back to what s/he was doing while perceiving or doing something else is given a hard time. Why? Well, because this user is not supplied with a direct link to the information X, which was used while Y happened, upon provision of that Y.
Here is how Zeitgeist solves the riddle: In the above video, Y is a tweet, which can be interpreted as an event because it has a timestamp. Zeitgeist takes the tweet, looks up the contemporaries, and provides them. Thereby, Zeitgeist enables the user to re-find X within a click from Y.
In the shown prototype implementation, Zeitgeist searches for the user’s activities within a five minute radius around the tweet’s timestamp and delivers those files, which have been involved in the activities. So Zeitgeist’s answer can be read like this: “While Ryan sent out his tweet, you were working on actions.py and listening to music.”
Stay tuned!
Seif: “Hey computer, I want to see what I was doing when Ryan Paul wrote his tweet about how to turn on Gwibber’s multicolumn view!”
Computer: “Well, tell me what files you were using then or where they are located!”
Seif: “Mmh, I was programming and listening to music.”
Computer: “That can be a lot of files, man. Tell me more!”
Seif: “OK, I just remembered that python file I was working on and I now also remember one artist and one song title of the music I was listening to at that time.”
Computer: “Great! Now tell this information to me, one at a time, and I’ll provide you the files.”
Seif: “Urgs!”
Ryan Paul and Seif Lotfy having Fun with Gwibber and Zeitgeist
The below video demonstrates that Seif’s computer can do better, if it has Zeitgeist running.
So, the above dialogue becomes considerably shorter to look something like this:
Seif: “Hey computer, I want to see … !”
Computer: “Here you are!”
Seif: “Thanks!”
Implicit Relating via the WHILE-Operator in a User’s Query
Today, a user who wants to get back to what s/he was doing while perceiving or doing something else is given a hard time. Why? Well, because this user is not supplied with a direct link to the information X, which was used while Y happened, upon provision of that Y.
Here is how Zeitgeist solves the riddle: In the above video, Y is a tweet, which can be interpreted as an event because it has a timestamp. Zeitgeist takes the tweet, looks up the contemporaries, and provides them. Thereby, Zeitgeist enables the user to re-find X within a click from Y.
In the shown prototype implementation, Zeitgeist searches for the user’s activities within a five minute radius around the tweet’s timestamp and delivers those files, which have been involved in the activities. So Zeitgeist’s answer can be read like this: “While Ryan sent out his tweet, you were working on actions.py and listening to these two songs.”
Albrecht Schmidt presented our work [4] at the Personal Informatics Workshop at CHI 2010 and blogged about it. It was him, who coined the distinction of implicit and explicit human-computer interaction (HCI) [5], which is now commonly used in HCI: “In contrast to explicit interaction, implicit interactions are based not on explicit actions by the user (directed at a computer), but more commonly on users’ existing patterns of behavior.” [6, p.191], referring to [5]. Differentiating between implicit and explicit HCI proves productive as an analogy for investigating different forms of relating a user’s information items, as enabled by experience-trace infrastructures, such as Zeitgeist and ContextDrive.
Implicit and Explicit HCI
“Implicit human-computer interaction is an action, performed by the user that is not primarily aimed to interact with a computerized system but which such a system understands as input.” [5]. This is opposed to explicit HCI, where “the user tells the computer at a certain level of abstraction (e.g., by command-line, direct manipulation using a GUI, gesture, or speech input) what she expects the computer to do.” [5]. An example of explicit HCI is using a direct-manipulation graphical user interface (GUI) to drag a file from one folder to another, in order to have the file become located in the target folder. Implicit HCI, on the other hand, requires computers to observe and interpret what the user is doing, via sensors, which may be implemented as physical sensors (e.g., GPS) or digital sensors (e.g., file usage monitor). Thereupon, a computer can offer information such as “you presented conceptReport-v1.1, while being at this customer’s site last time on april 12”. Consequently, monitoring user behavior is a basic prerequisite for implementing implicit HCI.
Implicit Creation of Working Sets
I’ll introduce implicit relating as a specialization of implicit HCI, which means it inherits the goals of implicit HCI. The purpose of implicit HCI is to reduce mental load on the user’s side by reducing attention-intensive efforts directed at explicitly handling a computer’s UI, in order to get something done. Similarly, implicit relating of a user’s information items aims at reducing explicit user actions towards grouping her information items, for example, in order to represent “the stuff she is currently working on” as working sets. Working sets, as an example of information collections of implicitly related items may be related to, but are still different from to be archived information collections [1]. This can be illustrated by comparing (a) paper piles on your desk with (b) folders in your rack: Grouping criteria and purpose of (a) and (b) are simply different.
Working Sets
In my last post (final section, 4.), I referred to re-finding working sets as one of the new user experiences enabled by experience-trace infrastructures. The notion of a working set is borrowed from Federico’s GNOME 2008 UX hackfest post and his October 2009 post on this topic. The motivation behind this notion is that working on rather complex topics usually involves multiple information items, such as code, emails, design documents, videos, and web resources. Plus, completing complex tasks is seldom achieved within a single uninterrupted time period of activity dedicated to that task. Hence, the need to again and again resume work, which often implies re-using at least some of the involved information items. Beyond, working sets commonly evolve over time, be it shrinking or expanding. Nowadays, it is left to the user to explicitly create and maintain folders to group working-set items, for the purpose of efficiently re-finding them, when resuming an activity.
Scenario
Consider the following common situation: When continuing work with others, one often needs to refer back to items used earlier in the common activity thread. This could be the UI mockup discussed two days ago, along with the IRC discussion related to it, and the existing code you looked at, which would need to be changed, in order to implement the discussed mockup.
As already said above, today, in order to conveniently go back to these items for task resumption, you would have needed to group them together before. So what? Well, first, this grouping is all too often “forgotten”, for various reasons. One of them is that it is perceived secondary to one’s actual task work, i.e. on top of “getting things done”. Let alone the fact that today’s means turn linking together an image, a chat discussion, and code pointers into a rather clumsy process. Additionally, anticipating what one might need in the future is hard in principle.
If not having grouped the needed working-set items beforehand, one is forced to hunt each of them down again, by means of explicit HCI, i.e. to re-find them by browsing and searching. Even if re-finding each relevant item might not be a big problem on its own, re-finding all of them consumes time and potentially interrupts your work flow. Plus, recalling all of the relevant items, if you achieve to do so, implies considerable effort, i.e. memory load.
Presentation of Working Sets
Instead, in the above scenario, your computer could unobtrusively present the relevant working-set items as soon as you are chatting with your collaborator again. Zeitgeist enables this by interpreting the collaborator match, and possibly more matches, between the before and now situations, as an indicator that you might want to continue working on one of the topics you had recently treated together. The presentation of an implicitly created working set, i.e. a set of implicitly related items, necessitates monitoring user behavior, just as is required to implement implicit HCI. In the short scenario above, this is monitoring of using images, of chat conversations, and of programming.
All of this serves the general goal of having information bits and pieces “at hand” when needed. That is the vision behind the early-prototype video by Seif Lotfy below, which implements the saying “stuff you use together belongs together”, which we coined at GUADEC 2009. The corresponding calculations in the Zeitgeist engine are for now performed ad-hoc in nearly real-time. One of the remaining challenges is to suitably draw borders among different working sets, accounting for their dynamic evolvement over time, and distinguishing working sets from what the user does “on the side”.
The more we achieve to solve these issues, the better the employment of such implicitly created working sets will work towards faster, less effortful, and so more convenient re-finding and access of previously used information items relevant to the user’s respective activity. Referring to paper-based work at your physical desk again, the above described implicit information organization can be understood as automatically putting paper documents into piles, informed by how each of them is used, by their different arrangements on your desk over time, and by the, possibly multiple, occurrences of the documents being involved in your workflow.
Design Issues
Automatically presenting a working set upon recognizing similarities between earlier situations and the current one, which, by the way, is a case of implicit output, releases the user from the need to explicitly browse or search for the items without requiring him to explicitly create and maintain working sets. Thereby, in the ideal case, the user doesn’t need to recall the relevant information items, but instead just needs to recognize them and choose accordingly from the presented working set. This needs to be complemented by means of querying for working sets, for example by defining a time scope.
Even though Zeitgeist provides support for the above discussed means for implicit creation of working sets, these dynamic collections might, in some cases, still become too large to be dealt with easily by the user. Therefore, our visualization of a working set will reflect a gradual and dynamic relevance notion, i.e. show how much a certain item belongs to a working set and/or to the currently recognized situation. In addition, we are experimenting with the concept of a life time of items with respect to inclusion into a working set, in combination with the relevance concept. This could be visually mirrored as having items fade out over time, if not being used, and fade in again, if being re-used. If users want more control, pinning items onto the working-set canvas will prevent fading behavior.
Beyond the implicit means of a gradual relevance notion and an item life-time, there are also explicit means of letting a user manipulate a working set and its visualization, in order to keep them treatable. These are blacklisting, filtering, and maybe defining size limits. Furthermore, we will investigate letting users partition a working set by arranging its items on a canvas or by employing tags or folders. The two latter naming means would also allow several working sets to be presented at once, reflecting, for example, the case of the same group of people working on different tasks “in parallel”.
Users often experience a loss of control when confronted with dynamic structures, such as dynamic menus. This needs to be avoided with respect to the rather dynamic concept of working sets. Users should not fear “losing” working-set elements or not being able to later re-find and access items they had earlier seen in working sets. This goal could be achieved by automatically turning working sets into persistent collections, for example, daily. These periodic backups would be “lossless” with respect to once included items. Working sets can even be re-computed at any time, as long as no events are deleted from the Zeitgeist log. By default, deletion of essential events therefore only happens upon user request. Consequently, users will not be required to explicitly keep working sets.
Additionally, we will try designing the computation and visualization of working sets such that users will not be forced to organize or re-organize these sets at specific occasions or times. Instead, users should feel and be in control over when to further classify information from working sets. This is because of the known negative effects of imposing interrupts on a user’s work flow, which are perceived as unnecessary overhead on the user’s part. Another rationale for the above design goal is the often reproduced finding of [2], that untitled “piles” are helpful means to represent working-set like information.
As opposed to the world of our physical desktops, the computer world enables us to have the same information item in multiple “piles”, so working sets, at the same time. Beyond, Zeitgeist can also keep track of the evolvement of working sets and the information they “contain”, in order to enable a user to refer back to different versions of them over time. A related challenge of making working sets usable is to design their behavior and user interface in a way, which lets the user distinguish among copy and reference semantics for the “contained” information items.
References
[1] William Jones, Jaime Teevan (2007). Personal Information Management. University of Washington Press, Seattle and London.
[2] Thomas W. Malone (1983). How do people organize their desks? – Implications for the Design of Office Information Systems. ACM TOIS, Vol. 1, No. 1 (January 1983), pp. 99-112.
[3] Thorsten Prante, Richard Stenzel, Kostanija Petrovic, Victor Bayon (2004). Exploiting Context Histories: A Cross-Tool and Cross-Device Approach to Reduce Compartmentalization when Going Back. Proceedings of Informatik 2004, Ulm, September 20-24, pp. 314-318.
[4] Thorsten Prante, Jens Sauer, Seif Lotfy, Albrecht Schmidt (2010). Personal Experience Trace: Orienting Oneself in One’s Activities and Experiences. CHI 2010 Workshop on Personal Informatics, April 10, Atlanta, Georgia, USA.
[5] Albrecht Schmidt (2000). Implicit Human Computer Interaction through Context. Personal and Ubiquitous Computing, Vol. 4, No. 2&3 (June 2000), pp. 191-199.
[6] Andrew D. Wilson (2008). Sensor- and Recognition-Based Input for Interaction. Chapter 10 of: Andrew Sears, Julie A. Jacko (2008). The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications (2nd Ed.), CRC Press, New York, USA.
Albrecht Schmidt presented our work [4] at the Personal Informatics Workshop at CHI 2010 and blogged about it. It was him, who coined the distinction of implicit and explicit human-computer interaction (HCI) [5], which is now commonly used in HCI: “In contrast to explicit interaction, implicit interactions are based not on explicit actions by the user (directed at a computer), but more commonly on users’ existing patterns of behavior.” [6, p.191], referring to [5]. Differentiating between implicit and explicit HCI proves productive as an analogy for investigating different forms of relating a user’s information items, as enabled by experience-trace infrastructures, such as Zeitgeist and ContextDrive.
Implicit and Explicit HCI
“Implicit human-computer interaction is an action, performed by the user that is not primarily aimed to interact with a computerized system but which such a system understands as input.” [5]. This is opposed to explicit HCI, where “the user tells the computer at a certain level of abstraction (e.g., by command-line, direct manipulation using a GUI, gesture, or speech input) what she expects the computer to do.” [5]. An example of explicit HCI is using a direct-manipulation graphical user interface (GUI) to drag a file from one folder to another, in order to have the file become located in the target folder. Implicit HCI, on the other hand, requires computers to observe and interpret what the user is doing, via sensors, which may be implemented as physical sensors (e.g., GPS) or digital sensors (e.g., file usage monitor). Thereupon, a computer can offer information such as “you presented conceptReport-v1.1, while being at this customer’s site last time on april 12”. Consequently, monitoring user behavior is a basic prerequisite for implementing implicit HCI.
Implicit Creation of Working Sets
I’ll introduce implicit relating as a specialization of implicit HCI, which means it inherits the goals of implicit HCI. The purpose of implicit HCI is to reduce mental load on the user’s side by reducing attention-intensive efforts directed at explicitly handling a computer’s UI, in order to get something done. Similarly, implicit relating of a user’s information items aims at reducing explicit user actions towards grouping her information items, for example, in order to represent “the stuff she is currently working on” as working sets. Working sets, as an example of information collections of implicitly related items may be related to, but are still different from to be archived information collections [1]. This can be illustrated by comparing (a) paper piles on your desk with (b) folders in your rack: Grouping criteria and purpose of (a) and (b) are simply different.
Working Sets
In my previous post (final section, 4.), I referred to re-finding working sets as one of the new user experiences enabled by experience-trace infrastructures. The notion of a working set is borrowed from Federico’s GNOME 2008 UX hackfest post and his October 2009 post on this topic. The motivation behind this notion is that working on rather complex topics usually involves multiple information items, such as code, emails, design documents, videos, and web resources. Plus, completing complex tasks is seldom achieved within a single uninterrupted time period of activity dedicated to that task. Hence, the need to again and again resume work, which often implies re-using at least some of the involved information items. Beyond, working sets commonly evolve over time, be it shrinking or expanding. Nowadays, it is left to the user to explicitly create and maintain folders to group working-set items, for the purpose of efficiently re-finding them, when resuming an activity.
Scenario
Consider the following common situation: When continuing work with others, one often needs to refer back to items used earlier in the common activity thread. This could be the UI mockup discussed two days ago, along with the IRC discussion related to it, and the existing code you looked at, which would need to be changed, in order to implement the discussed mockup.
As already said above, today, in order to conveniently go back to these items for task resumption, you would have needed to group them together before. So what? Well, first, this grouping is all too often “forgotten”, for various reasons. One of them is that it is perceived secondary to one’s actual task work, i.e. on top of “getting things done”. Let alone the fact that today’s means turn linking together an image, a chat discussion, and code pointers into a rather clumsy process. Additionally, anticipating what one might need in the future is hard in principle.
If not having grouped the needed working-set items beforehand, one is forced to hunt each of them down again, by means of explicit HCI, i.e. to re-find them by browsing and searching. Even if re-finding each relevant item might not be a big problem on its own, re-finding all of them consumes time and potentially interrupts your work flow. Plus, recalling all of the relevant items, if you achieve to do so, implies considerable effort, i.e. memory load.
Presentation of Working Sets
Instead, in the above scenario, your computer could unobtrusively present the relevant working-set items as soon as you are chatting with your collaborator again. Zeitgeist enables this by interpreting the collaborator match, and possibly more matches, between the before and now situations, as an indicator that you might want to continue working on one of the topics you had recently treated together. The presentation of an implicitly created working set, i.e. a set of implicitly related items, necessitates monitoring user behavior, just as is required to implement implicit HCI. In the short scenario above, this is monitoring of using images, of chat conversations, and of programming.
All of this serves the general goal of having information bits and pieces “at hand” when needed. That is the vision behind the early-prototype video by Seif Lotfy below, which implements the saying “stuff you use together belongs together”, which we coined at GUADEC 2009. The corresponding calculations in the Zeitgeist engine are for now performed ad-hoc in nearly real-time. One of the remaining challenges is to suitably draw borders among different working sets, accounting for their dynamic evolvement over time, and distinguishing working sets from what the user does “on the side”.
The more we achieve to solve these issues, the better the employment of such implicitly created working sets will work towards faster, less effortful, and so more convenient re-finding and access of previously used information items relevant to the user’s respective activity. Referring to paper-based work at your physical desk again, the above described implicit information organization can be understood as automatically putting paper documents into piles, informed by how each of them is used, by their different arrangements on your desk over time, and by the, possibly multiple, occurrences of the documents being involved in your workflow.
Design Issues
Automatically presenting a working set upon recognizing similarities between earlier situations and the current one, which, by the way, is a case of implicit output, releases the user from the need to explicitly browse or search for the items without requiring him to explicitly create and maintain working sets. Thereby, in the ideal case, the user doesn’t need to recall the relevant information items, but instead just needs to recognize them and choose accordingly from the presented working set. This needs to be complemented by means of querying for working sets, for example by defining a time scope.
Even though Zeitgeist provides support for the above discussed means for implicit creation of working sets, these dynamic collections might, in some cases, still become too large to be dealt with easily by the user. Therefore, our visualization of a working set will reflect a gradual and dynamic relevance notion, i.e. show how much a certain item belongs to a working set and/or to the currently recognized situation. In addition, we are experimenting with the concept of a life time of items with respect to inclusion into a working set, in combination with the relevance concept. This could be visually mirrored as having items fade out over time, if not being used, and fade in again, if being re-used. If users want more control, pinning items onto the working-set canvas will prevent fading behavior.
Beyond the implicit means of a gradual relevance notion and an item life-time, there are also explicit means of letting a user manipulate a working set and its visualization, in order to keep them treatable. These are blacklisting, filtering, and maybe defining size limits. Furthermore, we will investigate letting users partition a working set by arranging its items on a canvas or by employing tags or folders. The two latter naming means would also allow several working sets to be presented at once, reflecting, for example, the case of the same group of people working on different tasks “in parallel”.
Users often experience a loss of control when confronted with dynamic structures, such as dynamic menus. This needs to be avoided with respect to the rather dynamic concept of working sets. Users should not fear “losing” working-set elements or not being able to later re-find and access items they had earlier seen in working sets. This goal could be achieved by automatically turning working sets into persistent collections, for example, daily. These periodic backups would be “lossless” with respect to once included items. Working sets can even be re-computed at any time, as long as no events are deleted from the Zeitgeist log. By default, deletion of essential events therefore only happens upon user request. Consequently, users will not be required to explicitly keep working sets.
Additionally, we will try designing the computation and visualization of working sets such that users will not be forced to organize or re-organize these sets at specific occasions or times. Instead, users should feel and be in control over when to further classify information from working sets. This is because of the known negative effects of imposing interrupts on a user’s work flow, which are perceived as unnecessary overhead on the user’s part. Another rationale for the above design goal is the often reproduced finding of [2], that untitled “piles” are helpful means to represent working-set like information.
As opposed to the world of our physical desktops, the computer world enables us to have the same information item in multiple “piles”, so working sets, at the same time. Beyond, Zeitgeist can also keep track of the evolvement of working sets and the information they “contain”, in order to enable a user to refer back to different versions of them over time. A related challenge of making working sets usable is to design their behavior and user interface in a way, which lets the user distinguish among copy and reference semantics for the “contained” information items.
References
[1] William Jones, Jaime Teevan (2007). Personal Information Management. University of Washington Press, Seattle and London.
[6] Andrew D. Wilson (2008). Sensor- and Recognition-Based Input for Interaction. Chapter 10 of: Andrew Sears, Julie A. Jacko (2008). The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications (2nd Ed.), CRC Press, New York, USA.
