GUI-Kinect Stride: Meetings

From CSclasswiki
Jump to: navigation, search



--Thiebaut 15:30, 19 September 2012 (EDT)

  • First meeting
  • Wednesday 2:30 p.m. set as the default weekly meeting time.
  • Goal: design of a GUI for Processing project created by In Kyung (Inky) Lee for recording 3D choreographic work
  • Action Items for Julia
    • start going through the Java project and create a chart of the interaction of the different classes
    • start looking at Colin Ware's book Visual Thinking for Design. Use it as inspiration for what to do and what not to do when designing the GUI.
    • start creating a list of 5-college people (faculty, students) who could be good resources. We will invite them for a presentation of the tool at some point.
    • keep an eye for kinect-based projects that have common features with ours. Create a page cataloguing these projects. Always include pictures of movies, along with the address of the team, and, if possible, a list of the programmers/project leaders.


--Thiebaut 15:33, 3 October 2012 (EDT)

  • What we did: go over Julia's block diagram for the project. Good overall structure.
  • We removed the extra filters that were creating special effects that were hiding the raw Kinect data.
  • Action Items for Julia this week
    • Create the diagram of the program in Google docs (drawing). Add it to this wiki. Start a page new page for this.
    • Add markers around the location of the code we commented out so that it can be brought back easily.
    • Look-up Shiffman's page (NYU Prof), and go through his technique for displaying Kinect point clouds. Verify that you can create a simple demo with Kinect 3D data.
    • Verify that our project uses the same approach (same library). Questions to investigate: is Shiffman's library updated/upgraded from the one we are currently using?
    • In the page you will be creating for the block diagram. Write a 1-paragraph introduction explaining how our program is based on Shiffman's library (to be verified) and in which way it is an evolution of it (threads).
    • Playing with threads: your assignment: write a program that uses 2 new classes that are extensions of the Java Thread class. These threads will display 100 numbers on the screen. Thread 1 will display all the numbers between 1 and 100, and Thread 2 will display all the numbers between 101 and 200. Make the threads sleep between outputting each number. Make the sleep delay either 0 or 10 ms. Here's an idea of what your run() function might look like:
 public void run() {
       for ( int i = 0; i< 100; i++ ) {

             // display a number

             Thread.sleep( delay );

(you may have to protect your code with try/catch statements.)

    • Have a demo of your program ready for our next meeting! :-)

class demo {

    public void main( String args[] ) {
        int x = 3;


--Thiebaut 17:52, 11 October 2012 (EDT)

  • Good progress today.
    • Good first draft of the block diagram of the Java-Kinect project
    • Two different versions of thread working.
    • Shiffman's application to get frames from the Kinect install from scratch and working.
  • Action Items for next week:
    • Continue exploring Shiffman's program
    • Edit block diagram. Differentiate between "action" classes and containers of data. Make sure the relationship between draw() and the different blocks is explicit and correct (direction of arrows).
    • Start taking a look at ControlP5 library.
    • Start keeping an eye on Kinect projects and update the Kinect_Page Kinect page.


--Thiebaut 14:21, 25 October 2012 (EDT)

  • Action Item for DT: Find an example of research paper dealing with Kinect...
    • Efficient Model-based 3D Tracking of Hand Articulations using Kinect, by Iason Oikonomidis et al., Computational Vision and Robotics Lab., Institute of Computer Science, FORTH, and Computer Science Department, University of Crete, Greece, Aug. 2011.
    • Human Detection Using Depth Information by Kinect, by Lu Xia, Chia-Chih Chen and J. K. Aggarwal, Department of Electrical and Computer Engineering, U. Texas at Austin, June 2011.
  • Setup Google Alerts for Julia. Will get regular updates from Google on new development with Kinect: papers, web pages, movies, etc.
  • Action Item for JE: Play!, and generate documentation for how user can generate movies. What to click, where the movies end up, etc.
  • Action Item for DT + JE: start thinking of how to reduce size of movies... Too many 0s padded in the movie.


--Thiebaut 15:32, 2 November 2012 (EDT)

  • Installed drivers/libraries on Julia's Mac for running Kinect project with Eclipse
  • Action Item for DT: Get Janet Guo's thesis from Eitan.
  • Action Items for JE:
    • create a bug list for the GUI
    • start editing the project and fix the record mode: Clicking on record should 1) reset the buffer, 2) stop play, and 3) reset continuous play.
    • change the GUI so that the button shows loop instead of continuous as the caption.
    • investigate creating a 3-line console at the bottom of the GUI. This should be implemented by a separate class (say, ConsoleClass) that is static, and that maintains a list of status information lines that are sent to the console by various parts of the project. The class provides a general purpose method for posting status updates and keeps of history of the last n updates received (make n = 100).

class ConsoleClass {

    private ArrayList<String> buffer;
    private int maxNoLines;
    private GUIType textArea;   // whatever the textArea is on the actual GUI window
    static void init( GUIType x ) {
          buffer = new ArrayList<String>();
          maxNoLines = 1; // for now
          textArea = x;

    static void 
    static void post( String msg ) {
          buffer.add( msg );
          if ( buffer.size() > maxNoLines ) buffer.delete( 0 ); // remove oldest entry
          textArea.update( buffer ); // display contents of buffer in textArea of GUI
                                // (I have no idea what the GUI type is for something that displays text
                                // in ControlP5, but that should give you the idea...)


--Thiebaut 16:38, 29 November 2012 (EST)

  • GUI is tricky! We spent some time trying to figure out how to make the Play button reset automatically at the end of a movie, but ran into problems.
  • Decided to scrap the current GUI which doesn't look really good, and evaluate Qt Jambi.
  • Action Items for the next few weeks:
    • Start a new version of the Kinect project minus the current ControlP5 GUI. For right now just hiding the window for the GUI is probably enough, but if it seems feasible to completely remove it, so let's do so.
    • Go through the tutorials on Qt Jambi. Figure out what widgets would be good for our GUI. Create several mini projects with just Qt Jambi stuff. See if you can create something that uses several widgets that interact with a main application. Something like a calculator with 0 to 9 keys, a + and a - key, and a display widget to see the result. Should be challenging enough to get the sense of how things are going to work. Qt has a great graphical designer for creating windows with widgets in it. Use it!
    • Look for references/examples of project where Qt Jambi and Processing have been successfully mixed together. See if there are good explanations for making Qt Jambi work with Processing applets.
    • Create a simple Processing project that uses Qt Jambi and its widgets.
    • If that is too hard, it might be that we'd have to port the whole project to pure Java with Qt Jambi and not use Processing. Something that is part of our evaluation...
    • Have fun!




--Thiebaut 10:52, 22 January 2013 (EST)

Plan for 2nd semester

  1. Clear GUI and create a project with one window showing basic live point-cloud and an empty GUI window.
  2. Collaboratively create catalog of all features/actions that we want to have. Below is a non-exhaustive list we might want to consider editing:
    • save to file and compress
    • load from file and play (no live points)
    • load from file and overlay with live points
    • loop mode for playing files
    • move camera around by hand
    • record camera movements
    • play camera path (last, selected path?)
    • play prerecorded smooth camera paths
      • zoom
      • circle around center of room
      • horizontal circle
      • vertical circle
      • both horizontal and vertical
    • remove background or floor
    • more?
  3. Group features into logical groups
  4. Propose a GUI design
    • Suggest if menus are needed. Or tabs.
    • Keep it simple!
    • pick best widget (wether available or not)
  5. Start implementing one of the groups
  6. Keep documenting all progress in wiki.


  • Good meeting today. Cleaned up a bug. But discovered another one...
  • To do for next week:
    • Find a way to copy the projects automatically to the MacPro in FH343. See if could be a solution where you could replicate your Eclipse workspace on your Mac with the Eclipse workspace on the MacPro.
    • Review the changes we made to the Start/Stop button and make sure what we coded doesn't have holes in it.
    • Figure out a way to prevent fast double-clicks from blocking the system. One option is to add a boolean somewhere that let's the button's onClick() function interact with the PointCloud program whenever it is safe to do so.
    • Add a new widget (or two) to the GUI!


  • Discovery of the day: we need to implement an Finite State Machine (FSM) to interface the GUI to the application.
  • Figure out if there is a Java class already written for FSMs that would do exactly what we need. I know it's tempting to write one's own FSM, but people are using FSMs all the time and there should be a few libraries out there offering the basic operations we need. Here are some links I found that have interesting information:
  • For next week, add the FSM to the project and have it interact with the Play/Stop button, as well as the continuous button. Don't add any more buttons before we have these 3 work well.
  • DOCUMENT WELL! The FSM is going to be complicated. Figure out a way to keep track of each part of the state diagram. The diagram feature of Google docs is fine for drawing graphs. But that might become too busy/hairy for our graph with many conditions and many states. To be evaluated as we go along. Below is an example of a FSM for a traffic-light sequencer generated with Google docs.:


  • Below is a picture of the board after we explored some of the features of the FSM:


--Thiebaut 16:57, 23 February 2013 (EST)

  • Good meeting today. Managed to interface the play button to a simple FSM class, and the FSM to the PointCloud application.
  • To do for next week:
    • implement the simple FSM state diagram with 3 states presented today in Java.
    • test it
    • verify that it takes care of double clicks and clicking randomly on the button
    • implement the FSM control of the Continuous button in Java.
When coding the FSM, make sure to only translate the state diagram, and not add new logic to the code that is not present in the state diagram.


--Thiebaut 13:39, 1 March 2013 (EST)

  • Another good meeting today. The FSM looks solid, robust, and I have tried my best to break it, but JBE's coding didn't crash!
  • To do for next week:
    • Improve the GUI. Location of buttons. Frame around button. Color of button/label/background (Use Color Scheme Designer 3 for help picking good colors that work well together).
    • Work on the Wiki. Add descriptions to various sections that need additional material. Do not hesitate to reorganize the page.
    • Work on the record button
      • Should be able to record a movie of the live dancer
      • Should be able to record both the dancer superimposed on a pre-recorded movie that is playing in the background
        • Allow for synchronous start of both the play and record
        • Allow for asynchronous start of the recording while background movie is playing
        • Make sure that if two different sources are recorded together, an Id integer of a few bits identifying which source the point comes from is recorded along with the point.
    • Work on adding a delay to the record button. Suggestion: have several check boxes or radio buttons for 1 s, 2 s, 5 s, 10 s and 30 s for delay.


--Thiebaut 14:12, 8 March 2013 (EST)

  • Good meeting. Coded the Record button live, and it worked! The delay of 5 seconds worked too the first time. The FSM is a great addition to the project! The trick is to update it first, then code after!
  • TO DO for next week (Thursday at 3:00 p.m.)
    • Update the Wiki and complete the FSM diagrams
    • Do a survey of examples of GUIs with similar features, and report on each one in the Wiki. At least 3 good examples.
    • Update the color scheme for our GUI.
    • Add a new feature where we can stop or click on the GUI while the Record + delay is on going. Not sure what the best outcome of stopping the delay should be, but we should be able to make the FSM+GUI react while the delay is going on. Probably bringing back the original while loop should do it:

      while ( ! timeToStop ) {
      Thread.sleep( 100 ); //100 ms

      and find how to implement/update timeToStop.
    • Add the synchronous and asynchronous radio widgets.
    • Start putting together and abstract and a poster for CCSCNE and Grace Hopper. Inky's abstract is a good place to start.


--Thiebaut 16:35, 29 March 2013 (EDT)

  • First week after Spring Break!
  • Good meeting today with reviews of the GUI examples selected. Just make sure to comment on the features. Just the image is not enough. Doing research also means expressing an opinion, hopefully informed and accurate, about the qualities and shortcomings of other work in our area. With time and experience this process will result in sharper and more instructive comments. The good reflex, though, is to always ask oneself why something is good, what works, what can be improved.
  • To do for next week:
    • Complete the wiki page with missing features, i.e. survey of Kinect projects in the dance area.
    • Fix bug in play function of current PointCloud software
    • Generate first draft for a poster for Stride presentations. Contact Peter de Villiers with information about presentation and screen or projection facilty.
    • Add additional nonfunctional GUI widgets to GUI for improved look during presentation ;-)


--Thiebaut 14:09, 5 April 2013 (EDT)

  • The Kinect has a new GUI widget to control the tilt. Very cool. Just make sure not to position the Kinect on top of monitor or narrow perch... it might suffer... Oops!
  • The kinect tilt angle is recorded and restored when the program restarts. Cool. Can be operated from keyboard.
  • Figured out how to make the labels refresh and text strings not overlap each other in the GUI applet: just call background(0) in the GUI draw() function.  :-)
  • Nice progress from last week. Clear design. Good choice of colors.
  • Just not sure why the mysterious bug disappeared mysteriously...
  • To do for next week
    • Add frame counter and frame rate counter.
    • Think of how the tilt feature could be use creatively. A wave feature?
    • Work on poster
      • add Smith logo
      • add Dept. Computer Science
      • add names
      • add "Stride"
      • add margins around. At least an inch.
      • could add section on similar projects
      • add link to the Stride/wiki page. Might want to create a or similar URL shortener link for it. Will make recording URL for viewers easier.
      • rework the FSM image: fatter lines.
      • rework the Threads image: (see diagram we drew on notpad)
      • use GUI from our app rather than some other project
      • use a real point-cloud captured with our system. Search Inky's space for pictures.


  • Last TO-DO list of the semester!
  • Fix the wave controls
    • finish implementing the class that controls the kinect motor.
    • set min wave value
    • set max wave value
    • implement wave start/stop: do we need and FSM for this.
    • figure out where the "Wave" string should appear on the screen
  • For the other features, simply add a new section to the wiki titled something like "What to do next" and describe how one could continue improving the GUI.
  • Make sure the wiki is clean and up-to-date.