
I just completed teaching a MOOC on Computational Investing via coursera.org. I did some things right and a lot of things wrong. Here are my lessons learned from the first round. I’m very excited now about the second go at this, and the ability to make the course even better. Some of these items are coursera specific, but many apply more broadly.
Some people will be upset: Be prepared
I’m not talking about the students.
Your course will be closely scrutinized. Some folks at your institution may be critical of the idea. Many outside your institution will be critical as well.
One critique MOOCs are susceptible to is an accusation of “dumbing down” or “oversimplification.” Sebastian Thrun’s statistics course was attacked by others in this way. He responded appropriately, which is to say his course is aimed at a particular educational goal (namely as preparation for later robotics courses), and also that he appreciated the feedback and he would use it to improve the course.
Many of these attacks arise from a belief that MOOCs are promoted as “identical” to college course content; Or that the course is “just as rigorous” as graduate course CS XXXX at Georgia Tech. I personally never intended my MOOC to serve that role. My goal is to use MOOCs as a way to connect to willing learners on topics I’m very excited about.
That being said, I do believe that we can produce and deliver “rigorous” content via MOOCs, and many are working on that. But not all MOOCs have that purpose.
Set expectations for the students
In your course description and your video introduction be sure to make it very clear who your intended audience is. I would also remind students of this frequently in the early modules of the course.
You probably want to discourage folks who don’t have the tools to succeed (for instance in my course, programming experience). If they take the course they will not be happy, they will complain on the forums. They will also contact your TA and consume his or her valuable time.
You might offer these folks an alternative: Invite them to watch the course videos without the assignments. Coursera has a “preview” mode for this.
You also want to avoid having folks in the course who are “over qualified.” They’ll get bored and complain about the simplicity of the content. Their comments on the forums and blogs may tend to undermine the value of the course to others.
Get good help and delegate: Focus on the content
Ask your support team to handle as much as possible. This will enable you to focus on creating great content.
I received great support from Instructional Designer Fatimah Wirth, Ph.D., and from TA Sourabh Bajaj. They handled many details that would have overwhelmed me otherwise. (If you’re reading this Fatimah and Sourabh, thanks!)
Why slide decks are important
MOOCs offer a significant number of alternative ways for creating and delivering content. There isn’t a strict need for a powerpoint or keynote deck for every module.
When I teach “in person” I usually use the whiteboard a lot. I walk into the class with a few ideas about what I will cover, and because I know a lot about the topic (usually) I am able to provide an informative lecture. I found that I can’t do this very easily via MOOC.
Slide decks can help. I leverage slide decks in my MOOC workflow and my larger plan as follows.
First, I rely on the deck as a sort of “script” for each video. I’m always a bit “hopped up” on camera and it is nice to have the slides there to keep me on track.
Next, the deck enables me to easily recreate the video if necessary later. In fact I’m anticipating to redo a lot of my videos in response to student suggestions. If you don’t keep the slide deck “script” you might forget some of the content. In particular the deck helps you recall how that lecture fits in between two others.
Finally, the students often like to be able to view the slide decks for review.
Creating content will take a lot more time than you think it will
For me about 1 hour for each 15 minutes of video.
When creating content, figure out a workflow and stick to it
If you follow my approach of slide decks as “scripts” for your videos, you’ll be creating a lot of slide decks. I settled on the following work flow:
- Don’t worry about drawings or diagrams at first.
- Create the slide deck, use placeholder slides (just titles) for drawings you will add later.
- Create the drawings. I use penultimate on an iPad. It’s a lot faster than creating a “serious” diagram using a more formal tool. I feel that in this application the ability to churn out diagrams and revisions quickly outweighs potential concerns about quality. The image at the top of this post was created in this way.
- Add the drawings in to your slide deck.
- Create your video using the slide deck as a script.
Keep videos short: 5 to 12 minutes
Keep in mind that your students will be watching your content on mobile devices at lunch or during breaks, perhaps even on the subway. Studies have shown that 5 to 12 minutes is a digestible nugget’s-worth of knowledge.
When planning my videos I aim for about 10 slides. Even then I often find that the content I thought would take only 5 minutes takes 15 minutes.
Here endeth the lesson: keep track of time while taping
If you find yourself running long, just close the lesson and move the rest of that content to the next slide deck / video. As a rule of thumb, once I hit 8 minutes, I’m looking for the next logical end point.
This enables you to keep the video at the right length and keeps you from having to reshoot it. Just say something like “That’s all we have time for now. Next: We’ll learn more about X and Y.”
If you do that, remember to correct your slide deck accordingly after the video.
Continuity: Avoid mentions of dates, times and content order
If you say something like “in the next module we will learn X” then you can never move the present video to a different order in your course.
Now sometimes you DO want to do this, and it’s OK; just be sure about it. My personal approach is to organize my course as modules of 3 or 4 videos, and to allow myself to talk about content order within a module, but not across modules.
Interact on the forums a lot, and personally
One of the leading criticisms of MOOCs, and a valid one I think, is the lack of face-to-face one-on-one interaction between the educator and the student. The forums offered by MOOC providers are a first answer to this. Certainly the students of the MOOC interact there, but the teacher and the TAs should be there too.
I did this a lot at the beginning of the course, but as my workload increased (due to other non-MOOC responsibilities) I wasn’t able to do this as much. My impression is that the excitement for the students about the course dimmed a bit towards the end because of this. I can say that the students really value having the lead instructor participate him or herself. I also suggest to have your TA or TAs spend as much time as possible interacting with students on the forums.
Send email to the students for important announcements
I initially just posted significant announcements on the course announcement page. But many students didn’t know they should check there and were frustrated that they missed information about deadlines and so on.
So my suggestion is to start by posting all announcements by email, and to always email the major announcements. You can, later, reduce the number of announcements that you also send by email, but I would suggest to warn the students first, e.g., “In the future, homework details will be posted here: http…”
Ideas for interaction
Here’s something I tried that was very popular:
Invite the students to post questions for you on the forums. Then suggest that other students “vote up” the questions they most want the instructor to respond to. Devote a video or two to answering the questions.
These were my most popular videos.
Other possibilities include the use of a google hangout or live webinar
Think carefully about grading, especially peer review
Coursera offers a method for grading assignments in which students grade each other. Each student is supposed to assess 5 other students’ submissions. It worked OK, but there were a number of downsides:
First, it takes effort on your part to produce a detailed rubric. In order to have consistent grading across a wide spectrum of students you need very clear cut rules. Here’s the rubric for the peer reviewed assignment from my course.
Next, it takes time. All of the deadlines have to be stretched out to allow time for grading. It also takes the time of the students, which they may not be happy about. On the other hand, depending on the subject, this may be time well spent.
Finally, some students do not execute the peer review well. This leads to difficulties that may be hard to resolve, especially for a large class.
Grading can be easier for computer science-based courses
If the project is to develop software that can do X, then you can construct a grading method whereby the student is posed several questions that should be run through the program. In the case of my course, which is investing based, an example might go like this: Suppose you invested $100 in IBM on Sept 7, 2006, what would the value be on Sept 7, 2012?
More to come
If more lessons learned occur to me, I’ll post them here. Send me questions if you like.
Hakim Aksel
January 12, 2013
Great stuff professor Balch. Even though I stopped computational investing I at week 4, I really appreciated your enthusiasm. I should have watched the introduction video more carefully, and realize that I was overqualified for the content. However, I believe that even people like me benefited from QSTK and the class wiki on QSTK, what a nice tool for Data Analysis.
Tucker Balch
January 12, 2013
Thanks Hakim!
John
January 12, 2013
I went all the way through, enjoyed the course, and learned quite a bit. I’m now taking the Introduction to Computational Finance class and I have been intrigued at how different the approaches are to largely the same material. Both classes took different approaches and angles, with your class focusing more on automated trading / computer decision making and this new one covering how a Quant woudl use R to make investment decisions. Anyway, my point being that there is much to be learned from both courses and different value to be gained from each. I look forward to taking the second class when you offer it and getting deeper into the machine learning aspects :-)
Nick Iversen
January 13, 2013
I did the course and I liked everything except for one important thing: no sample answers to the Python assignments. After writing crappy code that got the job done I wanted to see what an expert’s code looked like. I WOULD HAVE LEARNED MUCH MORE ABOUT PYTHON from getting that feedback.
Tucker Balch
January 13, 2013
Hi Nick,
That’s an important comment. We’re going to see if we can address that going forward.
Marina
January 17, 2013
I have already taken about 10 Coursera courses. I enjoy each of them very much. I am very grateful for Coursera teachers and developers. That’s what I call the “taste of life” :-)) Unless you were here, I would not be able to learn so many interesting and helpful things. Thank you!
BUT THE MOST UPSETTING AND DANGEROUS THING IN Coursera are those 5-10 minutes mini-lectures. Many people who listen to this lectures have a full time employment and family duties (e.g., I am a nurse, I have 4 children at hope to care for and a mother with Altzheimer). I listen to Coursera while driving to work (one hour in each direction every day) because I have no other opportunity. Try to push the “next” button on your mobile phone every 5-10 minutes while driving 100 km/h! The button does not always pushes from the first time. You have to push it again and again.
I am careful, I don’t want to kill anybody, and I don’t want my children to go to orphanage. So I usually stop aside the road and press the damn button. But it’s dangerous to stop too. I have to stop every 5-10 minutes during one hour lecture. Suppose, you have 50.000 people in the class. Suppose 1% of them listen to Coursera while driving as I do. (I assume that much more people do it while driving, as it’s very convenient and for many people it’s hard to imagine other possible time). It means that you put in danger the lives of 500 people (and the people who drive by them) every time you make a break.
Of cause, the easiest thing to do it to say: “please don’t listen to it while driving” and you get off the responsibility. But people would just go on with this notion, because it’s convenient. Because, otherwise, we won’t get the desired education. We don’t have money to afford it. If I have a choice to put my life in danger or to abandon online education, I choose to put my life in danger. It’s my responsibility. I my opinion life without learning is meaningless.I hope, I won’t crash the car, but study, and get a better job in the future. It’s a game of chance and I take the bet.
But must I bet on my life? What can you do to help me?
You can make the 1-2 hour long lectures. You can put this this full lectures together with the short lectures. Short lectures meant for those who listen to you while coffee break and long are meant for those who listen to you while driving, while running and etc.
I know personally a girl who felled of the walker and broke her hand, because she was running and listening to the on line lecture in “model thinking”. The mini-lecture stopped and she tried to move it forward. It’s quiet tricky on iphone – you have to go “back” and press the link again. She tried it, but she felled down from the walker and broke her hand.
I beg you. Please, make the lectures long! And, please, tell it to the other lecturers.
Tucker Balch
January 17, 2013
Wow, interesting post! I think this problem is more on the delivery technology side. It should be possible to queue up a number lectures to listen to.
Marina
January 17, 2013
I tried to do it, but I could not find out how.
PabTorre
January 19, 2013
Profesor Peng implemented something that could be solution for this in his Computing for Data Analysis class. Every week, he posts a duplicate of the material in coursera as a youtube playlist, allowing for those who prefer to watch on that platform for whatever reason to do so.
Ron Amundson
February 3, 2013
It is on the delivery side, and considering fairly consistent stats that a lot of students don’t care for audio files, I understand why coursera doesn’t provide this as a downloadable option. On the other hand, I echo the above posters issues with short files while driving, exercising, working around the house etc. As a result I generally download the weeks videos, strip the audio and combine it into a single much larger mp3 file. Its pretty easy to do with a audio editor, but its probably non-obvious for a lot of people.
Michael
January 17, 2013
Thank you for teaching this course.
I thought your course was excellent and I signed up to take it again so that I am solid for Part 2. The MOOC model is great when you have a full-time job, kids and after school activities.
Keep up the great work!
Michael
PabTorre
January 19, 2013
Prof. Balch. Thank you for a very fun class. i found that your class presented interesting challenges and helped me deepen my understanding of Python as well as to expand my ideas of what is possible to do with a computational approach in finance.
A couple suggestions for the next run.
You should recommend an introductory Python class as a prerequisite… in my case, taking Introduction to programming: the fundamentals (also in coursera) was definitely a strong foundation that allowed me to focus my learning on the concepts that you were introducing.
Expand support of qstk to run on an IDE that is friendlier to noob’s… I was able to do a lot more in your class by using the matlab-like interface of Spyder… i use vi every day on my work when interfacing with unix systems and it still feels very alien to me…
Getting qstk to install easily on windows would also be a big step… for better or worse it is the world’s most popular OS, and many students complained in the forums about the troubles of running virtual machines… while others (like myself) spent a lot of time implementing hacks to have it run in Windows, which involved the less than pleasant experience of editing environment variables for trial and error… :)
Post optional assignments that allow the more advanced students to go deeper into key concepts, such as the functionalities found in pandas and numpy… or challenges related to real time application of trading concepts that you present on yoru class… those could keep the more advanced crowd entretained, and stop them from getting bored and complaining in the forums.
Could you explain how to use the event profiler’s output as input to other functions? I am very much interested in using the data that is displayed in the event profilers output pdf as input for the portfolio optimizer :)
Again thank you for a very fun, hands on class. It was a great learning opportunity and I’ll be l joining the next iteration to see how you apply the many improvements that you point out above (plus the extra lessons on technical analysis).
Tucker Balch
January 19, 2013
Thanks for the helpful feedback. We’re addressing a number of those issues in the next run. It will take a number of iterations to make it perfect.
John
January 20, 2013
One comment about the course would be that it would have been nice to use QSTK more… except for the event study, I suspect we didn’t even scratch the surface of what y’all have put into QSTK. :-)
Hakim Aksel
January 21, 2013
Actually I stopped following the course and focused on using QSTK via the wiki and the .py files. You can look for quant strategies on the web, usually coded in matlab, and then tweak them with your own code using QSTK
Tucker Balch
January 23, 2013
Glad to hear about that use for QSTK.
susan cook
January 21, 2013
Professor,
Having just joined, I have yet to actually learn the intricacies of this system, However, I do take online classes traditionally and for credit. I have a young son and a complicated background that make online learning more optional than a campus type setting. My Grandfather was a Math Professor at your school which is what caught my attention and brought me to your blog. Reading your blog i think you have some really definative and good Ideas. The student peer review grading is a bad Idea to me, and your system is set up with some difficulty in always getting to the heart of discussion which comes from letting students create as many threads as they want and not keeping it more contained and isolated to subject and focus. I am use to using blackboard so it could just be the duality of using two seperate systems but I think the coursera system has some unnecessary complications. As far as Dumbing down a class, it would be hard not to considering the students may be beginners, they are not getting real credit, and they may not have that much time to devote to discussions that don’t influence the grade. I think also when it talks of meeting up in your towns and what not it becomes a social agenda rather than educational.
I mostly see myself benefitting by being able to learn from a myriad of different well established educational facilities, and let’s face it once I learn something regardless of what venue I garner the information from I then retain the power of said knowledge even without a certificate or credit, learning will help better me and my families future. I appreciate your comments and that you take the time you do to teach.
Susan Michelle Cook nee (Faulkner)
Ralph Faulkner was my Grandfather, he taught at GA Tech until he retired, so I have great respect for your institution.
Peter Tran
February 6, 2013
I’ve enrolled for the Spring course. I should be disciplined enough to finish it, since it covers two areas that I love – investing and programming. As extra incentive (or disincentive depending on your POV), I may sign up with STICKK (http://www.stickk.com/) and state my goal to finish your course.
Looking forward to end of Feb!