• The Power of storytelling, Ursula Le Guin, and Black Lives Matter

    Darren Maczka

    Robbing one of one's stories is a form of violence; instead we must listen.

    For a while now, I’ve been musing about the concept of power (the social kind, not V*I), as it pertains to my research. Before that, a more central focus of interest around the time of my switch to the Engineernig Education program was that of narrative, inspired and shaped largely by Janet Murray’s Hamlet on the Holodeck. Mixed up in there has been an interest in interplay of gender and language, which led me to Ursula Le Guin’s The Left Hand of Darkness.

    It was then quite serendipitous when I came across a blog post by Maria Popova weaving all these ideas together. I had been sitting on this idea for a post for a while, not quite sure where to take it beyond “isn’t it nice that all of these things that have been increasingly meaningful to me are brought together so neatly by Maria Popova”, but over the past few weeks with yet more murders of Black men at the hands of police officers and subsequent demonstrations and shows of civil unrest I’ve been thinking about the power of telling ones story, and the oppression (also a form of violence, as Paulo Freire argues) of suppressing one’s story. Many White people in the U.S. do not necessarily view recent events in this way. Rather, they seem to have a troubling detachment from the history of our country, a history that Le Guin summarizes beautifully:

    My country came together in one revolution and was nearly broken by another.

    The first revolution was a protest against galling, stupid, but relatively mild social and economic exploitation. It was almost uniquely successful.

    Many of those who made the first revolution practiced the most extreme form of economic exploitation and social oppression: they were slave owners.

    The second American revolution, the Civil War, was an attempt to preserve slavery. It was partially successful. The institution was abolished, but the mind of the master and the mind of the slave still think a good many of the thoughts of America.

    The exploitation of Black lives runs deep through the history of our nation, yet many White people continue to condemn any form of protest to this exploitation that they do not deem “acceptable” (i.e. any form that succeeds in broadcasting the personal stories of pain, fear, and distrust of authority), yet every July 4th continue to proudly celebrate the violence they perpetrated in response to “stupid, but relatively mild social and economic exploitation.”

    Ultimately, these shows of violence can be seen as acts of storytelling. That is, after trying unsuccessfully share the narrative of oppression to enact meaningful change through lobbying government, having it fall on deaf ears while Black lives continue to be stolen by institutional authority, the only recourse is to share that story in a way impossible for the oppressors to ignore.

    This aspect of the conflict is poignantly demonstrated by the Black Obituary Project as a direct response to not only the taking of Black lives by state-sanctioned violence, but also the larceny of those lives’ stories by White media outlets that focus on any past criminal record, no matter how irrelevant, and the possibility the victim had a weapon, no matter how unfounded, rather than what that live was, and what it meant to their community.

    I certainly don’t know what all the answers are to the present unrest we have been experiencing, but I do know that to move forward we need to start listening to each others stories more closely, and with intent. This is not a “their problem”, this is our problem, or perhaps more optimistically, our challenge. The exploitation of Black lives that will always be a part of the ugly and uncomfortable foundation of our country, we can’t change that, and we must not forget that. While continuing to keep an eye on the past, how can we envision a future in which together we insure that this exploitation is contained in the past?

  • Rethinking bias

    Darren Maczka

    There are at least two ways we use the word "bias", one of them comes with its own implicit assumptions of worldview and epistemology. But we often forget this.

    This upcoming week in class we will be discussing implicit bias, the subconscious attitudes and stereotypes that affect our choices and behavior towards other people. This prompted me to finally try and write out a post about what “bias” is, and what assumptions, or even, implicit biases, are activated when we use the word “bias” in our every-day conversation. There are several definitions of “bias”, the two that we use often in everyday speech are

    1. a tendency to have a particular outlook due to personal beliefs rather than “objective” facts, often considered the antonym of “impartial”
    2. treating a person or groups of people unfairly, associated with prejudiced

    I think the two definitions are similar in some ways which I will explore later, first I want to dig into the first definition a bit.

    Recently I’ve been re-reading Rethinking History, by Keith Jenkins. It’s a great read if you really want to question some assumptions you might have about how it is we come to know about history, what history is, and what it isn’t. To this end, Jenkins critiques the common believe that history is meant to sift out the “facts” of what “really” happened in the past, often using “evidence”. I put those words in quotes because it turns out, they’re actually quite problematic. The problem can be boiled down to an analogy Jenkins uses: a geographer, a sociologist, and a historian all looking out the same window at the same landscape.

    A geographer, sociologist, and historian walk to a bar with a single window. What do they see?

    Fist of all, we can note that none of the three can see the whole picture, the framing is in this case, quite literally, accomplished by the window frame. Second, we can imagine that each of the three will find different elements of the scene noteworthy: the geographer may note differences in agricultural vs. forested land, the sociologist my notice what family structures are common in the homes, population distributions, etc. The historian may be keen on observing how any of these other qualities change over time, what conditions might precede certain changes, etc.

    The point is, all three of them are making “correct” observations, but all different observations. Though this is before Jenkins brings up the discussion of “bias”, we might ask, would we consider each of those different views out the window “biased”?

    A slight aside

    You may see where this is going: “bias” only makes sense under the underlying assumption that there is an objective Truth out there for which any one of us could be expected to observe the same thing. This is in fact, not how we observe the world to work. But this idea that there is an objective reality out there, and we can discover it through observation forms the philosophical underpinnings of the scientific method, something that has been enjoying quite a shining reputation for providing the plethora of modern-day conveniences that we just could not live without. We could have a lively, and productive debate on whether or not science really has provided us convenience (and if so, who it has provided that convenience for… and who has it not), but for the time being let’s just say that science has been successful because it “works” in the sense that if has allowed us to achieve a particular goal of understanding the properties of the world around us to the extent that we can manipulate that world to better suit our (but who is “us”?) needs.

    Formally, the philosophical underpinnings of science that I eluded to earlier is known as empiricism. It defines a particular way of viewing the nature of reality and knowledge: that reality exists, and we can come to know the true nature of that existence through making observations and collecting evidence.

    Back to bias

    Jenkins makes a rather profound observation: “bias” is an artifact of an empiricist worldview. It functions as a mechanism to ensure internal consistency in the underlying assumptions about the way the world works. Consider this scenario: Alice and Bob both believe that a single objective reality exists that we are all living in, an we can come to know the truth of this reality by making observations. Both Alice and Bob’s own sense of self and belonging in society depend on this believe, so there’s a lot riding on it. A corollary to the assumption that there is an objective reality that be observed is that it doesn’t matter who does the observing (baring any relativistic effects, let’s assume both Alice and Bob are traveling at the same relative velocity). If evidence can be collected through observation, and that evidence tells us something about reality, then anyone collecting the same evidence should come to the same conclusions regarding reality. So Alice and Bob happen to be in the same place and the same time, making the same observation. Perhaps they are interested in the color of the sky. “My, what a nice blue sky that is,” declares Alice. Bob, somewhat sheepishly and somewhat confused asks “yes, but I’m sure you mean ‘celeste’, not ‘blue’.”

    Assuming Alice is about about her ‘blue, not celeste’ conclusion we are faced with two logical conclusions:

    1. The underpinning assumptions about the nature of reality, that it is objectively observable, are incorrect, or
    2. Reality is just as objectively real as ever, there must be something wrong with the people observing it.

    Well, it wouldn’t make much sense for a worldview to self-contradict itself out of existence, so 1 can’t be true. That leaves us 2, something is going on in the observers to prevent them from making truthful conclusions about reality. Let’s label that effect “bias”.

    The point, already

    What I hope this point illustrates is the claim Jenkins made about “bias”, that it is a construct that serves to maintain internal consistency with a particular worldview, namely an empiricist worldview. That is, rather than being a product of the world itself, “bias” is a product of the way we choose to view it. Jenkins then criticisms the use of “bias” to describe other worldviews, e.g. “feminists are biased.” It makes no sense to say that, he claims, because it implies that empiricism isn’t just a particular worldview but is THE way reality works. Using “bias” like this prioritizes and normalizes empiricism. Jenkins concludes

    thus, the problem of bias is specifically an empiricist one but because it is the dominant approach then its problems get distributed as if they were everyone’s. But they are not. Of course – and this must be stressed – other discourses have their own problems of internal coherence, etc., but bias is not the way they are expressed.

    I particularly like Jenkins word choice: bias is an expression of an empiricist worldview. The point then, is that by casually using the word “bias” in our daily discourse, we are implicitly operating from within an empiricist frame of reference. While operating within this frame has proven to be incredibly successful (when we define ‘success’ in certain ways, from certain peoples perspectives) in the realm of science, it would be naive to assume this way of viewing the world should work ALL the time. This isn’t to say that empiricism is bad, or other ways of conceiving of the world are “better”, as Jenkins states, these other philosophies, including for example those that form the basis for feminism, have their own constructs to ensure internal consistency, they’re just not called “bias”.

    You want the truth? Who’s truth do you want?

    One of Jenkins’ central claims through the book is that rather than trying to answer “what is history?” we should be trying to answer “who is history for?” This idea can be extended to other fields of inquiry as well, it means changing the question “what is the truth” to “who is this truth for?”. Who is science for? Who is feminism for? Who is grad school for? Who is DiversityEdu for? What this allows us to do is move beyond arguments of who is “really” correct, which tend not to end with anyone satisfied, and to embrace the idea that just like the geologist, sociologist, and historian looking out the window, we can all have different narratives describing what we see, we are all “correct”, and by taking account of all those narratives we can begin to paint a more complete picture of what we all see.

    Wait, so if “bias” is just a made-up thing…

    I’m not saying that we should just stop using phrases like “implicit bias” when describing peoples unconscious assumptions and stereotypes, and I’m definitely not saying that because “bias” is a made-up construct that it some how makes the effects of something like “implicit bias” less real, or less legitimate.

    When I first had the idea to write this post it immediately after getting frustrated at a number of political stories about which politicians lied the most. Now in many cases, I’m sure politicians do lie, in the sense they say things that they know are contrary to their own observations of the world. But I suspect many times politicians arrive a different conclusions not because some are lying and some are telling the truth (which would assume an objective reality), but that each may be truthfully describing their own reality, and it just might be that different realities conflict with one another.

    And that’s ok. I suspect that to make progress in the quest for social justice (who is “social justice” for?) we can not limit ourselves to empiricist views of the world. We must be open towards other ways making sense of our experiences, and how we come to see reality through them. We must be aware that just like “bias”, any other worldview will have its own internal-coherence mechanism, and we need to be aware of it, and its function. Because we can’t fall into the trap of assuming a particular worldview is in fact reality itself. That never ends well for anyone.

  • Word vs. LaTeX and other meaningless comparisons


    This is also not a Word vs. LaTeX piece. But I would like to unpack what seems to be a fundamental missunderstanding made by many of the pro-Word ‘Word vs. LaTeX’ folk. LaTeX is not a word-processor. I will say that again to aid retention: LaTeX is not a word-processor. Trying to compare LaTeX vs. Word is not even like trying to compare apples to oranges–at least those are both a type of fruit–it would be more like trying to compare a symphony orchestra to a kidney. Or something. I’m having trouble coming up with a good analogy since there is little analogy between Word and LaTex. The point of this post is to help increase awareness that the monolithic word-processor model is not the only model for document preparation, and it isn’t even a particularly good model for many modern needs. This highlights the real problem with the so called word-processing wars: the assumption that a monolithic word-processor is The One True Way of document preparation and the tendency to conclude that any tools that do not fit The One True Way are somehow less “efficient.”

    LaTeX is a typesetting engine as well as a set of commands that describe how to typeset text. Both MS Word and LaTeX are part of a document preparation system, MS Word includes a typesetting engine and a set of commands to describe how to typeset text, but it also includes more. More is not necessarily better. But this distinction is what makes some arguments so inappropriate. In response to a fundamentally flawed “research study” that concluded Word was “more efficient,” LuAnne Thompson and Angie Pendergrass tweeted

    Two problems here (or from Microsoft’s viewpoint, raging successes): Did I mention LaTeX isn’t a word-processor? It’s also not a text editor, so a text editor is necessary for using LaTeX as part of a document preparation workflow. Spell checking is a process usually done while composing text in a text editor, not during typesetting, which is what LaTeX does. If you want spell checking capabilities your system-wide spell-checker (is it true that you still need to download a separate utility to get system-wide spell checking in Windows? That seems archaic.) should work with any text editor. The second problem/win-for-microsoft: “word … is the only way to go”. As if MS Word is the only way to use a spell-checker. Since MS Word has a spell-checker built into it.

    It’s not really about LaTeX vs. Word

    Replace Word with Apple’s Pages, or LibreOffice Writer. They’re all the same, following the Microsoft model of software design: combine all conceivable functionality a user might need into a single program, give it a [WIMP] user interface to ostensibly make all the features discoverable. This design philosophy was fueled in part by Microsoft’s business model: swallow up any third-party utilities (e.g., spell checkers and grammar checkers and make them part of word)

    A monopoly of thought

    At its core this issue is not so much a monopoly of tool as it is a monopoly of thought regarding what documents are for and how they are prepared.

    The Cathedral and the Bazaar

    The engineering workflow

    The workflow for which LaTeX was designed is one which values finding the best tool for the job, where “best” might mean something slightly different for everyone, thus this workflow is about people making choices that are right for them. I choose what text editor to use to interact with my text, I choose what bibliography manager to use (as long as it can export a BibTex file, and most can),

    The tools are not perfect

    LaTeX syntax is verbose and can be confusing at times. The BibTeX format has some issues. 99% of the time LaTeX does The Right Thing when positioning floats, but when it doesn’t it’s a pain to micro-manage.

  • Workflows vs. worldviews and other false dichotomies

    Darren Maczka

    I’ve recently been thinking more about workflows and tools for writing since migrating from Electrical and Computer Engineering where the LaTeX workflow was fairly ubiquitous to Engineering Education in which the ubiquitous workflow involves MS Word or similar. This is not going to be a LaTeX vs. MS Word piece. There are plenty of those out there, though this post was inspired by one in particular (with a bit of an inflammatory title) by Alex Bond who made a ostensibly pragmatic argument as to why he chose to stick with MS Word (for now).

    For the most part, I thought Alex’s piece was fairly written: while he chooses to use MS Word he makes it clear that one way isn’t objectively better, rather his argument centers around “what works for you”. Especially in the comments and responses it is clear that he keeps an open mind towards other tools and the conversation itself. I do have a couple minor quibbles: an apples to oranges comparison in his Cost & Access section and the repeated implication that a plain text workflow is the new kid on the block when plain text workflows have been pretty well established since at least the 1970s. This post is not about those quibbles. This post is about a far more serious piece of his argument:

    But in my mind, there are two arguments: the practical (A is tangibly better than B), and the philosophical (A is better than B because of ethical, moral, or philosophical reasons). These are both important discussions to have, but in this post, I’m going to focus on the first.

    Specifically, Alex’s claim that he will focus on practical reasons why one might pick one workflow over the other and leave the philosophical debate for another time (if at all?). I disagree with the premise of his argument. Practice is deeply informed by philosophy in both cases. It is fine to argue that a legitimate reason someone might choose to use the MS Word workflow is “because everyone uses it”. It is not fine, and extremely dangerous, to believe that that argument is not philosophical: by making it you are implicitly buying in (literally) to the Microsoft philosophy that created an environment in which “everyone uses it” in the first place. Microsoft’s philosphy with Word is that everyone should use it and thus it is designed in such a way to encourage that, and more of a concern, prevent people from using anything else by making it extremely difficult to collaborate with anyone not using MS Word.

    There must be a word for the tendency to ignore philosophy/worldview when yours happens to be the dominant one, it is certainly a kind of privilege, but “worldview privilege” feels clunky and I’m not sure it’s getting at the issue. Let me provide two examples, incidentally both having to do with Richard Stallman.

    The first was in an answer to a Quara question. In his answer, Phillip Remaker wrote

    Richard Stallman is a staunch idealogue [sic] prone to outrageous and exaggerated statements in order to get attention for his cause. He is not to be taken too seriously.

    There are two sentences in the above quote. The first can generally be taken as accurate, though the word choice is problematic and I will get to that in a bit. It’s the second sentence, especially as it is placed after the first that is the real problem. By putting those two sentences together in the way that he did, Phillip is leading the reader to make two assumptions:

    1. Other contemporary computing icons such as Bill Gates or Steve Jobs are not staunch ideologues, this is some special quality of Richard Stallman.
    2. People who have strong views grounded in ideology should not be taken seriously.

    In my comment to that answer[^1] I addressed both of these. Actually, addressing the first point makes it clear that the second point is invalid.

    Personal ideologies

    To suggest that neither Bill Gates nor Steve Jobs were guided by an ideology is flat out naive. Apple’s walled-garden philosophy is that the best user experience is obtained when one company (namely Apple) has tight control over both the hardware and software that the user can access. Microsoft’s philosophy has historically been that users will use Microsoft software if they are aware of no alternatives. In addition , both companies operate under a business philosophy that suggests the best way to keep customers is to make them dependent on your own product and lock them into a continuous update cycle. I am not arguing that any of these philosophies are objectively “bad” or “good”, but I am arguing that it is dangerous to ignore the philosophical underpinnings that has lead to the organizational structure of both Microsoft and Apple and more importantly to the design of their respective software products and user experience. Of course when people describe the strong philosophical backings of Bill Gates and Steve Jobs, rather than “staunch idealogue”, they might use “business savey”, or “perfectionist”, both avoiding the negative connotations of the dogmatic views associated with “staunch ideologues”. Likewise, when Bill Gates or Steve Jobs expressed their views in an emotional way it was because they are/were “passionate”, not “prone to outrageous statements”, and Steve Job’s abundant use of the word “magic” to describe Apple products was never “exaggerated”, it was just “good marketing”.

    National ideologies

    I suspect the reasons why language like “staunch ideologue” and “prone to exaggerated statements” are used with Stallman, but “business savey”, “perfectionist”, “passionate”, and “good marketing” are used with Gates and Jobs has to do with yet another ideology, that of free-market capitalism in the U.S. and the association with wealth and success. Both Gates and Jobs made a whole lot of money off by staying true to their “staunch ideologies”, and so in the eyes of U.S. culture they are both “successful”, and so their “staunch ideologies” must be the “right kind”. In contrast, Stallman quit his job at MIT because he was concerned remaining an employee there would conflict with his ideologies as he pursued his passion for free-as-in-freedom software. To be sure, Stallman makes some money, just a few orders of magnitude less than Gates and Jobs.

    Pragmatism and ideology

    While I don’t intend this post to get into the workflow wars my observations have been that most, if not all, arguments (or favoring) for the MS Word workflow tend to hing on “everyone else uses it”. As I said before, that is a fine reason as any, but don’t kid yourself for a second into thinking that argument doesn’t reek of ideology. It is literally the argument that Microsoft’s business model is designed to elicit, it is very much grounded in philosophy and thus, it is a philosophical argument. Of course posts that argue for the MS Word workflow tend not to talk much about the philosophy, and why should they have to? Everyone just agrees to it. It’s the same reason quantitative researchers don’t need to be explicit about their post-positivist worldview in their writing. The reader knows they are coming from a post-positivist worldview because it’s quantitative research.

    On the other hand, many arguments for the text-based workflow do explicitly mention philosophy, such as this excellent one. If nothing else I would urge you to view the explicit discussion of philosophy as a sign of honesty and openness, you’re not going to find an “everyone else is doing it” argument there.

    Either way, use what you are comfortable with and what works for you. But don’t say your decision isn’t ideological.


    Steve Jobs and Bill Gates (522695099) by Joi Ito from Inbamura, Japan - Steve Jobs and Bill Gates on Flickr. Licensed under CC BY 2.0 via Commons.

    Richard M Stallman Swathanthra 2014 kerala by Ranjithsiji - Own work. Licensed under CC BY-SA 4.0 via Commons.


    [^1]: which has received 30 upvotes so far!

  • Teaching Diversity in Tech


    Recently while on vacation I found myself in a conversation with a friend about language and word use, specifically how certain words can elicit a strong emotional response in some people, and how that response is dependent not only on the word itself, but the context in which it is heard. This lead to thinking about the more imperfective scenario of environments in which subtle use of language can affect how welcome one feels. For example, the classroom.

    First, a personal anecdote:

    At my undergraduate institution in the early 2000s, like other institutions at the time, it was not uncommon to hear a pervasive heternormative, homophobic rhetoric in athletic environments. In the case of male teams, the language also included high levels of misogyny. It was just one of those things that was “part of the culture” and wasn’t really questioned much.

    I am fairly certain that no desire to ostracize and exclude was intended, but it should come as no surprise that that is in fact what happened. I say that not to make excuses, but to differentiate that scenario from others in which language is used with the explicit intent to cause harm.

    Like other new college students, that was a time of self-discovery for me, a time to hash out the parts of my identity that were mine and not those dictated for me by my parents or society. It is no coincidence then that during that time I found myself both beginning the coming out process as a gay individual and also a member of the varsity swim team.

    I was on the team for about a year and a half before I left. I quit for a number of reasons: time commitment, interests in exploring other academic pursuits, making my own life choices, etc. The homophobic and misogynist language may not have driven me away, but it did nothing to make me feel welcome and valued.

    The cognitive dissonance could be quite intense. We were often told that we were all valued, all part of a team. Team social events were common and participation in events was encouraged by the coaches (during recruitment we were all encouraged to spend some unsupervised time with the team. This entailed going to a party, supposedly to get to know the team and see how fun they were. I just remember feeling uncomfortable). But then in the next breath words like “pussies” and “homos” were used to describe the opposing team, with the assumption that this would get us pumped up. For a young, not-quite-out gay male also dealing with all the self-confidence issues common to many new college students it was a confusing environment to be in.

    Years later I learned that another gay swimmer, much more confident than myself, came out to the coach and the negative language immediately stopped. It was just a simple case of the coach not being aware that the language he was using could be hurtful and confusing, and making assumptions about his athletes.

    Back to the STEM classroom.

    During the same time as all this was playing out I was beginning my degree in computer systems engineering. Unlike athletics, the engineering classroom didn’t have the same reputation for systemic homophobic and misogynist language, but it was clear that something must be going on, or else why was the gender distribution among electrical and computer engineers so male dominated compared to the gender distribution of the university at large? Also, unlike athletics in which it is now almost expected to hear about yet another athlete coming out and being accepted by his or her team, engineering classes are still predominately male. It’s difficult to assess whether the distribution of sexual orientations in engineering classes is any different from the larger population, but nothing about the curriculum is explicitly welcoming to people who might not fall into the narrow definition of what the typical engineer looks like.

    The problem, as I see it, isn’t so much that engineering educators don’t want to be inclusive, but that there seems to be an unspoken assumption that ones personal life is separate from and should not affect ones professional life. While this may sound nice on paper it isn’t an accurate reflection of reality, especially when the “professional” life we are talking about is undergraduate coursework which often coincides with a time of tremendous personal development. Also unlike Athletics, there’s no clear path towards inclusion. It’s not as simple as removing exclusionary language, but rather adding something. But what?

    Over the years, I’ve been on both sides of the proverbial podium.

    As a student I’ve struggled with finding my place in a field that doesn’t provide me with a lot of roll models outside the portion of my identity that is “white male” (which is pretty huge, I don’t mean to downplay the significance of being surrounded by people in engineering that look like me, but it’s the only experience I have to work with). As a teacher I’ve often wondered if I’m doing everything I can to create an inclusive classroom environment for everyone. At the very least, I worry about inadvertently creating an unwelcoming environment for some identities that I may not be sensitive to.

    It’s one thing to tell a class explicitly, “I want to foster an environment in which everyone feels welcome and validated, if you ever feel marginalized for any reason, please bring it to my attention” and quite a different thing to make each and every student feel that they are welcome, valued and can safely express their views and identity.

    Let’s get back to a specific example from my own experience.

    Computing and gender and sexual identity is an interesting case study. Back when “computer” meant a person performing computations, most were women. Women have a long history as computer programmers, going back to 1842 and Ada Lovelace, generally regarded as the first computer programmer at a time when computational machines where purely theoretical. So there are women roll models in computer engineering, and Alan Turing is just one of several examples of queer people in computing.

    The absence of discussions of diversity and identity in the typical undergraduate engineering experience isn’t for lack of examples then, but rather, the people of engineering just aren’t really talked about. We may use people’s names, as when we talk about the Turing machine, but there’s no room to talk about Turing the man. While at its core engineering is an intrinsically human endeavor, in the classroom we have done our best to strip it of its humanity leaving only equations, theories, and design guidelines.

    I suspect partly to blame were educational guidelines dictating content coverage. With so much content to cover, and so little time available there’s nothing left over for discussing issues not specified by the guidelines. Luckily, the guidelines may be changing for the better. In 1996 ABET adopted a new set of standards called Engineering Criteria 2000 (EC2000), which, in addition to stating the importance of a solid technical background, emphasize the necessity of developing more professional skills such as communication and awareness of ethical and contextual issues.

    Unfortunately, the typical institutional response to adopting these new guidelines is to create additional required courses, such as “Engineering Ethics” and shift around content in others to keep the credit count down to a (not necessarily) reasonable number.

    So even if accreditation agencies like ABET were to explicitly list diversity and inclusion issues as required topics in an engineering curriculum we might expect an institutional response in the form of additional courses like Women in Engineering, Queer History of Computing, People of Color in Mathematics, etc. Ignoring the impossibility of adding even more credit hours to an already overflowing 4 year program the problem with this approach is, I think, the same one that has kept the engineering classroom from being as diverse as the university campus it is on.

    By neatly partitioning “diversity and inclusion” into separate courses they would be removed from the rest of the “traditional” curriculum (much like computers were removed from the classroom and quarantined in computer labs in the 1980s). Learn about the significance of a Turing machine at 9am and learn about Turing the man at 10am in another class. Some learners would no doubt make the connection and go on to explore the intricacies of how personal identity might affect ones engineering work, but many would view this the same way I’ve heard people talk about the “Engineering Ethics” course: something extra, kind of cheesy case studies, doesn’t seem relevant to my own personal and professional identity.

    Clearly, I don’t think separating all these important topics into additional courses is the way to go, but the converse: having no courses at all but rather a vast open environment which individuals explore on their way to completing a degree, is… well, right now just a dream. But in the mean time, I think we’d be off in the right direction if we made a point of acknowledging that engineering is a human tool created by individual humans and used by humanity.

    When we talk about the power of abstraction, spend some time on a discussion of just how radical Ada Lovelace’s thoughts that computers could be used for art were when, in the 1800s the only role of computational machines was to perform numerical calculations.

    When learning about compilers and feeling that rush when you run your first program

    print("hello world")

    knowing that that high level human-readable command was translated into machine code that the processor could understand and open up a discussion about a time when humans had to write in machine code directly until Grace Hopper realized that the power of the computer included the ability to translate human-readable language into machine language automatically and by inventing the first compiler opening up computer programming to a much larger audience.

    And of course, when it comes time to discuss the theory of computation, reflect on the significance of Alan Turing’s suicide, the British parliaments refusal to grant a full posthumous pardon of his charges of “gross indecency” until 2013, and how any contemporary examples of discrimination, exclusion and marginalization may impact the future of engineering, which is after all, the future of humanity.

    Are there any technical topics from your own discipline that could be enriched by acknowledging the human component of engineering?

  • Writing In Public: Finishing a MS Thesis paper is hard


    Those of you who know me know that I’ve been ostensibly working on an Ph.D. MS degree for a very long time. Some might say too long. Others would say just the amount of time I needed to figure out where my passions lie and make appropriate mid-course corrections to my career trajectory. Now that I have made appropriate corrections I’m faced with finishing up a MS thesis paper and defending it in a couple of months so I can start a new chapter with a clear conscience (and the satisfaction of actually finishing something). Despite all the helpful advice to “just get it done, you’ll be so much happier!” I have found “just getting it done” to be incredibly challenging. Recently, Ennis suggested that I “write in public“ as motivation.

    I have a love/hate relationship with my MS research. A lot of the hate frustration comes from a mismatch of values between myself and my advisor. The parts of the project I really enjoyed and am proud of were the overcoming the electronic design challenges and just getting the damn thing to work reliably (enough). Unfortunately, that wasn’t the point of the research project, just a necessary prereq and as such I never wrote up a paper about the parts of the project I was most interested in and have continued to feel like those parts have been more or less forgotten/devalued by my advisor, though I know at a rational level this isn’t the case and in the land of academia, non-published work is generally regarded as being useless.

    .a history of self-sabotage

    After some reflection I’ve noticed a pattern of repeatedly turning my educational/career trajectory away from things that interest me, possibly from the subconscious thought that jobifying an interest may suck the fun out of it.

    A Series of Rather Unfortunate Events

    The high level objective of the project was to design a tracking algorithm that would estimate the position and velocity of a moving, noisy object in the water. Ultimately, we were to also design a control algorithm that would move our own vehicles in a way relative to the moving object that was “optimal” in some vaguely defined sense (vaguely defined on purpose to avoid the hassles of conducting research involving classified information). For reasons stemming from an organizational breakdown, funding ran out before we were able to develop a full solution, as a result all other team members were moved to new projects and it was left for me to turn what I had into a Ph.D. topic. After several years, I realized this was an unrealistic goal and decided to “simply” turn existing work into an MS thesis.

    .motivation by association

    I have also come to learn that I can be highly motivated to work on and complete a project that I’m not necessarily passionate about if I’m working with a team of passionate individuals who depend on my contributions. While I had ethical qualms with the project from the start, I was able to get my work done in a timely fashion in part because my motivation came from wanting to help my team mates succeed. When I no longer had teammates, that source of motivation dried up.

    Finding an anchoring idea

    I knew my final paper should tell a story, but for the longest time I didn’t have a clear idea of just what that story would be. I built some stuff. I wrote some code. I wrote some more code. I got things to mostly work. There were problems quantifying how well everything worked though, and comparing my final solution to other possible solutions.

    I learned that there was a rather interesting history to the problem I was attempting to solve, a search for scholarly articles on “bearings-only trajectory motion analysis“ brings up articles referencing work as early as the 1960s. I found it immediately self-validating to find that many of the troubles I had with finding a reliable solution were indicative to this particular class of problem.

    State estimation with measurements from a towed linear hydrophone array

    Here’s the general problem, with some more technical details. We wish to estimate the state (position and velocity) of a moving object, modeled as $$x = \begin{bmatrix} x & y & \dot{x} & \dot{y} \end{bmatrix}\prime$$

    \[ x[k+1]=Ax[k] + \Gamma\omega[k] \]


    \[ A= \begin{bmatrix} 1 & 0 & T & 0\\ 0 & 1 & 0 & T\\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1 \end{bmatrix} \]

    \[ \Gamma = \begin{bmatrix} T^{2}/2 & 0\\ 0 & T^{2}/2\\ T & 0\\ 0 & T \end{bmatrix} \]

    where $$k$$ is the current time step and $$T$$ is the sampling interval. The random variable $$\omega[k]$$ models process noise, in this case fluctuations in target acceleration, and is considered to be zero-mean with covariance $$Q[k]$$:

    \[ \begin{aligned} \E{\omega[k]} &= 0\\ \E{\omega^{2}[k]} &= Q[k] \end{aligned} \]

    All this means is that the object we are trying to track has a velocity that fluctuate randomly, but on average, it remains constant. This is a pretty simple motion model and not terribly interesting from an analysis standpoint, anything that moves in a straight line at constant speed follows this model. Things get quite a bit more interesting if the object is able to change direction at will but we’re not concerned with that case for the purposes of my work.

    Keep in mind, $$x$$ is entirely unknown, and a function of time, and we wish to estimate it. To do this, we must measure some aspect of the state. The sensor we build measures the angle from itself, to the noisy object. We use $$z[k]$$ to denote the measured value of state $$x$$ at time $$k$$.

    \[ z(x) = h(x) + \nu(x) \]

    Where $$\nu$$ is zero-mean Gaussian noise and

    \[ \begin{aligned} h(x) &= \begin{cases} \gamma(x) - \halfPI, & 0 \leq \gamma(x) \leq \pi\\ \frac{3}{2}\pi-\gamma(x), & \pi < \gamma(x) < 2\pi \\ \end{cases}\\ \gamma(x) &= \arctan\left( \frac{(x-q)^{\tran} e_N }{(x-q)^{\tran} e_E } \right) - \psi \end{aligned} \]

    The two important bits of this equation are that

    1. the measurement is the angle to the target, a non-linear function of the state x, and
    2. For a particular value of $$z(x)$$ the object could be along any point of two rays extending from the sensor.

    The Common Approach

    One very common way to implement state estimation is using a recursive Bayesian estimation(RBE) framework. If we can make certain assumptions about our system, then we can use a Kalman Filter, an application of RBE.

    Unfortunately, my system doesn’t quite meet the requirements to use a Kalman filter:

    1. The system and measurement models are not both linear (the measurement equation is non-linear)
    2. The measurement noise model is Gaussian with state-dependent variance.

    The first snafu is one that is quit common and is usually addressed with the extended Kalman filter (though, as I found out, the nature of the non-linearity in my system presents some challenges, even for the EKF). The second problem is apparently not as commonly dealt with which led a project colleague to develop a generalized EKF (GKF) to handle this situation.

    But does it work?

    .research in reverse

    In retrospect, I could have saved lots of frustration and unknown hours of looking for bugs that didn’t exist if I had done a literature review first. Due to the nature of funding in our lab, we typically first design and build hardware, program software, build a working prototype and collect field data before establishing what the actual research topic is and finally commence the writing phase. It would probably be worthwhile to find a way to circumvent the time pressures imposed by funding dates to do a comprehensive literature review before trying to fix problems that may have been addressed by others.

    My task was to implement this new GKF and show that it performed better than a standard EKF implementation for our particular system.

    Since the EKF and thus the GKF don’t have closed-form solutions, performance analysis is done numerically. I did this by running Monte-Carlo simulations of both configurations and plotting the mean estimation error for each filter. The conclusions I’ve had have always been a little weak: I present a plot something like

    error plot

    and vaugly say that the gkf appears to have better error performance. The percentages in the ledgend represent the divergence rate: the ratio of trial with estimates that converged to the true value. This number also appears to support the claim that the gkf has better performance (a higher success rate), but as of yet I haven’t discussed statistical significance. Is a 5% increase really a significant change? How likely is it that it could just be the result of chance?

    These are the questions I’m addressing now by performing more rigorous statistical analysis and hypothesis testing of the results.

  • Computer culture transforming education?


    And in teaching the computer how to think, children embark on an exploration about how they themselves think. – Seymour Papert, From Mindstorms

    Mindstorms: Children, Computers and Powerful Ideas was written by Seymour Papert in 1980 at the dawn of the personal computer revolution. I had the pleasure of reading an excerpt from the book published in The New Media Reader for a class. I immediately identified with what he wrote about programming as a tool to think about how to think. He predicted that the future ubiquity of computers might spark a revlotion in the learning process. Unfortunately, I was also struck by how little of what he hoped for has come to pass, even over 30 years since the writing, while his predictions of the increase in affordability, ubiquity and power of personal computers has come to be a reality. His critics have largely been correct in their predictions that computers would be used to replace exiting teaching practices of drills and tests rather than be exploited to change the way learning and teaching was done. But why?

    The challenges were not technical, Papert demonstrated practical applications of his ideas with the LOGO programming language and a mobile drawing device known as “the turtle” in several successful pilot cases in the early 80s, during which time computers were slowly trickling into the classroom. So why didn’t this ideas take hold?

    Papert’s vision for a computer driven transformation in education probably sounded quite insane to many when he started writing about children using computers for learning and creativity in the 1960s, when computers still filled rooms and were hardly “personal.” Yet, he predicted with advances in technology and manufacturing, computers would become so cheap that every child could own one.

    He saw the computer as a tool to think about thinking. To program a computer to do a useful task, one must have the ability to reason about the task, break it down into discrete steps, think about how they would complete the task, and then think about how the computer can be told to complete it using the commands it can understand. This idea is nothing new to any programmer alive today: knowing the words of a programming language is about as useful as knowing just the vocabulary of a spoken language. To turn words into meaningful sentences the speaker must be aware of context, nuance, and above all, be able to model, if even subconsciously, how the receiver of her words will be interpreted by a listener.

    Unfortunately, while most people today walk around with more computing power in their pocket than landed on the moon in 1969, they don’t see their personal computers the way Papert does. Computers (and I’m including mobile devices) have become so ubiquitous, with so many layers of software between the hardware and the user that most users aren’t thinking of the device in their hands, or in front of them on their desk as a powerful, general purpose computational machine. They view computers as a phone, or an address book, or a word processor (which, let’s be clear, most offer little more functionality than a typewriter with a “save” feature).

    Yes, computers and the internet have transformed the way we interact with each other, with our data, and have transformed the way we do business. But they have yet to have any kind of large scale impact in the way we learn or think. While we started to integrate computers in the classroom in the early 80s it didn’t take long before the system reacted to this intrusion and attack on the status quo by confining computers to the computer lab, out of the classroom, physically separated from the learning environment which Papert hoped they would transform. In the labs computers are used mainly to take notes, solve equations, plot data, or simulate a typewriter to write papers. All tasks we were already doing, albeit much slower.

    MOOCs are certainly a new experience, made possible by ubiquitous computing and the internet, but while MOOCs have certainly allowed many people to attend classes who might otherwise not be able to attend a brick and mortar institution, they leverage the awesome technology now available as just a tool to allow easy scaling of the lecture/exam model, rather than exploiting the potential to engage our creativity and extend and challenge the way we think and solve problems.

    Even with all this technology at our finger tips, we are still stuck with the method of education invented at a time when books were so expensive that a university could afford to buy a copy, and professors would read the book back to a classroom of students. It should come as no surprise that our word “lecture” comes from the Latin lectus¸ past participle of legere, “to read”.

    Papert recognizes this tendency to use new technology to mimic old technology in a piece he wrote as an appendix to a Proposal to the National Science Foundation in 1972.

    There must in the world be tens of thousands of people struggling to understand what happens in a classroom where children are asked to do sums with pencil on square paper. Some of them try to improve matters by having the children do the same sums on computer terminals.

    Unfortunately, this is the world we now live in, one which Papert warned us about if we didn’t act quickly:

    Time is of the essence. New technologies will diffuse into the schools and in the absence of available well-tried methods, untried ones will become established. And if this happens society will have to pay for one more expensive case of that bind which paradoxically places the major cost of innovation in the undoing of the old rather than the construction of the new. – http://www.papert.org/articles/AnEvaluativeStudyofModernTechnology.html

    What ended up happening was just what many skeptics feared. Like a live organism fighting off a foreign threat, the school environment quickly quarantined computers, removing them from the classroom and isolating them as their own topic:

    Thus, instead of cutting across and so challenging the very idea of subject boundaries, the computer was now defined as a new subject; instead of changing the emphasis from impersonal curriculum to excited live exploration by students, the computer was now used to reinforce standard hierarchical thinking. What had started as a subversive instrument of change was neutralized by the system and converted into an instrument of consolidation. The development of computer labs seems to me a kind of immune response by School to a foreign body; the logic of the process was to bring the intruder back into line with School’s ways. – The Children’s Machine

    Now the big question: how do we fix it? We are so comfortable using computers to grill, test, and practice it’s hard to convince anyone to do anything else with them. Attacked by the “QWERTY syndrome“ from one side, and shrinking budgets on the other, those of us who hope for innovation fear we are fighting an uphill battle. Better funded schools would help, certainly, as well as a reduction in [the emphasis on standardized tests][problems-created-by-standardized-testing].

    There’s more freedom to get creative and take more risks in teaching at the college level, but by that time we’re faced with first convincing learners to unlearn bad habits and assumptions about how to solve problems, which is a tough task for any educator, even more challenging given the time constraints faculty have when juggling their research, teaching and service responsibilities.

    So this is another one of those “I don’t know what the best solution is”, but I have a feeling it’s going to take many of us working together to shake up the system a bit, integrate computer science as a core component at all educational levels and tackle some of the new pedagogical challenges that computers/technology have presented.

  • Samples, Statistics, Central Limit Theorem, oh my!


    Continuing the theme from last time, I’d like to take a closer look at another final project from this past semester’s ECE2524. Just like the project I reviewed last time, the central limit theorem calculator works as advertised, but it requires quite a bit of interactivity which makes it difficult to use in a script. Providing a good non-interactive interface is particularly important for this project: one of the assumptions needed to use the results of the central limit theorem (CLT) is that the sample size is greater than 30 or 40. Depending on the data, a sample could contain hundreds of measurements and the current program prompts the user for every single one.

    Oh, statistics. Notorious for adding confusion and frustration to the lives of many a college student, I want to first praise the decision to focus a final project on a topic like the Central Limit Theorem. Writing a program to solve a problem is a great way to gain a better understanding of a new and complex problem space, while at the same time providing practice in the technical skills and tools to solve it.

    ([Central Limit Theorem]

    Quickly summarized, the CLT states that under certain conditions, statistics generated from samples taken of an arbitrary population can be assumed to be taken from a Normal (Gaussian) distribution. This is a powerful theorem:

    1. many populations we are interested in are not best represented as a Normal distribution, and what’s more, the true population distribution is often unknown.
    2. A lot of mathematical theory and tools exist to work with statistics of Normal distributions.

    When we are working with a data sample, we are often concerned with out well the sampled data represents the population which we are interested in. A [confidence interval] is used to express the degree of uncertainty (conversely, confidence) that a sample statistic is good estimate of a population parameter.

    The Code

    The original implementation depends on quite a few prompts for data making the program difficult to use in a script. Instead of interactive prompts, Unix developers favor command line options as a means of supplying initial parameters to a program. Currently the program prompts for:

    • sample size
    • each data point
    • standard deviation
    • a confidence level

    From this the program computes and displays the confidence interval calculated from the given data.

    First, note that some of the data requested from the user is redundant:

    • the sample size is a function of the data set
    • the standard deviation is a function of the data set

    If the program expects to read one sample per line of input, the number of input lines is the sample size, so we shouldn’t prompt the user for that information. The standard deviation of the sample can be easily calculated from the data as well, so we don’t need to prompt for that. Finally, as I mentioned, it is limiting to expect a user to manually enter data samples, often will be interested in running calculations on many samples, each with potentially hundreds or even thousands of data points. This data will often be coming from another file, or another program. If we modify our program to read data samples from standard input we can easily read data from just about any source.

    Let’s convert the void addSampleData(vetor<double> &sample, int sampleSize) function to vector<double> readSamples(istream &is). I’ll put this code in a file named sample.cpp and create a corresponding file sample.hpp with the function declaration.

    ([move semantics] Don’t worry about returning an object from the function like this, with C++11 move semantics the data isn’t copied, instead the ownership of the object is transferred from the function to the caller.)

    /* sample.cpp */
    #include <iostream>
    #include <vector>
    using namespace std;
    vector<double> readSample(istream& is) {
        vector<double> samples;
        string line;
        char *endptr;
        while (getline(is, line)) {
            double s;
            s = strtod(line.c_str(), &endptr);
        return samples;

    Note, I’ve left out error checking around strtod for clarity. Conveniently, we never have to deal with an explicit ‘size’ variable, when we need the sample size we’ll just call std::vector::size. Because our function takes a reference to an istream as an argument it will happily read in data from any source that can be made to look like an input stream, whether it be from standard input, a file provided on the command line, or even a network socket. A simple program that just reads in samples from standard input might look like

    #include <iostream>
    #include <algorithm>
    #include <vector>
    #include "sample.hpp"
    using namespace std;
    int main(int argc, char* argv[]) {
        std::vector<double> sample;
        double mean, var;
        sample = readSample(std::cin);
        mean = accumulate(sample.begin(), sample.end(), 0.0) / sample.size();
        vector<double> res(sample.size());
        transform(sample.begin(), sample.end(), res.begin(), [mean](double d) { return (d-mean)*(d-mean); } );
        var = accumulate(res.begin(), res.end(), 0.0) / (sample.size()-1);
        cout << mean << ", " << var << endl;
        return EXIT_SUCCESS;

    Notice the use of accumulate and transform provided by <numeric> and <algorithm>. Since computing the arithmetic mean and variance are common calculations, we might want to break these out into functions. While we are using a std::vector<double> to represent a sample, we want to impose as few assumptions as possible on input data for our functions, they should operate on anything that can be iterated over, so we’ll follow the same structure used by functions provided by `‘ and accept templated iterators as arguments.

    /* stat.hpp */
    #ifndef _STAT_HPP_
    #define _STAT_HPP_
    #include <algorithm>
    #include <numeric>
    namespace stat {
        template<typename InputIterator>
        double mean(InputIterator first, InputIterator last) {
            return std::accumulate(first, last, 0.0) / distance(first, last);
        template<typename InputIterator>
        double variance(InputIterator first, InputIterator last, double mu, int bias=1) {
            std::vector<double> res(std::distance(first, last));
            std::transform(first, last, [mu](double d) { return (d-mu)*(d-mu); });
            return std::accumulate(res.begin(), res.end(), 0.0) / (distance(first, last) - bias);

    The int bias argument is used to provide flexibility to use our function to compute both population variance (bias=0) and sample variance (bias=1).

    Now that we have nice, modular and orthogonal functions to calculate mean and variance, what might we want a program that reads in samples to do with them?

    Confidence Intervals

    Watch Confidence intervals on Khan Academy and see [What is a Confidence Interval?] for an understanding of what a confidence interval means.

    To compute a confidence interval of a statistic (e.g. mean) we need

    We can then calculate the margin of error and finally the [confidence interval].

    ([symbols] $$\mu$$ : population mean

    $$\sigma$$ : population standard deviation

    $$\bar{x}$$ : sample estimate of $$\mu$$

    $$s$$ : sample estimate of $$\sigma$$

    $$n$$ : sample size )

    If we want the confidence interval of the mean, then our statistic is the mean, which our program already calculates from the sample data. The standard error is

    \[ SE = s/\sqrt(n) \]

    Where $$s$$ is the sample estimate of $$\sigma$$ which our program already calculates in the form of the variance with stat::variance setting bias=1

    A nearly complete program to calculate a confidence interval of a sample might look like

    #include "stat.hpp"
    int main(int argc, char *argv[]) {
        vector<double> sample;
        double smean, svar, cstdev, cval;
        char* endptr;
        cval = NAN;
        int opt, ssize;
        cval = NAN; /* TODO: calculate critical value */
        sample = readSample(cin);
        ssize = sample.size();
        smean = stat::mean(sample.begin(), sample.end());
        /* With default delim=1 this calculates the best
         * estimate of the sample distribution variance given
         * this sample */
        svar = stat::variance(sample.begin(), sample.end(), smean);
        /* standard error */
        cstdev = sqrt(svar/ssize);
        /* margin of error = critical value * standard error */
        double merror = cval * cstdev;
        /* confidence interval = sample statistic ± Margin of error */
        cout << smean << "\t" << merror << "\t" << smean-merror << "\t" << smean+merror << endl;

    We use our readSample() to read a list of sample data from standard input, from this we also get the sample size. Our two stat:: functions take care of calculating the sample statistics and the standard error is just a simple calculation from these values.

    But how do we find the criticle value, cval? For an integer confidence level $$c$$, the critical value is x such that

    \[ CDF(x) = .5 - \alpha/2 \]

    Where $$\alpha = c/100$$ and $$CDF(x)$$ is the cumulative distribution function for the sample distribution, which due to the central limit theorem we can assume to be Normal. Unfortunately, solving for ‘x’ isn’t straightforward, which is why in practice we just use a look-up table (of course, the numbers in the table have to come from somewhere, but we’ll come back to that later). Just like for the resistor color code calculator, it makes a lot of sense to create a look-up table in the code. Not only does this increase transparency, it puts all the data in one, easy to varify and modify place. If the assumptions behind the central limit theorem are met, and our sample size is greater than 30, we can assume our sampling distribution is normal and use a z-table to look up our critical values.

    ([small sample size] If our sample size is less than 30 our sample statistics are best modeled coming from a Student’s-t distribution and we would need to use a t-table instead )

    #include <cmath>
    typedef struct {
        /* because checking floating point equality is tricky, store the
         * confidence as int(confidence_percent*100) */
        int conf;
        double zval;
    } ztable_entry;
    static ztable_entry z_table[] = {
        { 99, 2.58  },
        { 98, 2.33  },
        { 95, 1.96  },
        { 90, 1.645 },
    #define Z_TABLE_LEN  sizeof(z_table)/sizeof(z_table[0])
    /* return the critical value for a given confidence interval */
    /* If `conf` is not found in the table, NaN is returned */
    double critval(int conf) {
        for(size_t i=0; i < Z_TABLE_LEN; ++i) {
            if (z_table[i].conf == conf) {
                return z_table[i].zval;
        return NAN;

    The remaining piece of our program is to somehow get the desired confidence level from the user. In the original project the program prompted the user interactively. While this works and is user-friendly, it isn’t scripting friendly, and considering you will probably want to do something with the interval once you have it, and that the sample data could very well be coming from another program, we want this to be as scriptable as possible, so we’ll follow the Unix convention and get the confidence level from a command line parameter, using getopt to handle option parsing. I won’t go into details of using getopt here, inspect the source for details.

    Now we can use our program to calculate confidence intervals for various data sets. I generated some example data based on Problem 1 on the stattrek confidence interval page in a file named weight_data

    $ ./stats < weight_data
    180, 29.2793
    $ ./cint -c 95% < weight_data
    180     1.81475 178.185 181.815

    The output data contains tab delineated values of the sample mean, the margin of error, and then the sample mean ± the margin of error. The results closely match those given at stattrek (note that the sample data doesn’t have exactly the same statistics as those used in the sample problem!)

    My modified code is available on GitHub.

  • Decrease Complexity, Increase Usability


    Every semester I am impressed by the variety of final projects I get for ECE2524. Games are always popular (and always fun to play), and so the non-game projects tend to stand out. Two from the Spring 2014 were a utility to convert resistor values to color codes and a program that calculates the sample mean and variance of a set of data points. Both of these programs are great examples of the “do one thing and one thing well“ rule that Unix developers like to follow. In both cases the central algorithm is relatively straight forward as well, and so a “strong single center“ design is easy to define. It is for these reasons that I’d like to look at each project in more detail than I was able to during the review process. I’ll start with the resistor color code calculator.

    A Crash course in resistor values

    Resistors are measured in units called ohms. The resistance on an axial-lead resistor is coded in colored bands printed on the resistor body. To read the value of a resistor orient it so that the gap between bands is to your right, then read from left to right. The first through penultimate band before the gap translate directly into decimal coded values and the last band before the gap determines a multiplier value.

    ([aside] this sequential ordering will come in handy when writing a program to convert resistor values)

    color value multiplier
    black 0 100
    brown 1 101
    red 2 102
    orange 3 103
    yellow 4 104
    green 5 105
    blue 6 106
    violet 7 107
    grey 8 108
    white 9 109
    gold -
    silver -

    Notice the relationship between the value and multiplier of a particular color. This makes the chart a little easier to remeber (just use your favorite neumonic device to remember the color sequence “black, brown, red, etc.” and the rest follows. Given a color band sequence of “orange red brown” would give you “3” and “2” from the first two bands, for a value of “32”, then multiplied by “10” from the brown third band for a final value of 320Ω. If the third color were orange instead of brown then the value would be 32kΩ.

    A program to convert numbers to color sequences

    What would we want a program that calculates resistor color codes to do for us? Some possibilities:

    • check the color code for a single resistor value during breadboarding
    • convert a list of resistor values (perhaps from a bill of materials file) to color codes to help in selecting appropriate parts before breadboarding a project.
    • convert color code values to resistances to aid in sorting resistors back into appropriate storage bins.
    • some task we can’t imagine until we are faced with it.

    Let’s compile the original source code and see what we can do

    $ clang -Wall -Wextra -o res main.c

    There are no warnings, which are great, let’s run the program with a couple different resistor values.

    $ ./res 360
    First Band: Orange
    Second Band: Blue
    Third Band: Brown
    $ ./res 47000
    First Band: Yellow
    Second Band: Violet
    Third Band: Orange

    The output is easy to read which makes it easy to confirm it is correct, however, it is a little more verbose than it needs to be.

    The Rule of Silence states that we should only output what is critical to interpret the data, in this case, that is the sequence of three colors. A hint that the current output is verbose is provided by thinking how you would communicate a resistor color code to someone else: would you say “grab a resistor with an orange first band, blue second band and brown third band”, or would you say, “grab a orange, blue, brown”? Anyone who has spent some time breadboarding quickly learns that the shorter version is easier, and provides all the information needed in the context. We know we’re talking about resistor color bands (that’s the point of the program), and we know they appear in a sequence on the resistor and we expect the sequence to be mirrored in the output of our program, so

    $ ./res 360
    $ ./res 47000

    provides all the information we need. Also consider how much easier it is for your eyes to parse the output, imagine refering to an outpur format like that over and over again as you are working on a project. It’s quicker for your brain to pull out the relevent information and so it will be less draining to use a tool that produces output like that over an extended period of time than the original output. Making this change to the code involves removing the three printf statements that print the extra text. I have tagged this revision v0.alpha1.

    But what if we want to convert more than a single value? The power of computers and programs comes from their ability to perform the same action over and over again, quickly and reliably. Even relatively simple calculations become tedious for us to perform manually if we have more than a handful, so we would like our program to handle this for us. The two most common methods for passing a list of values to a program are through its command line arguments, or by writing to its standard input. The current program doesn’t handle either of these methods:

    $ ./res 360 47000 
    Usage: ./res value

    And considering a file named values with

    $ ./res < values
    Usage: ./res value

    Taking a look at the code, the sequence of parsing a value, performing the calculations and producing output are all defined inline in the main function. We could put the whole block in a loop but it will be easier to work with if we first refactor to create a function:

    int valueToColor(const char* value);

    We will use the int return value as an error status following the convention of using a value of 0 to mean success and a non-zero value to indicate something went wrong. For now we will only distinquish between these two cases and in the event of an error just print out the usage message. I have refactored the code into functions and tagged this revision v0.alpha2.

    Now we are in a good position to apply a standard line-oriented filter interface pattern:

    int main(int argc, const char* argv[]) {
        if (argc > 1) {
            /* for each argument in argv */
                /* parse string */
                /* run algorithm */
                /* print output */
        } else {
            /* for each line of input */
                /* parse line of input */
                /* invoke algorithm */
                /* print output */

    The interpretation of positional command line arguments as raw values instead of file names is a little non-conventional, I’ll come back to that later.

    I will use getline(3) to read lines of input, following the example usage provided in the man page. Making these changes, tags as v0.alpha3, allows for the following usage scenarios:

    $ ./res 360 47000
    $ ./res <values

    Now we’re faced with the challenge of parsing out individual records from the output. Since most Unix text tools are line-oriented, it is common practice and preferable to make output formatted as one record per line whenever reasonable. In this case, it is very reasonable. To make this job easier, and make the output format clear and easy to change in the future, I am going to modify the getColor* functions to return string constants instead of calling printf from within the functions. Then the output will be performed by a single printf statement in the valueToColor function:

    int valueToColor(const char* value) {
        /* conversion and error checking */
        printf("%s %s %s\n", getColor1and2(digit1), getColor1and2(digit2), getColor3(val));
        return 0;
    $ ./res <values
    orange blue brown
    yellow violet orange

    I’ve also change the output to be all lowercase, as the use of capital letters wasn’t adding any information. This revision is tagged v0.alpha4.

    Now we have the flexibility of converting an arbitrarily long list of resistor values into their color code equivalents. Why would we want to do this? Consider a simple lowpass filter schematic I created in gsched:

    lowpass filter schematic

    I can generate a bill of materials file by running

    $ gnetlist -g partlist1 -o output.bom active_lowpass.sch

    which creates a file named output.bom with

    ..refdes        device  value   footprint       quantity
    C1      CAPACITOR       0.1u    unknown 1
    C2      CAPACITOR       0.1u    unknown 1
    R1      RESISTOR        10k     unknown 1
    R2      RESISTOR        10k     unknown 1
    RA      RESISTOR        10k     unknown 1
    RB      RESISTOR        5.5k    unknown 1
    U1      OPAMP   unknown unknown 1
    $ grep 'RESISTOR' test/output.bom | cut -f 3

    The values are in metric notation (k means kilo, which means to multiply the value by 103), our program can’t handle this notation just yet, for the time being I have created a program called metric that reads metric formatted strings on standard input and converts them to pure numeric values on standard output:

    $ grep 'RESISTOR' test/output.bom | cut -f 3 | ./metric

    now this is in a format our res program understands

    $ grep 'RESISTOR' test/output.bom | cut -f 3 | ./metric | sort -rn | ./res | uniq -c
          3 brown black orange
          1 green green red


    I numerically sorted the output of our res program and piped the output to the uniq command to count the number of unique lines, from this we can easily read that we need three “brown black orange” resistors and one “green green red” resistor to build this circuit.

    This demonstrates the power of designing your utilities to fit the filter interface pattern, and to follow certain conventions such as “one record per line”. If you do that, you can easily use your program in pipelines, possibly to solve a problem completely different form the one that originally lead you to write the program in the first place!

    Transparency and Clarity

    The first thing to note, when we convert between color code and resistor values we refer to the chart above to look up values, assemble digits into numbers and then multiply by a value. A relatively simple calculation, and it would be natural to make our program mimic this procedure.

    If we inspect the code as it is, we see a couple of large if/else if blocks:

    const char* getColor1and2(int i) {
        if (i == 0)
            return "black";
        else if (i == 1)
            return "brown";
        else if (i == 2)
            return "red";
        else if (i == 3)
            return "orange";
        else if (i == 4)
            return "yellow";
        /* more */


    const char* getColor3(long i) {
        if (i / 100000000 > 0)
            return "violet";
        else if (i / 10000000 > 0)
            return "blue";
        else if (i / 1000000 > 0)
            return "green";
        else if (i / 100000 > 0)
            return "yellow";
        /* more */

    In the first case, it is easy to see that all the if/else if structure is doing is returning a color corresponding to a integer value. In both cases the relationship between colors and values is embedded into the logic of the code. The Rule of Representation favors moving complexity from logic to data structures, plus we’re used to looking at table when performing this calculation by hand, so let’s put a data table in our code:

    typedef struct {
        const char* color;
        int sig;
        double multiplier;
    } colorcode_t;
    static colorcode_t color_table[] = {
        { "black",  0,  1   },
        { "brown",  1,  1e1 },
        { "red",    2,  1e2 },
        { "orange", 3,  1e3 },
        { "yellow", 4,  1e4 },
        { "green",  5,  1e5 },
        { "blue",   6,  1e6 },
        { "violet", 7,  1e7 },
        { "grey",   8,  1e8 },
        { "white",  9,  1e9 },
    #define COLOR_TABLE_LEN  sizeof(color_table)/sizeof(color_table[0])

    This is a good application of the SPOT rule: all of our data is defined in one, and only one, easy to find location. Before we had literal color strings sprinkled across two functions with duplicate occurrences of each string and obfuscating the relationship between multiplier values and digit values.

    Now, our getColor1and2 function can be renamed to getColor:

    const char* getColor(int i) {
        return color_table[i].color;

    For clarity, I’m omitting a bounds checking on i. Notice that the logic has been completely replaced with a simple array index retrieval. The Rule of Optimization tells us we shouldn’t concern ourselves with speed, but it’s worth pointing out that this function is now O(1) while the if/else if method was O(N).

    Let’s take a look at getColor3 now. What is the relationship between the multiplier colors and the value? With the data in table form, it’s apparent that if the multiplier is in base 10 exponential form then the exponent determines the color in exactly the same way any single digit value does. For example, ‘360’ is ‘36 * 101‘, and the color for the multiplier is “brown”, for value 1.

    One last helper function

    void frexp10(double x, int *digits, int *exp) {
        char s[10];
        snprintf(s, sizeof(s), "%.2E", x);
        digits[0] = s[0] - '0';
        digits[1] = s[2] - '0';
        digits[2] = s[3] - '0';
        *exp = strtol(s+6, NULL, 10);

    The "%.2E" format string will convert the double into a string representation d.ddE±dd where d is a single digit between 0 and 9. We extract the three digits from the significand and convert the exponent to an integer value. Look what happens to our valueToColor function when we make use of this function:

    ([todo] this would be more orthogonal if we returned a data structure with the color information, rather than calling printf from within this function.)

    int valueToColor(const char* str) {    
        char* endptr;
        int digits[3];
        int exp;
        double value;
        value = strtod(str, &endptr);
        frexp10(value, digits, &exp);
        printf("%s %s %s\n", getColor(digits[0]), getColor(digits[1]), getColor(exp-1));
        return 0;

    ([note] We need to subtract 1 from the exponent our frexp10 gives us since the format given by "%.2E" coerces the number into scientific notation format d.dd)

    This is our “strong single center“: the well-defined computation that is at the core of our program. Our main function is just a light wrapper around this function providing support for iterating over lines of input or arguments of the argv array.

    My final revision makes use of the code from the metric program to handle metric formated strings such as "4.7k" and "1M", so the program will be able to handle resistor values in standard formats, such as those used in the bill of materials:

    $ grep 'RESISTOR' test/output.bom | cut -f 3 | sort -rn | ./res | uniq -c
          3 brown black orange
          1 green green red

    The use of the color_table and refactoring the algorithm code into resistor.c sets the stage for a dual program that converts color codes to numerical values.

    One more thing

    There’s one last rule we could bring into play, though for this particular application it doesn’t save us much, but the same idea could prove invaluable in another setting.

    The Rule of Generation suggests we should “write programs to write programs”. What part of the program seemed the most formulaic and seemed to have some redundant information?

    static colorcode_t color_table[] = {
        { "black",  0,  1   },
        { "brown",  1,  1e1 },
        { "red",    2,  1e2 },
        /* more */

    The table is tedious to write up, the second column is simply the 0-indexed line number and the third column can be generated from the second. Now, in this case, we know we only have 9 values, 11 if we account for “gold” and “silver”, and this information isn’t likely to change, so it wasn’t that bad to write by hand. But, you may find yourself in a situation that requires a large look-up table, with numerous pre-computed values, each one an additional potential source for a typo.

    Traditionally, awk has been the utility of choice for this type of code generation, though I have to admit, I don’t use it regularly and had to do a good bit of review before coming up with this working script:

    #!/usr/bin/awk -f
    BEGIN   {  }
    NR == 1 { 
               print "#define COLOR_TABLE_START_INDEX " $value "\n"
               print "static colorcode_t color_table[] = {"    
                expo=NR - value - 1
                printf ("   { %10s, %2d, %2.0e },\n", "\"" $1 "\"", expo, exp(expo*log(10)))  
    END     { 
                print "};\n"
                print "#define COLOR_TABLE_LEN  sizeof(color_table)/sizeof(color_table[0])" 

    Now, given a text file color_codes.txt containing

    black 0 

    We can process this input with our awk script to generate the C-code for our lookup table:

    $ awk -f color_codes.awk < color_codes.txt
    static colorcode_t color_table[] = {
       {    "black",  0, 1e+00 },
       {    "brown",  1, 1e+01 },
       {      "red",  2, 1e+02 },
       {   "orange",  3, 1e+03 },
       {   "yellow",  4, 1e+04 },
       {    "green",  5, 1e+05 },
       {     "blue",  6, 1e+06 },
       {   "violet",  7, 1e+07 },
       {     "grey",  8, 1e+08 },
       {    "white",  9, 1e+09 },
    #define COLOR_TABLE_LEN  sizeof(color_table)/sizeof(color_table[0])

    To incorporate this step into our build process we just add the appropriate entries to the Makefile

    resistor.o: color_table.h
    color_table.h: color_codes.txt color_codes.awk
        awk -f color_codes.awk < $< > $@

    and #include "color_table.h" in our resistor.c source file in place of the hand-written table and update our lookup function to use COLOR_TABLE_START_INDEX:

    const char* getColor(int i) {
        size_t idx = i - COLOR_TABLE_START_INDEX;
        if ( idx >= COLOR_TABLE_LEN )
            return "\0";
        return color_table[idx].color;

    we’re good to go. The use of COLOR_TABLE_START_INDEX is to handle non-zero starting values, for instance, if we wanted to add “silver” and “gold” with values “-2” and “-1” we could change our text file to

    silver -2

    run make, and just like that our program handles values like 5.2Ω with grace and ease.

    $ ./res 5.2
    green red gold

    (aside:[the fall of awk] It’s likely that awk‘s highly focused domain and relatively complex syntax is a reason its use has dropped in favor of general purpose languages like Python and Ruby.)

    Like I said, in this particular case, it’s unlikely we’ll ever have to regenerate color_table.h again, but this technique can be useful in situations for data is more likely to change. Also, keep in mind that the C code we wrote is valid against the C99 standard and should compile and work on any system with a C99 compiler: Linux, OS X, Windows, etc. However, awk is a Unix utility and will be available on POSIX compliant systems like OS X, but not necessarily other systems, in particular, Windows does not come with awk. awk has fallen out of popularity in favor of more powerful languages such as Python or Ruby, each of which could be used to solve this problem as well, even though they don’t have awks focus of working with field delimited records, and you are still faced with the challenge of dealing with users who don’t have your chosen utility installed. If you do use a trick like this, but still want to distribute your source code for as many platforms as possible, you will have to break the “don’t include generated files” rule and add color_table.h to your repository.

  • Assessment, Evaluation, and Final Grades


    I still feel like I’m recovering from the final couple weeks of the semester, and I still have one more task to complete as part of my teaching responsibilities before I don’t take a break and roll right in to finishing up a M.S. thesis. So the time is as good as any to reflect on these past two weeks, and since these were my final last two weeks teaching EC2524, maybe a chance to reflect on my experience as a whole.


    Grading has always been hard for me. It is ridiculous to assume that a single number can represent an individual’s understanding of a topic and I have a hard time doing things that I think are ridiculous. It would be easier if so much weight wasn’t placed on final grades.

    Anyway, I’m in danger of veering off on a tyrant about grading and that wasn’t what I wanted to do. The week of exams has always felt a bit overwhelming. I usually set the last day of classes as a due date for final project working prototypes at which point there are a few days of peer review, then possible responses to peer review before final final projects are due on the last day of exams. Then I have under two days to evaluate all of the projects (there were 29 this semester) and get final grades calculated and submitted.

    This last round was probably the first time that I’ve made it through the final couple days of intense grading without pulling my hair out, spending precious time agonizing over how to distribute points, or realize that I actually still had assignments from the semester in need of grading. I owe this improvement in part to experience and understanding the importance of getting all other assignment grades done and recorded before starting to look at projects, but also in part to the knowledge gained in Assessment Techniques in Engineering Education (ENGE 5404), my first ENGE class which I completed this semester.

    As a bit of an afterthought assignment, after preparing and giving final project presentations, we were asked to write a short paper explaining the difference between assessment and evaluation.

    We had been discussing assessment all semester long, and while I’m sure we talked about what assessment was at some point, we hadn’t really discussed evaluation, how it is different from assessment and how the two relate. The differences between the two are relatively straight forward, as I found, but I gained something with that added clarity that I think helped with the looming project evaluations I would be buried under for the next couple days.

    From ABET:

    Assessment is one or more processes that identify, collect, and prepare data to evaluate the attainment of student outcomes. Effective assessment uses relevant direct, indirect, quantitative a and qualitative measures as appropriate to the outcome being measured. Appropriate sampling methods may be used as part of an assessment process.

    Evaluation is one or more processes for interpreting the data and evidence accumulated through as assessment processes. Evaluation determines the extent to which student outcomes are being attained. Evaluation results in decisions and actions regarding program improvement.

    Or, the condensed version: the focus of assessment is to collect data about how the teaching/learning process is functioning and using that to improve while the goal of evaluation is to assign a number or grade that represents what a student has learned (or how effectively an instructor has taught). Assessment is a formative activity, it should be actively done throughout the semester, while evaluation is a summative process, it aggregates data that has been collected and turns it into something final.

    Ok, so why am I spending so much time on this, when I wanted to talk about how I ended up evaluating final projects? Well, grading has never been something I look forward to. In fact, it’s probably the one part of teaching that I’ve really disliked. Some of my dislike I think is just due to the nature of the task. It tends to be repetitive. If all the assignments are the same it can be boring, if all the assignments are different it can be really challenging to come up with a grading rubric that is consistent and fair. There is always a part of grading (especially final grades), that are subjective. It requires a degree of confidence on the part of the grader to assign a number that corresponds to how well a person has mastered material. Especially in my early years of teaching, I was often self conscious about final grades out of fear that my choices would be questioned, or that while trying to be fair and consistent across a group of students, I’d make a mistake, treat someone differently than someone else, and be called out on it.

    But I also don’t like the finality of grades, especially final grades. I feel its important to provide an opportunity to discuss work that doesn’t demonstrate proficiency and to make corrections and resubmit it. The system I use for programming assignments allows for continuous re-submissions (though it is far from perfect, in particular it doesn’t really encourage having a conversation with the instructor about why something was incorrect, something I would like to address in a future version), but there is limited opportunity to do something similar with final project grades.

    But another thing that writing about the differences between assessment and evaluation brought to the forefront of my thoughts was that while I don’t particularly like evaluation, for the reasons states, I actually enjoy assessment. To me it is a much more constructive, optimistic process. It’s not about judging what someone already knows, but about helping them learn more effectively, and that is something that really interests me. I have spent a lot of time thinking about and developing the system I currently use for homework submissions in part because it saves me the drudgery of repeated evaluation of programming assignments (and yes, in the context of the assignment, this is an evaluation, though that data can and is used by a larger assessment process), but also because I am experimenting with how to provide more useful, real time feedback so that learners can correct their own mistakes when it just isn’t feasible in a class of 71 for me to have a conversation with each individual about every stage of an assignment. What I can do, is look at the data that is generated by the automatic system, things like average completion time, average score on the first submission, average number of submissions before a perfect score, and even find the specific parts of an assignment that were most challenging, and use that to mold the direction we take in class discussions and future assignments. I think that is a critical piece of effective teaching (though I am by no means good at it yet!), and is one reason I tend to be critical and skeptical of rigid lesson plans and lecture notes that haven’t changed in 10 years.

    So ideally, I’d like to think of the data that I generate as a part of evaluating final projects part of assessment, that is, I’d like to be fairly critical and use a scale in which 100% means “I don’t see any obvious room for improvement” and having it be OK if the maximum score is an 80%. In an assessment scenario, a max score of 80% (or 60%, or %40, etc.) just means the instructor/learners need to rethink parts of what they’re doing to improve, the severity of the problem might reflected in the actual value. As long as there is an opportunity to improve I’m not too concerned about low maxes and low averages on a particular assignment. Also, due to the size of the class and my own time constraints, the final projects are the only assignments that I am able to really read through every line of code and judge for things like style, composition and technique. These are things I talk about and show examples throughout the semester, but without getting regular feedback I can’t expect everyone to pickup on the many nuances and reflect this in their code, so I want a way of saying “hey, this program works as expected, but the code itself has a number of serious problems”, and I want that to be reflected in the final project score, but I want a consistent mechanism in place so that that type of response isn’t detrimental to a students final grade in the course since I know I wasn’t able to provide feedback like that earlier in the semester.

    A project evaluation method with a focus on assessment

    All in all, I want to approach the final projects as an assessment tool that allows me give honest, critical feedback, and an algorithm for generating a final grade that takes into account the limitations of the grading/feedback system used throughout the semester. Here’s what I came up with.

    First, I wrote up a number of statements I wanted to evaluate, modeled after what I was asking for in the peer review assignment.

    ## Usage
    - The README contained everything I needed to easily use the program.
    - The program compiles/runs without errors.
    - The program worked as advertised.
    ## Style
    - The code is cleaning divided into modules and multiple files.
    - Variable and function names are meaningful
    - Comments are used where appropriate.
    ## Philosophy
    - The program most closely follows the `` interface pattern.
    - This choice of pattern is a good one for this application.
    - This program follows the Rule of Silence and Least Surprise.
    - This program follows the Rules of Modularity and Composition.
    - This program follows the Rules of Representation and Simplicity.

    I used a 4-point Likert scale for each item, values 1-4 corresponding to “disagree”, “slightly disagree”, “slightly agree”, and “agree”. To decouple my data entry from the feedback presentation, and to facilitate number crunching, I entered the data in a YAML formatted file for each project:

        score: ~     # [0-4]
        comment?: ~ 
        score: ~     # [0-4]
        comment?: ~
        score: ~     # [0-4]
        comment?: ~
        score: ~     # [0-4]
        comment?: ~
        score: ~     # [0-4]
        comment?: ~
        score: ~     # [0-4]
        comment?: ~
        matches: ~   # (filter|cat-like|ed-like|compiler)
        score: ~     # [0-4]
        comment?: ~ 
        score: ~     # [0-4]
        comment?: ~
        score:       # [0-4]
        comment?: ~   
        score:       # [0-4]
        comment?: ~
        score:       # [0-4]
        comment?: ~
    overall?: ~

    Each item had a place for a comment which I used to explain my score decision and often suggest improvements. The comments (text after the # symbols, only served as a reminder to me of the format that data should be in, though eventually I might think of a good way to use something like that for automatic validation of data as I enter it.

    The feedback was in the form of a peer review response, just like other reviews left by other students. For example, my review of the Text Analyzer project.

    The final piece, which reduced a lot of stress and made the experience rather painless and even enjoyable was a bit of ruby code for generating a final project grade:

    def project_scores(data)
      scores = [ :usage, :style, :philosophy ].inject({}) { |result,key| result[key] = collect_scores(data[key.to_s]); result } 
    def weighted_score(scores, weights)
      weights.zip(scores).inject(0) { |sum, (weight,value)| sum += weight*value }
    def grade_scale(scores)
      [ 60, 30, 10 ].permutation(3).collect { |p| weighted_score(scores,p) }.max.round(2)
    section_scores = [ 4*3, 4*3, 4*5].zip(scores.values).collect { |den, scores| scores.inject(0) { |r,s| r += s.to_f } / den }
    final_grade = grade_scale(section_scores)

    Scores are grouped into the three categories, “usage”, “style”, and “philosophy”, the average of each category is calculated producing an array of three numbers, weights of 60%, 40% and 10% are assigned to each category such that the final score is maximized. Thus, the poorest performing category will be weighted the least while the best performing category will be weighted the most. A good score can still be achieved even if there were serious problems in one of the categories.

    Having this in place allowed me to be more objective and consistent with evaluating each item. In the past, when I had attempted similar rubrics, but without a clear algorithm for generating a grade, my scores would inconsistently become more subjective if I noticed my responses for a number of items were negative, thinking to myself “the simple sum of these numbers isn’t going to accurately reflect what I think a final score for this project should be, so maybe I’ll be less critical for some items.”

    Now, this current system is certainly not perfect, and there were still moments of stressful subjectivity because I left out items like “This program does one thing and one thing well” which would add an extra dimension to account for really excellent, clearly defined ideas but perhaps with a number of implementation issues, or “This program was enjoyable and easy to use” which would add a dimension to capture a bit of the hard work that was put into designing a good user experience.