Sunday, August 31, 2014

The Uncommon Sense in Toyota Way



In the previous blog, I talked about the key values in Toyota Way, which in my opinion, are nothing more than common sense. I have been instinctively practicing these values in my small team, which consists of a dozen of people. I can talk to each member of the team, making sure I respect each of them by listening to them, observing them and giving them concrete help in their jobs. however, I would have no idea how to practice these values if the team was a little bigger (say containing more than 20 people). 

This truly shows how uncommon Toyota Way is. Most companies have some sort of Way that was developed by their founders, but over the time, their Way got watered down or changed. The fact that Toyota is able to ingrain Toyota Way in its DNA when it evolves into an international company when it has to assimilate different cultures is amazing!

Why Toyota is able to do so? I think one reason lies in its TPS system. Just-In-Time approach was created out of necessity: Japan is a small country with limited resources. However, as Toyota grows into an international country, Toyota continues this approach to intentionally force problems out of surface and challenges its employees to become better problem solvers and to do continuous improvement. It takes great courage and strong will for a company (or a person) to intentionally and constantly challenge itself and force self-growth.

Another reason must lie in Toyota’s way of choosing and developing leaders. Toyota believes in genchi genbutsu (go and see the actual situation): it believes true good solutions can only come from people who have intimate knowledge of the work. Because TPS creates challenges constantly, potential leaders can stand out from the crowd by proving themselves by solving challenges. Various mentors are assigned to leaders as they move up to make sure they live Toyota Way. The book said a US leader has mentors continuously throughout his 20 more years’ career in Toyota.  In a sense, it is like the apprenticeship system: core values are passed down from masters to apprentices.

It is extraordinary that Toyota is able to keep true to Toyota Way. I can’t image how in practical terms it is able to do so. 

The Common Sense in Toyota Way



If you work in software industry, and you haven’t lived in caves for the past few years, you must have heard “lean”. I was surprised to know that “lean” was inspired by Toyota Way, so I picked up the book The Toyota Way: 14 Management Principles from the World's Greatest Manufacturer , and to my bigger surprise, I found I have been intuitively following many principles of Toyota Way without knowing it. When I had time to reflect, I still believe there is nothing extraordinary about the Toyota Way, it is some simple common sense.

 

Problem-solving

Toyota Way puts a lot of emphasis on problem-solving: using the right approach to solve the right problem; it stresses not only about achieving goals, but about using the right approach to achieve goals. 

In my opinion, Toyota has the clearest and best definition of leadership qualities. Many companies  define leadership as some sort of abstract, fuzzy qualities that somehow enable a leader to inspire and coach employees, but there are no guidelines on how a leader can do so. I’ve encountered some weird perceptions of leadership, such as “if a leader is hands-on, he is not a good leader, because he doesn’t know how to delegate”, or “a leader is to pump motivation into employees, so employees will be motivated to do better jobs.” The problem with such perceptions is that it is as if all a leader has to do is to feed chick-soup to employees, so that employees will naturally know how to do their jobs better. Chick-soup can’t full a stomach, when facing difficulties, employees need guidelines on solutions (or on figuring out solutions), not empty words of “just hanging on there, you are doing a good job, you can do better”. I actually have attended several leadership training, where we were showed the same video where an American football coach shouting similar “motivational” words to a football team.
Toyota defines the leadership qualities as:


  1. Active and open-minded observing of the work of the organization
  2. Active listening to hear what people are really saying
  3. Systems thinking
  4. Clearly defining problems and identifying the root cause
  5. Creatively identifying countermeasures to the true root causes
  6. Planning
  7. Translating plans into action with clear accountability
  8. Taking the time and energy for deep reflection to identify further opportunities for improvement
  9. Understanding the actual strengths and weaknesses of each person
  10. Motivating and influencing people across the organization (with no direct authority) toward common objectives
  11.  Being able to teach others all of the above


1-7 are all about necessary qualities and the right approach to solve problems; 8 is to take steps to prevent problems from happening again or making further improvements; 9, 10, 11 are about people skills, with the ultimate goal of teaching people the right approach to solve problems. A Toyota leader can’t teach what he cannot do himself. 

The common sense here is that by exposing problems and solving problems, you can achieve improvement continuously. Yet, it is not a common sense embraced by some companies; you can see this in their innovation programs.  Often, such innovation programs value novel, outlandish ideas and discard ideas on improving existing processes and tools as too small and not fancy enough. Maybe I am too conservative, maybe some future “Jobs”-like geniuses are able to take advantage of such innovation programs and make a name for themselves, but I believe companies shouldn’t pin their hope on spotting big-bang ideas that can change the world overnight, instead they should cultivate an innovative culture where common people are motivated to solve daily problems and make continuous improvement. 

The uncommon thing for Toyota is that it takes extreme measures to intentionally create and expose problems. Toyota Production System (TPS) is designed to make problems visible to challenge people so that they grow and become better problem solvers. The two components of TPS are Just-in-Time and Jidoka (自働, automation with intelligence to stop itself when there is a problem).


Taiichi Ohno (大野耐一), the father of TPS, compares TPS to a boat traveling on a river flowing over rocks. The water was like inventory. With more water, the rocks are hidden below the surface and become invisible to the boaters. As the water level falls, however, the rocks will break the surface, and the boat must stop until the rocks are eliminated or crash. The rocks signify problems (equipment downtime, quality problems, communication problems, and on and on), and in traditional mass production, they are covered up by inventory: it seems to be smooth sailing despite all the problems. With just-in-time inventory, problems are painfully visible and must be solved otherwise production stops.

Genchi Genbutsu (現地現物, go and see the actual situation)

Toyota believes good solutions can only come from people who know how the real work is done. Good solutions can’t come from management sitting in the office, reading charts and giving out commands. Akio Toyota, the current president of Toyota says in 2009 that “Job titles are not important.  In the end, the people who know the gemba (where the actual work is done) are the most respected.”

At its core, genchi genbutsu reflects the deep respect for the core value-added workers of the company. Those who understand the value-added work and contribute to it are the ones that are respected and advance. 

Respecting “common” people is easier said than done. Many companies talk about trusting and empowering employees, but in practice, decisions are often made by management who are far removed from gemba and have little knowledge about the situation. Decisions thus made are often not good, in fact they may even be harmful and have to be forced down upon employees. At its core, these companies are arrogant and have little respect for the core value-added workers.    

Genchi genbutsu ensures that:
  1.  There is a constant growing roster of future leaders emerging from gemba.
  2.  The changes are driven by the people who are closest to the problem, which results in better problem solving, sustainable solutions, and the possibility of continuous improvement.
  3. People at every level are aligned with the company’s goal.

Kaizen (改善, daily improvement)

Toyota believes nothing is perfect and everything can be improved. Through genchi genbutsu, leaders and team members identify wasteful things in their daily jobs, and figure out ways to improve.

Again this is something that even a kindergarten can understand: if you continue to make improvements, even though improvements may be small, you will become better. But what is uncommon about Toyota is that it places a great stress on how it achieves improvements so that they are sustainable. Shortcuts to reach goals are not allowed. 

There is one example in the book: a potential leader was hired from outside. He was asked to take a position several levels down from his previous position and he agreed. The potential leader was asked to spend time on the assembly line and come up with Kaizen ideas to improve the assembly line. He noticed that workers were using primitive wrenches which sometimes resulted in quality issues in cars. He proposed to purchase more sophisticated wrenches for workers. This idea was rejected by his mentor. He returned to gemba and further observed and reflected. He finally came up with two root causes: 1) Workers were not well trained. Using primitive wrenches, they should listen for a different sound when they turned the wrench to a certain level. 2) The tools were not well maintained, the cost of maintaining them when they broke down was expensive. Based on these root causes, the potential leader devised a training program to train workers to listen for the sound; and he devised a preventive maintenance program.  

This example illustrates some key principles of Toyota:
  1. Genchi genbutsu. The potential leader was asked to step down a few levels and gain experiences from gemba; he was asked to stay on the floor and observe how work was done. 
  2. Identifying root causes, using the right problem-solving approach to achieve kaizen. 

Another example: a leader in a US Toyota plant noticed some innovative system used in a Japan Toyota plant. He knew that he was not the leader of gemba and was not the right person to decide whether this system was good for his US Toyota plant. So he assembled a team composed of hourly workers and one salary worker and sent the team to spend two weeks in the Japan Toyota plant. The team came back to US and tried to adopt the system. They soon run into problems because some differences between the two plants. The US team improved the system based on their situation and made a series of continuous improvements, the resulting system was more automated and more precise. The Japan plant heard of the improvement and came to learn from the US team.  
Notice the improvements were made by a team composed of hourly workers – the “common” people.

 

Spirit of challenge

Toyota believes each challenge is an opportunity (危机=机遇). Through 2008 ~ 2011, Toyota was hit by 3 crises:
  1. 2008, the great recession. 
  2. 2009, the recall crisis in US. Toyota was portrayed as killers on the road and trying to cover up, the president of Toyota, Akio Toyota, had to testify before the Congress. 
  3.  2011, the great east Japan earthquake and Tsunami.
For each crisis, Toyota has tried to turn it into opportunities for identifying problems and opportunities. For example:

  1.  In the great recession crisis, Toyota identified overproduction as the root cause. It had seen signs of oil price increasing and should have decreased the production of large vehicles; instead, it continued building until the crisis hit. 
  2.  In the recall crisis, Toyota believed the responses to the crisis were neither timely nor appropriately, because they came from Japan, not from the US leadership team who was in gemba. Therefore, Toyota should develop the US leadership team to be more independent and autonomous. 
  3.  In the earthquake and Tsunami crisis, Toyota discovered that it knew little about suppliers to suppliers.

Respect

I believe this is the most fundamental value of Toyota Way. “Respecting people” is something that is paid most lip service, but often not practiced at all. In my opinion, true respecting people means developing people to have problem-solving capabilities, trusting them to come up with solutions to solve problems in their jobs, giving them power to own processes, tools and metrics. Yet, in many companies, management devises metrics to monitor people, such metrics seldom reflect the true situation (see my article The Original Sin of Software Metrics ); and sometimes management further devises processes and tools to control people to move towards metrics. In doing so, it kills the spirit of innovation among people, making true improvement impossible.

Running Mahout on Windows



Why is Windows so difficult to please? I overcame a lot of difficulties to build Hadoop on Windows. Just as I was ready to breathe a sigh of relief, boom, more troubles: running a simple Mahout example failed. 

My Mahout location is: c:\EclipseProjects\Libraries\mahout-distribution-0.8\

Running a simple Mahout program failed (using Cygwin):


$ cd c:/EclipseProjects/Libraries/mahout-distribution-0.8/bin
$ sh mahout seqdirectory -i C:/EclipseProjects/mahout-cookbook/chapter02/Lastfm-ArtistTags2007/original -o C:/EclipseProjects/mahout-cookbook/chapter02/Lastfm-ArtistTags2007/sequence
hadoop binary is not in PATH,HADOOP_HOME/bin,HADOOP_PREFIX/bin, running locally
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/ProgramDriver at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:105)
Caused by: java.lang.ClassNotFoundException:org.apache.hadoop.util.ProgramDriver



This suggested that Mahout couldn’t find Hadoop. Mahout distribution is shipped with a Hadoop Jar: mahout-distribution-0.8\lib\hadoop\hadoop-core-1.1.2.jar, this Jar should be included into Mahout classpath. 

Open C:\EclipseProjects\Libraries\mahout-distribution-0.8\bin\mahout, and add this line into classpath:

# add release dependencies to CLASSPATH
  for f in $MAHOUT_HOME/lib/*.jar; do
    CLASSPATH=${CLASSPATH}:$f;
  done
 
  CLASSPATH=${CLASSPATH}:$MAHOUT_HOME/lib/hadoop/hadoop-core-1.1.2.jar;
 

Rerun the example, it failed again with:

Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-apple\mapred\staging\apple1283258319\.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
 


There are many threads on web discussing this issue, almost all come to the conclusion it is not solvable and it has to be worked around by changing the source code. So I download Hadoop 1.1.2 source code, and imported hadoop-1.1.2\src\core into Eclipse. The culprit is org.apache.hadoop.fs.FileUtil. Add try{} catch{} in this method:



public static void setPermission(File f, FsPermission permission)
                    throws IOException {
             FsAction user = permission.getUserAction();
             FsAction group = permission.getGroupAction();
             FsAction other = permission.getOtherAction();

             // use the native/fork if the group/other permissions are different
             // or if the native is available
             if (group != other || NativeIO.isAvailable()) {
                    execSetPermission(f, permission);
                    return;
             }

             try {
                    boolean rv = true;

                    // read perms
                    rv = f.setReadable(group.implies(FsAction.READ), false);
                    checkReturnValue(rv, f, permission);
if (group.implies(FsAction.READ) != user.implies(FsAction.READ)) {
                          f.setReadable(user.implies(FsAction.READ), true);
                          checkReturnValue(rv, f, permission);
                    }

                    // write perms
                   rv = f.setWritable(group.implies(FsAction.WRITE), false);
                    checkReturnValue(rv, f, permission);
if (group.implies(FsAction.WRITE) != user.implies(FsAction.WRITE)) {
                           f.setWritable(user.implies(FsAction.WRITE), true);
                          checkReturnValue(rv, f, permission);
                    }

                    // exec perms
                    rv = f.setExecutable(group.implies(FsAction.EXECUTE), false);
                    checkReturnValue(rv, f, permission);
                    if (group.implies(FsAction.EXECUTE) != user
                                 .implies(FsAction.EXECUTE)) {
                           f.setExecutable(user.implies(FsAction.EXECUTE), true);
                          checkReturnValue(rv, f, permission);
                    }
             } catch (IOException ioe) {
                    LOG.warn("Java file permissions failed to set " + f + " to "
                                 + permission + " falling back to fork");
                    execSetPermission(f, permission);
             }
       }
 

Compile this file into hadoop-core-FileUtil-1.1.2.jar and copy it under mahout-distribution-0.8\lib\hadoop\, add the jar into Mahout classpath:
# add release dependencies to CLASSPATH
  for f in $MAHOUT_HOME/lib/*.jar; do
    CLASSPATH=${CLASSPATH}:$f;
  done
 
  CLASSPATH=${CLASSPATH}:$MAHOUT_HOME/lib/hadoop/hadoop-core-1.1.2.jar;
  CLASSPATH=$MAHOUT_HOME/lib/hadoop/hadoop-core-FileUtil-1.1.2.jar:${CLASSPATH};
 
At last, my first Mahout example was able to run!