Why we need to fight unitedly and what we should learn from Wuhan

I know it is hard to be spared from covid19 news but please pardon me for another one. The virus, now officially declared as a global pandemic, has turned the world upside down in the past months. More than 200K people have been infected and 10K people have died, with the number drastically increasing day by day. It is a pity that the containment of the virus has failed in the western world, but we should still keep our chin up and fight unitedly to mitigate the consequences. As a person who grew up in Hubei, studied in Wuhan for college and now lives in the US, I have some words to share.


First, we should take this virus extremely seriously because people (both young and old) do die from it. There have been a lot of great articles about why social distancing is critical. The next couple of weeks would be critical for mitigating the consequence of the virus. Without strong measures, the number of infections increases by 10x every 8 days. What this means is that by April 19, there will be 90M infections in the US alone if no strong measure is taken. Please also refer to this great article for more information.


Second, we need to be strategical and to learn from past lessons. In particular, I want to share what happened in Wuhan in the early days of the outbreak. As the first location of the outbreak, Wuhan suffered a lot, but it looks like the western world has very little knowledge of the mistakes that were made in Wuhan. Here are some pieces of advice based on what I know:


  1. Keep calm. Don’t rush to hospitals. Stay away from panic groups. Use tele-diagnostics options if possible. In the early days of the Wuhan outbreak, a lot of people who got common code rushed to hospital but the waiting hours could be extremely long because the hospital system was crowded. Many people got infected in the tiring waiting process during which their immune system was impaired. Hospitals need to provide an appointment system to minimize the number of people who go to hospitals at the same time. People who have symptoms should trust the system and don’t panic.
  2. Staying at home is not sufficient, you need also protect your families, especially if you have any symptoms. The very first measure of Wuhan city government was to advise people to self-isolate at home. What happened after that? The situation got much worse because people who stayed at home infected their families. When young kids or old people got infected, then people had to visit hospitals in person, which transmitted the virus to other families. If you have symptoms, please distance yourself with your families and do whatever you can to minimize interaction with them. There would be some inconveniences but it would be life-saving. The government needs to take strong actions to provide shelters for people who have symptoms for isolation. People who have been in contact with infected persons or have symptoms should go to the shelters instead of staying at home. Extended testing needs to be made to guide who should stay at home and who should go to the shelter.
  3. Don’t think that only old people get the virus and young people don’t have responsibilities. A common misconception is that young people won’t get serious symptoms from the virus so won’t be in trouble. If the hospital is crowded, nobody could be spared. Besides, young people do die from the disease. Widespread of the virus also makes it more likely for older people to get infected and the virus may also mutate in the process.
  4. Most importantly, be gentle, nice and supportive to other people. I know this is a difficult time and a lot of people are frustrated, confused and panicked. However, as a human community, we shouldn’t point guns to each other during this difficult time. Certain government officials want to take advantage of the situation and promote their political agenda. This is NOT a time for that.


I want to end the article with a positive note. With a strong measure, we could reverse the course of the virus spread. Yesterday Wuhan has reported 0 new case. Wuhan people have achieved it with 2 months of efforts, so every country and community should be able to do the same. The critical thing is to stay united and learn from each other.


God bless every human in the world!



A Better Future for Everyone

I recently finished the book the war on normal people written by Andrew Yang, who is a democratic presidential candidate for the 2020 election. The normal people in the title refers to the silent majority of Americans who haven’t received higher education and have suffered in the recent technological development and globalization. As a “techie” who lives in Silicon Valley, I am very thankful for Andrew to share his thoughts and opinions. This book is a great way for people like me to have a realistic view of America, a view Andrew developed after he worked many years to foster entrepreneurship outside Silicon Valley.

My first question when I saw the title was: Aren’t we living in a peaceful time? Who is waging the war to normal people? According to Andrew, we are in a “war” created by the nature of capitalism to achieve efficiency and the new technology that favors high-skilled workers at the cost of the normal people. Andrew feels that normal people lack the ability to stand on their own foot in this “war.” As a result, the government needs to step in and to give normal people a hand. The solution Andrew proposed is the  the universal basic income (or so-called freedom dividend), which is to give $1000 to every American citizen. While there may be a better solution, Andrew has got the problem right — there is a massive job shift in terms of both skill requirements and locations. Unfortunately, not everyone could adapt to the shift. 

More than 5 million manufacturing workers lost their jobs after 2000, and it was the emotion behind this massive job loss that sent Trump to the white house in the 2016 election. In the first part of the book, Andrew focuses on where and why the jobs have gone. There are two reasons for the job loss — automation and globalization. The former allows employers to replace a lot of human workers with machines, the latter allows employers to outsource many jobs to countries of lower labor costs. The American workers now need to compete with both machines and cheap labors overseas.

The underlying driving force is the desire for companies to maximize profit for shareholders. In spite of Andrew’s grudges, I think private companies should not be scrutinized for this. After all, it is the same force that propels the machine of the market to operate and it is the government’s duty to set the rules of the market and private companies are not and should not be wealth fare programs.

Another point mentioned in the book is that fewer jobs are created by the new technology than those that were eliminated. The example it cites is that while Walmart employes over +1 million works, new tech giants like Google or Facebook employes a much fewer number of people to reach the same scale. Although arguably this is true, the point is less clear if we consider the total number of job opportunities they provide. Google or Facebook are also platform companies that enabled many more job opportunities. If you count contractors, gig-workers and content creators, the number of job opportunities created by these platforms is much larger than the number of full-time employees they hire. Besides, more than 4 million enterprises rely on Google and Facebook and they provide many more job opportunities. 

Although the total number may not change, there are still significant changes in jobs. On the one hand, new jobs require much higher skills than the ones they replace. On the other hand, New jobs appear in different geographical locations than the old ones.

While globalization and automation are shifting the nature of jobs, there is no mechanism to help people, especially less educated people, to adapt to the shift. Two decades have passed since the job losses started. As two decades is a long time and if the adaption still doesn’t happen, we couldn’t expect the trend will automatically be reversed. The losers are deprived of their representative rights in the market. They will eventually stop looking for job opportunities. 

It is right that the government needs to step in to help bridge the gaps. However, instead of distributing money unconditionally, the government should give stronger incentives and guidance for people to adapt to the change.

First, the government should invest in education and ensure it is affordable and accessible. Since new jobs generally require higher skills. Equal education is the key to ensure every person has equal access to the new opportunities created by technologies. STEM education in the US is considerably lagging. Student debt is a serious issue. The quality gap of education in public and private schools needs to be narrowed. Also, adult education should be subsidized by the government because, arguably, adults who decide to continue education need more incentives and encouragement than kids. 

Second, the government should help bridge the gap among different geographical areas. Currently residing in a place outside Silicon Valley and New York means a huge loss of job opportunities. When I graduated, I couldn’t find a good job in Boston so had to come to Silicon Valley. Most of my friends had similar experiences. Given that Boston is already a talent hub and a college city, the situations of other cities could only be worse. For this issue, I recommend another book The New Geography Of Jobs, which is authored by Enrico Moretti and also discusses the geographical shift of jobs in the united states. 

One joke is that why silicon valley becomes the tech hub is that VCs like the good weather of the area. Although this is a joke, it is certainly true that VCs have a preference for certain places over others. If VC lacks the incentives to go to other cities, the government should step in and either provide venture fundings or adjust the legislation to give VC more incentives to invest in other communities. 

At the same time, for those who want to relocate to a technological hub, the government should also provide the necessary help. This is tricky because we need to make sure this won’t cause too much pressure to the communities outside the existing technical hub. 

Although I don’t agree with the plan of giving freedom dividends unconditionally, I do think the government should play a more active and positive role to create a better future for every person in the upcoming new technological society. I am glad to hear that Andrew is on the way to find a solution and his campaign has already increased people’s awareness of the issue a lot. Best wishes to Andrew for his journey.



Why people write less nowadays?

When I was a kid, I used to write a lot. I wrote diaries daily to snapshot my thoughts and articles weekly to express my opinions. Remember that this was the time when computers were rare and when I could only write using pens and papers.

Nowadays, I have much better writing equipment yet I am much less prolific. I have been thinking to write an article for a long time. Unfortunately, I have been procrastinating, and my last post has been a few years ago. I also observed similar trends among my close friends. I feel this is a contemporary trend. When was the last time you write a love letter, a travel diary? People write much less than before, and even when they write, most of the writings are utilitarian, i.e., they convey facts but no emotions. I am talking about the writings that force authors to squeeze every emotion and creativity from the bottom of their brain to touch the audience’s soul.

There are simply so many distractions nowadays. We are such a deeply connected species — smartphones feel like our external body parts and everyone is online 24×7. Smartphones are great for killing times but unfortunately not so good tools for writers. Computers are much more useful for report writing, which requires extensive searches of information, but less so for pure literature writing. Besides, more and more smartphone applications are grabbing people’s attention by dividing information into small pieces to fit people’s fragmented calendar. As a consequence, people are accustomed to superficial thinking and spend less time on meditative writing.

It doesn’t mean that there is nothing we could do. A simple anti-dose for this is more self-discipline. We are lured but thankfully are not forced to live in a fragmented way. Just block your calendar, lock your phone and pick up your pen from today. This is not a perfect solution but it works if executed well. Good luck and enjoy being a writer!

Jing Conan Wang



On Paradigms of Problem Solving

In  the 19th century, engineering was almost a synonym of mechanical engineering. A typical way to solve problems at that time was 1. analyzing the mechanical structure, 2. building the machine with gears and wrench and 3. using steam engine to drive the machine.


Steam engine was a revolutionary innovation as it introduced a “paradigm” for problem solving. If you have a mechanic description of your problem, steam engine can take care of the rest. There were tons of derivative innovations following this “paradigm”, such as plane, steamship, submarine, automobile and so on. (some of them used internal combustion engine, an improvement of steam engine). Theoretically, steam engine and its derivatives can solve any mechanical problem. What people need to do is to describe the problem in a “mechanical language.”


In contrast, nowadays people have been accustomed to resort to computers when they have problems. Computer is another revolutionary innovation because it provides a similar paradigm. A computer is basically a calculator that can do additions very fast. Since all arithmetic operations can reduce to additions, computer provides us the ability to solve any arithmetic problems that can be described by computer languages.  The paradigm for computer is 1. formulating an arithmetic model, 2. proposing an algorithm and 3. running the algorithm with computers.


Steam engines and computers share some common points. Both of them provide a good solution for a fundamental problem: steam engines deal with the problem of “generating rotary movement with a strong force” and computers deal with the problem of “doing additions in a fast way”.


We solve problems by paradigms, namely we divide a problem into a large sequence of “fundamental problems” and solve them with a “problem solver”. The figure below shows the paradigm for steam engines and computers.

Why we need paradigm?

Paradigms can save us time. The real world is too complex and we cannot do everything well. A reasonable way is to solve a small set of problems perfectly and to transform the problem we want to solve into these problems.


Why we need to be cautious about paradigms?

Every innovator needs to be cautious about paradigms. Every paradigm has its own limitation. A paradigm can be very good for some problems and very bad for some others.


It is awesome to use steam engines (or its derivatives) to solve transportation problems. However, it will be very inefficient to use it to solve communication problems.

In the pre-information age when most communications were done by mails, a lot of people tried to improve communication efficiency by inventing faster mail vans.  They innovated in their familiar paradigm, but none of them is remembered by us. The problem of communication efficiency was not solved by faster mail vans but computers and the Internet.


Similarly, computers also have drawbacks. Whenever we face a problem nowadays, we subconsciously resort to computers  (by writing apps or algorithms). However, computers are powerful, but definitely are not omnipotent.


For example, short battery life is the bottleneck of electric cars. Tesla uses normal chemical batteries provided by Matsushita and its secret sauce is its power management algorithms. Although Tesla has done a good job, its solution still follows the paradigm for computers. The battery life problem is an energy problem, which cannot be modeled as arithmetic operations. The fundamental problem here is “to store a unit of energy”, which hasn’t been solved well yet.



Another hilarious example is this mad mother who created a app to lock his son’s phone when he doesn’t respond. The problem here—-“improving the mother-son relationship”– is definitely not an problem that computers are good at.


Summary of “The Most Powerful Idea in the World: A Story of Steam, Industry, and Invention”

Recently I finished reading the book The Most Powerful Idea in the World: A Story of Steam, Industry, and Invention. The author’s key statement is that industrial revolution was first and foremost, a revolution in invention, and the industrial revolution took place in England because “its unique combination of law and circumstances.” Here are some interesting sentences from the book.

  1. Before the eighteenth century, inventions were either created by those wealthy enough to do so as a leisure activity (or to patronize artisans to do so on their behalf), or they were kept secret for as long as possible. In England, a unique combination of law and circumstance gave artisans the incentive to invent, and in return obliged them to share the knowledge of their inventions.
  2. Papin was an industrial scientist before there was an industry to employ him, which made him, in consequence, completely dependent on patronage.
  3. The most powerful pumps in use in seventeenth-century England were operated by waterwheels, but nothing obliged rivers and streams to be convenient to mines; finding an alternative machine that could overcome water’s tendency to seek the lowest level of any excavation meant that vacuum was no longer a purely philosophical concept.
  4. IN ITS ORIGINAL MEANING, the word “patent” had nothing to do with the rights of an inventor and everything to do with the monarch’s prerogative to grant exclusive rights to produce a particular good or service.
  5. In Darcy vs. Allein, Chief Justice Popham ruled that Darcy’s grant was forbidden on several grounds, all of which violated the common law. Crown could not grant a patent for the private benefit of a single individual who had shown no ability to improve the “mechanical trade of making cards,” because by doing so it barred those who did. In other words, the court recognized that the nation could not grant an exclusive franchise to an individual unless that individual had demonstrated some superior “mastery” of a particular trade.
  6. The term of the patent was not to exceed fourteen years, a figure that makes sense only in terms of the artisans for whom Coke was so solicitous. Since the traditional seventeenth-century apprenticeship lasted seven years, a term of fourteen years would allow at least two cycles of apprentices to have been trained in the new industry, and therefore a generation of artisans to demonstrate their mastery of the new art.
  7. The first, the so-called civil law tradition, is a direct successor to the jurisprudence of the Roman Empire, and it dominates most of the legal systems of continental Europe; the second is the institution known as the common law, used in Britain and its former colonies.
  8. As Coke put it, under the common law, every man’s house is his castle, not because it is defended by moats or walls, but because while the rain can enter, the king may not; under the civil law, the king is bound by nothing at all.
  9. Recognition of a property right in ideas was the critical ingredient in democratizing the act of invention. However imperfectly, Coke’s patent system, combined with Locke’s labor theory of value, offered a protected space for inventive activity. The protected space permitted, in turn, the free flow of newly discovered knowledge: the essence of Francis Bacon’s program. Once a generation of artisans discovered they could prosper from owning, even temporarily, the fruits of their mental labor, they began investing that labor where they saw the largest potential return. Most failed, of course, but that didn’t stop a trickle of inventors from becoming a flood
  10. An adult human is able to convert roughly 18 percent of the calories he consumes into work, while a big hayburner like a horse or ox is lucky to hit 10 percent—one of the reasons for the popularity of slavery throughout history.
  11. One can make a water mill more powerful, but one cannot, in any measurable way, reduce its operating expenses. The importance of this can scarcely be underestimated as a spur to the inventive explosion of the eighteenth century. So long as wind, water, and muscle drove a civilization’s machines, that civilization was under little pressure to innovate. Once those machines were driven by the product of a hundred million years of another sort of pressure, innovation was inevitable.
  12. It is almost irresistibly tempting to see Watt’s life as the embodiment of the entire Industrial Revolution. An improbable number of events in his life exemplify the great themes of British technological ascendancy. One, of course, was his early experience with the reactionary nature of a guild economy, whose raison d’être was the medieval belief that the acquisition of knowledge was a zero-sum game; put another way, the belief that expertise lost value whenever it was shared. Another, as we shall see, was his future as the world’s most prominent and articulate defender of the innovator’s property rights. But the most seductive of all was Watt’s simultaneous residence in the worlds of pure and applied science—of physics and engineering. The word “residence” is not used figuratively: The workshop that the university offered its new Mathematical Instrument Maker was in the university’s courtyard, on Glasgow’s High Street, a bare stone’s throw from the Department of Natural Philosophy.
  13. Like an ever-growing percentage of his countrymen in the newly United Kingdom, Watt had acquired the tools necessary for scientific invention—the hands of a master craftsman, and a brain schooled in mathematical reasoning—without the independent income that could put those tools to work exclusively for the betterment of mankind.
  14. Watt needed capital. Investment capital, however, wasn’t easy to find in 1765 Britain; and it was a lot harder than it had been fifty years earlier. The reason was one of the greatest financial bubbles in history, the collapse of the South Seas Company.
  15. Though the most famous inventors are associated in the popular imagination with a single invention—Watt and the separate condenser, Stephenson and Rocket—Watt was just as proud of the portable copying machine he invented in 1780 as he was of his steam engine; Stephenson was, in some circles, just as famous for the safety lamp he invented to prevent explosions in coal mines as for his locomotive.
  16. Inventors are significantly more thing-oriented than people-oriented, more detail-oriented than holistic. They are also likely to come from poorer families than non-inventors in the same professions. No surprise there; the eighteenth-century Swiss mathematician Daniel Bernoulli,11 who coined the term “human capital,” explained why innovation has always been a more attractive occupation to have-nots than to haves: not only do small successes seem larger, but they have considerably less to lose.
  17. If the most important invention of the Industrial Revolution was invention itself, the automation of precision has to be one of the top three.
  18. Micrometers, devices for measuring very small increments, were then only about thirty years old; James Watt himself had produced what was probably the world’s first in 1776, a horizontal scale marked with fine gradations and topped with two jaws, one fixed and the other moved horizontally by turning a screw.
  19. The availability of patent protection was, predictably, motivating inventors to make more inventions; it was also motivating them to frustrate competing inventions from anyone else.
  20. Nearly fifty years later, the first description of the spinning jenny (“jenny” is a dialect term for “engine” in Lancashire) appeared in the September 1807 issue of The Athenaeum, in which readers learned that the first one was made “almost wholly with a pocket knife.
  21. Prior to the introduction of the jenny, Britain’s spinning was performed largely by what we would call independent contractors: the original cottage industrialists, taking raw materials from manufacturers who “put out” for contract the production of finished fabric.
  22. One of the more obdurate rules of economics, however, is that, given their capital demands, factories are preferable to more flexibly “outsourced” labor only if they are more productive.
  23. A great artisan can make a family prosperous; a great inventor can enrich an entire nation.
  24. Smith argued that two conditions were necessary for labor to produce the maximum amount of wealth: perfect competition among sellers—everyone pursuing his or her selfish interest, the famous “invisible hand”—and the complete freedom of buyers to substitute one commodity for another.
  25. A family living alone grows its own wheat and bakes its own bread; it takes a village to support a baker, and a town to support a flour mill. Some critical mass of people was needed to provide enough customers to make it worthwhile to invest in ovens, or looms, or forges, and until population levels reached that critical level, overall growth was severely limited.
  26. Because knowledge is the sort of property that can be sold to multiple consumers without lowering the value to any of them—Romer termed it nonrivalrous.
  27. The remarkable growth of the Netherlands during the 1600s essentially stopped a century later, and the only persuasive reason is size, or rather scale. A small country can shelter the world’s largest banks, shipbuilders, and even textile manufacturers, but since it can protect inventors only from their own countrymen, growth that depends on the creation of new knowledge is fundamentally unsustainable, like a nuclear chain reaction with insufficient critical mass.
  28. that heat and motion are essentially the same thing. This was critical, and surprisingly slow in coming.
  29. Fitch’s steamboat was not, as many histories have it, the world’s first. In 1772, two ex–artillery officers in the French army, the Comte d’Auxiron and Charles Monnin de Follenai, received a fifteen-year exclusive license to run a steamboat along the Seine. Unfortunately, their first attempt, a marriage of a Newcomen engine to a Seine bâteau, was less than successful: the engine was so heavy it sank the boat. Slightly more successfully, in 1785, the Marquis de Jouffroy d’Abbans took a 140-foot boat mounting a Newcomen-style engine out on the Saône from Lyon. He did make it all the way back to the dock, where cheering crowds met it—just in time, before the engine’s vibrations destroyed the boat.
  30. Evans was a visionary and a pioneer. But despite his prediction that “the time will come, when people will travel in stages moved by steam engines from one city to another almost as fast as birds can fly,” his greatest contribution to the history of steam locomotion was almost incidental: his decision to share the design of his boiler and high-pressure steam engine with his compatriots in Britain.
  31. Fusible plug is a small lead cylinder inserted into a predrilled hole in the wall of the engine’s boiler—a hole that, in a properly operating engine, would always be underwater. If, however, the water level in the boiler were to fall low enough to become dangerous, the heat would melt the lead plug,Trevithick’s engine, the first driven by high-pressure steam, earned him a considerable claim on the title “father of railways,” but the birth of steam locomotion was still a decade or so in the future. More important, though less romantic, was another of Trevithick’s innovations, one that was nearly as large an improvement over the first high-pressure design as that had been over the Boulton & Watt separate condensing

The Box

Recently I read The Box, a book about the history of container ships. This is book was recommended by Bill Gates — “you won’t look at a cargo ship in quite the same way again after reading this book.”  It indeed changed my view of the shipping industry and here is a summary of my thoughts.

Influence of Containerization

An immediate result of containerization is a sharp decline in international transportation costs, which resulted in an unprecedented globalization process and business paradigm shift.

Globalization is not a new phenomenon — the world economy was already highly integrated in the nineteenth century. However, the globalization caused by containership is quite different because it fundamentally changed the production process itself. 

Containerization significantly reduced the shipping cost among coastal cities between America and East Asia, which has abundant cheap and skilled laborers. As transportation costs decline, manufacturers could outsource their manufacturing overseas. Many American businesses only do research & design in the US and delegate the manufacturing to Original Equipment Manufacturers (OEMs) in East Asia. This new type of industrial paradigm would not be possible without container ships.

As a consequence, geographical disadvantage becomes a more serious problem. Because American consumers live in coastal cities, it no longer makes sense to manufacture in inland cities as the shipping costs by sea routes are so cheap. Doing business in those inland cities becomes much harder because of the overseas competitions. 

In east Asia, coastal cities also absorb all the foreign investment and markets. Guangdong and Jiangxi are two Chinese provinces that are adjacent to each other. However, the GDP per capita of Guangdong is almost twice that of Jiangxi. The reason is only that Guangdong is coastal while Jiangxi is landlocked.

To reduce the gap, inland cities have to invest heavily in transportation infrastructure to reduce the shipping cost, which is very challenging. 

Influence of Deregulations

The U.S. government played an interesting role in the history of containerization. The government regulations initially prohibited corporations to be involved in both land-based and sea-based transportations.

The initial goals of these regulations were to prevent monopoly and to ensure a fair price for consumers. However, the goodwills of lawmakers turned out to be a huge obstacle and made the cooperation among the shipping, railway, and trucking companies very challenging. Railroads and their customers could not negotiate long-term contracts setting rates. Trucks and railcars that had often been forced to return empty were able to be filled in on the return trip. 

Deregulation changed everything. In 1980, Congress freed interstate truckers to carry almost anything almost anywhere at whatever rates they could negotiate. 41,021 contracts were signed within five years and by 1988 U.S. shippers saved nearly one-sixth of their total land freight bill.

The ability to sign long-term contracts gave railroads incentive to adapt containerships. On average, it costs four cents to ship one ton of containerized freight one mile by rail in 1982 and that cost dropped 40 percent over the next six years, adjusted for inflation.

Although containers were supposed to help cargo move seamlessly among trains, trucks, and ships, it took 20 years since Malcolm McLean invented the first container for the industry to achieve the goal. The process could be much faster without government regulation. This interesting case is another example that shows that the government should keep itself away from the market most of the time. Governments are too slow to adjust themselves to the market due to bureaucracy, so the best way is to let the market speak for itself.



Tutorial to migrate from Bitbucket to Github

Install mercurial and hg-git

sudo apt-get install mercurial

sudo apt-get install mercurial-git

Note: The version of mercurial should be >= 2.8.

If the default version of mercurial in apt-get is < 2.8. You can install using pip

sudo pip install mercurial –upgrade

You need to create a repo on

Clone your bitbucket repo

hg clone hg-repo

Convert hg repo to git repo

Hg-Git can also be used to convert a Mercurial repository to Git. You can use a local repository or a remote repository accessed via SSH, HTTP or HTTPS. Use the following commands to convert the repository

$ mkdir git-repo; cd git-repo; git init; cd ..
$ cd  hg-repo
$ hg bookmarks hg
$ hg push ../git-repo

The hg bookmark is necessary to prevent problems as otherwise hg-git pushes to the currently checked out branch confusing Git. This will create a branch named hg in the Git repository. To get the changes in master use the following command (only necessary in the first run, later just use git merge or rebase).

$ cd git-repo
$ git checkout -b master hg

Push the Git repo to Github Server

cd git-repo;

git remote add origin <github-url>;

git push -u origin master;

cd ..;

I also write a script to do this automatically.



The Personal Analytics of My Evernotes

Jing Conan Wang

Aug. 09, 2013

I started to use Evernote since around 2011. Recently the number of notes in my Evernote account has surpassed 5000. To celebrate this milestone, I wrote some python scripts to visualize my evernotes.

The easiest way to get the Evernote data out is to use the official clients. Both the windows and the Mac Evernote official client can export data in ENEX format. Unfortunately, the development team of Evernote decides not to develop any linux client in the near future, which makes the data export in linux very hard. It may be possible to get the data out using the cloud APIs. But requesting an API key is too cumbersome for this small project.

Although it is the most convenient way, exporting data in ENEX format still has two pitfalls:

First, ENEX is a customized xml format, and it contains some characters, particularly ‘&nbsp’, that cannot be correctly recognized by lxml module in python. To address it, I wrote a script ( to convert ENEX files to regular JSON files.

Second, ENEX file doesn’t provide information about the notebook a note belongs to. Fortunately, the Windows client provides a command-line program ENScript.exe that can export notebooks separately. I wrote a powershell script (export_evernote.ps1) to export the notebooks into a folder, in which each ENEX file corresponds to a notebook. Again I used to convert the ENEX files in the folder into JSON files.

Each note is associated with two timestamps–1. the ‘Created’ time and 2. the last ‘Updated’ time.

Here is a plot of the number of notes I created in each year. Considering that only 7 months is covered, the total number for 2013 should be around 2000. From the plot, I was the most addicted to Evernote in 2012, during which I created over 2500 notes.

The following plot is the number of evernotes I updated in each year. The number of ‘updated’ notes was high in 2011, whereas the number is decreasing over the past two years. In 2011 and the first half of 2012,  I used a Mac Evernote client in my Macbook pro. After that, I switched to a Thinkpad x230 with Ubuntu 12.04, in which the only usable option is the web application ( Updating notes using Mac client is much easier than the web application, which may explain my decrease of note updates.


The following plot visualizes the number of notes I created in every month. There is a welcome note whose ‘created’ timestamp is Oct. 19, 2009. However, I signed up Evernote in Jan. 7, 2011. Why the ‘created’ time for this welcome note is Oct. 19, 2009? I guess this date is the birthdate of Evernote, and the ‘created’ timestamp was deliberately set as the birthdate of Evernote.

The following figure is the number of ‘updated’ notes in every month. Obviously, I suddenly stopped to update notes from May 2015, which matches the time I switched from Mac to Ubuntu. The updates were somewhat recovered since Mar. 2013, because I began to use Evernote as my GTD engines. I need to update my task lists and checklists.

The following two plots visualize the ‘created’ notes and ‘updated’ notes in week level. The number of ‘created’ and ‘updated’ notes during the 12th-14th weeks of 2012 are soaring.  During this time, I was busily preparing for the application of Google Summer of Code 2012. The efforts paid off, I was finally selected into Google Summer of Code 2012 in which I had a very wonderful experience.

Evernote provides a feature to tag notes. For each note, you can add as many tags as you like. Usually I don’t use this feature manually, but when I clip notes from my mobile phone and RSS reader, which I often do, tags will be added automatically. The following figure shows the time of tagged notes. The x-axis is the ‘created’ time and each y-coordinate corresponds to a tag.

I was heavily addicted to Google Reader (GR) when it was alive. In 2011, I often read GR in my iphone using MobileRSS, which added ‘MobileRSS’ tags when I clipped articles. Later, I was irritated when the app asked me to pay the second time after I upgraded to a newer IOS, ignoring that I had already purchased a pro version. As a result, I switched to Newsfy, which does not add tags automatically. After GR was shutted down in July, I switched to feedly together with many other disappointed GR users.

From 2012, I started to add tags manually. One missing feature in Evernote is to assign importance to each note like gmail. I emulated this feature by tag notes with @TOREAD, @✭ and @✭✭.

In Evernote, a notebook is a collection of individual notes. The following figure shows a stacked area graph of numbers of noted I created in each month. I started to use notebook feature systematically after April 2012. Before April, 2012, I mostly used evernote as an archive of web pages and dump most of notes into one notebook.  ‘Programming’ is one of the first notebooks I created, which was divided into more sophisticated categories later.

The ‘cybersecurity’ notebook, which is related to my research, dominated in May and the beginning of June of 2012. From May, 2012 to Aug 2012, my focus shifted to ‘GSOC’ , ‘Programming’ and ‘Python’, which dued to my participation of Google Summer of Code 2012.

There is a visible gap in Sep, 2012 when I had a vacation to recover from demanding work in the summer (that summer was very busy for me). I am a fan of classical music. Even in this busy summer, there were still a considerably large number of ‘Music’ notes. The ‘GSOC’ notebook disappeared after the google summer of code officially ended at Sep. 2012. Considering the number of created notes, the four notebooks I used the most are ‘Python’, ‘Linux’, ‘Music’, and ‘others’. The ‘others’ notebook was created in June, 2013 to store miscellaneous notes.

The following figure shows the number of ‘updated’ notes in every month for each notebook. An interesting observation is that few notes in the ‘others’ notebook is updated despite its considerable large size. This observation indicates that I rarely review the notes in ‘others’ notebook. This bad habit should be improved in the future.

The following two figures show the number of ‘created’ and ‘updated’ notes in every week for each notebook. For most notebooks, although the number of ‘created’ notes doesn’t change significantly, the number of ‘updated’ notes increased rapidly in the 12th week of 2013. The reason is that I categorized many notes manually in that week.

The blog is inspired by Stephen Wolfram’s blog: The Personal Analytics of My Life.

I fully agree with Mr. Wolfram that personal data is very useful and everyone should log their own life as much as possible. Evernote is a good tool to achieve this goal.

Surely there is more information I can dig out. But the information in this article provides a good starting point.  The source code of this project is available at:

If you are interested, you can try to analyze your Evernote data using these codes. Any suggestion, bugfix or improvement is welcomed.


Work Efficiently In the Information Age


Python Program Configuration