tag:blogger.com,1999:blog-308199942024-03-13T14:51:02.601-04:00Niketan Pansare's blogSome goals are so worthy that it's glorious even to fail.Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.comBlogger98125tag:blogger.com,1999:blog-30819994.post-68938626828695472962018-04-13T15:57:00.004-04:002019-02-27T13:42:57.534-05:00Installing Apache SystemML on the IBM Cloud<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="tr_bq">
To use the latest released version of Apache SystemML on the IBM Cloud, please install it using the following commands in the jupyter shell:<br />
<br />
<br />
<div dir="ltr" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
1. Install the latest version of SystemML (for example: 1.2.0):</div>
<blockquote class="tr_bq" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
!pip install systemml</blockquote>
<div dir="ltr" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
</div>
<div dir="ltr" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
2.. Please execute the following command in the notebook after installation:</div>
<blockquote class="tr_bq" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
!ln -s -f ~/user-libs/python3/systemml/systemml-java/systemml-1.2.0-extra.jar ~/user-libs/spark2/systemml-1.2.0-extra.jar<br />!ln -s -f ~/user-libs/python3/systemml/systemml-java/systemml-1.2.0.jar ~/user-libs/spark2/systemml-1.2.0.jar</blockquote>
<div dir="ltr" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
</div>
<div dir="ltr" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
3. Then restart the kernel.</div>
<div dir="ltr" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
</div>
<div dir="ltr" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
4. check if the correct version was installed:</div>
<blockquote class="tr_bq" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
!pip show systemml</blockquote>
<div dir="ltr" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
<br /></div>
<div dir="ltr" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
5. <b>(Important)</b> Finally, check if the correct version is being picked up:</div>
<blockquote class="tr_bq" style="background-color: white; color: #121212; font-family: Arial, Helvetica, sans-serif; font-size: 14px; outline: none;">
from systemml import MLContext<br />ml = MLContext(spark)<br />ml.version()</blockquote>
<br /></div>
</div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-83418237237650997632018-01-16T17:51:00.001-05:002018-01-16T17:54:33.646-05:00Notes from the Micro-MBA course (Part 1)<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
In my previous blogposts <a href="http://niketanblog.blogspot.com/2013/10/notes-on-entrepreneurship-part-1.html">1</a> <a href="http://niketanblog.blogspot.com/2013/10/notes-on-entrepreneurship-part-2.html">2</a> <a href="http://niketanblog.blogspot.com/2014/03/notes-on-entrepreneurship-part-3.html">3</a>, I summarized the notes from entrepreneurship course offered by Rice Center for Engineering Leadership. This blogpost will extend the previous blogposts for larger organization based on the notes from the Micro-MBA class at IBM Almaden Research Center. As a disclaimer: this blogpost should be viewed as my personal biased opinions based on limited understanding of the subject, rather than any official position of IBM or the <a href="https://www.youtube.com/watch?v=1WsD1iYMH8g">instructor</a>. Also, I will skip many topics for brevity.<br />
<br />
The Micro-MBA course can divided into following three sub-topics:<br />
<ol style="text-align: left;">
<li>Economics</li>
<li>Finance</li>
<li>Accounting</li>
</ol>
<h2 style="text-align: left;">
Economics</h2>
The fundamental assumption of economics is of <i>scarcity</i>: People have unlimited wants but limited resources, namely:<br />
<ul style="text-align: left;">
<li>Labor (human time and work)</li>
<li>Natural resources (raw materials)</li>
<li>Capital (man-made materials needed for the production of other goods such as buildins and machinery).</li>
</ul>
<br />
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-r2r8kdTKoX4/Wl05HWXi4qI/AAAAAAAARBI/gTECYdZL1QYegcBOrOprVK9_97GkT7mtgCLcBGAs/s1600/scarcity.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="327" data-original-width="537" height="242" src="https://4.bp.blogspot.com/-r2r8kdTKoX4/Wl05HWXi4qI/AAAAAAAARBI/gTECYdZL1QYegcBOrOprVK9_97GkT7mtgCLcBGAs/s400/scarcity.jpg" width="400" /></a></div>
<br />
Hence, we have to <i>chose</i> how to use the limited resources to maximize our wants and needs. An economist studies the choices to following three fundamental questions:<br />
<br />
<ul style="text-align: left;">
<li>WHAT (goods and services) to produce ?</li>
<li>HOW to produce them ?</li>
<li>WHO will receive the goods and services produced ?</li>
</ul>
<div>
<h3 style="text-align: left;">
WHO will receive the goods and services produced ?</h3>
The answer to the third question depends on what are the economic policies of the country. In a country with <i>centrally planned economy</i> such as Cuba and North Korea (and Soviet Union from 1917 to 1991), the government decides how the goods and service produced be allocated. On the other hand, <u>in <i>market economy</i>, the individuals/households/firms who are most willing and able to buy the goods and services receive them (i.e. based on principle of supply and demand)</u>. Note, most countries today follow <i>mixed economy</i>, where most economic decisions results from the interactions of the buyers and sellers in the market, but in which the government plays a significant role in the allocation of resources [1].<br />
<br />
<table border="1"><tbody>
<tr><th></th><th>Centrally planned economy</th><th>Market economy</th></tr>
<tr><td>Productive efficiency (produced at lowest cost)</td><td></td><td>Better</td></tr>
<tr><td>Allocative efficiency (reflects consumer preferences)</td><td></td><td>Better</td></tr>
<tr><td>Equity (fair distribution of goods and services)</td><td>Better</td><td></td></tr>
<tr><td>Freedom</td><td>Better for freedom from uncertainty</td><td>Better for freedom of choice</td></tr>
<tr><td>Main proponent</td><td>Karl Marx, who posited in Das Kapital that market competitions exploits labor and ultimately fosters monopolies. </td><td>Adam Smith, on the other hand claimed that competition and the self-interested pursuit of profit causes the firms and the individuals to behave in ways that are ultimately socially beneficial.</td>
</tr>
</tbody></table>
</div>
</div>
</div>
<br />
<h3 style="text-align: left;">
WHAT (goods and services) to produce and HOW to produce them ?</h3>
As described above, in a market economy, self-interested pursuit of profit motivates people to decide what goods and service to produce. The profit is high if the consumer is willing to buy the goods and service for higher price that the cost of goods and service. Every economic decision that a consumer makes takes account of utility (the amount of benefit to the consumer) and comes with an opportunity cost [3]. <i>Opportunity cost</i> is the benefit, profit, or value of something that must be given up in order to have something else.<br />
<h3 style="text-align: left;">
Opportunity cost</h3>
Let's understand opportunity cost by examining the problem of whether I should have done <a href="http://niketanblog.blogspot.com/2009/09/meta-phd.html">PhD</a> after my Masters. After my Masters, I had two options:<br />
<ul style="text-align: left;">
<li>Accept the job offer for Software Development Engineer (SDE) position in Microsoft SQL Server Data Mining team. The salary of this position is $ <i>x</i> per year.</li>
<li>Do 5-year PhD program at Rice university. Rice University gives PhD students a stipend of $ <i>y</i> per year. After PhD, I got an offer from IBM Research with a salary of $ <i>x + z </i>per year<i>.</i> </li>
</ul>
Assuming the compensation at Rice, Microsoft and IBM remain constant, the opportunity cost associated with doing a PhD after <i>n</i> years is $ <i>x*n - (x + z)*(n - 5) - 5y.</i> The below figure (generated by crude approximation) shows that it makes sense to do PhD if <i>n</i> > 10.5, where the opportunity cost becomes 0.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Jla6qN56FJU/WTTtekDDNlI/AAAAAAAAPmM/g0khy3HD6cckd0UvXEk-8K3ato0B6ilXACLcB/s1600/opportunity_cost.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="534" data-original-width="857" height="248" src="https://1.bp.blogspot.com/-Jla6qN56FJU/WTTtekDDNlI/AAAAAAAAPmM/g0khy3HD6cckd0UvXEk-8K3ato0B6ilXACLcB/s400/opportunity_cost.png" width="400" /></a></div>
<br />
The above example ignores the rate of increase of compensation of a Masters vs PhD student and also the intangible costs such as experience at Microsoft, potential cutting-edge role at IBM, guidance under Chris, etc.<br />
<h3 style="text-align: left;">
Specialization and Voluntary Trade</h3>
In the book <a href="https://en.wikipedia.org/wiki/The_Wealth_of_Nations">The Wealth of Nations</a>, Adam Smith highlights that two key points:<br />
<ol style="text-align: left;">
<li>Workers produce more when they occupy <i>specialized</i> roles, so businesses can offer higher quality products at lower prices. </li>
<li><i>Trade</i> benefits both buyer and seller (i.e. lower opportunity cost for both of them). Note, this is only true <i>iff</i> both the buyer and seller are engaged in the trade <i>voluntarily</i> and if both have complete <i>information. </i>The counter-examples for voluntary exchange are slavery, human trafficking and fraud, where government regulations are necessary.</li>
</ol>
To summarize, a corporation should attempt to:<br />
<u>Step 1</u>: <b>specialize</b> such that it is the lowest cost provider<br />
<u>Step 2</u>: apply <b>opportunity cost</b> in decision making, risk assessment (risk vs return) and pricing analysis (for example: if supply is high, price is $1 less than second cheapest seller and if supply is low, the price is $1 more than second highest bidder/buyer).<br />
<u>Step 3</u>: <b>exchange</b> between two opportunity cost such that both parties are better off (i.e. producer sells to make profit and the consumer buys as it is more expensive to make it).<br />
<br />
<h3 style="text-align: left;">
Monetary system</h3>
Except countries like Germany, South Korea, Switzerland, Norway, etc, the government spends more than its revenues. The difference is called as deficit spending or budget deficit or simply deficit.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-NJyXxYjpM2I/Wl2Ys3U4QCI/AAAAAAAARB4/zB1FsNgO3MgtnQp8VOYy1ZOaR4-MfkvXgCLcBGAs/s1600/deficit.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="919" data-original-width="1252" height="467" src="https://4.bp.blogspot.com/-NJyXxYjpM2I/Wl2Ys3U4QCI/AAAAAAAARB4/zB1FsNgO3MgtnQp8VOYy1ZOaR4-MfkvXgCLcBGAs/s640/deficit.JPG" width="640" /></a></div>
To pay for the deficit, the Treasury borrows the money from the banks by issuing them treasury bonds. Treasury bonds offer interests and promise by the government to pay the principal and hence are considered as safest investments. However, in recent times, countries like Ukraine, Argentina, Greece, Zimbabwe and Russia were either defaulted or had to restructure their debts (see <a href="https://en.wikipedia.org/wiki/List_of_sovereign_debt_crises">wikipedia</a> for the exhaustive list). The big three rating agencies - Standard & Poor's, Moody's and Fitch - rate countries on their ability to pay interest, refinance and repay them.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-OYGimFhKv3Q/Wl2gBR_u51I/AAAAAAAARCI/7HdeANO3Pz4v-TzaOn1hVHsMYqDH4m3WwCLcBGAs/s1600/SnPRating.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="556" data-original-width="1026" height="345" src="https://2.bp.blogspot.com/-OYGimFhKv3Q/Wl2gBR_u51I/AAAAAAAARCI/7HdeANO3Pz4v-TzaOn1hVHsMYqDH4m3WwCLcBGAs/s640/SnPRating.JPG" width="640" /></a></div>
<br />
Then through Open Market Operations (or OMO), the banks gets to sell the treasury bonds (or mortgage-backed securities) to the Federal Reserve (or Fed for short) at a profit. Interesting point to note here: Banks give Fed the treasury bonds and Fed adds "credit" to the bank reserves. As countries' central bank, Fed has unique power to create credit out of thin air without having money to back the credit up ... often referred to as "printing money".<br />
<br />
Other than buying treasury bonds, Fed has another tool at its disposal: the <b>reserve requirement</b> (or cash reserve ratio or liquidity ratio), which refers to percentage of cash that must be kept at Fed or in the vault by a commercial. If a bank makes too many loans or if there is a sudden demand for withdrawals, then amount on money in the bank's vault falls below the reserve requirements. In this case, the bank first attempts to borrow money from other banks at the <b>fed funds rate</b> and then the remaining amount from the Fed at the <b>discount rate</b>. An equivalent interbank lending rate to feds funds rate in UK is LIBOR rate - the London Interbank Offered Rate (set by the British Banker's Association).<br />
<br />
In practice, there is no standard fed funds rate, instead it is the weighted average of the interest rate banks charge each other overnight to meet the reserve requirements. The Federal Open Market Committee (FOMC) meets eight times a year to set the target fed funds rate and then by buying/selling of treasury bonds, it attempts to achieve the target fed funds rate. The Fed funds rate affect the prime rate (for setting home equity lines of credit, auto loans, personal loans and credit card rates) and 11th District cost of funds index (for setting adjustable-rate mortgages).<br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="144"><div class="MsoNormal">
<br /></div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
Jan 2018</div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
Dec 2017</div>
</td>
<td style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
Jan 2017</div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="144"><div class="MsoNormal">
<a href="https://www.bankrate.com/rates/interest-rates/wall-street-prime-rate.aspx">Prime rate</a> reported by the Wall Street Journal's bank survey</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
4.5</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
4.25</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
3.75</div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="144"><div class="MsoNormal">
Discount rate</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
2</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
1.75</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
1.25</div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="144"><div class="MsoNormal">
Fed funds rate</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
1.5</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
1.5</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
0.75</div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="144"><div class="MsoNormal">
11<sup>th</sup> District cost of funds</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
0.746</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
0.737</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 119.7pt;" valign="top" width="141"><div class="MsoNormal">
0.603</div>
</td>
</tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-5AMwmMN8tts/Wl5f3_Ko_lI/AAAAAAAARCk/03MzqF3xdHEiQFMV79giMxYaYWxt00YRQCLcBGAs/s1600/fed%2Bfunds%2Brate.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="895" data-original-width="1299" height="440" src="https://1.bp.blogspot.com/-5AMwmMN8tts/Wl5f3_Ko_lI/AAAAAAAARCk/03MzqF3xdHEiQFMV79giMxYaYWxt00YRQCLcBGAs/s640/fed%2Bfunds%2Brate.jpg" width="640" /></a></div>
<br />
Treasury then can use the money it received by selling the bonds to fund the deficit. Bank can use the credit it received from Fed and the money deposited by its customers minus the reserve requirements to make loans. This is called as fractional reserve lending and it works as follows. Let's assume that reserve requirements is 10% and X deposits $1000 in bank A. The bank A keeps $100 in its vault and lends remaining $900 to Y who needs it to fund her business. Y has an account in bank B and deposits $900 in her account which she intends to remove as and when needed. Bank B keeps $90 in its vault and lends remaining $810 to C who in turns deposits in bank W. Now, when A, B and C check their accounts in their respective banks, they will see that they have $1000, $900 and $810 respectively, that is the initial $1000 has grown to $2710 in the economy. This system fails if there is a mass panic and A, B and C all decide to remove the amount at once (commonly referred to as bank run).<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-wJje7kc0I0o/Wl5iwMf9MRI/AAAAAAAARCw/AMOx18q21OQq_FSzV3rop1xZzA8eGKE4gCLcBGAs/s1600/economy.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="500" data-original-width="650" height="307" src="https://3.bp.blogspot.com/-wJje7kc0I0o/Wl5iwMf9MRI/AAAAAAAARCw/AMOx18q21OQq_FSzV3rop1xZzA8eGKE4gCLcBGAs/s400/economy.png" width="400" /></a></div>
<div style="text-align: left;">
Before we move ahead, it is a good idea to classify different types of banks:</div>
<div style="text-align: left;">
</div>
<ol style="text-align: left;">
<li>Supranational banks:</li>
<ol>
<li>The International Monetary Fun (IMF) ensures the stability of the international economic system and has 187 member countries.</li>
<li>World bank provides financial help to combat poverty, malnutrition and lack of communications and infrastructure to the developing countries.</li>
</ol>
<li>Central banks such as Federal Reserve, European Central Bank, Bank of England, Reserve Bank of India (RBI), Swiss National Bank and The People's Bank of China (PBC), set monetary policy described in the below section.</li>
<li>Commercial bank's business model is to charge higher interest rate for loan than they pay depositors.</li>
<li>Investment banks provide three functions: Merger and Acquisition (M&E) facility for corporations to raise debt or equity, Institutional Client Services which includes Fixed Income, Commodities and Currencies (FICC) and Asset/Wealth Management for high net worth clients.</li>
</ol>
<br />
<h3 style="text-align: left;">
Government intervention</h3>
The government keeps the economy growing (close to long-run trend rate of 2.5%), limits unemployment and keep inflation low (inflation target of 2%) by fiscal and monetary policies:<br />
<ol style="text-align: left;">
<li><b>Fiscal policy</b>: is set by the US Congress, the President and the Treasury Secretary, which involves changing government spending and taxation. </li>
<li><b>Monetary policy</b>: is set by the central bank (i.e. Federal Reserve in US), which involves influencing the demand and supply of money, primarily through the use of interest rates.</li>
</ol>
Inflation refers to a general increase in the price of goods and services and is caused by increase in the supply of money. Though it is hard to define/measure precisely, it is commonly measured by consumer price index (or CPI) which is weighted average change in the prices of consumer goods and services. Government love inflation but just to the right amount (~2%) as it makes debt cheaper (as dollar becomes cheaper), increases taxes (as prices increases) and increases exports. On the other hand, fixed income retirees and importers (such as oil) hate inflation.<br />
<br />
A country is in recession when two successive quarters, or a six months, show a decrease in real GDP. Hence, during recession, we get a fall in the inflation rate. However, deflation is when we get a negative inflation rate i.e. falling prices. Since the second world war, recessions have not led to deflation – just a lower inflation rate. A depression is a severe recession.<br />
<br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td colspan="3" style="border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 217.5pt;" valign="top" width="280"><div align="center" class="MsoNormal" style="text-align: center;">
<br /></div>
</td>
<td colspan="2" style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 102.85pt;" valign="top" width="102"><div align="center" class="MsoNormal" style="text-align: center;">
<b>Fiscal Policy</b></div>
</td>
<td colspan="3" style="border-left: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 158.45pt;" valign="top" width="199"><div align="center" class="MsoNormal" style="text-align: center;">
<b>Monetary Policy</b></div>
</td>
</tr>
<tr>
<td colspan="2" rowspan="2" style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 136.75pt;" valign="top" width="175"><div align="center" class="MsoNormal" style="text-align: center;">
<br /></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 80.75pt;" valign="top" width="105"><div align="center" class="MsoNormal" style="text-align: center;">
Set by:</div>
</td>
<td colspan="2" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 102.85pt;" valign="top" width="102"><div align="center" class="MsoNormal" style="text-align: center;">
US Congress,
President and Treasury Secretary</div>
</td>
<td colspan="3" style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 158.45pt;" valign="top" width="199"><div align="center" class="MsoNormal" style="text-align: center;">
Federal Reserve</div>
</td>
</tr>
<tr>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 80.75pt;" valign="top" width="105"><div align="center" class="MsoNormal" style="text-align: center;">
Measured by</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 48.9pt;" valign="top" width="35"><div align="center" class="MsoNormal" style="text-align: center;">
<b>Tax</b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 53.95pt;" valign="top" width="67"><div align="center" class="MsoNormal" style="text-align: center;">
<b>Spending</b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 45.85pt;" valign="top" width="60"><div align="center" class="MsoNormal" style="text-align: center;">
<b>Interest Rates</b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 41.05pt;" valign="top" width="47"><div align="center" class="MsoNormal" style="text-align: center;">
<b>Treasury Bonds</b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 71.55pt;" valign="top" width="92"><div align="center" class="MsoNormal" style="text-align: center;">
<b>Reserve
requirements</b></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 57.3pt;" valign="top" width="73"><div align="center" class="MsoNormal" style="text-align: center;">
<b>Inflation</b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 79.45pt;" valign="top" width="102"><div align="center" class="MsoNormal" style="text-align: center;">
Prices ↑</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 80.75pt;" valign="top" width="105"><div align="center" class="MsoNormal" style="text-align: center;">
Consumer Price
Index (CPI)</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 48.9pt;" valign="top" width="35"><div align="center" class="MsoNormal" style="text-align: center;">
↑</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 53.95pt;" valign="top" width="67"><div align="center" class="MsoNormal" style="text-align: center;">
↓</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 45.85pt;" valign="top" width="60"><div align="center" class="MsoNormal" style="text-align: center;">
↑</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 41.05pt;" valign="top" width="47"><div align="center" class="MsoNormal" style="text-align: center;">
Sell</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 71.55pt;" valign="top" width="92"><div align="center" class="MsoNormal" style="text-align: center;">
Increase</div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid windowtext 1.0pt; mso-border-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 57.3pt;" valign="top" width="73"><div align="center" class="MsoNormal" style="text-align: center;">
<b>Recession</b></div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 79.45pt;" valign="top" width="102"><div align="center" class="MsoNormal" style="text-align: center;">
Low output and unemployment</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 80.75pt;" valign="top" width="105"><div align="center" class="MsoNormal" style="text-align: center;">
GDP/GNP, Unemployment
rate</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 48.9pt;" valign="top" width="35"><div align="center" class="MsoNormal" style="text-align: center;">
↓</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 53.95pt;" valign="top" width="67"><div align="center" class="MsoNormal" style="text-align: center;">
↑</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 45.85pt;" valign="top" width="60"><div align="center" class="MsoNormal" style="text-align: center;">
↓</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 41.05pt;" valign="top" width="47"><div align="center" class="MsoNormal" style="text-align: center;">
Buy</div>
</td>
<td style="border-bottom: solid windowtext 1.0pt; border-left: none; border-right: solid windowtext 1.0pt; border-top: none; mso-border-alt: solid windowtext .5pt; mso-border-left-alt: solid windowtext .5pt; mso-border-top-alt: solid windowtext .5pt; padding: 0in 5.4pt 0in 5.4pt; width: 71.55pt;" valign="top" width="92"><div align="center" class="MsoNormal" style="text-align: center;">
Decrease</div>
</td>
</tr>
</tbody></table>
<div>
<br /></div>
For example: to reduce inflationary pressures,<br />
<div>
<ul style="text-align: left;">
<li>a fiscal policy will involve higher taxes and lower spending. The advantage of using fiscal policy is that it will help to reduce the budget deficit, but is often difficult to implement due to political reasons.</li>
<li>a monetary policy (in this case termed as tight monetary policy) will attempt to <i>reduce the supply of money</i> by raising the interest rates and borrowing rates among banks, selling government bonds (also called as "open market operations" or OMO) and by increasing the rate of minimum reserve to be kept by commercial banks towards central bank. </li>
</ul>
Extreme situations:<br />
<div style="text-align: left;">
</div>
<ul style="text-align: left;">
<li><b>Buying too many treasury bonds:</b> During extreme recession or to avoid deflation, Fed buys lots of government debt so as to inject more money into economy and encourage growth. This is called as <b>Quantitative Easing</b> as was pioneered by the Bank of Japan in 1990s. In 2009 financial crisis, Fed and Bank of England purchased over $2 trillion and $520 billions government debt respectively. </li>
<li>Comprehensive Financial Reform Acts: </li>
<ul>
<li>After 1929 stock market crash, <a href="https://en.wikipedia.org/wiki/Glass%E2%80%93Steagall_legislation">Glass Steagall Act</a> was enacted which separated investment banking from commercial/retail banking to prevent bank run. This act was repealed in 1999 by the Gramm-Leach-Bliley Act allowing the banks to invest depositors' funds in unregulated derivatives. This led to banks becoming too big to fail and requiring bailout in 2008-2009 to avoid another depression.</li>
<li>After 2008-2009 housing crisis, <a href="https://en.wikipedia.org/wiki/Dodd%E2%80%93Frank_Wall_Street_Reform_and_Consumer_Protection_Act">Dodd-Frank Act</a> was passed which increased regulations on unregulated derivatives trading and created Consumer Finance Protection Bureau that took actions against predatory student lending practices and payday loan scams. Much of these regulations might be rolled back <a href="https://en.wikipedia.org/wiki/Financial_CHOICE_Act">Financial CHOICE Act</a> is enacted.</li>
</ul>
<li><a href="https://en.wikipedia.org/wiki/Stagflation">Stagflation</a> (= Inflation + Recession at the same time) is a condition of slow economic growth and relatively high unemployment accompanied by a rise in prices, or inflation. It occurs due to supply shock (such rapid increase in price of oil) and due to policies that harm industry while growing money supply too quickly. Venezuela is an example of stagflation where if you fix inflation, you create depression and if you fix recession, you create hyperinflation.</li>
<li>When inflation rate increases by ~ 50% monthly, it is referred to as <b>hyperinflation</b>. Germany after Treaty of Versailles (322% inflation rate), Hungary in 1945 (19,000% inflation rate) and Zimbabwe in 2008 (230 million % inflation rate) had some of the examples of hyperinflation.</li>
</ul>
Economist track the economic growth, unemployment and inflation using the following three measurements:<br />
<br />
<b>1. Gross Domestic Product (GDP):</b> is the value of all the finished goods and services (hence product) produced within a country's border (hence domestic) in a specific time period, usually a year.<br />
<div>
- Since it is "gross", the depreciation in the capital asset is not deducted. If it where deducted, then it becomes "net".</div>
<div>
- Since it is "domestic", it does not take into account the country's earning outside its geographical boundaries, or foreign remittances. Gross National Product (GNP) is GDP + income earned by residents from overseas investments - income earned within the domestic economy by overseas residents.</div>
<div>
- Since it is "product", the intermediate goods are not taken into account. For example, wheat sold for final consumption to consumers will be included into GDP, but not the amount of wheat sold to bakeries for the production of bread.<br />
<div>
- GDP is commonly used as an indicator of the economic health of a country and can be computed in three principal ways: from total output, from income and from expenditure.</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-uYNqKpC2KhY/Wl1LfOmif-I/AAAAAAAARBc/eWoWTkkqBmcpZDYa8HTh6Z0k2yvksOAjQCEwYBhgL/s1600/GDP.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="703" data-original-width="1592" height="281" src="https://4.bp.blogspot.com/-uYNqKpC2KhY/Wl1LfOmif-I/AAAAAAAARBc/eWoWTkkqBmcpZDYa8HTh6Z0k2yvksOAjQCEwYBhgL/s640/GDP.JPG" width="640" /></a></div>
<div>
<br /></div>
<div>
<div>
<b>2. Unemployment rate</b>: is the number of unemployed people is divided by the number of people in the labor force times 100. Note, it defines unemployed people as those who are willing and available to work, and who have actively sought work within the past four weeks. Hence, little kids don't count, neither do people who are unable to work due to their age or disability or who chose not to work. The below figure shows the unemployment rate per country in 2013:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-Vcfa0clOeVo/Wl1WHEua4ZI/AAAAAAAARBo/8XhffWNoCBcqHzD16dBxq23VKmDxQd5KACLcBGAs/s1600/unemployment%2Brate.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="561" data-original-width="1082" height="329" src="https://4.bp.blogspot.com/-Vcfa0clOeVo/Wl1WHEua4ZI/AAAAAAAARBo/8XhffWNoCBcqHzD16dBxq23VKmDxQd5KACLcBGAs/s640/unemployment%2Brate.JPG" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>3. Inflation rate:</b> measured by consumer price index (or CPI).<br />
<br />
<h2 style="text-align: left;">
Reference:</h2>
1. Economics - Hubbard and O'Brien.<br />
2. https://www.investopedia.com/<br />
3. The Book of Money - Conaghan and Smith.<br />
4. Macroeconomics: Crash Course: https://www.youtube.com/watch?v=d8uTB5XorBw<br />
5. The Monetary System Visually Explained: https://www.youtube.com/watch?v=23DNe0cJhcU<br />
6. https://www.thebalance.com/open-market-operations-3306121<br />
7. https://www.bankrate.com/rates/interest-rates/prime-rate.aspx<br />
8. https://www.thebalance.com/dodd-frank-wall-street-reform-act-3305688</div>
</div>
</div>
</div>
</div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-73471549110293007042017-11-30T14:09:00.001-05:002017-11-30T17:44:06.898-05:00How to debug the unicode error in Py4J<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Many Java-based big data systems such as SystemML, Spark's MLLib, CaffeOnSpark, etc that use the Py4J bridge in their Python APIs often throw a cryptic error instead of detailed Java stacktrace:<br />
<blockquote class="tr_bq">
py4j.protocol.Py4JJavaError: < exception str() failed >
</blockquote>
</div>
For more details on this issue, please refer to <a href="https://docs.python.org/2.7/howto/unicode.html#the-unicode-type">Python's unicode documentation</a>.<br />
<br />
Luckily, there is a very simple hack to extract the stack trace in such a situation. Let's assume that SystemML's MLContext is throwing the above Py4J error: ml.execute(script). In this case, simply wrap that code in the following try/except block:<br />
<script src="https://gist.github.com/niketanpansare/f79488679804f60351e38faf9cb30e89.js"></script>
</div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-36523285372940174992017-10-16T19:16:00.000-04:002019-02-01T19:58:51.727-05:00Git notes for SystemML committers<div dir="ltr" style="text-align: left;" trbidi="on">
Replace <span style="background-color: #9fc5e8;">niketanpansare</span> below with your git username:<br />
<br />
1. Fork and then Clone the repository<br />
<blockquote class="tr_bq">
https://github.com/<span style="background-color: #9fc5e8;">niketanpansare</span>/systemml.git</blockquote>
<br />
2. Update the configuration file: systemml/.git/config<br />
<blockquote class="tr_bq">
[core]<br />
<span style="white-space: pre;"> </span>symlinks = false<br />
<span style="white-space: pre;"> </span>repositoryformatversion = 0<br />
<span style="white-space: pre;"> </span>filemode = false<br />
<span style="white-space: pre;"> </span>logallrefupdates = true<br />
[remote "origin"]<br />
<span style="white-space: pre;"> </span>url = https://github.com/<span style="background-color: #9fc5e8;">niketanpansare</span>/systemml.git<br />
<span style="white-space: pre;"> </span>fetch = +refs/heads/*:refs/remotes/origin/*<br />
[branch "master"]<br />
<span style="white-space: pre;"> </span>remote = origin<br />
<span style="white-space: pre;"> </span>merge = refs/heads/master<br />
[remote "upstream"]<br />
<span style="white-space: pre;"> </span>url = https://github.com/apache/systemml.git<br />
<span style="white-space: pre;"> </span>fetch = +refs/heads/*:refs/remotes/upstream/*<br />
[remote "apache"]<br />
<span style="white-space: pre;"> </span>url = https://gitbox.apache.org/repos/asf/systemml.git<br />
<span style="white-space: pre;"> </span>fetch = +refs/heads/*:refs/remotes/apache/*<br />
<br /></blockquote>
The above configuration states that: "origin" points to your fork and "upstream" points to the github mirror of the main repository, which is referred to as "apache".<br />
<br />
Optional but recommended: Update the <span style="background-color: white; font-family: "menlo"; font-size: 13px;">~/.gitconfig </span>file:<br />
<blockquote class="tr_bq">
[user]<br />
name = Niketan Pansare<br />
email = npansar@us.ibm.com<br />
[alias]<br />
systemml-pr = "!f() { git fetch ${2:-upstream} pull/$1/head:pr-$1 && git checkout pr-$1; }; f"</blockquote>
<br />
3. Now, let's you created a branch '<span style="background-color: #ffe599;">foo</span>' with <span style="background-color: #ea9999;">3</span> commits. You can <a href="https://help.github.com/articles/creating-a-pull-request/">create a PR</a> via <a href="https://github.com/apache/systemml">github website</a><br />
<br />
4. Once the PR is approved and if you are the committer, you can merge the PR:<br />
<blockquote class="tr_bq">
git checkout <span style="background-color: #ffe599;">foo</span><br />
git reset --soft HEAD~<span style="background-color: #ea9999;">3</span> && git commit<br />
git pull --rebase apache master<br />
git checkout master<br />
git pull apache master<br />
git merge <span style="background-color: #ffe599;">foo</span><br />
gitk<br />
git log apache/master..<br />
git commit --amend --date="$(date +%s)"<br />
git push apache master</blockquote>
<div>
Let's discuss the above commands in more detail:<br />
- git checkout <span style="background-color: #ffe599;">foo</span>: ensures that you are working on the right branch.<br />
- git reset --soft HEAD~<span style="background-color: #ea9999;">3</span> && git commit: squashes all your commits into one single commit in preparation for the merge.<br />
- git pull --rebase apache master: rebases your local master branch with the apache.<br />
- git checkout master: switch to the local master branch<br />
- git merge <span style="background-color: #ffe599;">foo</span>: merge the branch foo with local master<br />
- gitk: to double-check visually if local and remote branches are all in synch and if the commit is not creating weird loops.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
- git log apache/master..: double-check if the log makes sense and if you have added "Closes #PR." at the end of the log and correct JIRA number as the header.<br />
- git commit --amend --date="$(date +%s)": if not, allows you to modify your log message.<br />
- git push apache master: go ahead and push your commit to apache :)<br />
<br />
5. If you are a committer and want to merge a PR (let's assume PR <span style="background-color: #6fa8dc;">100</span>), you need to create a local branch using the following command:<br />
<blockquote class="tr_bq">
<span class="s1">git systemml-pr<span class="Apple-converted-space"> <span style="background-color: #6fa8dc;">100</span></span></span><br />
git checkout pr-<span style="background-color: #6fa8dc;">100</span></blockquote>
Also, if you had to squash the commits as described above, please use the below command to amend the author:<br />
<br />
<blockquote class="tr_bq">
git commit --amend --author="FirstName LastName <contributor@blah.com>"
</blockquote>
<br /></div>
6. If you have modified the docs/ folder, you can update the documentation:<br />
<blockquote class="tr_bq">
git
subtree push --prefix docs apache gh-pages</blockquote>
<br />
Before, updating the documentation, you can sanity test on your branch by:<br />
<blockquote class="tr_bq">
git
subtree push --prefix docs origin gh-pages</blockquote>
<br />
If the above command fails, you can delete the gh-pages branch and re-execute the command:<br />
<blockquote class="tr_bq">
git
push origin --delete gh-pages</blockquote>
<br />
Advanced commands:<br />
1. Cherry-picking:<br />
<blockquote class="tr_bq">
git log --pretty=format:"%h - %an, %ar : %s" | grep Niketan<br />
git checkout gpu<br />
git cherry-pick</blockquote>
<div>
<br /></div>
<br />
Notes from other committers:<br />
Deron: <a href="https://gist.github.com/deroneriksson/e0d6d0634f3388f0df5e#integrate-pull-request-with-one-commit-into-apache-master">https://gist.github.com/deroneriksson/e0d6d0634f3388f0df5e#integrate-pull-request-with-one-commit-into-apache-master</a><br />
Mike: <a href="https://gist.github.com/dusenberrymw/78eb31b101c1b1b236e5">https://gist.github.com/dusenberrymw/78eb31b101c1b1b236e5</a><br />
<style type="text/css"> p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures} </style></div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-60482033100131423862015-11-13T15:26:00.000-05:002015-11-15T00:36:48.194-05:00Using Google's TensorFlow for Kaggle competition<div dir="ltr" style="text-align: left;" trbidi="on">
Recently, Google Brain team released their neural network library 'TensorFlow'. Since Google has a <a href="http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html">state-of-the-art Deep Learning system</a>, I wanted to explore TensorFlow by trying it out for my first Kaggle submission (<a href="https://www.kaggle.com/c/digit-recognizer" target="_blank">Digit Recognition</a>) . After spending few hours getting to know their jargons/APIs, I modified their <a href="http://tensorflow.org/tutorials/mnist/pros/index.md">multilayer convolution neural network</a> and came 140th (with 98.4% accuracy) ... not too shabby for my first submission :D.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-7Ix7FaevRds/VkZAzS2GNdI/AAAAAAAAOvI/9QuV8cFH1MM/s1600/140.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="19" src="http://4.bp.blogspot.com/-7Ix7FaevRds/VkZAzS2GNdI/AAAAAAAAOvI/9QuV8cFH1MM/s320/140.JPG" width="320" /></a></div>
<br />
If you are interested in outscoring me, apply <a href="http://scikit-learn.org/stable/modules/cross_validation.html">cross-validation</a> on the below python code or may be consider using ensembles:<br />
<pre class="python" name="code">from input_data import *
import pandas
class DataSets(object):
pass
mnist = DataSets()
df = pandas.read_csv('train.csv')
train_images = numpy.multiply(df.drop('label', 1).values, 1.0 / 255.0)
train_labels = dense_to_one_hot(df['label'].values)
#Add MNIST data from Yan LeCun's website for better accuracy. We hold out test, just for accuracy sake, but could have easily added it :)
mnist2 = read_data_sets("/tmp/data/", one_hot=True)
train_images = numpy.concatenate((train_images, mnist2.train._images), axis=0)
train_labels = numpy.concatenate((train_labels, mnist2.train._labels), axis=0)
VALIDATION_SIZE = 5000
validation_images = train_images[:VALIDATION_SIZE]
validation_labels = train_labels[:VALIDATION_SIZE]
train_images = train_images[VALIDATION_SIZE:]
train_labels = train_labels[VALIDATION_SIZE:]
mnist.train = DataSet([], [], fake_data=True)
mnist.train._images = train_images
mnist.train._labels = train_labels
mnist.validation = DataSet([], [], fake_data=True)
mnist.validation._images = validation_images
mnist.validation._labels = validation_labels
df1 = pandas.read_csv('test.csv')
test_images = numpy.multiply(df1.values, 1.0 / 255.0)
numTest = df1.shape[0]
test_labels = dense_to_one_hot(numpy.repeat([1], numTest))
mnist.test = DataSet([], [], fake_data=True)
mnist.test._images = test_images
mnist.test._labels = test_labels
import tensorflow as tf
sess = tf.InteractiveSession()
x = tf.placeholder("float", [None, 784]) # x is input features
W = tf.Variable(tf.zeros([784,10])) # weights
b = tf.Variable(tf.zeros([10])) # bias
y_ = tf.placeholder("float", [None,10]) # y' is input labels
#Weight Initialization
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def bias_variable(shape):
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial)
# Convolution and Pooling
def conv2d(x, W):
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def max_pool_2x2(x):
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
# First Convolutional Layer
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
x_image = tf.reshape(x, [-1,28,28,1])
h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)
# Second Convolutional Layer
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)
# Densely Connected Layer
W_fc1 = weight_variable([7 * 7 * 64, 1024])
b_fc1 = bias_variable([1024])
h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
# Dropout
keep_prob = tf.placeholder("float")
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)
# Readout Layer
W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])
y=tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)
cross_entropy = -tf.reduce_sum(y_*tf.log(y))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
tf.initialize_all_variables().run()
for i in range(20000):
batch = mnist.train.next_batch(50)
train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
print "test accuracy %g"%accuracy.eval(feed_dict={x: mnist2.test.images, y_: mnist2.test.labels, keep_prob: 1.0})
prediction = tf.argmax(y,1).eval(feed_dict={x: mnist.test.images, keep_prob: 1.0})
f=open('prediction.txt','w')
s1='\n'.join(str(x) for x in prediction)
f.write(s1)
f.close()
</pre>
The above script assumes that you have downloaded train.csv and test.csv from <a href="https://www.kaggle.com/c/digit-recognizer/data">Kaggle's website</a>, <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/tutorials/mnist/input_data.py">input_data.py</a> from TensorFlow's website and installed <a href="http://pandas.pydata.org/pandas-docs/stable/install.html">pandas</a>/TensorFlow.<br />
<br />
Here are few observations based on my experience playing with TensorFlow:<br />
<ol style="text-align: left;">
<li>TensorFlow <b>does not have an optimizer</b>: </li>
<ol>
<li>TensorFlow statically maps an high-level expression (for example "matmul") to a predefined low-level operator (for example: <a href="https://github.com/tensorflow/tensorflow/blob/d6357a5849db980df51d00d8a9ff874cda2faeb3/tensorflow/core/kernels/matmul_op.h">matmul_op.h</a>) based on whether you are using CPU or GPU enabled TensorFlow. On other hand, <a href="http://researcher.watson.ibm.com/researcher/view_group.php?id=3174">SystemML</a> compiles a <a href="https://github.com/SparkTC/systemml/blob/master/system-ml/src/main/java/com/ibm/bi/dml/parser/BinaryExpression.java">matrix multiplication expression</a> (X %*% y) into one of <a href="https://github.com/SparkTC/systemml/blob/master/system-ml/docs/devdocs/MatrixMultiplicationOperators.txt">many matrix-multiplication related physical operators</a> using a sophisticated optimizer that adapts to the underlying data and cluster characteristics.</li>
<li>Other popular open-source neural network libraries are <a href="http://caffe.berkeleyvision.org/">Caffe</a>, <a href="http://deeplearning.net/software/theano/">Theano</a> and <a href="http://torch.ch/">Torch</a>. </li>
</ol>
<li>TensorFlow is <b>a parallel, but not a distributed system</b>:</li>
<ol>
<li>It does parallelize its computation (across CPU cores and also across <a href="http://tensorflow.org/how_tos/using_gpu/index.md">GPUs</a>):
<a href="http://3.bp.blogspot.com/-Pn9pI7D3CMM/VkY6vHrOw8I/AAAAAAAAOu4/g0npiLwizEo/s1600/NumCores.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="40" src="http://3.bp.blogspot.com/-Pn9pI7D3CMM/VkY6vHrOw8I/AAAAAAAAOu4/g0npiLwizEo/s320/NumCores.JPG" width="320" /></a> </li>
<li>Google has released only the single-node version and kept distributed version in-house. This means that the open-sourced version does not have a parameter server. </li>
</ol>
<li>TensorFlow is <b>easy to use, but difficult to debug</b>:</li>
<ol>
<li>I like TensorFlow's Python API and if the script/data/parameters are all correct, it works absolutely fine :)</li>
<li>But, if something fails, the error messages thrown by TensorFlow are difficult to decipher. This is because the error messages point to a generated physical operator (for example: tensorflow.python.framework.errors.InvalidArgumentError: ReluGrad input), not to the line of code in the Python program.</li>
</ol>
<li>TensorFlow is <b>slow to train and not yet robust enough</b>:</li>
<ol>
<li>Here are some initial numbers by <a href="http://alex.smola.org/">Alex Smola</a> comparing TensorFlow to other open-source deep learning systems: <br /> <a href="http://3.bp.blogspot.com/--AA1HiBpMto/VkeDZbPAFkI/AAAAAAAAOvc/A4A3QiBj_30/s1600/TensorFlow.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="115" src="http://3.bp.blogspot.com/--AA1HiBpMto/VkeDZbPAFkI/AAAAAAAAOvc/A4A3QiBj_30/s320/TensorFlow.JPG" width="320" /></a></li>
</ol>
</ol>
</div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-57331721377552381352015-02-17T12:40:00.003-05:002015-11-14T21:39:15.992-05:00Plotting as a useful debugging tool<div dir="ltr" style="text-align: left;" trbidi="on">
While writing the code for bayesian modeling, you will have to test
the distribution (prior, likelihood or posterior). Here are two common
scenarios that you might encounter:<br />
<ol>
<li>You want to test the function that generates <span style="text-decoration: underline;">random deviates</span>. For example: you have derived a conjugate formula for a parameter of your model and want to test whether it is correct or not.</li>
<li>You want to test a probability <span style="text-decoration: underline;">density function</span>. For example: likelihood or posterior function that you might want to run rejection sampler on.</li>
</ol>
In both these cases, you will start by making sure the property of
the distribution (for example: range, mean, variance) are correct. For
example: if the parameter you are sampling is variance, then you will
have “assert(returnedVariance > 0)” in your code. Then, the next
obvious test should be visual inspection (trust me, it has helped me
catch more bugs than I would by traditional programming debugging
techniques/tools). This means you will plot the distribution and see if
the output of your code makes sense.<br />
We will start by simplifying the above two cases by assuming standard
normal distribution. So, in the first case, we have access to “rnorm”
function and in second case, we have access to “dnorm” function of R.<br />
Case 1: In this case, we first collect random deviates (in “vals”) and then use ggplot to plot them:<br />
<blockquote class="tr_bq">
<pre name="code" class="python">
library(ggplot2)
vals = rnorm(10000, mean=0, sd=1)
df = data.frame(xVals=vals)
ggplot(df, aes(x=xVals)) + geom_density()
</pre>
</blockquote>
<br />
The output of above R script will look something like this: <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-7ZcEU_YtiL8/VON8k7aWI2I/AAAAAAAAOVQ/qM0H9oankHM/s1600/RnormPlot.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-7ZcEU_YtiL8/VON8k7aWI2I/AAAAAAAAOVQ/qM0H9oankHM/s1600/RnormPlot.jpg" height="320" width="320" /></a></div>
Case 2: In this case, we have to assume a bounding box and sample inside
that to get “x_vals” and “y_vals” (just like rejection sampling):<br />
<blockquote class="tr_bq">
<pre name="code" class="python">
library(ggplot2)
x_vals=runif(10000,min=-4,max=4)
y_vals=dnorm(x_vals, mean=0, sd=1)
df = data.frame(xVals=x_vals, yVals=y_vals)
ggplot(df, aes(x=xVals, y=yVals)) + geom_line()
</pre>
</blockquote>
<br />
The output of above R script will look something like this:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-v42-xG-fWAs/VON8z4K2lKI/AAAAAAAAOVY/0pysAOluzoE/s1600/DNormPlot.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-v42-xG-fWAs/VON8z4K2lKI/AAAAAAAAOVY/0pysAOluzoE/s1600/DNormPlot.jpg" height="320" width="320" /></a></div>
Just as a teaser to a post that I will post later, we can use the script
somewhat similar to that of case 1 to study the characteristics of a
distribution:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-arO0IuBF0aY/VON89ozUi5I/AAAAAAAAOVg/XtG2VMmU50o/s1600/GammaDist.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-arO0IuBF0aY/VON89ozUi5I/AAAAAAAAOVg/XtG2VMmU50o/s1600/GammaDist.png" height="320" width="320" /></a></div>
</div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-47000090849684670082015-02-17T12:30:00.003-05:002015-11-14T22:00:56.571-05:00How to setup “passwordless ssh” on Amazon EC2 cluster<div dir="ltr" style="text-align: left;" trbidi="on">
Often for running distributed applications, you may want to setup a
new cluster or tweak an existing one (running on Amazon EC2) to support
passwordless ssh. For creating a new cluster from scratch, there are lot
of cluster management tools (which is beyond the scope of this
blogpost). However, if all you want to do is setup “passwordless ssh”
between nodes, then this post might be worth your read.<br />
The script below assumes that you have completed following three steps:<br />
<br />
<b>Step 1.</b> Created RSA public keypair on each of the machine:<br />
<blockquote class="tr_bq">
<pre name="code" class="bash" style="font-family: monospace;"><span style="color: #7a0874; font-weight: bold;">cd</span> ~
<span style="color: #c20cb9; font-weight: bold;">ssh-keygen</span> <span style="color: #660033;">-t</span> rsa</pre>
</blockquote>
<pre class="bash" style="font-family: monospace;"></pre>
Do not enter any paraphrase, instead just press [enter].<br />
<br />
<b>Step 2.</b> Suppressed warning flags in ssh-config file:<br />
<blockquote class="tr_bq">
<pre class="bash" style="font-family: monospace;"><span style="color: #c20cb9; font-weight: bold;">sudo</span> <span style="color: #c20cb9; font-weight: bold;">vim</span> <span style="color: black; font-weight: bold;">/</span>etc<span style="color: black; font-weight: bold;">/</span>ssh<span style="color: black; font-weight: bold;">/</span>ssh_config
StrictHostKeyChecking no
<span style="color: #007800;">UserKnownHostsFile</span>=<span style="color: black; font-weight: bold;">/</span>dev<span style="color: black; font-weight: bold;">/</span>null</pre>
</blockquote>
<pre class="bash" style="font-family: monospace;"> </pre>
<b>Step 3.</b> Copied the key pair file “MyKeyPair.pem” to master’s home directory:<br />
<blockquote class="tr_bq">
<pre class="bash" style="font-family: monospace;"><span style="color: #c20cb9; font-weight: bold;">scp</span> <span style="color: #660033;">-i</span> <span style="color: black; font-weight: bold;">/</span>local-path<span style="color: black; font-weight: bold;">/</span>MyKeyPair.pem <span style="color: black; font-weight: bold;">/</span>local-path<span style="color: black; font-weight: bold;">/</span>MyKeyPair.pem ubuntu<span style="color: black; font-weight: bold;">@</span>ec2-master-public-address.compute-<span style="color: black;">1</span>.amazonaws.com:~</pre>
</blockquote>
<br />
Assuming that above three steps have been completed, run this script on
the master to enable passwordless ssh between master-slave and/or
slave-slave nodes: <br />
<pre name="code" class="python">
#!/bin/bash
# Author: Niketan R. Pansare
# Password-less ssh
# Make sure you have transferred your key-pair to master
if [ ! -f ~/.ssh/id_rsa.pub ]; then
echo "Expects ~/.ssh/id_rsa.pub to be created. Run ssh-keygen -t rsa from home directory"
exit
fi
if [ ! -f ~/MyKeyPair.pem ]; then
echo "For enabling password-less ssh, transfer MyKeyPair.pem to master's home folder (e.g.: scp -i /local-path/MyKeyPair.pem /local-path/MyKeyPair.pem ubuntu@master_public_dns:~)"
exit
fi
echo "Provide following ip-addresses"
echo -n -e "${green}Public${endColor} dns address of master:"
read MASTER_IP
echo ""
# Assumption here is that you want to create a small cluster ~ 10 nodes
echo -n -e "${green}Public${endColor} dns addresses of slaves (separated by space):"
read SLAVE_IPS
echo ""
echo -n -e "Do you want to enable password-less ssh between ${green}master-slaves${endColor} (y/n):"
read ENABLE_PASSWORDLESS_SSH
echo ""
if [ "$ENABLE_PASSWORDLESS_SSH" == "y" ]; then
# Copy master's public key to itself
#cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
for SLAVE_IP in $SLAVE_IPS
do
echo "Checking passwordless ssh between master -> "$SLAVE_IP
ssh -o PasswordAuthentication=no $SLAVE_IP /bin/true
IS_PASSWORD_LESS="n"
if [ $? -eq 0 ]; then
echo "Passwordless ssh has been setup between master -> "$SLAVE_IP
echo "Now checking passwordless ssh between "$SLAVE_IP" -> master"
ssh $SLAVE_IP 'ssh -o PasswordAuthentication=no' $MASTER_IP '/bin/true'
if [ $? -eq 0 ]; then
echo "Passwordless ssh has been setup between "$SLAVE_IP" -> master"
IS_PASSWORD_LESS="y"
fi
fi
if [ "$IS_PASSWORD_LESS" == "n" ]; then
# ssh-copy-id gave me lot of issues, so will use below commands instead
echo "Enabling passwordless ssh between master and "$SLAVE_IP
# Copy master's public key to slave
cat ~/.ssh/id_rsa.pub | ssh -i ~/MyKeyPair.pem "ubuntu@"$SLAVE_IP 'mkdir -p ~/.ssh ; cat >> ~/.ssh/authorized_keys'
# Copy slave's public key to master
ssh -i ~/MyKeyPair.pem "ubuntu@"$SLAVE_IP 'cat ~/.ssh/id_rsa.pub' >> ~/.ssh/authorized_keys
# Copy slave's public key to itself
ssh -i ~/MyKeyPair.pem "ubuntu@"$SLAVE_IP 'cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys'
fi
done
echo ""
echo "---------------------------------------------"
echo "Testing password-less ssh on master -> slave"
for SLAVE_IP in $SLAVE_IPS
do
ssh "ubuntu@"$SLAVE_IP uname -a
done
echo ""
echo "Testing password-less ssh on slave -> master"
for SLAVE_IP in $SLAVE_IPS
do
ssh "ubuntu@"$SLAVE_IP 'ssh ' $MASTER_IP 'uname -a'
done
echo "---------------------------------------------"
echo "Sorry, prefer to keep this check manual to avoid headache in Hadoop or any other distributed program."
echo -n -e "Do you see error or something fishy in above block (y/n):"
read IS_ERROR1
echo ""
if [ "$IS_ERROR1" == "y" ]; then
echo "I am sorry to hear this script didn't work for you :("
echo "Hint1: Its quite possible, slave doesnot contain ~/MyKeyPair.pem"
echo "Hint2: sudo vim /etc/ssh/ssh_config and add StrictHostKeyChecking no and UserKnownHostsFile=/dev/null to it"
exit
fi
fi
echo -n -e "Do you want to enable password-less ssh between ${green}slave-slave${endColor} (y/n):"
read ENABLE_PASSWORDLESS_SSH1
echo ""
if [ "$ENABLE_PASSWORDLESS_SSH1" == "y" ]; then
if [ "$ENABLE_PASSWORDLESS_SSH" == "n" ]; then
echo -n -e "In this part, the key assumption is that password-less ssh between ${green}master-slave${endColor} is enabled. Do you still want to continue (y/n):"
read ANS1
if [ "$ANS1" == "n" ]; then
exit
fi
echo ""
fi
for SLAVE_IP1 in $SLAVE_IPS
do
for SLAVE_IP2 in $SLAVE_IPS
do
if [ "$SLAVE_IP1" != "$SLAVE_IP2" ]; then
# Checking assumes passwordless ssh has already been setup between master and slaves
echo "[Warning:] Skipping checking passwordless ssh between "$SLAVE_IP1" -> "$SLAVE_IP2
IS_PASSWORDLESS_SSH_BETWEEN_SLAVE_SET="n"
# This will be true because ssh $SLAVE_IP1 is true
#ssh $SLAVE_IP1 ssh -o PasswordAuthentication=no $SLAVE_IP2 /bin/true
#if [ $? -eq 0 ]; then
if [ "$IS_PASSWORDLESS_SSH_BETWEEN_SLAVE_SET" == "n" ]; then
echo "Enabling passwordless ssh between "$SLAVE_IP1" and "$SLAVE_IP2
# Note you are on master now, which we assume to have
ssh -i ~/MyKeyPair.pem $SLAVE_IP1 'cat ~/.ssh/id_rsa.pub' | ssh -i ~/MyKeyPair.pem $SLAVE_IP2 'cat >> ~/.ssh/authorized_keys'
else
echo "Passwordless ssh has been setup between "$SLAVE_IP1" -> "$SLAVE_IP2
fi
fi
done
done
echo "---------------------------------------------"
echo "Testing password-less ssh on slave slave"
for SLAVE_IP1 in $SLAVE_IPS
do
for SLAVE_IP2 in $SLAVE_IPS
do
# Also, test password-less ssh on the current slave machine
ssh $SLAVE_IP1 'ssh ' $SLAVE_IP2 'uname -a'
done
done
echo "---------------------------------------------"
echo "Sorry, prefer to keep this check manual to avoid headache in Hadoop or any other distributed program."
echo -n -e "Do you see error or something fishy in above block (y/n):"
read IS_ERROR1
echo ""
if [ "$IS_ERROR1" == "y" ]; then
echo "I am sorry to hear this script didn't work for you :("
echo "Hint1: Its quite possible, slave doesnot contain ~/MyKeyPair.pem"
echo "Hint2: sudo vim /etc/ssh/ssh_config and add StrictHostKeyChecking no and UserKnownHostsFile=/dev/null to it"
exit
fi
fi
</pre>
<br />
Here is a sample output obtained by running the above script (src code available <a href="https://www.dropbox.com/s/a7rdqy2e94yblsc/enablePasswordLess.sh" target="_blank">via link</a>):<br />
<blockquote class="tr_bq">
<pre name="code" class="python">ubuntu@ip-XXX:~$ ./enablePasswordlessSSH.sh
Provide following ip-addresses
Public dns address of master:ec2-XXX-29.compute-1.amazonaws.com
Public dns addresses of slaves (separated by space):ec2-XXX-191.compute-1.amazonaws.com ec2-XXX-240.compute-1.amazonaws.com ec2-XXX-215.compute-1.amazonaws.com ec2-XXX-192.compute-1.amazonaws.com ec2-XXX-197.compute-1.amazonaws.com
Do you want to enable password-less ssh between master-slaves (y/n):y
Checking passwordless ssh between master -> ec2-XXX-YYY.compute-1.amazonaws.com
bunch of warning and possibly few "Permission denied (publickey)." (Ignore this !!!)
---------------------------------------------
Testing password-less ssh on master -> slave
<span style="color: red;">Don't ignore any error here !!!</span>
---------------------------------------------
Sorry, prefer to keep this check manual to avoid headache in Hadoop or any other distributed program.
Do you see error or something fishy in above block (y/n):n
Do you want to enable password-less ssh between slave-slave (y/n):y
---------------------------------------------
Testing password-less ssh on slave <-> slave
<span style="color: red;">Don't ignore any error here !!!</span>
---------------------------------------------
Sorry, prefer to keep this check manual to avoid headache in Hadoop or any other distributed program.
Do you see error or something fishy in above block (y/n):n
</pre>
</blockquote>
<br />
Warning: The above code does not check for passwordless ssh for
slave-slave configuration and sets it blindly even if passwordless ssh
is already enabled. It might not be big deal if you call this script few
times but won’t be ideal if the cluster needs to dynamically modified
over and over again. Still, to modify this behavior, look at the line: <span style="color: #3366ff;">IS_PASSWORDLESS_SSH_BETWEEN_SLAVE_SET=”n”</span><br />
<br /></div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com1tag:blogger.com,1999:blog-30819994.post-64184309562132299282014-03-08T00:09:00.003-05:002014-03-27T15:32:18.960-04:00Notes on entrepreneurship (Part 3)<div dir="ltr" style="text-align: left;" trbidi="on">
Few updates since the last time I wrote the blog:<br />
1. At the end of the entrepreneurship course, the <a href="http://www.vigilant-app.com/" target="_blank">website</a> and <a href="https://itunes.apple.com/us/app/vigilant-personal-safety-sos/id713529402?ls=1&mt=8" target="_blank">an iOS app</a> was implemented and I got an <b>A grade</b> ... yaay !!!<br />
<br />
2. Since I was out-of-country during the final presentation to judges (VCs, Startup founders, etc), I created a youtube video to motivate this project and my teammate explained the remaining logistics of our project [10].<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="//www.youtube.com/embed/D9-bHdrZDtQ" width="420"></iframe>
<br />
<br />
3. Vigilant team won 3rd place in <a href="http://hackrice.challengepost.com/submissions/20465-vigilant" target="_blank">HackRice 2014 competition</a> for developing Android version along with SVM classifier [3] to predict threat level based on location and time of the day. We used crime data from Houston PD for 5 years, so yes this feature only worked for Houston :(<br />
<br />
4. I was lucky enough to get selected to participate in <a href="http://rcel.rice.edu/ignite2014" target="_blank">Ignite Conference 2014</a> [4]. To get the ball rolling on the first day, we were asked to build as tall structure as possible in given time frame with marshmallow on the top using noodle sticks, tape and thread [5]. The next day, we got to visit <a href="http://www.khoslaventures.com/" target="_blank">Khosla Ventures</a> and <a href="https://squareup.com/" target="_blank">Square</a> (and few other startups). On last two days, we had talks from <a href="http://gsm.ucdavis.edu/ignite-speakers" target="_blank">several guest entrepreneurs</a> where they spoke about their entrepreneurship journey. Here are few of the lessons I learnt (directly/indirectly from this conference):<br />
<b>4.a What ideas/projects to work on ?</b><br />
- One of the most common advice you will receive is "fail fast, fail often and fail cheap". However, I suggest not to take it verbatim and understand that the emphasis is not on failing but learning. So, you should not go with the mindset that if I am anyways planning to fail early, let me start with a dumb idea. In fact, the mindset should be "test early/cheaply; if you do fail, don't get hung over it, instead learn quickly why did you fail; and then adapt". Let me put this in much broader context in terms of one of the most important lesson I have learnt in research and entrepreneurship: <u>though most people will judge you by your bank balance and your citations, you (as a researcher or entrepreneur) should try to evaluate yourself by your ability to keep failures small and useful</u>.<br />
- Don't hesitate or hold yourself back if you are passionate about an idea that appears to be "<a href="http://en.wikipedia.org/wiki/Black_swan_theory" target="_blank">black swan</a>". But makes sure you "know what you know and know what you don't know". Then, evaluate whether things you don't know will halt the project, if so, find an expert who can help you with that.<br />
- Before you think of quitting your job and start a venture, make a list of non-glamorous/routine stuff about startup you will have to do. Though many of these might be solvable by capital, if you do intend to go the lean startup way, ask yourself: Are you mentally flexible to get over your ego and do these task if need be ?<br />
<b>4.b Building a team and network:</b><br />
- Build team that is smarter than you. And make sure you give back and focus on their growth too, else you will soon face with talent retention problem.<br />
- Here are some of the key points to build and keep great teams: Minimize unnecessary bureaucracy [14], exercise merit-based promotion strategy, treat others with respect and most importantly, with respect to improvement of your startup, encourage everyone to pitch-in their ideas rather than giving everyone list of to-do items [15].<br />
- Hire a really good sales team you can find. This is especially true in internet technology though it has scalability challenges. Why? People would much rather prefer to deal with a sales person rather than internet widget when they are paying huge money for the given service. This also means don't wait until your product testing phase to hire a sales team. Rather start with integrated sales-development team and use sales part of the team as to validate/refine the hypothesis by communicating with customers during the product development phase. Best case scenario, the founder is that person ... that first assumes the role of salesman, talks to clients/customers, figures out requirements, then by himself or by hiring really good team develops the product, markets it, and then keeps on iterating the whole process again [8].<br />
- Networking is extremely critical. It usually takes time, else it seems inauthentic/insincere. So, don't use the number of LinkedIn connections/visiting cards in your wallet as measure of networking, instead count number of people who genuinely want to help you and vice versa. Other way to put it: networking is not about how many people you are acquainted to but rather how many mentors, benefactors and friends you have.<br />
<b>4.c Managing yourself:</b><br />
- It is extremely important to find ways to manage stress and take care of your health. Also, if your better-half and family understands the challenges of startups and supports/love you, the stress/difficulties reduces exponentially.<br />
- Especially in software development phase, use productivity rather than hours worked as measure.<br />
- Know the difference between persistence and stubbornness. Remember persistence (without stubbornness) requires learning and pivoting given new information/lessons, rather than doing same things over and over again expecting different results [6].<br />
- When working in team, your idea is only good to the degree to which you can explain it. If your team doesn't understand it, you can almost be sure that your customer won't be able to understand it. This skill will probably require an entirely new blogpost (which I might write later), but for now here are three things you can do:<br />
a. Read books about presentation/explanation/UI design: <a href="http://www.amazon.com/Design-Everyday-Things-Donald-Norman/dp/0465067107/ref=pd_sim_b_1" target="_blank">Design of everyday things</a>, <a href="http://www.amazon.com/Dont-Make-Me-Think-Usability/dp/0321344758/ref=pd_sim_b_5" target="_blank">Don't make me think</a>, <a href="http://www.amazon.com/The-Back-Napkin-Expanded-Edition/dp/1591842697/ref=pd_sim_b_16" target="_blank">Back of Napkin</a>, <a href="http://www.amazon.com/The-Art-Explanation-Products-Understand/dp/1118374584/" target="_blank">Art of explanation</a>, <a href="http://www.amazon.com/Whiteboard-Selling-Empowering-Through-Visuals/dp/1118379764/ref=pd_sim_b_3" target="_blank">Whiteboard selling</a>, <a href="http://www.amazon.com/slide-ology-Science-Creating-Presentations/dp/0596522347/ref=pd_sim_b_17" target="_blank">slide:ology</a> and <a href="http://www.amazon.com/Things-Designer-People-Voices-Matter/dp/0321767535/ref=pd_sim_b_2" target="_blank">100 things every designer needs to know</a>.<br />
b. Learn empathy (i.e. relating to team/developers/sales and especially customers).<br />
c. Be specific. If you don't break up tasks into smaller items and clarify the purpose/big picture, people (either your team or investor) might feel overwhelmed and can sometime get defensive. One particular thing that every member on your team needs to be very crystal clear about is "minimum viable product".<br />
<br />
Let me recap the high-level steps [16] described until now in this blog series:<br />
<b><br /></b>
<b>Step 0:</b> Define your means: "what I know", "what do I have" and "whom do I know".<br />
<br />
<b>Step 1: </b><u>Ideation</u>: See <a href="http://niketanblog.blogspot.com/2013/10/notes-on-entrepreneurship-part-1.html" target="_blank">part 1 of this series</a> for high-level tips.<br />
<u>Step 1.1:</u> Being little more specific about step 1: Create/Refine 60 second<b> elevator pitch</b>: The key idea is if you meet a key investor/evangelist/strategic partner in an elevator and if he asks you what are you working on, you should be able to summarize your business idea before the elevator reaches certain floor and that person has to leave. Here are few suggestions about creating your elevator pitch:<br />
- Sit down and write your elevator pitch without too much thinking. First iteration will always be awful.<br />
- Iterate, rehearse, iterate again and then rehearse again (and keep doing this) until your elevator pitch becomes second nature to you.<br />
- Talk to your customers and experts in that field and try to incorporate their feedback.<br />
- Be honest and don't promise "world peace". Remember, your words represents your thoughts/character and the investor is investing in you as much as he/she is investing in the idea.<br />
- There is no one way to do this, so learn-by-example; aka do youtube/google search for "elevator pitch" and listen to them [7].<br />
- Use analogy to simplify the problem.<br />
- If possible, explain how the problem affects you, your family or someone you know.<br />
- The first and last sentence are extremely important and connecting them helps to create a more effective delivery.<br />
Here is one of the iteration of my elevator pitch:<br />
<blockquote class="tr_bq">
Studies show that in the US alone, one out of four women get sexually assaulted in their life-time. This number is much higher when they travel to developing countries like India. Though this problem is highly complex, we believe that a fraction of it can be solved by technology-based deterrents that act as first line of defense when in danger. We present one such solution: Vigilant. Vigilant is an affordable and hassle-free personal safety solution that connects you to friends, family and authorities during emergency with just a click of a button. The goal of Vigilant is to make everyones life safer with the help of smarter software and hardware.</blockquote>
<b>Step 2:</b> <u>Opportunity evaluation</u>:<br />
Step 2.1: Use high-level checklist described in <a href="http://niketanblog.blogspot.com/2013/10/notes-on-entrepreneurship-part-1.html" target="_blank">part 1 of this series</a>.<br />
Step 2.2: Create a <b>business model canvas</b>: Before you read anything further, you should definitely watch this <a href="http://www.youtube.com/watch?feature=player_embedded&v=QoAOzMTLP5s" target="_blank">2 min video</a> describing the business model canvas. Also, read the book <a href="http://www.amazon.com/Business-Model-Generation-Visionaries-Challengers/dp/0470876417" target="_blank">Business Model Generation</a> to understand why and how to create a business model canvas. If you don't have time to read through the book, here is few websites that allows you to create one by step-by-step instructions: <a href="http://www.zoomstra.com/foundersworkbook/" target="_blank">zoomstra</a>, <a href="https://www.launchpadcentral.com/" target="_blank">launchpad central</a>, <a href="https://turbostart.co/" target="_blank">turbostart</a>, <a href="http://businessmodelpress.com/" target="_blank">wordpress plugin</a>, <a href="https://mural.ly/" target="_blank">collaboration website</a>. You can also go through following lectures/tutorials to learn how to create one: <a href="http://udacity.com/course/ep245" target="_blank">Steve Blank's udacity class</a>, <a href="http://www.alexandercowan.com/business-model-canvas-templates/" target="_blank">Alex's blog</a>.<br />
Step 2.3: Early customer validation/requirement gathering using one or more of the <b>feedback/analytics tools</b> (some are applicable only at later stages): These tools help you quantify/find out: what people think they want, how much they think they want to pay and how much they will really pay.<br />
- False buy page or dummy (but fully functional) <a href="http://www.vigilant-app.com/shop/" target="_blank">e-commerce page for buying clickers</a> [1].<br />
- Contact form plugin in wordpress.<br />
- Feedback button in app.<br />
- Site Traffic widget/plugin in wordpress.<br />
- <a href="https://www.facebook.com/vigilantapp" target="_blank">Facebook likes</a>.<br />
- <a href="https://docs.google.com/forms/d/1EK8xOm_Mn99FTM_qdkRzu0ZcXiWYBC8NDnteBLitSPw/viewform" target="_blank">Google forms for suggestions from early adopters</a>: Here are few suggestions for coming up with good questions (ones that I didn't know earlier): <a href="http://kevindewalt.com/2013/02/02/accelerate-your-customer-development-how-to-quickly-get-dozens-of-interviews/" target="_blank">Kevin's blog</a>, <a href="http://dschool.stanford.edu/use-our-methods/" target="_blank">Stanford's videos for customer discovery</a>, <a href="http://leanstartup.pbworks.com/w/page/54918676/Customer%20Interview%20Templates%20and%20Resources" target="_blank">Learn startup customer development templates</a>, <a href="http://businessmodelalchemist.com/2012/09/test-your-value-proposition-supercharge-lean-startup-and-custdev-principles.html" target="_blank">Alex's blog</a>, <a href="http://www.slideshare.net/sblank/customer-discovery-23251533" target="_blank">Steve's suggestions</a>, <a href="http://www.kaushik.net/avinash/experimentation-and-testing-a-primer/" target="_blank">Kaushik's suggestions</a>.<br />
- Other tools that you can use for market research are <a href="http://www.google.com/insights/consumersurveys/home" target="_blank">Google's customer survey tool</a>, <a href="http://customerdevlabs.com/2012/08/21/using-mturk-to-interview-100-customers-in-4-hours/" target="_blank">Amazon's mechanical turk</a>, <a href="http://qualtrics.com/" target="_blank">Qualtrics</a>.<br />
- Pamphlet with google url shortener[2] and QR codes across campus.<br />
- Other tools that allow to do customer validation: <a href="https://validately.com/" target="_blank">validately</a>, <a href="http://www.foundersuite.com/" target="_blank">foundersuite</a>.<br />
<br />
<b>Step 3:</b> Build a great team !!!<br />
- See suggestions given above in point <b>4.b</b> and <b>4.c</b>.<br />
- There are few websites that help you find your founders: <a href="http://founderdating.com/" target="_blank">FoundersDating</a>, <a href="http://www.cofounderslab.com/" target="_blank">CoFoundersLab</a> or attend a <a href="http://www.meetup.com/" target="_blank">meetup</a> or hackthons or <a href="http://startupweekend.org/" target="_blank">startup weekend</a>.<br />
<br />
<b>Step 4</b> (or 5): Implementation, marketing, validation, refining business model canvas and keep on iterating.<br />
This step requires above mentioned feedback/analytics tool as well as <b>marketing/branding tools</b>:<br />
- Checkout <a href="http://www.fiverr.com/" target="_blank">fiverr</a> for really cool and cheap marketing ideas.<br />
- Make sure you install SEO plugin for your wordpress. I am using <a href="https://wordpress.org/plugins/all-in-one-seo-pack/" target="_blank">this plugin</a>, but there are equally good plugins in the market.<br />
- Create a marketing message/keywords based on above feedback tools or <a href="http://www.google.com/trends" target="_blank">google trends</a> or <a href="https://adwords.google.com/o/KeywordTool" target="_blank">google keyword tool</a>. Then keep refining it with respect to location based on <a href="http://www.google.com/insights/" target="_blank">google insights</a>. Similarly, there are several analytics tools that will help you in marketing/branding like <a href="http://www.google.com/analytics/" target="_blank">Google Analytics</a>, <a href="https://developer.amazon.com/sdk/analytics.html" target="_blank">Amazon Analytics</a>, <a href="https://heapanalytics.com/" target="_blank">Heap Analytics</a>, <a href="http://www.appannie.com/" target="_blank">AppAnnie</a>, <a href="http://www.distimo.com/" target="_blank">Distimo</a>, etc.<br />
- Advertise on <a href="http://www.google.com/adwords/" target="_blank">Google</a>, <a href="https://www.facebook.com/advertising/" target="_blank">Facebook</a>, <a href="https://adcenter.microsoft.com/" target="_blank">Bing</a> or <a href="https://www.linkedin.com/ads/start" target="_blank">LinkedIn</a>. If you prefer traditional marketing tools, you can look at <a href="http://next.srds.com/for-agencies" target="_blank">srds</a>.<br />
- Read <a href="http://www.amazon.com/The-Art-Explanation-Products-Understand/dp/1118374584/" target="_blank">Art of explanation</a> and make a user-friendly video about your product. You can use one of the following tools to create the video: <a href="http://www.sparkol.com/products/videoscribe" target="_blank">Sparkol</a>, <a href="http://explainify.com/" target="_blank">Explainify</a>, <a href="http://www.loosekeys.tv/" target="_blank">LooseKeys</a>, <a href="https://aws.amazon.com/elastictranscoder/" target="_blank">Amazon's Elastic Transcoder</a>, <a href="http://www.techsmith.com/camtasia.html" target="_blank">Camtasia</a>.<br />
For exhaustive list of tools, refer to <a href="http://steveblank.com/tools-and-blogs-for-entrepreneurs/" target="_blank">Steve Blank's list</a>, <a href="http://ycombinator.com/lib.html" target="_blank">YCombinator's list</a>, <a href="http://startupweekend.org/resources/" target="_blank">Startup Weekend's list</a>, <a href="http://startuptools.pbworks.com/w/page/17974963/FrontPage" target="_blank">PBWork's list</a>, <a href="https://docs.google.com/spreadsheet/ccc?key=0AsSKjTzap8WkdHVvLVhZaG1EUmQyVWdCTHV5WThZbXc&usp=sharing" target="_blank">HBS list</a>.<br />
<br />
<b>Step 5</b> (or 4): <b>Funding your startup</b>:<br />
Step 5.1: Research on which type of investor you want to approach: angel investors [12], venture capitalist, <a href="http://en.wikipedia.org/wiki/Comparison_of_crowd_funding_services" target="_blank">crowdsourcing</a> or <a href="http://en.wikipedia.org/wiki/Super_angel" target="_blank">super angels</a>. Also, be perfectly clear about amount of funding you need and valuation of your company as this will determine amount of equity (or other form of compensation) you will have to give to the investor. There are other parameters to consider as well like how much involvement you want from investor. Here is small figure to understand the startup funding lifecycle [11]:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-4wsHVJNCSz0/UydrJBG4q5I/AAAAAAAANqU/mKPXGjag3Rg/s1600/700px-Startup_financing_cycle.svg.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-4wsHVJNCSz0/UydrJBG4q5I/AAAAAAAANqU/mKPXGjag3Rg/s1600/700px-Startup_financing_cycle.svg.png" height="265" width="400" /></a></div>
<br />
Step 5.2: Now research on the specific investors you want to target and classify them as either "ideal/ambitious" or "can live with/moderate". Start with investor's website and understand their funding philosophy. Then look into ventures they have funded and see how they are doing [13].<br />
Step 5.3: Prepare your <b>pitch deck:</b><br />
- See tips from from <a href="http://niketanblog.blogspot.com/2011/09/presentation-on-online-aggregation-for.html" target="_blank">my previous blogpost</a> about presentation.<br />
- Here is a really good 2-min <a href="http://www.youtube.com/watch?v=Q9pTXwsZqzM" target="_blank">video about pitching</a> from Guy Kawasaki. Note the 10 slides at 10min 4 seconds.<br />
Step 5.4: Remember there are other things more important than just capital for your startup. Ask advice and hopefully gain a mentor or board of director. Sometimes they can also refer you to another funding agency.<br />
<br />
Since this is often the most confusion aspect of startup for a techie, let me try my best to explain and emphasize the <u>difference between validation and planning</u> [8, 9]. Usually, planning involves deep thinking so as to perform forecasting, whereas validation starts with brainstorming the hypothesis and then validating it through rapid customer feedback loop. Planning which is associated with <a href="http://en.wikipedia.org/wiki/Waterfall_model" target="_blank">waterfall-like model</a>, starts with defining the specific goal and then coming up with strategies and detailed steps to move towards that goal. Validation, on other hand, is often associated with agile development/effectuation/lean startup methodology and then involves pivoting the goal based on customer/market feedback. Planning assumes that you know precisely what customer requirements are beforehand, validation doesn't. Since customer requirements are known, planning is suitable for large-scale established companies where the emphasis is on "execution" of those requirements. On other hand, since precise customer requirements are unknown in startups, emphasis is on "searching" of the requirements through validation of hypothesis [8]. This is why many entrepreneurs prefer a short business model canvas (which is a hypothesis generating and validating tool) over more comprehensive <a href="http://en.wikipedia.org/wiki/Business_model" target="_blank">business models</a>.<br />
<br />
<b>Reference:</b><br />
[1] Implementing e-commerce page on the wordpress took me no more than 30 minutes using the plugin WooCommerce. It allows you to track orders, generate reports, and even manage coupons.<br />
[2] Google's url shortener (and many more similar free services) helps you track the traffic through the given link. Here is a useful hack: Have multiple shortened urls for your webpage and use different ones for pamphlets in different geographical areas. Then use the analytics to do much more targeted ad campaign :)<br />
[3] For ML geeks, I must confess that the model suffers from selection bias as it only contains the data when crime occurred, not when the crime didn't occurred. Since you have to finish the project in a day according to rules of hackathon, most of which were spent on data gathering/cleaning, I had no time to work on this :(<br />
[4] One thing that surprised me was that even though this conference was in Silicon Valley, I was the only Computer Science graduate student there, all others were either MBAs or MDs or MD-PhDs or BioTech students.<br />
[5] My team came second and built the structure that was 26 inch tall. The idea is to start the first structure with marshmallow and keep building lower structure. Also, it is important to reinforce the structure (building supporting vertical sticks and choosing multiple sticks for lower pillars) so that it does not collapse. Most of the teams that failed started by building as tall structure as they could without marshmallow and then tried to put marshmallow on the top, which caused entire structure to collapse because of the weight. So the take-away point for entrepreneurs from this exercise is do bottom-up incremental iterations/updates with customer involvement so as to mitigate risk. Here is the pic of our structure:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-cCilEqnY7K8/UxpsvBhnT2I/AAAAAAAANok/_Oe8Muki8zY/s1600/Ignite.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-cCilEqnY7K8/UxpsvBhnT2I/AAAAAAAANok/_Oe8Muki8zY/s1600/Ignite.jpg" height="200" width="83" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
[6] From Mark Otero, CEO & Co-founder, Klicknation.<br />
[7] Daniel Pink in his book "To sell is human" suggest following template: "Once upon a time ___. Every day, ___. One day ___. Because of that ___. Because of that ___. Until finally ___".<br />
[8] Read <a href="http://steveblank.com/" target="_blank">Steve Blank's blog</a> for more detail.<br />
[9] <a href="http://www.effectuation.org/learn/effectuation-101">http://www.effectuation.org/learn/effectuation-101</a><br />
[10] In the video, the clip where people have duck-tape on their mouth is from another youtube video about raising voice against sexual violence, i.e. it was not created by me and hence don't give me credit if you want to cite it. I used the clip because it resonates with the motivation of my project :)<br />
[11] <a href="http://en.wikipedia.org/wiki/Seed_round">http://en.wikipedia.org/wiki/Seed_round</a><br />
[12] It might also be worthwhile to look into websites like <a href="https://angel.co/" target="_blank">angellist</a>.<br />
[13] Websites like <a href="http://www.crunchbase.com/" target="_blank">CrunchBase</a> and <a href="http://techcrunch.com/" target="_blank">Techcrunch</a> helps here. Other useful website you should definitely visit is <a href="http://www.glassdoor.com/index.htm" target="_blank">Glassdoor</a> to gauge the talent-retention ratio as well as willingness of upper-management to experiment new ideas. Why? It sometimes can be a metric to evaluate the founders and indirectly the investors. There are few ranking websites based on different criteria, one of them being <a href="http://www.forbes.com/sites/alexkonrad/2014/03/26/midas-list-2014-the-methodology/" target="_blank">Forbes Midas' list</a>.<br />
[14] For example: <a href="http://csgsa.rice.edu/" target="_blank">Rice Computer Science Graduate Student Association (CS GSA)</a> is a flat organization and there are coordinators that have responsibilities/duties. Everyone can pitch in and help others, but when it comes to arbitration/point of contact/failure responsibility, the assigned coordinator is the one in-charge. For people outside the organization, that person is Overall coordinator, think of it as a President of the club and for the current year, that's me (thank you very much) :)<br />
[15] Sure, some ideas might not align with the vision of the company and it may be dropped, but it has to be done in form of discussion rather than in an authoritative manner.<br />
[16] Note, these are high-level steps, even though you might find the list of tools overwhelming. For detailed step-by-step instructions regarding startup process, please read one of several startup related books available on <a href="http://www.amazon.com/b/ref=amb_link_398016502_1?ie=UTF8&node=7083914011&pf_rd_m=ATVPDKIKX0DER&pf_rd_s=auto-sparkle&pf_rd_r=1DBMZAT83116F6Q13WAZ&pf_rd_t=301&pf_rd_p=1711487982&pf_rd_i=startup" target="_blank">amazon</a> or listen to startup lectures on udacity/coursera/youtube or better attend an incubator program near you. </div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-49594408390734122542013-10-12T16:30:00.000-04:002013-10-13T15:39:42.405-04:00Notes on entrepreneurship (Part 2)<div dir="ltr" style="text-align: left;" trbidi="on">
In the first class, we divided ourselves into random groups, each with four people (mostly from different background). The idea was to pitch an startup idea within 10 minutes based on the means of our group members. This exercise was repeated again with different set of people. One thing I noticed after this exercise was even though it seems like nice idea to start with your means and build a startup based on that, people usually revert back to the idea that they are absolutely passionate about, irrespective of their means [8].<br />
<br />
As an assignment that week, we were supposed to take $5 and in span of two hours make as much money as we can. My group decided to go with "Grocery delivery service for professors" and each of us were supposed to ask professors in our department to help with that. I got to speak to only few professors in Computer Science department (as it was Friday) and only <a href="http://www.cs.rice.edu/~sc40/" target="_blank">Swarat</a> was on-board with the idea. My team-mates were unable to get any professors from their respective department, so they decided to go with another idea: "Personalized cards and their delivery". We made $10 in tips with the first idea and ~$13 with the second idea. The lesson from this exercise was <b>sales is hard, but probably the important part of a startup </b>[9].<br />
<br />
In the next class, each of us gave an elevator pitch of their idea for a startup. Let me put my idea with respect to <a href="http://niketanblog.blogspot.com/2013/10/notes-on-entrepreneurship-part-1.html" target="_blank">previous post</a>:<br />
1. <b>Means</b>:<br />
- What do I know ? Background in software development, research experience in building large-scale systems and knowledge of embedded systems.<br />
- What do I have ? Very low capital ... <a href="http://www.phdcomics.com/comics/archive.php?comicid=1215" target="_blank">student salary</a> :(<br />
- Whom do I know ? Software professionals in India and US (from my bachelors, masters and <a href="http://www.linkedin.com/in/niketan" target="_blank">job experience</a>), Marketing/Advertising professionals in India (my father's advertising firm and my MBA friends), Trustworthy partner in India (my best friend and brother Rohan), Research scientists (my advisor, my colleagues at Rice university [3], my collaborators, contacts from internships and also from my experience as President of <a href="http://csgsa.rice.edu/" target="_blank">Rice Computer Science Graduate Student Associatio</a>n). Other than couple of exceptions, until now I have been lucky enough to be surrounded by really nice people, which is why I believe this to be my strongest means [4].<br />
<br />
2. <b>Ideation</b>:<br />
In the <a href="http://niketanblog.blogspot.com/2012/12/my-view-on-recent-protest-in-new-delhi.html" target="_blank">previous blogpost</a>, I vented my frustration over sexual assaults cases in India as well as provided high-level suggestions (which I must admit I had no control over). So, I decided to use my means to develop something that might help improve the situation (in whatever little way possible).<br />
Unlike US, the commonly-accepted safety net in India is not government/police, but the social structure (i.e. your friends and family). Many of the personal safety solutions like pepper spray, taser or guns require state licenses in many countries and are even prohibited in few countries like Canada, China, Bangladesh, Singapore, Belgium, Denmark, Finland, Greece, Hungary, etc. Not to mention they are expensive and can often escalate the situation. This is why many people follow a self-imposed curfew, that is either not leave home after dark or have someone accompany you.<br />
With advent of smartphones, an obvious solution seems to be "have a personal safety app". Most of these apps have cool features such as sending GPS location to your friends and family but have a major flaw that make them non-functional: <u>they require you to take out your phone, open the app</u> and press an SOS button in it. Clearly, this is not what an average person would do in times of danger. Here are some of the feedback/comments on such apps [1]:<br />
- I think a "dangerous environment" is the last place I would want an expensive item like an iPhone prominently displayed.<br />
- ... app (might not be) readily available to access, on screen #8 somewhere on the 2nd or third row ...<br />
- Pulling out your iPhone in the face of an attacker? That's one sure way to escalate the situation. And he now knows you have a nice phone too.<br />
Reality check: Not only I am passionate about this problem, there does not exists a solution that works satisfactorily (i.e. "pain points") ... to me, they are just cosmetic app, the ones you buy for keepsake but does not serve any purpose to the society.<br />
<br />
Before discussing next point, let me take a small detour and tell you what metric I use to determine the success of this startup [2]: At the end of the course (or may be few months past that), I am able to make an app that works seamlessly in real-life situation and which I can recommend to my loved ones without any hesitancy.<br />
<br />
3. <b>Opportunity evaluation</b>:<br />
Like many computer programmers, I have an habit of developing software that do exactly what I want but not what my audience would need. To ensure that it doesn't happen, I sent out an survey asking people what they would like in during times of emergency. The demographics of participants were as follows:<br />
- 53% males and 47% females<br />
- 82% of participants were between 20-30 years old and 14% between 30-40 years old.<br />
- 59% from US, 16% from India and 8% from China<br />
- 24% had Bachelors degree, 47% had Masters degree, 27% were PhD students.<br />
- 32% earned between $10K-50K, 28% earned between $50K-100K, 16% earned above $100K and 20% were not currently earning.<br />
<br />
Here is the summary of responses:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-OuR6-QwUA9o/Ujoa-_SUsgI/AAAAAAAAHQE/HeR28CTOjjU/s1600/Screen+Shot+2013-09-18+at+4.43.58+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="196" src="http://4.bp.blogspot.com/-OuR6-QwUA9o/Ujoa-_SUsgI/AAAAAAAAHQE/HeR28CTOjjU/s640/Screen+Shot+2013-09-18+at+4.43.58+PM.png" width="640" /></a></div>
<br />
The above figures shows that majority of people (46%) wanted an external device like bluetooth-clicker that victim can press when he/she is in danger. This was a good indicator that I should go ahead and spend some time building such an app.<br />
<br />
The next step was to understand what features a user would want/need in times of emergency:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-mOyA83KwUWk/UjocGfO21sI/AAAAAAAAHQM/YDbTbcPCQX4/s1600/Screen+Shot+2013-09-18+at+4.49.29+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="180" src="http://1.bp.blogspot.com/-mOyA83KwUWk/UjocGfO21sI/AAAAAAAAHQM/YDbTbcPCQX4/s640/Screen+Shot+2013-09-18+at+4.49.29+PM.png" width="640" /></a></div>
<br />
Since some of these features require backend services (for sending emails/SMS, managing account/app) which need capital (I am not rich enough to maintain this kind of service on my own), I asked how much people are willing to pay for this kind of service. It was clear that most people preferred buy-once kind of a model:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-BMnYLrN7gtw/UjodHGSTo8I/AAAAAAAAHQU/NhmDrmt7h0g/s1600/Screen+Shot+2013-09-18+at+4.53.57+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="106" src="http://4.bp.blogspot.com/-BMnYLrN7gtw/UjodHGSTo8I/AAAAAAAAHQU/NhmDrmt7h0g/s640/Screen+Shot+2013-09-18+at+4.53.57+PM.png" width="640" /></a></div>
<br />
<br />
Now, the checklist of opportunity evaluation:<br />
a. Unique value proposition of my idea: "Affordable" and "hassle-free" way to connect to your friends, family and authorities in times of emergency ... with just a click of button.<br />
b. Is it defensible: Nope, anyone with strong programming experience can replicate the features of my app. In fact, in long run, that is exactly what I want ... lot of good apps and competition that eventually help reduce the number of sexual assaults and make the world little safer place.<br />
c. Is it profitable/sustainable: I really don't know :( ... It could be a product like dropbox, which people never thought they would want ... but once they got it, they can't imagine their life without it ... or it could be a total flop. The only way to be completely sure is by implementing it :)<br />
d. Clearly defined customer: Young women traveling abroad or working late, senior citizens and frequent travelers.<br />
e. Is it feasible and scalable: Yes, I used my experience in building large-scale cost-effective systems to build the backend. Also, my experience in C/C++ programming, knowledge of <a href="http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller" target="_blank">design patterns</a> and user-friendly <a href="https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/ProgrammingWithObjectiveC/Introduction/Introduction.html" target="_blank">Apple documentation</a> helped me: (a) learn the basics of Objective C in less than a week and (b) build the version 1.0 of the iOS app in less than a month.<br />
<br />
Finally, I must reiterate the core principle of opportunity evaluation for a startup: It is not possible to know a priori whether an idea will turn out to be good business or not. So, I decided to stick to "affordable loss" principle and develop the app with as low cost as I could. Few of the hacks I used to ensure "affordable loss" are as follows:<br />
- Moving most of the computing to clients rather than server (so as not to buy overly expensive servers).<br />
- Using pay-as-you-use services wherever I felt absolutely necessary (for example: Amazon web-services).<br />
- Using GIMP to develop my own logo (which I had to learn btw :P) rather than hiring a designer [6].<br />
- Using wordpress for the startup website rather than spending days perfecting the CSS to make it mobile-compatible or hiring a web-developer.<br />
- Focusing on minimum viable product (using Texas Instrument's SensorTag) rather than prematurely buying tons of bluetooth clicker from China [12].<br />
- Using in-app purchases rather than setting up credit-card system in my website to provide features which other services charge me (for example: SMS/Email) [10].<br />
- Not running after patents early on in the venture [5].<br />
- Buying readily-made icons set rather than designing them yourselves [7].<br />
- Choosing a hosting plan that has no hidden fees and that supports your choice of backend services.<br />
- Using gmail as support email (rather than one provided by hosting services) and adding feedback button in app as well as contact form on the website (with some kind of captcha [13]). One tip I have for new developers is try to minimize the number of clicks/typing in app for sending feedbacks, for example: pre-fill "to-" address as well as "subject line".<br />
- Utilizing the membership benefits of Apple/Google development program, i.e. off-loading testing [11], advertising/cross-promotion, expert feedback, reliable delivery of your software and version management.<br />
<br />
<b>References / footnotes:</b><br />
[1] http://reviews.cnet.com/8301-19512_7-57452393-233/iwitness-app-claims-to-be-ultimate-deterrent-to-crime/<br />
[2] Rather than use metrics like expected profit/revenue after first year or public offering or something on similar lines.<br />
[3] I already have got 4 other PhD students from Rice university on-board to develop Android/Windows version of the app. This is in lines with another principle of Effectual Entrepreneurship: Form partnerships.<br />
[4] The intent of this statement is not flattery, just an observation. Here is why I think so: though I never deliberately tried to enforce it, my circle of influence consists of three types of people: those who are genuinely nice, those who are smarter than me, and non-self-destructive people. Of course, the categories are not mutually exclusive and in fact people who are smarter are usually genuinely nice (probably because they only focus on self-upliftment, not on pulling others down). Here is a layman example most people can relate to: even in course with relative grading, a smart person knows that he/she is a student of global class and is not be threatened by his/her class-mates' progress ... which is probably why you will rarely find such a student shying away from group discussion (so as to gain advantage) or deliberately spreading false information to sabotage other's grades. May be things might not be such black-and-white in other fields, which is one of the reason why I absolutely love research and development.<br />
[5] A patent attorney who attended our class thought one of the feature of our app is patentable, which might be true. But there are 2 problems with going that way: (a) the real cost of patent is not in getting it, but in defending it, (b) It will limit the features that other programmers who are smarter than me can introduce in their personal safety app (which goes against my success criteria for this particular problem). To be completely honest, there are ways around low-capital issue for those startups who really want to get a patent: (1) file provisional patent yourself under $200 (just read about how to define the scope of your patent and also about court dates), (2) ask your parent organization or angel investor to file a patent for you in exchange for royalties or stake in the startup (for example: <a href="http://ott.rice.edu/home/" target="_blank">Rice university's OTT office</a>), (3) contact a patent troll to defend you patent.<br />
[6] For people with little more budget, there are websites like <a href="http://99designs.com/">http://99designs.com/</a> and <a href="https://www.elance.com/">https://www.elance.com/</a> that allows you to hire a free-lance designers/developers. A similar website for building mock prototypes of your products to show to investor is <a href="http://www.protomold.com/">http://www.protomold.com/</a>.<br />
[7] There is always a tradeoff between time and money, you just have to figure out what is your exchange rate for time ;) ... for example: if someone is providing you service that will save you 1 hour, how much are you willing to pay for that service.<br />
[8] Whether building your startup "based on your existing means" is better than "based on <a href="http://www.youtube.com/watch?v=KuNQgln6TL0" target="_blank">your passion</a> and then expanding your means as you go" or vice versa, I really don't know. There are obvious advantages for both and also obvious disadvantages when pushing the respective principle to extreme.<br />
[9] Even though you might think an idea is pretty good (and will benefit the customer), people are not willing to pay as much as you think ... probably because it's either suspicion that you are trying to dupe them or incorrect valuation of the product/market from your end or something else.<br />
[10] Though it might seem simple to just plug-in existing credit-card library and setup a php page with mysql backend, things get a little complicated when you start thinking about transactional semantics and the fact that people can buy new devices or deleting the app and similar situations.<br />
[11] In traditional company, a developer would be evaluated based on stupid metrics, such as bugs assigned, solved, etc. Though on paper they seem apt, they can have harmful side-effects for startup such as spending too much time perfecting a feature without any customer validation/feedback. So, instead of spending significant resources on testing, submit your app to Apple as and when you add new feature and indirectly ask them to test it :) ... Other way, for cheap testing is by using crowd-sourcing websites such as Amazon's mechanical turk (which I will ignore for this post).<br />
[12] Since the SensorTag took a while to be delivered to my home address (thanks to my apt complex rejecting the package), I decided to use an accessory that I already owned for version 1.0 (i.e. headphones) and introduce bluetooth feature in the next version.<br />
[13] There are lot of wordpress plugins that allow you to add captcha in your website in just few minutes.</div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-65905635718825927552013-10-01T15:57:00.000-04:002013-10-02T11:43:45.949-04:00Notes on entrepreneurship (Part 1)<div dir="ltr" style="text-align: left;" trbidi="on">
In this semester, I am taking ENGI 540, an entrepreneurship class offered by Rice Center for Engineering Leadership that focuses on lean startup and effectuation principles. It follows "reverse classroom" approach where we are supposed to go through <a href="http://www.amazon.com/Effectual-Entrepreneurship-Stuart-Read/dp/0415586445/ref=sr_1_1?s=books&ie=UTF8&qid=1379541528&sr=1-1&keywords=effectual+entrepreneurship" target="_blank">preparatory readings</a> and watch <a href="http://www.slideshare.net/guido23/entrepreneurship-101-22079010" target="_blank">online lectures</a> before every class. In the class, we discuss what we accomplished every week and also interact with guest entrepreneurs, venture capitalists and IP lawyers. I thought it might be nice to blog about my experience and share few insights I got from taking this class.<br />
<br />
In the first class and after reading through first few chapters of Effectual Entrepreneurship, I found following concepts very intriguing:<br />
1. Defining one's "<b>means</b>", which my professor defines as "who you are, what you know, what you have and who you know". The core idea in a startup is to start with your mean and create opportunities; sometimes expanding one's means through partnership.<br />
<br />
2. <b>Ideation</b>: One interesting habit that a budding entrepreneur can practice is make an idea journal, where you force yourself to come up with 1 new idea every week (or if you are ambitious every day). There are various tools available for idea generation like mind mapping, lateral thinking, brainstorming, ... (will leave this for some other blogpost). Here are some of the tips for generating business ideas:<br />
(2.a) Look into areas of change: Eg: opportunities in healthcare domain (because of ObamaCare) and energy field (because of increase in oil/gas prices & environmental impact). You can also analyze the consumption habits based on demographics [3], especially in market that is rapidly changing, i.e. BRIC countries (Brazil, Russia, India and China).<br />
(2.b) Pain points: Is there a problem that is bothering you that is not solved by any existing solutions ?<br />
(2.c) Cross-domain: Can you apply your expertise in one field into other ? For example: if you have expertise in end-user/consumer marketing, you can find an opportunity in a field which only focuses on corporate services (by delivering those services to end-user).<br />
<br />
3. <b>Opportunity evaluation</b>: It is not possible to know a priori whether an idea will turn out to be good business or not. Most successful entrepreneurs and investors claim there is only one way to see good business opportunity: Go ahead, implement it creatively with very low levels of investments and either find real customers who are willing to buy the product or service at reasonable price or locate partners who are willing to commit real resources to the ventures early on or ideally both [1]. Still there are few high-level checklist that you can use to check if your idea is good or not:<br />
(3.a) Does your idea have unique value proposition; if yes, what is it ?<br />
(3.b) Is your idea profitable and sustainable (i.e. profitable over long period of time) ?<br />
(3.c) Is your idea defensible ? People protect their idea either by patents/copyrights/trade-secrets, etc. Sometimes early entry into the market and strong user-base also makes an idea defensible.<br />
(3.d) Does it have a clearly defined customer ?<br />
(3.e) Other questions: Is it feasible ? Is it scalable ?<br />
<br />
4. Causal reasoning states: To the extent we can predict the future, we can control it; whereas effectual reasoning states: to the extent we can control the future, we don't need to predict it. So, instead of creating elaborate predictive business model before beginning a startup (which can be frightening), it might be a nice idea to follow the effectual strategy: focus on what is controllable about the future and only commit what you can afford to lose (i.e. "<a href="http://www.effectuation.org/sites/default/files/documents/affordable-loss.pdf" target="_blank"><b>affordable loss</b></a>" principle [2]). For extremely risk-averse people who might prefer inaction, I would like to cite Sarasvathy's quote: "20 years from now, you will be disappointed with things you didn't do than things you did do".<br />
<br />
<b>Reference:</b><br />
[1] Effectual Entrepreneurship - Saravathy et al.<br />
[2] A good application of affordable loss principle for iPhone development is given in <a href="http://www.youtube.com/watch?v=jAGdUogSnh4" target="_blank">this youtube video</a>.<br />
[3] Procter & Gamble changed the delivery mechanism of shampoo in India ... small packets instead of large bottles.</div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-65681079838748307862013-05-13T16:49:00.000-04:002013-05-15T12:09:25.562-04:00Religion and Science<div dir="ltr" style="text-align: left;" trbidi="on">
When I first decided to write a blogpost on this topic, the first thought that struck me was "What can I write about religion and science that isn't already articulated by others ?". This was quickly replaced by another thought "If it is so, why majority of people have twisted views about them ?". The answer was pretty simple: In public venues or in mainstream media, people with extreme views are the ones who are most vocal, whereas moderates often seem unconcerned, callous or worse condescending.<br />
<br />
Since moderates are often characterized as <u>being tolerant</u> (which is a good thing), let's try to understand what does it really mean to be tolerant. Is it same as being indifferent ? If you are tolerant, should you be vocal about it or is it sufficient to say "you believe what you want to believe, but don't bother me" [12] ? Should it be a choice or a requirement ? Would you be tolerant if you chose not to tolerate the intolerants (often called as terrorists, extremists, fascists, nazi, etc) [1]. Wait, it gets even more complex: Who defines the term "intolerant" ? For Islamic extremists, it's Western world stopping them from enforcing their interpretation of <a href="http://en.wikipedia.org/wiki/Sharia">sharia law</a>. For Western world, it's cowardly and inhuman attacks on civilians by these terrorists. For some, it's bible thumpers trying to dictate how they should live their life; for others, it's atheists (or may be liberals), who are ruining their family or moral values. The list goes on and on and is equally applicable in my beloved homeland India and gracious guest country USA (and I suspect every diverse nation in the world). Unfortunately, unlike <a href="http://archive.org/details/opensocietyandit033120mbp">Karl Popper</a> I don't have a globally applicable answer to these questions [10]. A local answer is to be tolerant in terms of freedom of speech, but keep people responsible for their actions in terms of secular humanitarian laws [2]. <u>Bottom-line</u>: in a civil society, the simplest working principle seems to be that we educate ourselves (enough to be intellectually capable of deciding right from wrong), embrace the diversity, be tolerant of opposing views and let the law judge the people who we feel are intolerant. It might also help if we abandon our cloak of complacency, engage in serious and meaningful debates/discussions and elect representatives who share our thoughts. Before I come back to the original topic, <a href="http://www.thedailyshow.com/watch/wed-april-24-2013/exclusive---bassem-youssef-extended-interview-pt--2?xrs=share_fb">let me explain the dangers of being intolerant</a>: If you chose to accept so-called "little intolerance", you will soon find yourself governed by "little intolerant" elected officials and as history has showed us, they will soon be replaced by "little more intolerant"elected officials (who would win the elections by saying the former aren't intolerant enough) and the cycle will continue.<br />
<br />
Since there are different definitions of the same term [3, 4], for sake of this blogpost, let me explain my interpretation of these terms.<br />
1. You are a <u>theist</u> if you believe that God created the world and takes active interest in the world. For example: if you are a theist, in times of trouble you believe that God will intervene to provide you with guidance or in someway affect the outcome. How far and how often God intervenes depends on degree of your theist beliefs.<br />
2. Almost all theists also follow one of many <u>organized religions</u>, which are based on <i>revelations</i> given by prophets or reincarnated Gods. It is important to note few things about organized religions:<br />
<ul style="text-align: left;">
<li>There is a clear distinction between prophets and ordinary people, former being first-class citizens who are the only people that have access to these revelations.</li>
<li>Since revelations are the only mechanism for finding or validating "religious truths", it makes <u>scientific inquiry impossible.</u> This is because as of this day, there is no clear way to distinguish between the revelations of first-class citizens and hallucinations/delusions of ordinary people. This is also true for spirituality. So, religion has a concept of <u>faith</u> which requires you to believe something because the first-class citizens said so [5].</li>
<li>These revelations can be categorized into two parts: information about God (and hence the universe) and information about how to conduct ourselves in the world. Often, the former (for simplicity, let's call it knowledge) is used as main reason to stick to the latter (let's call it moral guiding principles or human values) in our day-to-day life. By definition, knowledge is considered to be absolute and universally-applicable [6] whereas the moral guiding principles are treated as relative [7]. The consensus among many intellectuals is that the <a href="http://en.wikipedia.org/wiki/Scientific_method">scientific method</a> is the <u><i>best available tool</i></u> for exploring and validating knowledge. For moral guiding principles, people often use either religion or philosophy or spirituality or law or introspection or advice from friends and family [8].</li>
<li>Most revelations are described in holy books and are often accompanied by a back story. Many anti-theists criticize the revelations because of validity of the back story rather than applicability of revelations themselves. Some theists believe them to be exact historical account whereas others theists believe them to be embellished so as to get the message through to the general public [9]. In these back stories, there is often mentions of so-called miracles and many use the argument that specification of these <a href="http://www.youtube.com/watch?v=TmxAGhC-gLU">miracles is a proof</a> of validity of the revelations and hence God [11]. However, non-reproducibility and often embellished centuries-old second-hand accounts make the likelihood of these miracles highly improbable, if not impossible, and hence they are not accepted as facts by the scientific community.</li>
<li>Even though it is often suggested that all religions say the same thing [12], many organized religions are very clear on one particular requirement: "To adhere to religion X, you must ignore teachings of all other religions and consider them to be pagans". So, ipso facto all religions don't say the same thing, it is what we chose to believe to promote tolerance. I, on the other hand, think we should celebrate pluralism and not be afraid of the fact that different religions say different things. The only reason we ever need to be tolerant is coexistence and survival of human race.</li>
<li>Many religions are based on the concept of two-world: physical world (where we live) and transcendent world (which is termed as heaven, paradise, etc). Some religions that have concept of rebirth and karma (eg: Buddhism and Hinduism), also have concept of enlightenment (nirvana or moksha), which is often referred to as state of being, rather than a physical location (like heaven). The hidden assumption in two-world philosophy and most religions is that human beings and earth has undivided attention of God and hence has special and central place in this universe. Science makes no such assumption and in fact many scientists believe human race and earth to be a product of a wonderful accident.</li>
</ul>
3. Science on other hand is based on "scientific method", which Oxford dictionary defines as "a method or procedure that has characterized natural science since the 17th century, consisting in systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses". Since there are already enormous amount on literature on this I won't discuss it in more detail. However, I do have few advices for young scientists:<br />
<ul style="text-align: left;">
<li>Make sure you understand how to differentiate between science and <a href="https://en.wikipedia.org/wiki/Pseudoscience">pseudoscience</a>.</li>
<li>Don't be afraid of uncertainty and not knowing everything. It's OK to say "I don't know", you will be surprised how often that phrase is used by <a href="https://www.youtube.com/watch?v=YltEym9H0x4">renowned scientists</a>.</li>
<li>Avoid the urge to be self-centric and to accept something to be fact because you feel it "deep inside" [12]. Also, don't make up stories, that have no scientific bases, to explain things you don't understand.</li>
<li>There is no such thing as faith in science. It is replaced by reasoning and experimentation. For example, when you read a research paper or a book, always exercise critical reasoning on it to make sure that the results are plausible. If you think something is fishy, conduct an experiment to validate its result and hypothesis [5]. This is also true when a professor explains you laws of motion or electricity or something else and claims it as science.</li>
<li>For reasoning and validation, stick to <a href="http://en.wikipedia.org/wiki/Occam's_razor">Occam's razor</a>, i.e. don't just jump to the easiest/quickest answer, instead put effort to find the answer which requires fewest assumptions. For experimentation, at minimal, understand the <a href="http://en.wikipedia.org/wiki/Correlation_does_not_imply_causation">correlation-causation fallacy</a> and also <a href="http://en.wikipedia.org/wiki/Statistical_hypothesis_testing">statistical testing</a>. One particular book that that you can start with is <a href="http://www.amazon.com/How-Solve-Aspect-Mathematical-Method/dp/4871878309">How to Solve it by Polya</a>.</li>
</ul>
Even though both religion and science started because of human curiosity to explain the universe [13], both employ completely different methods/practices: revelations and scientific method, respectively. To make sense of the debate between them, it is necessary to recognize that these methods are <u>inherently incompatible</u>. So, unless all the parties agrees on a common method for finding and validating facts/truths, I think the whole argument will keep on <a href="http://www.youtube.com/watch?v=6H9BuxeNro0">spiraling itself</a>. So, if a religious person choses to use the scientific method to convince me of the validity of the revelations, I welcome it; if not, I prefer not to engage in the discussion at all.<br />
<br />
Just like religious missionaries, let me take a moment to state why you should use scientific inquiry for finding facts and solving problems ;) Look around what scientific inquiry has enabled us to discover and invent: medicines, cars, internet, computers, phones, planes, trains, camera, music players, guitars, paints, bricks, electricity, bulbs, etc. Whether you are a student or a nobel prize winner, a poor person or a millionaire, a prisoner or the president, your experimental findings and ideas will be judged purely on its merit and nothing else ... not on your status or gender or age or or religion or race or caste or nationality. You are allowed to ... no, you are encouraged to come up with new ideas, ask questions, doubt and investigate widely held beliefs. In fact, I urge you to doubt everything in this blogpost, discuss it with your teacher/professor/colleagues/priest/friend/family, read books about both religion and science and come up with your own opinions :)<br />
<br />
Let me summarize this blogpost in a sentence: Whether religion or spirituality is inevitable and necessary part of of human life, I don't know; but I am absolutely sure that they don't have place in science classroom or textbook.<br />
<br />
References and footnotes:<br />
[1] The paradox of tolerance: http://jefflees.wordpress.com/2010/01/22/the-paradox-of-tolerance/<br />
[2] The legal argument about tolerance gets more complicated if majority decides to collude to deliberately oppress the few, which has happened far too often in the past. However, we won't get into that and let's rely on inherent human goodness and assume that there are constitutional provisions to disallow that.<br />
[3] http://en.wikipedia.org/wiki/God<br />
[4] The Cambridge companion to atheism<br />
[5] Scientists are expected to practice same level of skepticism to the theories of even the most accomplished scientists (be it Newton, Einstein, Darwin, Turing or Feynman), i.e. <a href="http://www.youtube.com/watch?v=fKyFkAfmwFA">science does not have authorities, just heroes</a>.<br />
[6] Even though I called knowledge to be absolute and universally-applicable, it was little misleading. Read research journals to find just how many previously accepted theories are replaced by new theories in light of new evidence, which essentially means knowledge and science is ever-evolving. Anyways, I urge you to read about <a href="http://en.wikipedia.org/wiki/Epistemology">epistemology</a> for understanding the nature and scope of knowledge.<br />
[7] Religious fundamentalists believe "how to conduct yourself" is absolute (i.e. strict adherence to revelations).<br />
[8] Recently, I encountered a person who was against organized religion, but claimed that spirituality is better tool than science for both exploring knowledge and for moral guiding principles. He didn't provide me with any evidence or rationale to support his hypothesis, so I disagree with him on that. It is also important to point out here that <a href="http://en.wikipedia.org/wiki/The_Moral_Landscape">Sam Harris</a> suggests that science can also determine human values.<br />
[9] Irrespective of which is true, one thing I submit to is that they probably were the best collections of ideas of their generations. However, in this age of globalization and democracy, I don't think if they are applicable.<br />
[10] At the international level, with so many dictatorships, monarchy, theocratic governments, territorial disputes and dogmatic/partisan politicians around, it is difficult to come up with one principle that will work every where. Non-intervention (or non-alignment) expects every country to make decision before you stop a genocide (example: EU and Bosnia). Respecting every nation's sovereignty means pulling out your troops from unstable governments (like Iraq and Afghanistan) or countries surrounded by superior military powers (like Kuwait, South Korea, Taiwan or Japan). Think non-violence and Tibet. So, this blogpost is only applicable to citizens of secular nations with access to freedom of speech.<br />
[11] If you believe in miracles, aren't miracles of science (pun intended) equally mind-blowing, for example: <a href="http://en.wikipedia.org/wiki/Airplane">flying objects</a>, <a href="http://en.wikipedia.org/wiki/Moon_landing">going to the heaven</a>, <a href="http://en.wikipedia.org/wiki/In_vitro_fertilisation">create new life</a>, <a href="http://en.wikipedia.org/wiki/Medicine">curing the sick</a>, <a href="https://www.google.com/">all-knowing oracle</a>, <a href="http://www.youtube.com/watch?v=ro0OYlcL7cc">walking on water</a>, <a href="http://www.dailymail.co.uk/sciencetech/article-2231674/Scientists-reveal-perfect-invisibility-cloaking-device-admit-Harry-Potter-cloak-way-off.html">being invisible</a>, etc.<br />
[12] <a href="http://www.youtube.com/watch?v=IpNRw7snmGM">Penn Jillette: Why tolerance is condescending ?</a><br />
[13] Of course, many religious people will disagree with this opinion and will point out that religion exists because God wanted it to exists, not because humans were fishing for answers.<br />
<br />
<span style="font-size: x-small;">For those people who followed the entire line of argument, I must admit that I deliberately skipped the underlying question about existence of God in non-religious/non-theists, atheist, agnostics settings.</span></div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com1tag:blogger.com,1999:blog-30819994.post-85417248178546261892013-03-27T19:49:00.001-04:002013-03-28T15:21:37.138-04:00Hadoop MapReduce - System's view<div dir="ltr" style="text-align: left;" trbidi="on">
Here are videos that I created to explain Hadoop's MapReduce framework from system's perspective. As a disclaimer, this video is at an abstraction that I liked, which means it might not be suitable for general Computer Science audience. If however, you have some experience with Hadoop, you should be fine. I have skipped over lot of Hadoop's features for simplicity and succinctness, like HDFS, Hadoop I/O, setting up/administering Hadoop cluster and writing Hadoop programs (and various high-level languages/tools that use Hadoop).<br />
<br />
1. Job Submission at the client side<br />
<iframe allowfullscreen="" frameborder="0" height="480" src="http://www.youtube.com/embed/YcOm6Xk6MD4" width="640"></iframe>
<br />
<br />
2. Job submission at the JobTracker side<br />
<iframe allowfullscreen="" frameborder="0" height="480" src="http://www.youtube.com/embed/jdWjyMVK8pU" width="640"></iframe>
<br />
3. Task Scheduling<br />
<iframe allowfullscreen="" frameborder="0" height="480" src="http://www.youtube.com/embed/SUv4x5hVKgU" width="640"></iframe>
<br />
4. Task Creation<br />
<iframe allowfullscreen="" frameborder="0" height="480" src="http://www.youtube.com/embed/twghAtrQVsw" width="640"></iframe>
<br />
5. Map execution<br />
<iframe allowfullscreen="" frameborder="0" height="480" src="http://www.youtube.com/embed/wZFa2M8PM_g" width="640"></iframe>
<br />
6. Reduce execution<br />
<iframe allowfullscreen="" frameborder="0" height="480" src="http://www.youtube.com/embed/XnRs4uaOeaA" width="640"></iframe>
<br />
Slides: <a href="http://www.cs.rice.edu/~np6/Presentations/HadoopMapReduce.pdf">PDF</a>, <a href="http://www.cs.rice.edu/~np6/Presentations/HadoopMapReduce.key">Keynote</a></div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-7335632043351426182013-03-14T15:49:00.001-04:002013-03-15T15:27:07.658-04:00Video tutorial on reinforcement learning<div dir="ltr" style="text-align: left;" trbidi="on">
This video tutorial is based on a talk I gave at my <a href="http://kavrakilab.org/comp650">Comp 650 class</a> and hence assumes that the audience knows a little about <a href="http://en.wikipedia.org/wiki/Markov_decision_process">Markov Decision Processes</a> and <a href="http://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process">Partially Observable Markov Decision Processes</a>. If you are unfamiliar with them, I recommend that you at least read through their wikipedia pages before going through this tutorial. Also, I suggest that you view this video on youtube, because the embedded video given below is cropped.<br />
<br />
<iframe width="640" height="480" src="http://www.youtube.com/embed/mwCmRCn1MYo?rel=0" frameborder="0" allowfullscreen></iframe>
<br />
If you have any suggestions, comments or questions, please feel free to email me :)<br />
<br />
<a href="http://cs.rice.edu/~np6/Presentations/LearningBasedMethodsMDP-POMDP.pdf">PDF slides</a></div><div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-45626043873804292602012-12-25T16:47:00.000-05:002012-12-25T22:20:28.803-05:00My views on recent protest in New Delhi<div dir="ltr" style="text-align: left;" trbidi="on">
According to 2011 census in India<sup>1</sup>, there are 586469174 women in India, i.e. 48.46% of India's population and 8.38% of total world's population (which is roughly around the total number of English and Spanish speaking people in the world).<br />
<br />
Out of these only 65.46% are literate<sup>2</sup> (as compared to 82.14% literacy rate in males), with the definition of literacy being very narrow.<br />
<br />
Crimes against women<sup>3</sup> include rape (20737 cases in the year 2007), dowry deaths (8093 cases), molestation (38734 cases), sexual harassment (10950 cases) and cruelty by husbands/relatives (75930 cases). To add to that, there is human trafficking (for sexual slavery or domestic servitude), female infanticide, discrimination, acid attacks, lack of health care and forced marriage. The number of rape cases has increased by 80% in 7 years from 2000 and is still increasing. In the year 2011, 26000 rapes were reported<sup>4</sup> throughout India; in fact it is the fastest growing crime in India. According to Thomas Reuters Foundation<sup>5</sup>, India is the fourth most dangerous country in the world for women behind Afghanistan, Congo and Pakistan.<br />
<br />
According to many, there are three main reasons behind this:<br />
1. Social taboo:<br />
We Indians (including me) are hypocrites when it comes to dealing with rape survivors<sup>6</sup>. They are outcasted to protect family's so-called honor and hence most of the rape cases aren't reported or are dropped. Also, in event of such crimes, most of us prefer not to intervene and help the victim (be it eve-teasing, dowry, rape, etc) and her family. This indifference in our attitude provides additional incentives to the rapist to commit such heinous crime.<br />
<br />
2. Severity of punishment:<br />
An unenforced law means nothing however stringent they are. So, I don't think severity of punishment acts as a deterrent to rapists. In fact, considering how our law enforcement agencies (both judicial and policing) are, the rapists might not be wrong to think they won't be apprehended at all. With such an alarming increase in rape cases, one might think it's reasonable to support increasing severity in special circumstances<sup>7</sup> and especially for repeat offenders. But, the point to note is, in India, rape is punishable up to life imprisonment<sup>9</sup> which I think is severe enough. The main problem is that it doesn't get enforced or it takes way too long.<br />
<br />
3. Lack of speedy and fair trials with severe punishment to rape shield violators:<br />
There already are provisions for speedy trials<sup>10</sup> and rape shields<sup>11</sup>, but don't get executed due to lack of resources and infrastructure. First, there are 26.3 million pending cases<sup>8</sup>. Then there aren't enough women officers (I believe there should be an option for the victim to talk directly to women officers in every police chowky, for reporting rape, molestation, etc). Lastly, even though India has one of the strongest rape shield laws, Chaudhry<sup>11</sup> says they are routinely violated or reduced to technicality in practice. It is imperative to punish any authority that violates the rape shield.<br />
<br />
Let me reiterate my point: an unenforced law will solve nothing. I see the protestors (which btw I salute for their efforts, barring the ones that are doing it for political gains and media attention/ratings) are focusing on the wrong issue: asking for death penalty law for rape.<br />
<br />
Why ? Researchers have shown again and again<sup>12</sup>, <u>certainty and not severity of punishment proves to be true deterrent for the crime</u>. In fact, increasing severity can turn to be counterproductive (i.e. may lead to more false negatives).<br />
<br />
Sure, increasing severity might help us, as the society, deal with the shock, but it will do nothing to help improve the safety of women. The law will be passed, authorities will keep on violating rape shield, there will still be little or no resources to handle the case quickly and we shall go to living our life once again forgetting this incident ever happened and at the same time congratulating ourselves for the job well done. If you do truly care about the situation, let's forget the quick fix and do a more difficult thing: Take a step back and dispassionately think how we can solve this problem once and for all. Here is what I could think of:<br />
- Punishment or/and suspension of authority breaking rape shield (or discouraging victims for registering complaints).<br />
- Increase in forensic and judiciary budgets to expedite trials.<br />
- More women cops and psychological experts to help deal with the trauma. Also, create a special department for dealing with these crimes or train existing police officers to deal with such scenarios.<br />
- Incentive programs for improving the female literacy rate.<br />
- Standing up for the crimes against women when we see it, while still keeping our paternal instinct in check so that every women gets to enjoy her liberty.<br />
<br />
It took me half a day to write this blogpost and I humbly admit I am missing lot of points. But, I am sure a more experienced panel can come up with better solution if they are not sidelined by politics or their own self-interest.<br />
<br />
References:<br />
1. Census of India: http://www.censusindia.gov.in/<br />
2. http://en.wikipedia.org/wiki/2011_census_of_India<br />
3. National Crime Records Bureau: http://ncrb.nic.in/cii2007/cii-2007/1953-2007.pdf<br />
4. http://en.wikipedia.org/wiki/Rape_in_India<br />
5. http://www.trust.org/documents/womens-rights/resources/2011WomenPollResults.pdf<br />
6. http://www.ndtv.com/video/player/news/determined-not-to-hide-her-identity-rape-survivor-bravely-tells-her-story/259248<br />
7. http://en.wikipedia.org/wiki/2012_Delhi_gang_rape_case<br />
8. http://www.hindustantimes.com/News-Feed/india/Nearly-30-million-cases-pending-in-courts/Article1-224578.aspx<br />
9. http://indiankanoon.org/doc/1279834/<br />
10. http://defensewiki.ibj.org/index.php/Right_to_a_Speedy_Trial#India<br />
11. http://www.firstpost.com/living/gurgaon-gang-rape-let-me-tell-you-all-about-this-girl-242185.html<br />
12. http://www.sentencingproject.org/doc/deterrence%20briefing%20.pdf</div>
<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com3tag:blogger.com,1999:blog-30819994.post-64554449881911582362012-08-12T12:51:00.000-04:002012-08-12T17:30:18.634-04:00Education: whether to skip school or not ?<div dir="ltr" style="text-align: left;" trbidi="on">
Couple of days back, one of my friend (Deepak) posted a <a href="http://nikhilgoyal.me/2012/08/09/stop-waiting-on-the-world-to-change/">post</a>, which lead to somewhat heated debate. Before, I go any further, I must admit I have not read the blog writer's (Nikhil's) book and judging from the title "<span style="background-color: rgba(255, 255, 255, 0.871094); font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 22.100000381469727px;">One Size Does Not Fit All: A Student’s Assessment of School</span>", he seems to be radical thinker (as suggested by Deepak). Also, the above mentioned <a href="http://nikhilgoyal.me/2012/08/09/stop-waiting-on-the-world-to-change/">post</a> might have assumed that the reader has already read his book and hence the post itself could be out of context for me.<br />
<br />
Personally, I felt the post was just a collage of motivational/self-help pieces, without a direct and measurable theme/message. People don't need cheer-leaders that say "here are pictures/quotes of Einstein, Jobs, Ford ... yaay ... now, go and become like them it"; they need a coach ... a teacher, who gives them a clearly defined way or approach to achieve it ... stating how much effort one needs to put into it and what are pros and cons of following that approach. For example, 10,000 hour rule of <a href="http://www.amazon.com/Outliers-Story-Success-Malcolm-Gladwell/dp/0316017930/ref=la_B000APOE98_1_1?ie=UTF8&qid=1344716501&sr=1-1">Gladwell's Outliers</a> or <a href="http://www.amazon.com/Getting-Things-Done-Stress-Free-Productivity/dp/0142000280/ref=sr_1_1?s=books&ie=UTF8&qid=1344717403&sr=1-1&keywords=getting+things+done">Allen's GTD</a>. Sure there are gray-areas, situations where people only need motivational speeches and not solutions (for example: <a href="http://www.amazon.com/Notes-Myself-Struggle-Become-Person/dp/0553273825/ref=sr_1_1?s=books&ie=UTF8&qid=1344717963&sr=1-1&keywords=notes+to+myself">Notes to myself</a>) ... and may be the above <a href="http://nikhilgoyal.me/2012/08/09/stop-waiting-on-the-world-to-change/">post</a> could have been intended for exactly that.<br />
<br />
Just a side-note (and not directing directly to the author of the post), I think writers of self-help books (or any non-fiction book for that matter), should try to aspire for journalistic integrity ... which means checking facts before you make a statement, validating your propositions and yes, being direct and clear. If you don't have time to research something (for example: if you are writing a time sensitive news article), or are making some assumptions (say, about the audience or the context of the article), add a disclaimer and make a reasonable statement based on <a href="http://en.wikipedia.org/wiki/Occam's_razor">Occam's razor</a>. And most importantly, be very critical of the implications of what you write. Sure, you are allowed to be wrong (in few if not all scenarios, for example: if you track scientific research, every once in a while an idea proposed few years back is discarded). In fact there is a <a href="http://www.quora.com/Mathematics/How-can-I-prove-randomness">mathematical argument</a>, that you cannot prove anything to absolute certainty. But it should at least pass a sniff test of critical thinking: <i>If you make your case to a rational person without any prejudice on that particular subject, is he/she going to dismiss it outright ?</i> An example of totally ridiculous argument that does not pass this sniff test is <a href="http://www.youtube.com/watch?v=hmUguHWtoOA">Zakir Naik on NDTV about Osama bin Laden</a>, which might not be violation of <a href="http://en.wikipedia.org/wiki/First_Amendment_to_the_United_States_Constitution">first amendment</a> (i.e. freedom of speech); but clearly can have damaging consequences, especially if audience is ill-informed and trusts the speaker.<br />
<br />
Now coming back to the debate (given in the below image ... the first line that is cut off is "<span style="background-color: #edeff4; color: #333333; font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; font-size: 11px; line-height: 14px;">Sorry Deepak, I don't see any tangible idea in his blog ... may be i am missing something ... sure,</span><span style="background-color: #edeff4; color: #333333; font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; font-size: 11px; line-height: 14px;"> </span>") with my friend, the sentence in the <a href="http://nikhilgoyal.me/2012/08/09/stop-waiting-on-the-world-to-change/">blog</a> that ticked me off was "<span style="background-color: rgba(255, 255, 255, 0.871094); font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 22.100000381469727px;">Drop the textbooks. Get out and do it</span>":<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-O5In6TjeRa4/UCbIzt5-KKI/AAAAAAAAFSk/4rUw2Xgcsqs/s1600/Screen+Shot+2012-08-11+at+4.02.15+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-O5In6TjeRa4/UCbIzt5-KKI/AAAAAAAAFSk/4rUw2Xgcsqs/s1600/Screen+Shot+2012-08-11+at+4.02.15+PM.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-RypHlWKdNAA/UCbJucAdwCI/AAAAAAAAFSs/EFiQEmmQge0/s1600/Screen+Shot+2012-08-11+at+4.07.38+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-RypHlWKdNAA/UCbJucAdwCI/AAAAAAAAFSs/EFiQEmmQge0/s1600/Screen+Shot+2012-08-11+at+4.07.38+PM.png" /></a></div>
After this debate/conversation, I felt like I was just a critic, bashing the post and nitpicking on a single possibly unintended sentence, rather than contributing to the subject. Hence, I decided to write this post to give my views on education.<br />
<br />
I am doing PhD, so there is going to be bias in support of education. So I ask you to consider the points discussed below with respect to your situation, taking into account my personal bias.<br />
<br />
I honestly believe formal education is easiest and fastest way to success (though many students don't feel like that's true due to amount of competition they see around). You can either work really hard during four years of graduation or struggle rest of forty years of your career to compensate for those four years. There are exceptions of course, for example, athletes who prefer/need to horn their skills than study (like <a href="http://en.wikipedia.org/wiki/Sachin_Tendulkar">Sachin Tendulkar</a>, <a href="http://en.wikipedia.org/wiki/Usain_Bolt">Usain Bolt</a>), entrepreneurs for whom opportunity cost is higher than education (like <a href="http://en.wikipedia.org/wiki/Bill_Gates">Bill Gates</a>, <a href="http://en.wikipedia.org/wiki/Steve_Jobs">Steve Jobs</a>, <a href="http://en.wikipedia.org/wiki/Richard_Branson">Richard Branson</a>) and people who are forced to discontinue their formal education due to certain circumstances (Nobel laureate <a href="http://en.wikipedia.org/wiki/Doris_Lessing">Doris Lessing</a>). Lot of books and <a href="http://www.forbes.com/sites/stevenbertoni/2012/01/11/four-career-tips-from-billionaires-who-never-graduated-college/">post by these entrepreneurs</a> advocate to try to be these exceptions, but I suggest caution. They are called "exceptions" for a reason; count number of people (either in your locality or in the world) who left their schools versus number of people who attended the schools, and then use your measure of success to classify them as successul or unsuccessful ... you will find your answer :)<br />
<br />
So, why do some smart people (like authors of books mentioned in above paragraph) suggest on leaving the schools ? Well, you need to read between the lines. If you have fairly concrete idea of your product/service (along with somewhat reasonable plan) and believe waiting to finish your education will cost you the competitive edge, then it might be a good idea to take a sabbatical and be an entrepreneur. Also, read <a href="http://www.youtube.com/watch?v=KuNQgln6TL0&feature=related">advices</a> and <a href="http://www.amazon.com/The-Startup-Owners-Manual-Step-By-Step/dp/0984999302/ref=pd_sim_b_2">books on startup</a>, understand the novelty, amount of hard work and dedication required in building a successful startup before you make your decision of leaving the school.<br />
<br />
Regarding Sean Parker's advice "<a href="http://www.forbes.com/sites/stevenbertoni/2012/01/11/four-career-tips-from-billionaires-who-never-graduated-college/">skip school, google your education</a>", it doesn't work. Government spends lot of money in educational system (such as deciding curriculum, giving grants, hiring/educating teachers, etc) and it will be a bad idea to not capitalize on it. I would modify his advice: "<i>continue your school, but also google your education</i>". There are lot of websites that give free courses/talks (like <a href="https://www.coursera.org/">coursera</a>, <a href="http://ocw.mit.edu/index.htm">MIT</a>, <a href="http://www.ted.com/talks">TedTalks</a> etc), utilize them. Read wikipedia, books (textbooks and other non-course books), research papers, newspapers and blogs, then exercise critical thinking and combine them with advices given by your teacher, to formulate your own opinions/views ... world really needs more informed people. Also, there are lot of classes/camps, that you can attend/audit to supplement your education. For example: I completed PG Diploma course in Embedded Systems from ECIL (a course offered to only graduates) during my second year of engineering at VJTI. Two things I learnt during this experience is (1) even though there are pre-requisites, educators are willing to overlook them if you show enough enthusiasm and determinism; (2) if you chose to break the rules, be ready to put twice the amount of effort your peers are putting. During my four years of engineering, every semester, I always had a parallel course or non-course project that I was working on. This helped me immensely after my graduation. My point is you don't have to succumb to conformity (there are ways to supplement your formal education), and if you really plan to do leave formal education, make sure you are doing it for right reason (and not because you are lazy or because you feel leaving studies will somehow magically make your life easier and especially not because someone cites you a quote from a successful person).<br />
<br />
Here is a high level intuition of why formal education seems tedious and boring: Society wants to filter out people who cannot survive the <a href="http://www.amazon.com/The-Dip-Little-Teaches-Stick/dp/1591841666/ref=sr_1_3?ie=UTF8&qid=1344789407&sr=8-3&keywords=dip">dip</a>. (The dip is a period of long struggle that is necessary to achieve the goal and be differentiated from general public). For example, if PhDs or medicine were not as difficult as they are, there would be abundance of doctors and PhDs, and the elite status they enjoy now would be gone; so the society wants to ensure that these so-called elites really have to struggle before they can be labelled into this category. Sure, there is other reasons too: Some degrees (PhDs, MDs) requires knowing lot of stuff, and hence the long graduation period. You can't become a doctor by just knowing how to apply a bandage or by memorizing prescriptions for common diseases. For sake of this argument, say if a person can, would you want your family member treated by such person ?<br />
<br />
That leads to the question of what are rational reasons to quit your studies (other than the case discussed above): If you think (or can forecast that) after surviving the dip, you don't like what's end of it, then it might be a good idea chose a different path. For example: during first year of medical school, if you don't like the life of doctor, it might be a good idea to drop out of medical school and do something that you are really passionate about. But, it is clearly a bad idea to drop out just because you feel medical school is hard or there is too much competition. Similarly, there are also rational reasons to <a href="http://www.quora.com/Graduate-School/Should-I-quit-my-CS-PhD">quit PhD</a>, but all of them should be accompanied by careful thinking and introspection. The <a href="http://www.amazon.com/The-Dip-Little-Teaches-Stick/dp/1591841666/ref=sr_1_3?ie=UTF8&qid=1344789407&sr=8-3&keywords=dip">book</a> also covers other scenarios such as dead-end, where you have no hope of moving ahead and are stuck in routine ... but I don't think education fits into that scenario.<br />
<br />
Note, if you believe you have dyslexia, bipolar or any such health issues (either physical or mental), then you need to completely ignore my advices and consult your family and/or therapist, as they will give you better suggestions than this blogpost.</div><div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com1tag:blogger.com,1999:blog-30819994.post-10892724143337962712012-07-16T16:17:00.000-04:002012-07-16T19:32:29.244-04:00Improving speech recognition using clustering-based adaptive model assignment<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: CMR12; font-size: 12pt;">
I came up with this idea while working on <a href="https://www.facebook.com/voca.desktop">Voca</a>. Since I don't intend to include this as part of my PhD thesis, write a research paper (since I don't have time to perform experimental evaluation) or make money out of it by patenting it (thanks to extremely slow Rice OTT process), I thought it might at least make a good blog entry.
</span>
<br />
<br />
<span style="font-family: CMR12; font-size: 12pt;">In this blogpost, I will discuss a novel approach to improve speech recognition </span><span style="font-family: CMR12; font-size: 12pt;">that in-turn will help improve the accuracy of Voca desktop app (if and when I decide to work on it). This approach can be applied to any speech-to-text software that is to be distributed to masses (eg: Siri).</span><br />
<h2 style="text-align: left;">
</h2>
<h2>
<span style="font-family: CMR12;"><span style="font-size: large;">1. Background</span></span></h2>
<div class="column" style="font-size: medium; font-weight: normal;">
</div>
<span style="background-color: white; color: purple; font-family: CMBX12; font-size: 14pt;">1.1 Introduction</span> <br />
<span style="font-family: CMR12; font-size: 12pt;">Automatic speech recognition (ASR) involves predicting the most likely sequence of words </span><span style="font-family: CMMI12; font-size: 12pt;">W </span><span style="font-family: CMR12; font-size: 12pt;">for an input audio data </span><span style="font-family: CMMI12; font-size: 12pt;">A. </span><span style="font-family: CMR12; font-size: 12pt;">To do so, the speech recognizers use the following optimization function[2]:</span><br />
\begin{align*}
\hat{W}&= {arg\,max}_{W} P(W | A) & \\
&= {arg\,max}_{W} \dfrac{P(W) P(A | W)}{P(A)} & \dots \text{ Bayes theorem} \\
&\propto {arg\,max}_{W} P(W) P(A | W) & \dots \text{ The denominator normalizes to a constant}
\end{align*}
<br />
<br />
<span style="font-family: CMR12; font-size: 12pt;">The term </span><span style="font-family: CMMI12; font-size: 12pt;">P </span><span style="font-family: CMR12; font-size: 12pt;">(W ) means the probability of occurrence of given sequence of
words and is defined by the </span><span style="font-family: CMTI12; font-size: 12pt;">language model. </span><span style="font-family: CMR12; font-size: 12pt;">At a high level, the language
model captures the inherent restrictions imposed by the english language.
For example: a language model might imply that the sentence “it is raining
heavily in houston” is more likely sentence that “raining it is in heavily
houston”. </span><br />
<br />
<span style="font-family: CMR12; font-size: 12pt;">The term </span><span style="font-family: CMMI12; font-size: 12pt;">P(A|W) </span><span style="font-family: CMR12; font-size: 12pt;">refers to the probability of an audio data given the
sequence of word and is captured by the </span><span style="font-family: CMTI12; font-size: 12pt;">acoustic model, </span><span style="font-family: CMR12; font-size: 12pt;">which at a high level
is probabilistic mapping of the acoustic sound to the words. The acoustic </span><span style="font-family: CMR12; font-size: 12pt;">model is usually found by training hidden markov model on a small subset
of users. The users that train the acoustic model are referred as </span><span style="font-family: CMTI12; font-size: 12pt;"><i>trainers</i> </span><span style="font-family: CMR12; font-size: 12pt;">in
subsequent sections.</span><br />
<br />
<span style="font-family: CMR12; font-size: 12pt;">To summarize, the speech recognizer </span><span style="font-family: CMTI12; font-size: 12pt;">searches </span><span style="font-family: CMR12; font-size: 12pt;">for the sequence of words,
that maximizes the above function using the acoustic and the language
model. The output of this search procedure is a tuple </span><span style="font-family: CMMI12; font-size: 12pt;">< </span><span style="font-family: CMR12; font-size: 12pt;">words </span><span style="font-family: CMMI12; font-size: 12pt;">W, </span><span style="font-family: CMR12; font-size: 12pt;">confidence </span><span style="font-family: CMMI12; font-size: 12pt;">>, </span><span style="font-family: CMR12; font-size: 12pt;">where confidence refers to the value of the above function for </span><span style="font-family: CMMI12; font-size: 12pt;">W.
</span><span style="font-family: CMR12; font-size: 12pt;">For example: </span><span style="font-family: CMMI12; font-size: 12pt;"><“it </span><span style="font-family: CMR12; font-size: 12pt;">is raining heavily in houston”, 0.9 </span><span style="font-family: CMMI12; font-size: 12pt;">></span><br />
<span style="font-family: CMR12; font-size: 12pt;">Thus, accuracy of speech recognizer thus highly depends on these two
models. Since only acoustic model varies with respect to the user, in our
further discussion we will only refer to acoustic model and assume the use of
a standardized language model (eg: <a href="http://sourceforge.net/projects/cmusphinx/files/Acoustic%20and%20Language%20Models/US%20English%20HUB4%20Language%20Model/">HUB4</a>).</span><br />
<span style="color: purple; font-family: CMBX12; font-size: 14pt;">1.2 Roles</span><br />
<div class="column">
<span style="font-family: CMR12; font-size: 12pt;">There are three types of people involved in speech recognition process:
</span><br />
<ul>
<li style="font-family: 'CMSY10'; font-size: 12.000000pt;">
<span style="font-family: CMBX12; font-size: 12pt;">Developers </span><span style="font-family: CMR12; font-size: 12pt;">write the code for the speech recognition software.
</span><br />
</li>
<li style="font-family: 'CMSY10'; font-size: 12.000000pt;">
<span style="font-family: CMBX12; font-size: 12pt;">Trainers </span><span style="font-family: CMR12; font-size: 12pt;">record their voice and give the audio files to the developers.
Developers then use these audio files to train the acoustic model.
</span><br />
</li>
<li style="font-size: 12pt;">
<div style="font-family: CMSY10;">
<span style="font-family: CMBX12; font-size: 12pt;">Users </span><span style="font-family: CMR12; font-size: 12pt;">are the general audience who uses the speech recognition software. The number of trainers is usually much less than number of
users.</span></div>
</li>
</ul>
</div>
<div style="text-align: left;">
<span style="color: purple; font-family: CMBX12; font-size: 14pt;">1.3 Steps in speech recognition process</span></div>
<div class="column">
<span style="font-family: CMR12; font-size: 12pt;">The steps in typical speech recognition API/software is as follows: </span><br />
<span style="font-family: CMR12; font-size: 12pt;">1. Before releasing the API/software:
</span><br />
<ol style="list-style-type: none; text-align: left;">
<li>
<span style="font-family: CMR12; font-size: 12pt;">(a) Developers encourages trainers to record their voice.
</span><br />
</li>
<li>
<span style="font-family: CMR12; font-size: 12pt;">(b) Developers use the audio files to train a universal model for all
the users. The developers may also use information provided by
experts to improve language model.
</span><br />
</li>
<li>
<span style="font-family: CMR12; font-size: 12pt;">(c) Developers bundle together the model and the search algorithm
and release the API/software.
</span><br />
</li>
</ol>
<span style="font-family: CMR12; font-size: 12pt;">2. After releasing the API/software: </span><br />
<span style="font-family: CMR12; font-size: 12pt;"> (a) The search algorithm in the API/software tries to find what the
user is saying by comparing the user’s acoustic signal with its
universal model using various statistical methods. </span><br />
<br />
<div class="column">
<span style="color: purple; font-family: CMBX12; font-size: 14pt;">1.4 Challenges</span><br />
<div class="column">
<span style="font-family: CMR12; font-size: 12pt;">However, speech recognition is inherently a hard problem due to issues like:
</span><br />
<ol>
<li style="font-family: 'CMR12'; font-size: 12.000000pt;">
<span style="font-size: 12pt;">Differences in speaking style due to accent, regional dialect, sex, age,
etc (Eg: the acoustic signature of a young chinese girl saying some
words will probably differ from an older german person speaking those
exact same words).
</span><br />
</li>
<li style="font-family: 'CMR12'; font-size: 12.000000pt;">
<span style="font-size: 12pt;">Lack of natural pauses to detect word boundaries.
</span><br />
</li>
<li style="font-family: 'CMR12'; font-size: 12.000000pt;">
<span style="font-size: 12pt;">Computational issues like search space, homophones (words that sounds
same), echo effect due to surroundings, background noise.
</span><br />
</li>
</ol>
<span style="font-family: CMR12; font-size: 12pt;">Lot of research effort has been put to tackle issues due to point 2 and 3.
To tackle issue 1, the most common solution is to train on as many people as
possible to create an somewhat universal acoustic model that captures these
differences (as mentioned in step 1b of section 1.3). This is because most
people do not want to spend hours training the acoustic model and would
prefer an out-of-the-box solution. The problem is this approach is that it
always creates a biased acoustic model that works well for some people, but
is highly inaccurate with many. </span><br />
<h2 style="text-align: left;">
<span style="font-family: CMR12;"><span style="font-size: large;">2. Summary</span></span></h2>
<div class="column">
<span style="font-family: CMR12; font-size: 12pt;">Instead of using one universal model that works for everyone, we propose
the following approach:
</span><br />
<blockquote class="tr_bq">
<span style="font-family: CMR12; font-size: 12pt;">Train </span><span style="font-family: CMMI12; font-size: 12pt;">k </span><span style="font-family: CMR12; font-size: 12pt;">different models in step 1b of section 1.3 (where is </span><span style="font-family: CMMI12; font-size: 12pt;">k </span><span style="font-family: CMR12; font-size: 12pt;">is
reasonably small number, for example: 100) such that each of
these models try to capture the speaking style of a particular
group of people. Then, recommend the model that best suits the
user depending on his/her acoustic similarity with each of the
above groups. </span></blockquote>
<br />
<div class="column">
<span style="color: purple; font-family: CMBX12; font-size: 14pt;">2.1 Why this idea will work in practice ?</span><br />
<ol style="text-align: left;">
<li><span style="font-family: CMR12; font-size: 12pt;">It is a well known fact that the accuracy of the speech recognizer is
higher if the trainer and the user are the same person.</span></li>
<li><span style="font-family: CMR12; font-size: 12pt;">As pointed out earlier, most users do not want to spend hours training
the model.</span></li>
<li><span style="font-family: CMR12; font-size: 12pt;">Some people are more acoustically similar than others. For eg: model
trained on young native american student will suit well for a similar
student, but might not suit well for an old Indian lady. Hence, if the
trainer and the user speak very similarly, the model will have higher
accuracy than an universal model.</span></li>
<li><span style="font-family: CMR12; font-size: 12pt;">We like to point out that users can be clustered into groups by the way
they speak and the number of these groups are finite. This is basically
a soft assignment, for eg: I would fall in the category American accent
with probability 0.2, but in category Indian accent with probability 0.7.
Thus this knowledge can be incorporated in the recognition process. </span></li>
</ol>
<div class="section">
<div class="layoutArea">
<div class="column">
<span style="color: purple; font-family: CMBX12; font-size: 14pt;">2.2 Related Works</span></div>
</div>
<div class="layoutArea">
<div class="column">
<span style="font-family: CMR12; font-size: 12pt;">An area that is close to our invention is speaker-adaptive modeling[1], where
the universal model is adapted to the user based on training on </span><span style="font-family: CMTI12; font-size: 12pt;">speaker-
dependent </span><span style="font-family: CMR12; font-size: 12pt;">features. This means, the parameters of acoustic models are split
into two parts:</span><br />
<ul style="text-align: left;">
<li><span style="font-family: CMR12; font-size: 12pt;">Speaker independent parameters</span></li>
<li><span style="font-family: CMR12; font-size: 12pt;">Speaker dependent parameters</span></li>
</ul>
<span style="font-family: CMR12; font-size: 12pt;">This division requires user to supply relatively smaller training set (which
need not be done during initial setup). However, it still has following difficulties</span><br />
<ul style="text-align: left;">
<li><span style="font-family: CMR12; font-size: 12pt;">Training is significantly compute-intensive task and might not be a
viable option for smart phones, netbooks or even laptops. Offloading
the training on to server will involve more network usage (which in turn
will increase the cost of speech recognition for user or/and developer).</span></li>
<li><span style="font-family: CMR12; font-size: 12pt;">Training might require expert-intervention. </span></li>
<li><span style="font-family: CMR12; font-size: 12pt;">It still is not an out-of-the-box solution as an off-line solution will require some training to make it speaker-adaptive.</span></li>
</ul>
<div style="font-size: 12pt;">
<span style="font-size: 12pt;">
</span></div>
<div style="text-align: left;">
<span style="font-family: CMR12; font-size: 12pt;">For </span><span style="font-family: CMMI12; font-size: 12pt;">n </span><span style="font-family: CMR12; font-size: 12pt;">users (where </span><span style="font-family: CMMI12; font-size: 12pt;">n </span><span style="font-family: CMSY10; font-size: 12pt;">≫ </span><span style="font-family: CMMI12; font-size: 12pt;">k), </span><span style="font-family: CMR12; font-size: 12pt;">Speaker-adaptive systems in effect will have
</span><span style="font-family: CMMI12; font-size: 12pt;">n </span><span style="font-family: CMR12; font-size: 12pt;">models and our </span><span style="font-family: CMTI12; font-size: 12pt;">clustering-based model assignment </span><span style="font-family: CMR12; font-size: 12pt;">approach will have </span><span style="font-family: CMMI12; font-size: 12pt;">k
</span><span style="font-family: CMR12; font-size: 12pt;">models. Since </span><span style="font-family: CMMI12; font-size: 12pt;">k </span><span style="font-family: CMR12; font-size: 12pt;">models are trained on the server, model training will also
benefit from expert-intervention without any additional cost to the user. It
is important to note that model assignment does not require any expert-
intervention and is not same as speaker-adaptive training. </span></div>
<h2 style="text-align: left;">
<span style="font-family: CMBX12; font-size: 17pt;">3. Detail discussion</span></h2>
<div style="text-align: left;">
</div>
<div class="column">
<span style="color: purple; font-family: CMBX12; font-size: 14pt;">3.1 Key conditions</span><br />
<ol style="text-align: left;">
<li><span style="font-family: CMR12; font-size: 12pt;">Every user can be represented as a mixture model over the acoustic
model. </span><span style="font-family: CMR12; font-size: 12pt;">Example: Let’s say that user </span><span style="font-family: CMMI12; font-size: 12pt;">U1 </span><span style="font-family: CMR12; font-size: 12pt;">is sampled with mixture proportion
</span><span style="font-family: CMSY10; font-size: 12pt;">{0.9, </span><span style="font-family: CMR12; font-size: 12pt;">0.1} over model </span><span style="font-family: CMMI12; font-size: 12pt;">m1 </span><span style="font-family: CMR12; font-size: 12pt;">and </span><span style="font-family: CMMI12; font-size: 12pt;">m2 </span><span style="font-family: CMR12; font-size: 12pt;">respectively. This means that if we
run recognition on the audio file generated by the user </span><span style="font-family: CMMI12; font-size: 12pt;">U1 </span><span style="font-family: CMR12; font-size: 12pt;">on both the
models, it is likely that model </span><span style="font-family: CMMI12; font-size: 12pt;">m1 </span><span style="font-family: CMR12; font-size: 12pt;">will produce output that is going to
be </span><span style="font-family: CMTI12; font-size: 12pt;">9 times more accurate </span><span style="font-family: CMR12; font-size: 12pt;">that model </span><span style="font-family: CMMI12; font-size: 12pt;">m2.</span></li>
<li><span style="font-family: CMR12; font-size: 12pt;">When we create a model, the job of the expert is to make sure that:</span></li>
</ol>
<ol style="list-style-type: none;">
<li>
<span style="font-family: CMR12; font-size: 12pt;">(a) Each model is representative of a distinct accent.</span><br />
<span style="font-family: CMR12; font-size: 12pt;">(b) Every user (on average) is expected to have high affinity to only
1 model.</span></li>
</ol>
<span style="font-family: CMR12; font-size: 12pt;">This means if we have a two sets mixture models </span><span style="font-family: CMSY10; font-size: 12pt;">{0.002, </span><span style="font-family: CMR12; font-size: 12pt;">0.99, 0.003}
and </span><span style="font-family: CMSY10; font-size: 12pt;">{0.33, </span><span style="font-family: CMR12; font-size: 12pt;">0.33, 0.33} for a random user, the former configuration
is preferable.</span><br />
<span style="font-family: CMR12; font-size: 12pt;"> 3. Corollaries based on above points:</span> <br />
<span style="font-family: CMR12; font-size: 12pt;"> (a) A model is simply a cluster of users. </span><br />
<span style="font-family: CMR12; font-size: 12pt;"> (b) A user can belong to multiple cluster.</span><br /> <br />
<span style="color: purple; font-family: CMBX12; font-size: 14pt;">3.2 Modified steps for speech recognition</span><br />
<div class="column">
<span style="font-family: CMR12; font-size: 12pt;">We propose a modified methodology that will improve the accuracy of the
speech recognition:
</span><br />
<span style="font-family: CMR12; font-size: 12pt;">1. Before releasing the API/software:
</span><br />
<ol style="list-style-type: none;">
<li>
<span style="font-family: CMR12; font-size: 12pt;">(a) Developers encourages trainers to record their voice.
</span><br />
</li>
<li>
<span style="font-family: CMR12; font-size: 12pt;">(b) Using the model described in section 3.3, we get </span><span style="font-family: CMMI12; font-size: 12pt;">k </span><span style="font-family: CMR12; font-size: 12pt;">models (For
example: </span><span style="font-family: CMMI12; font-size: 12pt;">k </span><span style="font-family: CMR12; font-size: 12pt;">= 100).
</span><br />
</li>
<li>
<span style="font-family: CMR12; font-size: 12pt;">(c) Chose one of these </span><span style="font-family: CMMI12; font-size: 12pt;">k </span><span style="font-family: CMR12; font-size: 12pt;">model as a default model. Also, set a value
for the parameter </span><span style="font-family: CMMI12; font-size: 12pt;">n </span><span style="font-family: CMR12; font-size: 12pt;">(For example: </span><span style="font-family: CMMI12; font-size: 12pt;">n </span><span style="font-family: CMR12; font-size: 12pt;">= 3).
</span><br />
</li>
<li>
<span style="font-family: CMR12; font-size: 12pt;">(d) for i = 1 to n do:
<br />
- Select a new phrase p<sub>i</sub> such that all <i>k</i> models have at least one audio file that have phrase p<sub>i</sub> or one very similar to the phrase p<sub>i</sub>.
Hint: During training process, ask every trainer to speak the phrase p<sub>i</sub>. Use edit-distance and some tolerance delta (= 5) to find similar phrases.
</span><br />
</li>
<li>
<span style="font-family: CMR12; font-size: 12pt;">(e) Developers bundle together the default model and the search algorithm and release the API/software.
</span><br />
</li>
</ol>
<span style="font-family: CMR12; font-size: 12pt;">2. After releasing the API/software:
</span><br />
<ol style="list-style-type: none;">
<li>
<span style="font-family: CMR12; font-size: 12pt;">(a) During installation (or during periodic re-tuning), for </span><span style="font-family: CMMI12; font-size: 12pt;">i </span><span style="font-family: CMR12; font-size: 12pt;">= 1 to </span><span style="font-family: CMMI12; font-size: 12pt;">n
</span><span style="font-family: CMR12; font-size: 12pt;">do:
</span><br />
<ol style="list-style-type: lower-roman; text-align: left;">
<li style="font-family: CMR12; font-size: 12pt;">
<span style="font-size: 12pt;">Ask user to speak the phrase </span><span style="font-family: CMMI12; font-size: 12pt;">p<sub>i</sub> </span><span style="font-size: 12pt;">and send the audio file to the
server.
</span><br />
</li>
<li style="font-size: 12pt;">
<div style="font-family: CMR12;">
<span style="font-size: 12pt;">For each model </span><span style="font-family: CMMI12; font-size: 12pt;">j, </span><span style="font-size: 12pt;">do:
</span></div>
<ol style="list-style-type: upper-latin;">
<li style="font-family: CMR12; font-size: 12pt;">
<span style="font-size: 12pt;">Do speech-to-text transcription for the given audio file.
</span><br />
</li>
<li style="font-family: CMR12; font-size: 12pt;">
<span style="font-size: 12pt;">Let </span><span style="font-family: CMMI12; font-size: 12pt;">p′<sub>i,j</sub> </span><span style="font-size: 12pt;">be the output phrase of the transcription.
</span><br />
</li>
</ol>
</li>
</ol>
<div style="font-family: CMR12; font-size: 12pt;">
<span style="font-size: 12pt;">Let </span><span style="font-family: CMMI12; font-size: 12pt;">ε<sub>i,j</sub> </span><span style="font-size: 12pt;">be a measure that calculates the error in transcription of phrase </span><span style="font-family: CMMI12; font-size: 12pt;">p<sub>j</sub> </span><span style="font-size: 12pt;">on model </span><span style="font-family: CMMI12; font-size: 12pt;">j.</span><span style="font-size: 12pt;">For example: </span><span style="font-family: CMMI12; font-size: 12pt;">ε<sub>i,j</sub> </span><span style="font-size: 12pt;">= Edit-Distance(p′<sub>i,j</sub>,p<sub>i,j</sub>). Note that, depending on the training data, an expert might chose to use a different metric (before releasing the API) such as acoustic similarity between a random audio file in the model and the audio file outputted by the use.</span></div>
</li>
<li>
<span style="font-family: CMR12; font-size: 12pt;">(b) Return the model to the user that has minimum error for all the
phrases </span>
</li>
<li>
<span style="font-family: CMR12; font-size: 12pt;">(c) The user uses the model that is send by the server rather than a
universal model. </span>
</li>
</ol>
<div style="font-family: CMR12; font-size: 12pt;">
<span style="color: purple; font-family: CMBX12; font-size: 14pt;">3.3. Model for clustering the trainers</span></div>
<div style="font-size: 12pt;">
</div>
<div class="column">
<div style="font-family: CMR12;">
<span style="font-family: CMBX12; font-size: 12pt;"><b>Step 1:</b> </span><span style="font-size: 12pt;">Create two sets:
</span></div>
<div style="font-size: 12pt;">
</div>
<ul style="font-family: CMSY10; text-align: left;">
<li><span style="font-family: CMR12; font-size: 12pt;">Set 1 (Set of models): Models that have been found to contain </span><span style="font-family: CMR12; font-size: 12pt;">similar users. Initial value = </span><span style="font-size: 12pt;">{}.</span></li>
<li><span style="font-family: CMR12; font-size: 12pt;">Set 2 (Set of trainers not yet clustered): Put all the trainers in </span><span style="font-family: CMR12; font-size: 12pt;">Set 2</span></li>
</ul>
<div style="font-family: CMSY10; font-size: 12pt; text-align: left;">
<span style="font-family: CMBX12; font-size: 12pt;"><b>Step 2:</b> </span><span style="font-family: CMR12; font-size: 12pt;">Train an acoustic model M using audio files of trainers in set 2.</span></div>
<div style="font-family: CMSY10; font-size: 12pt;">
<span style="font-family: CMBX12; font-size: 12pt;"><b>Step 3:</b> </span><span style="font-family: CMR12; font-size: 12pt;">Run recognition for the audio files of the trainers in set 2 </span><span style="font-size: 12pt;">⇒ </span><span style="font-family: CMR12; font-size: 12pt;">You
get </span><span style="font-family: CMMI12; font-size: 12pt;">ε<sub>i</sub> </span><span style="font-family: CMR12; font-size: 12pt;">(i.e. Edit-distance between output phrase of recognizer and the
true phrase). This means some of the audio files are used as held-out. It might also be interesting to have no held-out and see the error when the training and test data are the same.
</span></div>
<div style="font-family: CMSY10; font-size: 12pt;">
<span style="font-family: CMBX12; font-size: 12pt;"><b>Step 4:</b> </span><span style="font-family: CMR12; font-size: 12pt;">Find maximal subset of the trainers whose </span><span style="font-family: CMMI12; font-size: 12pt;">Zi </span><span style="font-family: CMR12; font-size: 12pt;">matches the trainers
for all the phrases </span><span style="font-size: 12pt;">⇒ </span><span style="font-family: CMR12; font-size: 12pt;">Train the model using these trainers and add it
to Set 2 (after a sanity test i.e. this model should be able to predict a random set of audio files that it was trained on)
</span></div>
<div style="font-family: CMSY10; font-size: 12pt;">
<span style="font-family: CMBX12; font-size: 12pt;"><b>Step 5:</b> </span><span style="font-family: CMR12; font-size: 12pt;">Put the remaining in Set 2 and goto Step 2. </span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br />
<br />
<h2>
<span style="font-family: CMBX12; font-size: 17pt;">4. References</span><div style="font-size: medium; font-weight: normal;">
</div>
</h2>
<div class="column">
<ol style="list-style-type: none;">
<li>
<span style="font-family: 'CMR12'; font-size: 12.000000pt;">[1] X. Huang and K.F. Lee. On speaker-independent, speaker-dependent,
and speaker-adaptive speech recognition. </span><span style="font-family: 'CMTI12'; font-size: 12.000000pt;">Speech and Audio Processing,
IEEE Transactions on, </span><span style="font-family: 'CMR12'; font-size: 12.000000pt;">1(2):150 –157, apr 1993.
</span><br />
</li>
<li>
<span style="font-family: 'CMR12'; font-size: 12.000000pt;">[2] Geoffrey Zweig and Michael Picheny. Advances in large vocabulary continuous speech recognition. volume 60 of </span><span style="font-family: 'CMTI12'; font-size: 12.000000pt;">Advances in Computers, </span><span style="font-family: 'CMR12'; font-size: 12.000000pt;">pages
249 – 291. Elsevier, 2004. </span><br />
</li>
</ol>
</div>
</div>
</div>
</div>
</div><div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-29946492784940275862012-02-02T19:34:00.000-05:002012-02-02T19:38:40.773-05:00Cross-validation on SVM-Light<div dir="ltr" style="text-align: left;" trbidi="on">
Before reading this blogpost you should read: <a href="http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf">Practical guide to libSVM</a>, <a href="http://en.wikipedia.org/wiki/Support_vector_machine">Wikipedia entry on SVM</a>, <a href="http://en.wikipedia.org/wiki/Cross-validation_(statistics)">Wikipedia entry on cross-validation</a>, and <a href="http://www.amazon.com/Pattern-Recognition-Learning-Information-Statistics/dp/0387310738">PRML - Bishop</a>'s chapter on SVM. In this post, I will not explain how to use open source SVM tools (like libsvm or svmlight), but would rather assume that you have used them and want some clever script to automate them. Also note that, the below script uses default parameters. So if you get poor results, you can add an outer for loop that iterates through your choice of parameters and supplies them to the program <u>./svm_learn</u>.<br />
<br />
To perform cross-validation on <a href="http://www.csie.ntu.edu.tw/~cjlin/libsvm/">libsvm</a>:<br />
<ol style="text-align: left;">
<li>Download and extract it</li>
<li>You need gnuplot installed on the machine you want to run libsvm. If you want to run it on the server, do 'ssh -X myuserName@myMachineName'</li>
<li>make</li>
<li>Go to Tools folder and run 'python easy.py my_training_file my_test_file'</li>
</ol>
To perform cross-validation on <a href="http://svmlight.joachims.org/">svmlight</a>:<br />
<ol style="text-align: left;">
<li>Download and extract it. There you will find two important binaries: svm_learn and svm_classify</li>
<li>Unfortunately the data gathering using svmlight is not easy, so I have coded following script. </li>
</ol>
<pre class="brush: html">#!/bin/bash
# Usage: svm_light_cross_validation full_data num_train_items num_test_items k_cycles results_file
RESULT_FILE=$5
echo "Running SVM-Light via cross validation on" $1 "by using" $2 "training items and" $3 "test items (Total number of cross-validation cycles:" $4 > $RESULT_FILE
MODEL_FILE="model."$RANDOM".txt"
TEMP_FILE="tempFile."$RANDOM".txt"
PRED_FILE="prediction."$RANDOM".txt"
DATA_FILE=$1
NUM_TRAIN=$2
NUM_TEST=$3
NUM_CYCLES=$4
TEMP_DATA_FILE=$DATA_FILE"."$RANDOM".temp"
TRAIN_FILE=$TEMP_DATA_FILE".train"
TEST_FILE=$TEMP_DATA_FILE".test"
TEMP_RESULT=$RESULT_FILE".temp"
SVM_PATTERN='s/Accuracy on test set: \([0-9]*.[0-9]*\)% ([0-9]* correct, [0-9]* incorrect, [0-9]* total)\.*/'
for k in `seq 1 $NUM_CYCLES`
do
sort -R $DATA_FILE > $TEMP_DATA_FILE
head -n $NUM_TRAIN $TEMP_DATA_FILE > $TRAIN_FILE
tail -n $NUM_TEST $TEMP_DATA_FILE > $TEST_FILE
echo "------------------------------------------" >> $RESULT_FILE
echo "Cross-validation cycle:" $k >> $RESULT_FILE
# first run svm with default parameters
echo "" >> $RESULT_FILE
echo "Polynomial SVM with default parameters" >> $RESULT_FILE
for i in 1 2 3 4 5 6 7 8 9 10
do
echo "order:" $i >> $RESULT_FILE
./svm_learn -t 1 -d $i $TRAIN_FILE $MODEL_FILE > $TEMP_FILE
./svm_classify -v 1 $TEST_FILE $MODEL_FILE $PRED_FILE > $TEMP_RESULT
cat $TEMP_RESULT >> $RESULT_FILE
sed '/^Reading model/d' $TEMP_RESULT > $TEMP_RESULT"1"
sed '/^Precision/d' $TEMP_RESULT"1" > $TEMP_RESULT
sed "$SVM_PATTERN$k poly $i \1/g" $TEMP_RESULT >> "better"$RESULT_FILE
done
echo "" >> $RESULT_FILE
echo "RBF SVM with default parameters" >> $RESULT_FILE
for g in 0.00001 0.0001 0.001 0.1 1 2 3 5 10 20 50 100 200 500 1000
do
echo "gamma:" $g >> $RESULT_FILE
./svm_learn -t 2 -g $g $TRAIN_FILE $MODEL_FILE > $TEMP_FILE
./svm_classify -v 1 $TEST_FILE $MODEL_FILE $PRED_FILE >> $TEMP_FILE
cat $TEMP_RESULT >> $RESULT_FILE
sed '/^Reading model/d' $TEMP_RESULT > $TEMP_RESULT"1"
sed '/^Precision/d' $TEMP_RESULT"1" > $TEMP_RESULT
sed "$SVM_PATTERN$k rbf $g \1/g" $TEMP_RESULT >> "better"$RESULT_FILE
done
done
rm $MODEL_FILE $TEMP_FILE $PRED_FILE $TEMP_DATA_FILE $TEMP_RESULT $TEMP_RESULT"1"
echo "Done." >> $RESULT_FILE
</pre>
<br />
Here is how you run this script:
<br />
<pre class="brush: html">./svm_light_cross_validation myDataInSVMLightFormat.txt 5000 2000 10 MyResults.txt
</pre>
<br />
myDataInSVMLightFormat.txt looks like "binaryClassifier(-1 or 1) featureNum:value ....". Example:
<br />
<pre class="brush: html">1 1:1.75814e-14 2:1.74821e-05 3:1.37931e-08 4:1.25827e-14 5:4.09717e-05 6:1.28084e-09 7:2.80137e-22 8:2.17821e-24 9:0.00600121 10:0.002669 11:1.19775e-05 12:8.24607e-15 13:8.36358e-10 14:0.0532899 15:3.25028e-06 16:0.148439 17:9.76215e-08 18:0.00148927 19:3.69801e-16 20:4.20283e-16 21:7.9941e-05 22:5.7593e-09 23:0.000251052 24:0.000184218 25:7.07359e-06
</pre>
<br />
After running this script you will get two important files: MyResults.txt and <b>betterMyResults.txt </b>(which is a space separated file with format: 'validation_cycle_num [poly or rbf] parameterValue accuracy'). You can then import betterMyResults.txt in R or Matlab and get some pretty graphs :)</div><div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com7tag:blogger.com,1999:blog-30819994.post-11131164193009308212012-02-01T18:10:00.002-05:002012-02-01T18:15:48.005-05:00Using VIM for "fast" data transformation<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Recently, I had to give my data to a lab-mate. I had done some pre-processing on <a href="http://archive.ics.uci.edu/ml/datasets/Twenty+Newsgroups">20 Newsgroup dataset</a> and converted it to following format:<br />
- Each line represented a document.<br />
- Words were replaced by an integer that denoted it's index in vocabulary.txt file<br />
- The line was append with a binary classifier (1 or -1) based on whether the document was in "comp" newsgroup or "sci" newsgroup.<br />
- The words were represented along with their frequency.<br />
For example, let's assume for now the word "technology" is 1002th word in the vocabulary. Then, if the word "technology" appeared 10 times in 12th document (in the folder <span style="background-color: white; text-align: -webkit-auto;">comp.graphics</span>), then 12th line in my file will be:<br />
1 1002:10 ...<br />
<br />
Here is a sample line in the file:
<pre class="brush: html">
1 3798:1 9450:1 12429:1 13352:1 14155:1 15858:1 22319:1 29652:1 31220:1 34466:2 35883:1 37734:1 40188:1 40683:1
1 90:1 1078:1 2101:2 3775:1 5183:2 5195:1 5747:1 7908:1 7963:1 9748:1 11294:3 14879:1 16006:2 18188:1 19742:1 20928:1 21321:1 21935:1 23613:1 25354:2 26721:1 29652:1 30407:1 34054:1 36546:2 38252:1 39376:2 40204:1
</pre>
Now, I had to convert this to following format:<br />
fileNum | wordIndex | frequency<br />
For example:
<pre class="brush: html">
3|3798|1
3|9450|1
3|12429|1
3|13352|1
3|14155|1
</pre>
I used following VIM tricks:<br />
1. First delete the binary classifiers (and also blank spaces at the end of the line):<br />
<br />
<pre class="brush: html">
:%s/^-1//g
:%s/^1//g
:%s/ $//g
</pre>
2. Then, comes the most important part: Insert line number before every word. <br />
To do this, I used following trick:
- First replace spaces by some arbitrary character, in my case '=' <br />
- Then, replace that character by the line number !!!
<pre class="brush: html">
:%s/ / =/g
:g/=/exec "s/=/ ".line(".")."|/g"
</pre>
The last command needs some explanation: </div>
<div class="p1">
It says, execute a command globally whenever you see '=' character: <b>:g/=/exec</b> some-command</div>
<div class="p1">
In our case the command is replacing that character by the line number. The line number of a given line can be found using VIM's internal function: <b>line(".")</b></div>
<div class="p1">
Hence, the input file is converted into following format:</div>
<pre class="brush: html">
3|3798:1 3|9450:1 3|12429:1 3|13352:1 3|14155:1 3|15858:1 3|22319:1 3|29652:1 3|31220:1 3|34466:2 3|35883:1 3|37734:1 3|40188:1 3|40683:1
4|90:1 4|1078:1 4|2101:2 4|3775:1 4|5183:2 4|5195:1 4|5747:1 4|7908:1 4|7963:1 4|9748:1 4|11294:3 4|14879:1 4|16006:2 4|18188:1 4|19742:1 4|20928:1 4|21321:1 4|21935:1 4|23613:1 4|25354:2 4|26721:1 4|29652:1 4|30407:1 4|34054:1 4|36546:2 4|38252:1 4|39376:2 4|40204:1
</pre>
<div class="p1">
3. Now, to some garbage cleaning and trivial replacement:</div>
<pre class="brush: html">
:%s/:/|/g
:%s/ / /g
:%s/^ //g
</pre>
<div class="p1">
4. Finally, replace spaces by a newline character:</div>
<pre class="brush: html">
:%s/ /!!!!!!/g where !!!!!! is press Control+V and then press Enter
</pre>
<div class="p1">
<span class="s1">Control-V is the special character escape key</span>. Remember the character '\n' won't work here.</div>
<div class="p1">
<br /></div>
<div class="p1">
And voila you are done !!!</div>
</div><div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-59266911925342300842012-01-17T18:24:00.000-05:002012-01-18T22:43:40.828-05:00Iris - siri like software<div dir="ltr" style="text-align: left;" trbidi="on">
Recently I have started a pet project <u>Iris</u><sup>1</sup> while doing background study for my recent paper.<br />
<br />
<b>Voice commands/response:</b><br />
Like <a href="http://www.youtube.com/watch?v=rNsrl86inpo">Siri</a>, it takes user commands and can respond using either Voice or Text.<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="http://www.youtube.com/embed/MiwRIJDkjW4" width="420"></iframe>
<br />
<br />
<b>Portable:</b><br />
Since it is written in Java, it can run on most OS, including Mac, Windows and Linux. I have only tested it on Mac Lion, so I welcome any beta testers using other OS. For now, some of the features like play itunes, open email, etc are only available on Mac. I plan to add these features on other OS as well.<br />
<br />
<b>Configurable:</b><br />
Most of the features such as use of VOICE or TEXT as feedback mechanism, as well as feedback message are configurable through the configuration file ''user_scripts/iris.config". You can also modify any of the predefined apple scripts (for eg: playSong.scpt) in that folder which will be called by Iris on the voice cammand. Also, for advance users, there are ten function commands that he/she can use to perform user-specific tasks.<br />
<br />
<b>Easy to install and use:</b><br />
All you need to do is download and extract the <a href="http://code.google.com/p/iris-super-pet/downloads/list">binary file</a> and run following command:<br />
<pre class="brush: html">
java -Xmx200m -jar bin/Iris.jar
</pre>
The above command asks Java to run the program Iris.jar (in bin folder) and also specifies that it should not use more than 200 MB of memory.<br />
<br />
<b>Lightweight:</b><br />
The whole setup takes about 20MB.<br />
<br />
<b>Client-side:</b><br />
The speech-to-text is done on your laptop/desktop and not on some remote server. This means you can use it even if have no internet connection. This also means that there is no possibility of some big brother monitoring your voice commands.<br />
<br />
<b>Open source:</b><br />
You are free to download and modify the <a href="http://code.google.com/p/iris-super-pet/source/checkout">source</a>. As I am working towards a paper deadline, I am looking for collaborators that would help me with following features:<br />
<ol style="text-align: left;">
<li>User-friendly: Improving voice commands</li>
<li>More accuracy: Expanding dictionary to include more words and also to add different 'user accents' for a given word.</li>
<li>Expanding grammar to include arbitrary spoken words. </li>
<li>Apple scripts to accessing contacts and recognizing contact names to assist users with various functionality.</li>
<li>Adding GUI interface rather than command line. Since all the responses go through OutputUtility.java, this is going to be easy.</li>
<li>New features: Support for RSS, Top News, <a href="http://developer.yahoo.com/weather/">Weather</a>, Popular city names, Dictionary/Theasurus, Web/Wikipedia search, Dictation (to text/messages/events/<a href="http://www.toodledo.com/info/api_doc.php">todo item</a>/alarm/reminders), Jokes, Motivational quotes, More Personalization.</li>
</ol>
If you have suggestions for improvement or want to collaborate with me, email me at 'NiketanPansare AT GMail'.<br />
<br />
Iris uses CMU's <a href="http://cmusphinx.sourceforge.net/sphinx4/">Sphinx4</a> for recognizing speech and <a href="http://freetts.sourceforge.net/docs/index.php">FreeTTS</a> for text-to-speech conversion.<br />
<br />
<b>References:</b><br />
<sup>1</sup> Iris is name of my friend's dog (well, no pun intended when I use the term 'pet project'). It is also the anagram of the word 'Siri' (a popular Apple's software for IPhone 4s).</div><div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com2tag:blogger.com,1999:blog-30819994.post-38423764980034584172012-01-02T00:02:00.001-05:002012-01-02T14:47:00.503-05:00C++ workflow<div dir="ltr" style="text-align: left;" trbidi="on">
Recently I had a memory bug that took me 2 days to figure out. <a href="http://valgrind.org/">Valgrind</a> gave no error message (well, to be honest, it did give out a warning that got lost in my log messages). <a href="http://www.gnu.org/software/gdb/">gdb</a> pointed to completely random place that led me to believe STL's <a href="http://www.cplusplus.com/reference/iostream/stringstream/">stringstream</a> was the culprit. Since there was no way I could have definitively say it's not the cause, I couldn't rule out that possibility.<br />
<br />
My impatience got the best of me and I foolishly rewrote that part of code using a less elegant solution using arrays. Now gdb started pointing to STL's <a href="http://www.cplusplus.com/reference/stl/vector/clear/">vector.clear()</a> method and believe it or not I was using <a href="http://www.cplusplus.com/reference/string/string/">std::string</a> to store in it, not some user defined class …. I had it … there is no way in the world I am going to believe that gdb was right (thanks to my confidence in <a href="http://www.amazon.com/Effective-Specific-Improve-Programs-Designs/dp/0321334876">Effective C++</a> and std::vector < str::string >). But, not knowing the bug in a 'single threaded' 'command-line based' C++ code with access to gdb and pretty printing was driving me crazy. My <a href="http://niketanblog.blogspot.com/2008/04/random-c-tricks-gdb-valgrind-and.html">previous blogpost</a> didn't help either :(<br />
<br />
So, on the same evening, armed with my music player and hot cup of tea, I decided to ignore the bug and beautify my code with better comments … geek alert !!!
That's when it hit me, due to my recent modification, I was writing past the array bounds in a for loop. The funny thing is it is one of the few <a href="http://valgrind.org/docs/manual/quick-start.html">memory bugs</a> that valgrind does not report in its final assessment and my best guess is that I was writing over vector's header location (and vector only complains about that during its next operation (in my case was clear()) ... which for all intensive purposes is random).<br />
<br />
Embarrassed by this stupid mistake that wasted my two days, I felt humbled to revisit my C++ work-flow.
<br /> <br />
So, here is a sample workflow: <br />
- <b> svn update</b> <br />
- Code using your favorite editor (Make sure you configure it to conform to your style rules). <br />
- Set DEBUGFLAGS in your makefile that does static analysis (<b>make my_proj</b>). <br />
- Test it on local machines using '<b>make my_test</b>' <br />
- Run valgrind if necessary and check against below given valgrind checklist <br />
- <b>make style</b> <br />
- <b>make doc</b> <br />
- <b>svn update</b> <br />
- <b>svn commit -m "some meaningful message"</b> <br />
<br />
The key principle is:
<h2>
Automate whenever possible (with emphasis on reusability)</h2>
<ol style="text-align: left;">
<li>Never ever start coding without an <a href="http://en.wikipedia.org/wiki/Apache_Subversion">SVN</a> (or cvs or git, etc.). It will save you lot of headaches if your machine crashes or if you want to revert back to a previous version.
<br />
If the project has multiple coders/testers/collaborators, it helps if you have some kind of project-management tool like <a href="http://niketanblog.blogspot.com/2009/10/how-to-set-up-trac-svn-on-ubuntu.html">trac</a> or <a href="http://www.atlassian.com/software/jira/overview">jira</a>. Also, if you are going to release your code to public, you might want to consider project hosting sites like <a href="http://en.wikipedia.org/wiki/Google_Code">Google Code</a> or <a href="http://en.wikipedia.org/wiki/SourceForge">source forge</a>.
<br />
Though this might seem obvious, don't host the svn server on personal machine or at least make sure to take regular backups if you do.</li>
<li>Use a Makefile that takes care of all necessary tasks (compilation, cleaning, data generation, document generation, code beautification, etc) and a user-friendly README file. (Advance coders can consider <a href="http://www.gnu.org/software/automake/">automate</a>). Here is a sample Makefile of my current project:
<pre class="brush: html">
CC = g++
DEBUGFLAGS = -g -Wall -Wextra
MINUITFLAGS = `root-config --cflags` `root-config --libs` -lMinuit2 -fopenmp
LIBS = -lgsl -lgslcblas -lm -lstdc++
HEADER = Helper.h MathHelper.h LDA.h Configuration.h
SRC = main.cpp MathHelper.cpp Helper.cpp LDA.cpp
OUTPUT = spoken_web
spoken_web: $(SRC) $(HEADER)
$(CC) $(SRC) -o $(OUTPUT) $(DEBUGFLAGS) $(LIBS)
doc: $(HEADER) spokenweb_doxygen_configuration
doxygen spokenweb_doxygen_configuration
r_data: GenerateData.R
R --slave < GenerateData.R
style: $(SRC) $(HEADER)
astyle --options=spokenweb_astyle_rules $(SRC) $(HEADER)
clean:
rm -f *.o $(OUTPUT) *~ *.data parameters.txt metaFile.txt *.data
</pre>
</li>
<li>Use a document generator like <a href="http://www.stack.nl/~dimitri/doxygen/">doxygen</a>. It forces you to write much cleaner documentation and more importantly cleaner interface. Trust me it will help if you had to reuse your code in future.
<pre class="brush: html">make doc
</pre>
</li>
<li>Use code beautifier like <a href="http://astyle.sourceforge.net/">astyle</a> that not only shields you against inelegant colleague/coder, but also personalizes the code if you like to do '<a href="http://www.amazon.com/Code-Reading-Open-Source-Perspective/dp/0201799405">Code Reading</a>' (Don't you clean gym equipment after workout, well, why not do the same with your code). Here is sample style rules that I love:
<pre class="brush: html">
--style=java
--indent=spaces=2
--indent-classes
--indent-switches
--indent-preprocessor
--indent-col1-comments
--min-conditional-indent=0
--max-instatement-indent=80
--unpad-paren
--break-closing-brackets
--add-brackets
--keep-one-line-blocks
--keep-one-line-statements
--convert-tabs
--align-pointer=type
--align-reference=type
--lineend=linux
--suffix=none
</pre>
If your company follows other style guidelines or if you don't want to change existing formatting, remove '--suffix=none' from the style rules and once you are done, you can restore the original unformatted files (*.orig).
<pre class="brush: html">make style
</pre>
</li>
<li>Familiarize yourself with program analyzers and debugging tools, and have scripts that automate them in your coding process:
<ul style="text-align: left;">
<li>g++'s flags (See DEBUGFLAGS in my makefile)</li>
<li>Static analysis tools like cppcheck (For more cases, I am satisfied with the above g++'s flags)</li>
<li>Runtime memory analyzer tool like <a href="http://valgrind.org/">valgrind</a>. (Please see below given Valgrind checklist, and also my <a href="http://niketanblog.blogspot.com/2008/04/random-c-tricks-gdb-valgrind-and.html">previous blogpost</a>).</li>
<li>gdb: For stepping through the code for more comprehensive debugging</li>
</ul>
There are several paid tools that do much better job of analyzing your code that those mentioned above. There are </li>
<li>I like to follow standard directory structure:<br />
./ Makefile and configure scripts<br />
./src cc , h files (Use subdirectories and namespaces if necessary)<br />
./bin build directory <br />
./data data required for testing/running the code <br />
./test test drivers <br />
./doc documentation generated by doxygen</li>
<li>Maintain a list of code snippets (stored in Evernote or a personal notebook) or helper classes that you might have to use again. Note that the helper classes should have good interface and must be well documented (and robust). For example, I use <a href="http://www.gnu.org/software/gsl/">GSL</a> for writing sampling functions for bayesian programs, which has very ugly interface and requires some amount of book-keeping. So, I have coded a wrapper class that gives an R like interface which makes my code extremely readable, less error prone and I don't need to worry about any book-keeping or other numerical issues while calling them.
<pre class="brush: c++">
/// \defgroup StatDistribution R-like API for family of probability distributions
/// Naming convention:
/// I have followed the naming conventions of R programming language.
/// Please see its documentation for further details
///
/// Here is a brief intro:
/// Every distribution that R handles has four functions. There is a root name,
/// for example, the root name for the normal distribution is norm.
/// This root is prefixed by one of the letters
/// - d for "density", the density function
/// - r for "random", a random variable having the specified distribution
/// Example: For the normal distribution, these functions are dnorm, and rnorm.
///
/// Important note: Since most bayesian calculation is done in log space, (almost)
/// every density function is capable of outputing 'numerically robust' logarithmic value.
/// Hence, use dnorm(x, mean, sd, true) in your code rather than log(dnorm(x, mean, sd))
///
/// Suported continuous distribution:
/// - Uniform (unif - r,d)
/// - Univariate Normal (norm - r, d); Truncated normal (tnorm - r,d);
/// - Multivariate Normal (mnorm - r, d, conditional_r, conditional_rt) (--> Only m = 2, 3 supported)
/// - Beta (beta - r,d)
/// - Inverse Gamma (invgamma - r,d)
///
/// Suported discrete distribution:
/// - Discrete Uniform (dunif - r,d)
/// - Dirichlet (dirichlet - r,d)
/// - Multinomial (multinom - r,d)
/// - Poisson (pois - r,d)
/// @{
/// @name Continuous Uniform Distribution
///@{
/// Returns uniform variable [min, max)
double runif(double min, double max);
double dunif(double x, double min, double max, bool returnLog = false);
///@}
/// @name Discrete Uniform Distribution
///@{
/// Returns uniform variable [min, max) --> i.e. [min, max-1]
int rdunif(int min, int max);
double ddunif(int x, int min, int max, bool returnLog = false);
///@}
/// @name Univariate Normal Distribution
///@{
double rnorm(double mean, double sd);
double dnorm(double x, double mean, double sd, bool returnLog = false);
///@}
/// @name Multinomial distribution
/// Both prob and retVal should be of length sizeOfProb
/// size, say N, specifying the total number of objects that are put into
/// K (or sizeOfProb) boxes in the typical multinomial experiment.
/// If the array prob is not normalized then its entries will be treated as
/// weights and normalized appropriately. Furthermore, this function also allows
/// you to provide log probabilities (in which case, you need to set isProbInLogSpace to true)
/// Eg: double prob[3] = {0.1, 0.2, 0.7}; double retVal[3]; rmultinom(1, 3, prob, retVal, false);
///@{
void rmultinom(int size, int sizeOfProb, double* prob, unsigned int* retVal, bool isProbInLogSpace = false);
double dmultinom(int sizeOfProb, double* prob, unsigned int* x, bool isProbInLogSpace = false, bool returnLog = false);
///@}
</pre>
So to sample N(10,2), instead of having code like:
<pre class="brush: c++">
gsl_rng* rng_obj = gsl_rng_alloc(gsl_rng_mt19937);
double sampledVal = gsl_ran_gaussian(rng_obj, 2) + 10;
gsl_rng_free(rng_obj);
</pre>
I have to use following code (which if you have coded in R is more readable):
<pre class="brush: c++">
StatisticsHelper myStatHelper;
double sampledVal = myStatHelper.rnorm(10, 2);</pre>
</li>
</ol>
<ol style="text-align: left;">
</ol>
<h2>
Valgrind checklist:</h2>
First run valgrind's memcheck tool (<span class="Apple-style-span" style="color: #333333; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18px;">valgrind --tool=memcheck program_name</span>) and check:<br />
<ol style="text-align: left;">
<li>The heap summary (only returns memory leaks, not memory corruption errors) Eg:
<pre class="brush: c++">
==19691== HEAP SUMMARY:
==19691== in use at exit: 0 bytes in 0 blocks
==19691== total heap usage: 12,126 allocs, 12,126 frees, 938,522 bytes allocated
==19691==
==19691== All heap blocks were freed -- no leaks are possible
</pre>
If number of allocs and the number of frees will differ in o/p, the try <br />
<span class="Apple-style-span" style="color: #333333; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 18px;">valgrind --tool=memcheck --leak-check=yes --show-reachable=yes program_name</span> <br /> <br />
Steps to avoid memory leaks: <br />
- If you are assigning pointers to other pointers, be very explicit as to who is responsible for the deletion of the memory.<br />
- Delete every object created by new (especially in case of exceptions).
</li>
<li>Intermediate warning/error messages irrespective of the heap summary. Eg:
<pre class="brush: c++">
==15633== Invalid write of size 4
==15633== at 0x4163F3: LDAGibbsSampler::FillInSpeechToTextOutputs() (LDA.cpp:206)
==15633== by 0x41565A: LDAGibbsSampler::LDAGibbsSampler(int, int, double, double, std::string, double, double, int) (LDA.cpp:28)
==15633== by 0x402E26: main (main.cpp:34)
</pre>
Note, sometime one error can propagate and throw error messages at different places, so don't panic. <br /> <br />
The intermediate error messages can detect:
<ul style="text-align: left;">
<li><b>Illegal read / Illegal write errors</b> (like the message '<i>Invalid read of size 4</i>' above): It can mean any of the memory corruptions given below.
If the error message is not informative, run valgrind again with '--read-var-info=yes' flag.
<br />
Common memory corruption bugs like these are: <br />
- Writing/Accessing out of bounds memory (either directly or by array oriented functions like strcpy) <br />
- Using an unallocated object<br />
<pre class="brush: c++">
my_class* obj_ptr;
obj_ptr -> var1 = val; // or val = obj_ptr -> var1;
</pre>
- Using a freed object <br />
Note: <a href="http://valgrind.org/docs/manual/mc-manual.html">Memcheck</a> (tool of valgrind) does not perform bounds checking for stack or global arrays, so even if you don't get this error/warning message, it does not mean that there is no memory corruption errors.</li>
<li> <b>Use of uninitialised values</b> (either on stack or heap) in <b>user function</b>: Message looks like '<i>Conditional jump or move depends on uninitialised value(s)</i>'. To see information on the sources of uninitialised data in your program, use the '--track-origins=yes' option.</li>
<li><b>Use of uninitialised values</b> in <b>system calls</b>: Message looks like '<i>Syscall param write(buf) points to uninitialised byte(s)</i>'.</li>
<li><b>Illegal free/deletions:</b> (<i>Invalid free()</i>) <br />
- Calling free() (or delete) on the object that is already freed (or deleted). <br />
- Deleting object that is created on stack.
</li>
<li> <b>Mismatched operators</b>: (<i>Mismatched free() / delete / delete []</i>) <br />
- Calling free() on the object that is created by new operator. <br />
- Calling delete on the object that is created by malloc(). <br />
- Calling delete[] on the object that is created by the new operator. <br />
</li>
<li> <b>Overlapping source and destination blocks</b>: (Source and destination overlap in ...) <br />
- Look out functions like memcpy, strcpy, strncpy, strcat and strncat.
</li>
</ul>
</li>
</ol>
<br />
References:<br />
- http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml <br />
- http://www.yolinux.com/TUTORIALS/C++MemoryCorruptionAndMemoryLeaks.html <br />
- http://www.cprogramming.com/tutorial/memory_debugging_parallel_inspector.html <br />
<script type="text/javascript">
SyntaxHighlighter.all()
</script>
</div><div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-46643362181477688492011-10-11T14:59:00.001-04:002011-10-11T20:54:34.685-04:00How to analyze algorithms ?<div dir="ltr" style="text-align: left;" trbidi="on">
<h2>Important assumptions:</h2>
<ul style="text-align: left;">
<li> This process of analysis should be independent of programming language and machine architecture. So you can't just write your algorithm in C++ and compare it using running time of some other algorithm written in Java. </li>
<li>Also, since each algorithm can take as an input arbitrary data structure (eg: graph, array, binary object, tree, ...), there has to be some standard way to specify it. For sake of our analysis, we use input size (n) as the measure for our analysis. For eg: length of array, number of vertices or edges of the graph, number of nodes of a tree etc. </li>
</ul>
The detailed explanations of these two points is given in<a href="http://www.amazon.com/Introduction-Algorithms-Thomas-H-Cormen/dp/0262033844/ref=dp_ob_title_bk"> Introduction to Algorithms by Cormen, Leiserson, Rivest</a>, .. (CLR) <br />
<h2>Setup:</h2>
Input of size n ==> Algorithm that takes T(n) steps ==> some output <br /> <br />
The number of steps in the algorithm often depends on factors other than the input size. For eg: the number of steps for quicksort for the input [1, 2, 3, 4, 5] is different than as compared to the input [5, 4, 3, 2, 1]; even though both have same input size (and in this case same elements).<br />
Hence, we will try to find 3 functions (upper bound, lower bound and Theta bound) that can capture the effects of these factors:
$$ g_1(n), g_2(n) \text{ and } g_3(n) \text{ such that } $$
$$ T(n) \in O(g_1(n)), T(n) \in \Omega(g_2(n)) \text{ and } T(n) \in \Theta(g_3(n))$$
Many authors also write above equations as:
$$ T(n) = O(g_1(n)), T(n) = \Omega(g_2(n)) \text{ and } T(n) = \Theta(g_3(n))$$
<h2> Big-O notations:</h2>
Now, let's understand definition of above notations:
\begin{align}
\text{To prove that } & g_1(n) \text{ is upper bound of T(n), i.e.} T(n) \in O(g_1(n)) \\
\text{ find two constants } & c, n_0 \text{ such that } \\
& \forall n \geq n_0, T(n) \leq c g_1(n)
\end{align}
For lower bound, change ≤ sign to ≥ sign and for theta bound find both lower and upper bound:
$$ T(n) \in \Theta(g_3(n)) \text{ iff } T(n) \in O(g_3(n)) \text{ and } T(n) \in \Omega(g_3(n))$$
<br /> To understand these in more details, I suggest you solve problems from CLR chapter 3. Here are some example you can start with:
\begin{align}
2n^2 \in O(n^3) & \text{ with } c=1 \text{ and } n_0 = 2 \\
\sqrt{n} \in \Omega(lg n) & \text{ with } c=1 \text{ and } n_0 = 16 \\
\frac{n^2}{2} - 2n \in \Theta(n^2) & \text{ with } c_1=\frac{1}{4}, c_2=\frac{1}{2} \text{ and } n_0 = 8 \\
\end{align}
(Note: theoretically very large functions such as infinity^n or n^infinity can always be lower bounds. So, it is necessary to supply as tight bound as possible for any given algorithm. Mathematically, the notations for tight upper bound is <a href="http://en.wikipedia.org/wiki/Big_O_notation#Little-o_notation">small o</a> and for tight lower bound is small omega, but we will ignore them for sake of this discussion) <br />
<br /><h2>Common measures for algorithms:</h2>
Above three notations are often replaced by the terms 'worst case', 'best case' and 'average case' complexity. But, more precise definitions are as follows:
\begin{align}
\text{Worst case}& T(n) = max \{ T(x) | \text{ x is an instance of size n} \} \\
\text{Best case}& T(n) = min \{ T(x) | \text{x is an instance of size n} \} \\
\text{Average case}& T(n) = \sum_{\text{input }x} Prob(x) T(x)
\end{align}
This means, to find average case complexity of algorithm, you need to find probability of a given input and use it to find <a href="http://en.wikipedia.org/wiki/Expected_value">expectation</a> of T(n). <br /> <br />
<h2>Pseudo code to Big-O:</h2>
Now, let's address the key question: given a pseudo code, how do we find its lower bound (or upper bound or ...). Here is one possible way to find those bounds:
<ul style="text-align: left;">
<li> First, express the algorithm in recursive form </li>
<li> Determine appropriate metric for input size 'n' </li>
<li> Then, find recurrence relation for the algorithm </li>
<li> Solve the recurrence relation using either Substitution method, recursion tree method or the master theorem </li>
</ul>
<br />For example, the recurrence relations for popular algorithms are:
\begin{align}
\text{Binary search: } & T(n) & = T(\frac{n}{2}) + O(1) \\
\text{Sequential search: } & T(n) & = T(n-1) + O(1) \\
\text{Tree traversal: } & T(n) & = 2T(\frac{n}{2}) + O(1) \\
\text{Selection sort: } & T(n) & = T(n-1) + O(n) \\
\text{Merge sort: } & T(n) & = 2T(\frac{n}{2}) + O(n)
\end{align}
Try writing the pseudo code for any of the above algorithm and verify the recurrence relations. <br />
<br /><h2>Master theorem:</h2>
Master theorem can only solve the recurrence relation of the form: <b>T(n) = a T(n/b) + f(n)</b>.
Here is my modus operandi for master theorem:
\begin{align}
\text{If } f(n)=n^{log_{b}a}lg^{k+1}n, & \text{ then } T(n) = \Theta(n^{log_{b}a} lg^{k+1}n) \\
\text{Else if } n^{log_{b}a} \text{ is polynomially greater than } f(n), & \text{ then } T(n) = \Theta(n^{log_{b}a}) \\
\text{Else if } f(n) \text{ is polynomially greater than } n^{log_{b}a}, & \text{ then } T(n) = \Theta(f(n)) \\
\text{Else use Substitution method}&
\end{align}
<br />Here are solutions to two important recurrence relations, that cannot be solved using Master theorem:
\begin{align}
\text{Recurrence relation} & & \text{Complexity} \\
T(n) = T(n-1) + b n^k & => & O(n^{k+1})\\
T(n) = cT(n-1) + b n^k & => & O(c^n)
\end{align}
<br />For recurrence relation of other forms, refer to CLR for substitution and recursion tree method. <br />
(In short, for substitution method, you guess the solution and use induction to find the constants and to prove that the guessed solution is correct. <br />
Recursion tree method are often used to find the guess for the substitution method)
<br /><br /><h2>Some useful approximations:</h2>
For non-negative real b,c,d: <br />
\begin{align}
\text{Polynomial growth rate:}&
\sum_{i=1}^n i^c &= \Theta(n^{c+1}) \\
\text{Logarithmic growth rate:}&
\sum_{i=1}^n \dfrac{1}{i} &= \Theta(log(n)) \\
\text{Exponential growth rate:}&
\sum_{i=1}^n c^i &= \Theta(c^n) \\
\text{n-Log(n) growth rate:}&
\sum_{i=1}^n log(i)^c &= \Theta(n log(n)^c) \\
\text{Combination of above:}&
\sum_{i=1}^n log(i)^c i^{d} &= \Theta(n^{d+1} log(n)^c) \\
&\sum_{i=1}^n log(i)^c i^d b^{i} &= \Theta(n^{d+1} log(n)^c b^{n}) \\
\text{Stirling's approximation:}&
n! &= \sqrt(2 \pi n) (\frac{n}{e})^n (1 + \Theta(\frac{1}{n}))
\end{align}
</div><div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-63079073206865328582011-09-29T23:26:00.003-04:002011-10-11T17:31:56.519-04:00Presentation on Online Aggregation for Large MapReduce Jobs<div dir="ltr" style="text-align: left;" trbidi="on">
This post has two purposes: <br />
1. Give information about my Comp 600 presentation at Rice University <br />
2. Enumerate the lesson's learnt while preparing and giving the presentation.
<br />
<div dir="ltr" style="text-align: left;" trbidi="on">
<iframe allowfullscreen="" frameborder="0" height="300" src="http://player.vimeo.com/video/29806431?title=0&byline=0&portrait=0" webkitallowfullscreen="" width="400"></iframe>
<br />
(If the above videos does not work, click on: <a href="http://www.dailymotion.com/video/xle1r8_online-aggregation-for-large-mapreduce-jobs_tech">Dailymotion</a> or <a href="http://vimeo.com/29806431">Vimeo</a>)<br />
<br />
Also, if you have time, go through my slides or better still read my paper :)<br />
<a href="http://prezi.com/4u9xyqzhc7qy/online-aggregation-for-large-mapreduce-jobs/">- Slides </a><br />
<a href="http://www.cs.rice.edu/%7Enp6/Papers/OnlineAggregationForLargeMapReduceJobs.pdf">- Paper</a> <br />
<br />
<hr />
<h2>
FAQs during the presentation at VLDB: </h2>
<br />
1. What if the blocks are equally sized and spread across uniformly ?<br />
- We used exactly that configuration. The input blocks of wikipedia logs are approximately of same size and HDFS does good job of spreading the data across uniformly. Still, we found biased results in the beginning.<br />
<br />
2. Does your system replace general database ?<br />
- Nope. As of now, it only performs single table aggregation with group-by. For a general online database, see <a href="http://dl.acm.org/citation.cfm?id=1412335">DBO paper</a> by our group.<br />
<br />
3. Will you release your code ?<br />
- Yes, we intend to release our code as open-source. For current version, see http://code.google.com/p/hyracks-online-aggregation/<br />
<br />
4. Who do you see as potential user of the system ?<br />
- Any company that has massive data stored in HDFS (logically as single table) and want to perform aggregation in online fashion or only till certain accuracy. Note, that the table containing the data need not be pre-randomized.<br />
<br />
5. What does user of the system needs to know ?<br />
- Just how to write and run a hadoop job ... All he/she has to do is write a hadoop mapper and set appropriate parameters in job configuration. For details see appendix of our paper.<br />
<br />
6. You didn't explain too much statistics in the talk ... I want to know more <br />
- Yes, you are absolutely right. Infact, that was intentional :) ... If I have to give one-line summary of statistics here is it: Sample from a Multi-variate normal on value, processing time and scheduling time (which have correlation factor associated with between each of them) which takes in account the 'lower bound' information of processing time of the blocks that are currently been processed ... rather than simply sampling from the distribution over data.<br />
<br />
7. You don't talk about performance ... Isn't randomized scheduling policy bad for performance ?<br />
- We didnt talk about the performance, mainly because we didnot provide any comparative study for performance in our paper. Note that, our policy is not same as a strict randomized policy. We expect the master to maintain a randomize queue and start the timer when a worker asks for the block. If for some reason (say locality), worker doesnot want that block or master has some internal (locality-based) scheduling algorithm, he can leave the timer ON and go for next block in the randomized queue. This is necessary machinery for sampling theory. This will yield performance that is similar to locality-based scheduling. However, in our current implementation, we have not used this mechanism. We intend to do the performance study in our future work.<br />
<br />
<hr />
<h2>
Presentation tips: </h2>
Clearly my presentation has lot of room for improvement, for example: need to talk slower, more uniform layout of the slides, more interaction with audience etc. However, I have learnt a great deal while preparing and giving this presentation. I will summarize few points that I (and probably lot of graduate students) need to be careful about. For more detailed analysis of these points, I suggest you read <a href="http://www.amazon.com/Presentation-Zen-Simple-Design-Delivery/dp/0321525655">Presentation Zen</a>, <a href="http://www.amazon.com/slide-ology-Science-Creating-Presentations/dp/0596522347/ref=pd_bxgy_b_text_b">Slide:ology</a> and <a href="http://www.amazon.com/When-Scientist-Presents-Audio-Science/dp/9812839208/ref=sr_1_1?s=books&ie=UTF8&qid=1317502367&sr=1-1">When the scientist present</a>. <br />
<br />
<u><b>Step 1: Planning your presentation</b></u><br />
1. Create a relaxing environment<br />
- Go to a library, coffee place, garden (place where no one will disturb you)<br />
- Take time to relax and forget about daily chores.<br />
- Don't bring your laptop. In this step, you will create rough sketch of the presentation on paper (not slideware)<br />
- Lot of the time, changing locations helps you to be more creative.<br />
<br />
2. Make sure you understand the big picture of your presentation.<br />
Can you find 30-45 seconds answer to following questions:<br />
- What is the single most important "key" point that I want my audience to remember<br />
- Why should that point matter to the audience<br />
<br />
Also, make sure that your presentation title has keywords that highlights the big picture (as many people (if not all) will come to your talk either reading just the title (and abstract) or your name). This does not mean the title has to be too long; in fact shorter and thought-provoking title are preferred.<br />
<br />
3. Now, create a sketch of the presentation. This involves enumerating key ideas you will be presenting and also organising/ordering them. You can use either or all of the following tools:<br />
- Mindmap on whiteboard or paper<br />
- Post-it (use sharpie to make sure you don't misuse post-it)<br />
- Brainstorm the list of ideas + Grouping them<br />
- Sketching or drawing <br />
Nancy Duarte says most effective presentations involves telling a story that audience can connect and relate to, rather than just present the information/data. Also, best professors are good story-tellers. They just don't go through the material in the book, but put their own personality, character and experience into the material in form of a narrative, which is illuminating, engaging and memorable.<br />
<br />
The most useful point I have learnt from Presentation Zen is that the presentation consists of three parts:<br />
- <b>Slides</b> the audience will see (which you will create in Step 2)</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
- <b>Notes</b> only you will see (which you have created in this Step)</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
- <b>Handouts</b> that audience can take away (so that they don't have to jot down key definition or formula). This in worst case could simply be a copy of your paper.</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
This essentially means that the slides are not telepromptor, that you read aloud throughout your presentation.</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
</div>
<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<u><b>Step 2: Design the slides</b></u><br />
To succeed as a presenter, you need to think as a designer (from slide:logy).<br />
<br />
1. First few slides are very very important !!! <br />
- I like to give use them to tell the audience the big picture of the presentation.<br />
- It is also necessary that this has to be catch their attention, as we all know the attention-span of audience decreases exponentially after first few minutes. Don't use the first few minutes reading through the 'Outline' slide.<br />
- Use following principle: Tell them what you are going to present, present it, then tell them again what you presented :)<br />
<br />
2. Slides should be as simple as possible.<br />
- Don't use fancy, distracting background images. I prefer sticking to simple white background slide.<br />
- Good design has plenty of empty spaces.<br />
- Avoid typical bullet point presentations. They are boring and they do more harm than good.<br />
- Don't use 3D figures when 2D figures can suffice<br />
- Never ever clutter the slides with useless data, words, figures, images, etc<br />
<br />
3. 1 message per slide ... not more !!<br />
- Do you understand that message ?<br />
- Does your slide reflect that message ? You don't have to write down the message into your slide<br />
- Can I make this slide simpler and still make sure that the message is conveyed. In most cases, this is done by replace bunch of text either with an appropriate image or animation.<br />
- A lot of the time, entire message can be conveyed just by a quote from a reliable or important person. <br />
<br />
4. Try to find a good figure or animation to convey a complex idea, algorithm or system. Here are some tips on creating animation:<br />
<span style="font-size: x-small;">- Using same slide background, create many slides (by copy operation). Change only 1 detail per slide. When you keep moving from one slide to next, it would give an effect of animation, something like flip-book. However, this process creates a lot of slides (which is problematic in many cases). So, if you want to fit everything into 1 slide, you can create a flash object, gif file or avi file and embed it into the slide. Most people use Adobe tools to create professional looking flash files, but here is poor man's solution:<br />- Use flip-book style slide transition technique and desktop recorder (eg: recordmydesktop) to create a movie. It usually creates movie in ogv file format. <br />- Also, lot of time you need to create animation using statistical distribution. In that case, use animation library of R and create ogv movie using desktop recorder. (This library also allows you store the animation in gif, but I am having some problems with it).</span><br />
<blockquote>
<span style="font-size: x-small;"># R code for animation<br />library(animation)<br />ani.options(interval = 1, nmax = 20)<br />sd1 <- 41<br />for (i in 1:ani.options("nmax")) {<br /> x=seq(900, 1100,length=200)<br /> y=dnorm(x,mean=1000,sd=sd1)<br /> plot(x,y,type="l",lwd=2,col="red")<br /> ani.pause() ## pause for a while ('interval')<br /> sd1 <- sd1 - 2<br />}</span></blockquote>
<span style="font-size: x-small;">a. Convert OGV to AVI:<br />mencoder input.ogv -ovc xvid -oac mp3lame -xvidencopts pass=1 -speed 0.5 -o output.avi<br />b. Convert OGV to GIF (or swf):<br />ffmpeg -i input.ogv -loop_output 0 -pix_fmt rgb24 -r 10 -s 320x240 output.gif<br />"-loop_output 0" means loop forever.</span><br />
<br />
5. Is the content tailored to meet audience' expectation:<br />
- Employers/future collaborators will try to evaluate you rather than the topic.<br />
- Students (undergrad/first year grads/ ..) or people from different area want to understand the problem and intuition behind the solution. Hence, they are more interested in motivation section.<br />
- Co-author will look for recognition.<br />
- Sponsor will look for acknowledgement.<br />
- People who work in same area will try to see if how your system/solution compares to theirs (i.e. related work) or may be are there to look for a new problem (i.e. your future work). <br />
<br />
6. Other design tips:<br />
- Use contrast (using font type, font color, font size, shape, color, images) accentuate the difference. (If it is different, make it very different - Garr Reynolds). It helps audience to identify the message of the slide quickly.<br />
- Repetition: Use same image (or anchor) to represent the same concept throughout the presentation.<br />
- Alignment: Use grids, invisible lines. Audience needs to know the order in which to process the information.<br />
- Proximity<br />
- Hierarchy, Unity<br />
- Use high quality graphics. There are websites that create and sell them like <a href="http://www.istockphoto.com/">www.istockphoto.com</a>; or even give them for free like <a href="http://www.flickr.com/creativecommons">http://www.flickr.com/creativecommons</a>, etc.<br />
- Nothing distracts the audience more than a cluttered slide.<br />
- Make sure your font size is greater than 24 point (so that audience can read it from the projector)<br />
- Use comics ... it not only conveys the information efficiently, but also makes the presentation playful. Though many of my colleagues disagree with this, I believe most audience wants to have fun even if it is a research presentation .. so lighten the mood, be creative and "artsy" with your slides :)<br />
<br />
"The more strikingly visual your presentation is, the more people will remember it. And more importantly they will remember you" - Paul Arden.<br />
Here are some slides that are well designed:<br />
<a href="http://www.slideshare.net/jbrenman">http://www.slideshare.net/jbrenman</a><br />
<a href="http://www.slideshare.net/chrislandry">http://www.slideshare.net/chrislandry</a><br />
<a href="http://www.slideshare.net/gkawasaki">http://www.slideshare.net/gkawasaki</a><br />
<a href="http://www.slideshare.net/garr">http://www.slideshare.net/garr</a><br />
<br />
<u><b>Step 3: Rehearse and re-rehearse</b></u><br />
- Use both notes and slides.<br />
- Rehearse until you can remember and are comfortable with them. Imagine the worst case scenario that you lost your laptop, slides and all your notes ... and you have to give your presentation !!! ... If you can do that, you are ready :)<br />
- Here you also do the job of editing, if you feel certain slides are just fillers and can be removed ... be brutal and delete them.<br />
- Listen to <a href="http://www.ted.com/talks">http://www.ted.com/talks</a> for motivation on how to present.<br />
<br />
<u><b>Step 4: Delivery</b></u><br />
- Go to the presentation room 15 min before the scheduled time and make sure your slides are compatible, projector works, ... Also, bring a bottle of water, just in case.<br />
- Leave the lights on and face the audience. People are there to see you, not read through the slides. Use wireless presenter so that you don't need to hide behind the podium/monitor.<br />
- Slides are not telepromptor ... so don't keep staring at the monitor ... connect with the audience.<br />
<br />
Remember these two quotes:<br />
- Every presentation has potential to be great; every presentation is high stakes; and every audience deserves the absolute best - Nancy Duarte.<br />
- You are presenting yourself rather than the topic. So, negative impact of poor presentation far exceeds failure of getting your point across ... Jean-Luc Lebrun</div>
</div><div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0tag:blogger.com,1999:blog-30819994.post-49141253209916941042011-08-15T01:18:00.002-04:002011-08-15T01:24:00.852-04:00Thinking in Probability<div>This blogpost is based on a lecture I gave at Jacob Sir's classes yesterday. The intent of my lecture was to urge student to ask questions and read "stuff" beyond the textbook. To get the interest of the students, I started with 2 examples, which explained the dependent and independent events and also the need to stick to mathematical rigor rather than intuition.</div><div>
<br /></div><div>First example was classic Monte Hall problem, which suprisingly none of my students had heard about. This was fortunate because this meant everyone would have to think about it on their own, rather than provide a prepared answer (thought-through by someone else). So, here is the problem: </div><div>
<br /></div><div>There are three doors D1, D2 and D3; behind two of them there is a goat and behind the other is a car. The objective of the game is to win a car. So, you have to guess which door to chose ... Does it make difference whether you chose D1, D2 or D3 ? ... The consensus was NO, because the Prob(win)=0.33 for each of the door. To make the problem interesting the game show host (who know which door has car and which doors has goats), opens one of the remaining door that has goat. He now asks you based on this information whether you would like to switch your earlier choice. For example, earlier you chose D1 and the game show host opens D2 and shows that it has goat. Now you can either stick to D1 or switch to D3. The real question we are interested in is:</div><div>Does switching the door make more sense or staying with the same door ? or It does not matter whether you chose D1 or D3.</div><div>
<br /></div><div>Interesting but incorrect answers:</div><div>1. It does not matter whether you chose D1 or D3, because Prob(win) for each door is now 0.5</div><div>2. Staying with the door makes more sense, since we have increased the Prob(win) from 0.3 to 0.5</div><div>3. We really don't have plausible reason to switch, hence stay with the same door; especially since the game show host may want to trick us.</div><div>
<br /></div><div>The correct answer is switching doubles the Prob(win), hence it makes more sense.</div><div>Consider the case where you don't switch:</div><div>Prob(win) = Prob(D1 = car) = 0.33 (The probability does not change because of an event outside its scope)</div><div>Now consider the case where you switch:</div><div>Prob(win) = Prob(D1 = goat) = 0.66</div><div>
<br /></div><div>The other way to understand the problem is by enumerating</div><div>D1 : C G G</div><div>D2 : G C G</div><div>D3 : G G C</div><div>Swap: G C C => Prob(car) = 0.66</div><div>Stay: C G G => Prob(car) = 0.33</div><div>
<br /></div><div>For people who chose the third incorrect answer, I asked them not to use information not provided in the problem and show as much restrain as possible to use intuition over logic. Deductive thinking says go from Step 2 to Step 3 only if there is a valid theorem, axiom, or logical reason (not intuition).</div><div>
<br /></div><div>Let's move to the second problem: </div><div>Say I have a fair coin. I toss the coin four times and I get {Head, Head, Head, Head}, what will you bet on ?</div><div>Some students earlier based their answer on gambler's fallacy, stating that Tail is more likely since they have seen four consecutive heads. I then asked them to sit in circle and discuss with each other and then answer my question again. Through this exercise, I wanted them to learn the idea of collaboration to get an answer. And vola ... they came with the correct answer ... It doesnot matter whether we bet on head or tail, since the events are INDEPENDENT and hence each outcome is equally likely => Prob(T5 | H1, H2, H3, H4) = Prob(T5) = 0.5</div><div>
<br /></div><div>Finally, I gave them two different explanation of probability:</div><div>1. Deterministic view for probability (explained in <a href="http://niketanblog.blogspot.com/2009/11/god-does-not-play-dice.html">my earlier blogpost</a>)</div><div>2. <a href="http://en.wikipedia.org/wiki/Measure_(mathematics)">Measure theoretic approach</a></div><div>
<br /></div><div>After explaining the deterministic view, I emphasized the importance of probability while solving real life problem. For example, in coin tossing experiment, you are able to make some rational predictions, even though you don't know (hidden) parameters like initial angle of coin, weight and density of coin, shape of the coin, velocity of the coin, resistance due to air, characteristics of surface where it lands, etc ... Similarly, you can build simpler probabilistic model for complex scenarios like weather forecasting, stock prediction, traffic congestion, ...</div><div>
<br /></div><div>Reference:</div><div>1. http://niketanblog.blogspot.com/2009/11/god-does-not-play-dice.html</div><div>2. http://en.wikipedia.org/wiki/Monty_Hall_problem</div><div>3. http://en.wikipedia.org/wiki/Gambler's_fallacy</div><div>4. http://en.wikipedia.org/wiki/Measure_(mathematics)</div><div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com1tag:blogger.com,1999:blog-30819994.post-12111595708700539312011-03-15T17:31:00.003-04:002011-03-15T20:20:56.555-04:00Nice post about Sachin Tendular after India-SA matchRemember when you failed an examination. How many people recall that, your class, friends, relatives? You failed to make it to the IITs or IIMs. Who remembers. How many times have you had the feeling of being the best in your class, school , university, state….., you failed to get a visa stamped this quarter…, you missed a promotion this year…, how did it feel when you dad told you in your early twenties that you are good for nothing…..and now your boss tell you the same...<br /><br />You keep introspecting and go into a shell when people most of whom don’t matter a dime in your life criticize you, back bite you, make fun of you. You are left sad and shattered and you cry when your own kin scoffs at you. You say I am feeling low today. It takes a lot from us to come out of these everyday situations and move on. A lot??? really?<br /><br />Now here’s a man standing on the third man boundary in the last over of a world cup match. The bowler just has to bowl sensibly to win this game. What the man at the boundary sees is 4 rank bad bowls bowled without any sense of focus, planning or regret. India loses, yet again in those circumstances when he has done just about everything right.<br />He does not cry. Does not show any emotion. Just keeps his head down and leaves the field. He has seen these failures for 22 years now. And not just his class, relatives, friends but the whole world has seen these failures. We are too immature to even imagine what goes on in that mind and heart of his. That’s why I would never want to be Sachin.<br /><br />True, he has single handedly lifted to moods of this entire nation umpteen number of times. He has been an inspiration to rise above our mediocrity. Nobody who has ever lifted the willow even comes close to this man’s genius. His dedication and metal strength is unparallel. This is specially for those people who would have made fun of him again last night when India lost. They are people who are mediocre in their own lives. Who just scoff at others to create cheap fun. Who have lived in a small hole throughout their lives and thought they have seen the oceans.<br /><br />Think about the man himself. He is 37 years of age. He has been playing almost non stop for 22 years. The way he was running and diving around the field last night would have put 22 year olds to shame. The way he played the best opening quickies in the world was breathtaking. He just keeps getting better which is by the way humanly impossible. Its not for nothing that people call him GOD.<br />But still I don’t want to be in those shoes. We struggle in keeping our monotonous lives straight, lives which affect a limited number of people. Imagine what would be the magnitude of the inner struggle for him, pain both mental and physical, tears that have frozen with time, knees and ankles and every other joint in the body that is either bandaged or needs to be attended to every night, eyes that don’t sleep before a big game, bats that have scored 99 international tons and still see expectations from a billion people.<br /><br />And he just converts those expectations into reality. We watch in awe, feel privileged.<br />Well I think its time that his team realizes that enough is enough. They have an obligation, not towards their country alone but towards sachin. They need to win this one for him. Stay assured that he himself will still deliver and leave no stone unturned to make sure India wins this cup.<br />This is not just a game, and he is not just a sportsman. Its much more than this. Words fail here.....<br /><br />-- Anonymous<br /><br />(This post is not written by Harsha Bhogle even though the source page said it was)<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com2tag:blogger.com,1999:blog-30819994.post-89751672190256456572011-02-24T16:37:00.001-05:002011-02-24T16:42:41.376-05:00Setting and running Hadoop 0.20.2Step 1: Download and extract hadoop code<br />wget http://mirror.cloudera.com/apache/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz<br />Extract the files and go into that directory<br /><br />Step 2: Configure Hadoop<br />vim ~/.bash_profile<br />export JAVA_HOME=/usr/local/java/vms/java<br />export HADOOP_HOME=/home/oa/hadoop-asterix/hadoop-0.20.2<br /><br />vim ${HADOOP_HOME}/conf/masters<br />asterix-master<br /><br />vim ${HADOOP_HOME}/conf/slaves<br />asterix-001<br />asterix-002<br />asterix-003<br /><br /><br />vim ${HADOOP_HOME}/conf/core-site.xml<br /><property><br /> <name>fs.default.name</name><br /> <value>hdfs://10.122.198.195:54310</value><br /> <description>The name of the default file system. A URI whose<br /> scheme and authority determine the FileSystem implementation. The<br /> uri's scheme determines the config property (fs.SCHEME.impl) naming<br /> the FileSystem implementation class. The uri's authority is used to<br /> determine the host, port, etc. for a filesystem.</description><br /></property><br /><br /><br />vim ${HADOOP_HOME}/conf/mapred-site.xml<br /><property><br /> <name>mapred.job.tracker</name><br /> <value>10.122.198.195:54311</value><br /> <description>The host and port that the MapReduce job tracker runs<br /> at. If "local", then jobs are run in-process as a single map<br /> and reduce task.<br /> </description><br /></property><br /><br /><br />vim ${HADOOP_HOME}/conf/hdfs-site.xml<br /><property><br /> <name>dfs.replication</name><br /> <value>2</value><br /> <description>Default block replication.<br /> The actual number of replications can be specified when the file is created.<br /> The default is used if replication is not specified in create time.<br /> </description><br /></property><br /><property><br /> <name>dfs.name.dir</name><br /> <value>/mnt/hdfs/name_dir/</value><br /></property><br /><property><br /> <name>dfs.data.dir</name><br /> <value>/mnt/hdfs/data_dir/</value><br /></property><br /><property><br /> <name>dfs.datanode.address</name><br /> <value>0.0.0.0:54325</value><br /></property><br /><property><br /> <name>dfs.datanode.http.address</name><br /> <value>0.0.0.0:54326</value><br /></property><br /><br /><br />vim ${HADOOP_HOME}/conf/hadoop-env.sh<br />export JAVA_HOME=/usr/local/java/vms/java<br /><br />Make sure the name and data directory are properly setup using following script. This also confirms that you can do passwordless ssh from master to slave machines.<br />vim ${HADOOP_HOME}/commands.sh<br />cd /mnt<br />sudo mkdir hdfs<br />cd hdfs/<br />sudo mkdir name_dir<br />sudo mkdir data_dir<br />sudo chmod 777 -R /mnt/hdfs<br /><br /># Sync this file<br />parallel-rsync -p 6 -r -h ${HADOOP_HOME}/conf/slaves ${HADOOP_HOME} ${HADOOP_HOME}<br /><br />Then, run the above command.sh for every slave<br />user_name="ubuntu"<br />command="sh ${HADOOP_HOME}/commands.sh"<br />for slaves_ip in $(cat ${HADOOP_HOME}/conf/slaves) <br />do <br />ssh ${user_name}@${slaves_ip} ${command}<br />done<br /><br /><br />Step 3: Compile hadoop and sync slaves<br /># Compile source code<br />${ANT_HOME}/bin/ant<br />${ANT_HOME}/bin/ant jar<br />${ANT_HOME}/bin/ant examples<br /><br /># Sync slaves (see EC2 point 3 for installing parallel ssh)<br />parallel-rsync -p 6 -r -h ${HADOOP_HOME}/conf/slaves ${HADOOP_HOME} ${HADOOP_HOME}<br /><br />Step 4: Run hadoop<br />${HADOOP_HOME}/bin/hadoop namenode -format<br />${HADOOP_HOME}/bin/stop-all.sh<br />${HADOOP_HOME}/bin/start-all.sh<br /><br />Check logs/ datanode namenode. Also see if the nodes are up, by using (any normal browser or) links http://master-ip:50070<br />If you get "incompatible namespace error" in datanodes log, let me try deleting hdfs dir and restarting hdfs<br /><br />Step 4: Loading the HDFS<br />If you are loading from:<br />1. Local filesystem of master, use: either copyFromLocal or put<br />bin/hadoop dfs -copyFromLocal /mnt/wikipedia_input/wikistats/pagecounts/pagecounts* wikipedia_input<br /><br />2. Some other hdfs, use put<br /><br />3. Files on some other machine accessible via scp<br /># Configure following variables. Keep space between parenthesis of array and each item (no comma).<br /># If this script gives error like '4: Syntax error: "(" unexpected', try bash <script-name><br /># If that gives permission denied error, put name of directories instead of ${directories1[@]}<br />user_name="ubuntu"<br />machine_name="my_machine_name_or_ip"<br />file_prefix="pagecounts*"<br />hdfs_dir="wikipedia_input"<br />directories1=( "/mnt/data/sdb/space/oa/wikidata/dammit.lt/wikistats/archive/2010/09" "/mnt/data/sdc/space/oa/wikidata/dammit.lt/wikistats/archive/2010/10" "/mnt/data/sdd/space/oa/wikidata/dammit.lt/wikistats/archive/2010/11" )<br />for dir1 in ${directories1[@]}<br />do<br /> echo "--------------------------------------------" <br /> cmd="ssh ${user_name}@${machine_name} 'ls ${dir1}/$file_prefix'"<br /> echo "Reading files using: " $cmd<br /> for file1 in `eval $cmd`<br /> do<br /> file_name1=${file1##*/}<br /> echo -n $file_name1 " "<br /> scp oa@asterix-001:$file1 .<br /> bin/hadoop dfs -copyFromLocal $file_name1 $hdfs_dir<br /> rm $file_name1<br /> done<br />done<br /><br />Step 5: Run your mapreduce program<br />${HADOOP_HOME}/bin/hadoop jar ${HADOOP_HOME}/build/hadoop-hop-0.2-examples.jar wordcount tpch_input tpch_output<br /><br />Some tips for amazon EC2:<br />1. Lot of times due to resource allocation policies, EC2 shutdowns your virtual machine (hence the assigned network) and master/slaves/namenodes/datanodes goes into fault tolerance mode and restart the jobs. You can set infinite time for heartbeat to tackle this error (This works because EC2 restarts you virtual machine after some time and there is no "real" failure, just temporary lags) by setting following into ${HADOOP_HOME}/conf/hdfs-site.xml<br /><property><br /> <name>dfs.heartbeat.interval</name><br /> <value>6000</value><br /> <description>Determines datanode heartbeat interval in seconds.</description><br /></property><br /><property><br /> <name>dfs.heartbeat.recheck.interval</name><br /> <value>6000</value><br /> <description>Determines datanode heartbeat interval in seconds.</description><br /></property><br /><property><br /> <name>heartbeat.recheck.interval</name><br /> <value>6000</value><br /> <description>If dfs... doesnot work</description><br /></property><br /><property><br /> <name>dfs.socket.timeout</name><br /> <value>180000</value><br /> <description>dfs socket timeout</description><br /></property><br /><br />2. Login into EC2 machine:<br />EC2_KEYPAIR_DIR="/home/np6/EC2"<br />echo "\nEnter Public DNS of Master"<br />read AMAZON_PUBLIC_DNS<br />echo "If this doesnot work try, (exec ssh-agent bash) and then this command again"<br />ssh-agent<br />ssh-add ${EC2_KEYPAIR_DIR}/ec2-keypair.pem<br />ssh ubuntu@$AMAZON_PUBLIC_DNS<br /><br />3. Setting up java and other programs on slaves from master<br /># First install parallel ssh<br />user_name="ubuntu"<br />command="sudo apt-get install pssh"<br />for slaves_ip in $(cat ${HADOOP_HOME}/conf/slaves) <br />do <br />ssh ${user_name}@${slaves_ip} ${command}<br />done<br /><br /># Then install java (if you dont prefer openjdk)<br />vim ${HADOOP_HOME}/commands.sh<br />sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"<br />sudo apt-get update<br />sudo apt-get install sun-java6-jdk <br />sudo update-java-alternatives -s java-6-sun<br />echo "export JAVA_HOME=/usr/lib/jvm/java-6-sun" >> ~/.bashrc<br />echo "export HADOOP_HOME=/home/ubuntu/hadoop" >> ~/.bashrc<br />source ~/.bashrc<br />echo "Check if the version of java is correct:"<br />java -version<br /><br /># Sync this file<br />parallel-rsync -p 6 -r -h ${HADOOP_HOME}/conf/slaves ${HADOOP_HOME} ${HADOOP_HOME}<br /><br />Then, run the above command.sh for every slave<br />user_name="ubuntu"<br />command="sh ${HADOOP_HOME}/commands.sh"<br />for slaves_ip in $(cat ${HADOOP_HOME}/conf/slaves) <br />do <br />ssh ${user_name}@${slaves_ip} ${command}<br />done<br /><br />4. Setting up hadoop master and slaves for lazy person (I would recommend you follow above steps instead)<br />cd ${HADOOP_HOME}/conf<br />echo "\nEnter Public DNS of Master"<br />read AMAZON_PUBLIC_DNS<br />sed -e "s/<name>mapred.job.tracker<\/name> <value>[-[:graph:]./]\{1,\}<\/value>/<name>mapred.job.tracker<\/name> <value>${AMAZON_PUBLIC_DNS}<\/value>/" hadoop-site.xml > a1.txt<br />sed -e "s/<name>fs.default.name<\/name> <value>[-[:graph:]./]\{1,\}<\/value>/<name>fs.default.name<\/name> <value>hdfs:\/\/${AMAZON_PUBLIC_DNS}:9001<\/value>/" a1.txt > hadoop-site.xml<br />echo ${AMAZON_PUBLIC_DNS} > masters<br />echo "\nEnter Slave string seperated with space (eg: domU-12-31-39-09-A0-84.compute-1.internal domU-12-31-39-0F-7E-61.compute-1.internal)"<br />read SLAVE_STR<br />echo $SLAVE_STR | sed -e "s/ /\n/" > slaves<br /><br />Some other neat tricks:<br />1. Replace default java temp directory:<br />export JAVA_OPTS="-Djava.io.tmpdir=/mnt/java_tmp"<br /><br />2. Setting number of open file limit to 99999<br />sudo vi /etc/security/limits.conf<br />ubuntu soft nofile 99999<br />ubuntu hard nofile 99999<br />* soft nofile 99999<br />* hard nofile 99999<br />sudo sysctl -p<br />ulimit -Hn<br /><br />3. Checking the machines on the network<br />cat /etc/hosts<br /><br />or naming machines as masters and slaves: vim /etc/hosts<br />10.1.0.1 asterix-master<br />127.0.0.1 localhost<br />10.0.0.1 asterix-001<br />10.0.0.2 asterix-002<br /><br />Checking machines ip address<br />/sbin/ifconfig<br /><br />4. Configuring password-less ssh of master to slaves<br />slave_user_name="ubuntu"<br />for slaves_ip in $(cat ${HADOOP_HOME}/conf/slaves) <br />do <br />ssh-copy-id -i $HOME/.ssh/id_rsa.pub ${slave_user_name}@${slaves_ip}<br />done<br /><br />For more detailed step by step example, see <br />http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/<div class="blogger-post-footer"><!-- Search Google -->
<center>
<form method="get" action="http://www.google.co.in/custom" target="google_window">
<table bgcolor="#ffffff">
<tr><td nowrap="nowrap" valign="top" align="left" height="32">
<a href="http://www.google.com/">
<img src="http://www.google.com/logos/Logo_25wht.gif" border="0" alt="Google" align="middle"></img></a>
<br/>
<input type="text" name="q" size="31" maxlength="255" value=""></input>
</td></tr>
<tr><td valign="top" align="left">
<input type="submit" name="sa" value="Search"></input>
<input type="hidden" name="client" value="pub-7438668549819913"></input>
<input type="hidden" name="forid" value="1"></input>
<input type="hidden" name="ie" value="ISO-8859-1"></input>
<input type="hidden" name="oe" value="ISO-8859-1"></input>
<input type="hidden" name="safe" value="active"></input>
<input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFF3DB;LBGC:FFF3DB;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;"></input>
<input type="hidden" name="hl" value="en"></input>
</td></tr></table>
</form>
</center>
<!-- Search Google -->
</div>Niketanhttp://www.blogger.com/profile/04265098294100409965noreply@blogger.com0