tag:blogger.com,1999:blog-26724274731199231092024-03-06T23:39:20.460-08:00Marco CraveiroMarco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.comBlogger141125tag:blogger.com,1999:blog-2672427473119923109.post-29809822360203954132020-04-06T11:59:00.001-07:002020-06-16T14:29:32.164-07:00Nerd Food: Notes on Computational Finance, Part I: Introduction<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2020-04-06 Mon 20:45 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Nerd Food: Computational Finance, Part I: Introduction</title>
<meta name="generator" content="Org mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center;
margin-bottom: .2em; }
.subtitle { text-align: center;
font-size: medium;
font-weight: bold;
margin-top:0; }
.todo { font-family: monospace; color: red; }
.done { font-family: monospace; color: green; }
.priority { font-family: monospace; color: orange; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.org-right { margin-left: auto; margin-right: 0px; text-align: right; }
.org-left { margin-left: 0px; margin-right: auto; text-align: left; }
.org-center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
/* Languages per Org manual */
pre.src-asymptote:before { content: 'Asymptote'; }
pre.src-awk:before { content: 'Awk'; }
pre.src-C:before { content: 'C'; }
/* pre.src-C++ doesn't work in CSS */
pre.src-clojure:before { content: 'Clojure'; }
pre.src-css:before { content: 'CSS'; }
pre.src-D:before { content: 'D'; }
pre.src-ditaa:before { content: 'ditaa'; }
pre.src-dot:before { content: 'Graphviz'; }
pre.src-calc:before { content: 'Emacs Calc'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-fortran:before { content: 'Fortran'; }
pre.src-gnuplot:before { content: 'gnuplot'; }
pre.src-haskell:before { content: 'Haskell'; }
pre.src-hledger:before { content: 'hledger'; }
pre.src-java:before { content: 'Java'; }
pre.src-js:before { content: 'Javascript'; }
pre.src-latex:before { content: 'LaTeX'; }
pre.src-ledger:before { content: 'Ledger'; }
pre.src-lisp:before { content: 'Lisp'; }
pre.src-lilypond:before { content: 'Lilypond'; }
pre.src-lua:before { content: 'Lua'; }
pre.src-matlab:before { content: 'MATLAB'; }
pre.src-mscgen:before { content: 'Mscgen'; }
pre.src-ocaml:before { content: 'Objective Caml'; }
pre.src-octave:before { content: 'Octave'; }
pre.src-org:before { content: 'Org mode'; }
pre.src-oz:before { content: 'OZ'; }
pre.src-plantuml:before { content: 'Plantuml'; }
pre.src-processing:before { content: 'Processing.js'; }
pre.src-python:before { content: 'Python'; }
pre.src-R:before { content: 'R'; }
pre.src-ruby:before { content: 'Ruby'; }
pre.src-sass:before { content: 'Sass'; }
pre.src-scheme:before { content: 'Scheme'; }
pre.src-screen:before { content: 'Gnu Screen'; }
pre.src-sed:before { content: 'Sed'; }
pre.src-sh:before { content: 'shell'; }
pre.src-sql:before { content: 'SQL'; }
pre.src-sqlite:before { content: 'SQLite'; }
/* additional languages in org.el's org-babel-load-languages alist */
pre.src-forth:before { content: 'Forth'; }
pre.src-io:before { content: 'IO'; }
pre.src-J:before { content: 'J'; }
pre.src-makefile:before { content: 'Makefile'; }
pre.src-maxima:before { content: 'Maxima'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-picolisp:before { content: 'Pico Lisp'; }
pre.src-scala:before { content: 'Scala'; }
pre.src-shell:before { content: 'Shell Script'; }
pre.src-ebnf2ps:before { content: 'ebfn2ps'; }
/* additional language identifiers per "defun org-babel-execute"
in ob-*.el */
pre.src-cpp:before { content: 'C++'; }
pre.src-abc:before { content: 'ABC'; }
pre.src-coq:before { content: 'Coq'; }
pre.src-groovy:before { content: 'Groovy'; }
/* additional language identifiers from org-babel-shell-names in
ob-shell.el: ob-shell is the only babel language using a lambda to put
the execution function name together. */
pre.src-bash:before { content: 'bash'; }
pre.src-csh:before { content: 'csh'; }
pre.src-ash:before { content: 'ash'; }
pre.src-dash:before { content: 'dash'; }
pre.src-ksh:before { content: 'ksh'; }
pre.src-mksh:before { content: 'mksh'; }
pre.src-posh:before { content: 'posh'; }
/* Additional Emacs modes also supported by the LaTeX listings package */
pre.src-ada:before { content: 'Ada'; }
pre.src-asm:before { content: 'Assembler'; }
pre.src-caml:before { content: 'Caml'; }
pre.src-delphi:before { content: 'Delphi'; }
pre.src-html:before { content: 'HTML'; }
pre.src-idl:before { content: 'IDL'; }
pre.src-mercury:before { content: 'Mercury'; }
pre.src-metapost:before { content: 'MetaPost'; }
pre.src-modula-2:before { content: 'Modula-2'; }
pre.src-pascal:before { content: 'Pascal'; }
pre.src-ps:before { content: 'PostScript'; }
pre.src-prolog:before { content: 'Prolog'; }
pre.src-simula:before { content: 'Simula'; }
pre.src-tcl:before { content: 'tcl'; }
pre.src-tex:before { content: 'TeX'; }
pre.src-plain-tex:before { content: 'Plain TeX'; }
pre.src-verilog:before { content: 'Verilog'; }
pre.src-vhdl:before { content: 'VHDL'; }
pre.src-xml:before { content: 'XML'; }
pre.src-nxml:before { content: 'XML'; }
/* add a generic configuration mode; LaTeX export needs an additional
(add-to-list 'org-latex-listings-langs '(conf " ")) in .emacs */
pre.src-conf:before { content: 'Configuration File'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.org-right { text-align: center; }
th.org-left { text-align: center; }
th.org-center { text-align: center; }
td.org-right { text-align: right; }
td.org-left { text-align: left; }
td.org-center { text-align: center; }
dt { font-weight: bold; }
.footpara { display: inline; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.equation-container {
display: table;
text-align: center;
width: 100%;
}
.equation {
vertical-align: middle;
}
.equation-label {
display: table-cell;
text-align: right;
vertical-align: middle;
}
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
.org-svg { width: 90%; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2020 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
Welcome to the first part of what we hope will be a long series of
articles exploring <a href="https://www.quantlib.org/">QuantLib</a>, <a href="https://github.com/OpenSourceRisk/Engine">Open Source Risk Engine</a> (ORE), <a href="https://github.com/libbitcoin/libbitcoin-system">libbitcoin</a>
and other interesting finance-related FOSS (Free and Open Source)
projects. However, I'm afraid this will be a bit of a dull first post,
as we need to clarify our objectives before we can jump into the fray.
</p>
<div id="outline-container-orgc99cdfe" class="outline-2">
<h2 id="orgc99cdfe">About</h2>
<div class="outline-text-2" id="text-orgc99cdfe">
<p>
Even though I've been a software engineer in the financial sector for
over fifteen years, I've always felt I lacked a <i>deep understanding</i>
of the foundational concepts that make up the domain. As a
self-confessed reductionist, I find this state of affairs extremely
uncomfortable, akin to hearing a continuous loop of David Hilbert's
words: <i>wir müssen wissen, wir werden wissen</i><sup><a id="fnr.1" class="footref" href="#fn.1">1</a></sup>. The
situation <i>had</i> to be remedied, somehow, and the material you are now
reading is the proposed fix for my ailments. As to the methodology:
given I've had some success in applying the Feynman
Technique<sup><a id="fnr.2" class="footref" href="#fn.2">2</a></sup> to <a href="http://mcraveiro.blogspot.com/2015/08/nerd-food-neurons-for-computer-geeks.html">other complex domains</a>, it seemed only natural
to try to use it for this endeavour as well. Past experience also
demonstrated writing is an adequate replacement for <i>in vivo</i>
communication, which is just as well in this brave new world of social
distancing.
</p>
<p>
So that's that for the <i>why</i> and the <i>how</i>. But, just <i>what</i> exactly
are we researching?
</p>
</div>
<div id="outline-container-org038ff83" class="outline-3">
<h3 id="org038ff83">Scope</h3>
<div class="outline-text-3" id="text-org038ff83">
<p>
These posts shall largely amble where our fancy takes us, within the
porous boundaries of finance. Alas, we can hardly keep calling our
target domain "trading, accounting, crypto and a bit of quantitative
finance, when viewed through the lens of FOSS" - descriptive though
that might sound. We are Software Engineers after all, and if there is
one thing we do is to name things, <i>especially</i> when we lack
competence to do so<sup><a id="fnr.3" class="footref" href="#fn.3">3</a></sup>. In this vein, I decided to call this motley
domain of ours "Computational Finance". Should the name have merit,
I'm afraid I have little claim to it, as it was shamelessly stolen
from this passage in <a href="https://writings.stephenwolfram.com/2016/09/how-to-teach-computational-thinking/">Wolfram's writings</a>:
</p>
<blockquote>
<p>
Doctors, lawyers, teachers, farmers, whatever. The future of all these
professions will be full of computational thinking. Whether it’s
sensor-based medicine, computational contracts, education analytics or
computational agriculture - success is going to rely on being able to
do computational thinking well.
</p>
<p>
I’ve noticed an interesting trend. Pick any field X, from archeology
to zoology. There either is now a “computational X” or there soon will
be. And it’s widely viewed as the future of the field.
</p>
</blockquote>
<p>
These seemed like inspiring words for anyone embarking on a long and
uncertain journey, so we made them our own and, in turn, it gave us a
name to rally around. But what of its boundaries? One of the biggest
challenges facing the reductionist is that, in the limit, <i>everything
is interconnected with everything else</i>, for there is no natural
halting function. Thus, if you are not careful, all paths will
eventually lead you into the realm of quarks and particle physics,
regardless of your starting point. Now, that would not be an entirely
useful outcome. I have never found a completely satisfactory answer to
this question in any on my previous meanderings, but in general I tend
to follow an empiric approach and let <i>taste</i> be my
guide<sup><a id="fnr.4" class="footref" href="#fn.4">4</a></sup>. Granted, its probably not the scientific
solution you were expecting, but it seems that there are "intuitive"
boundaries in subjects, and when we hit one of those we shall
stop<sup><a id="fnr.5" class="footref" href="#fn.5">5</a></sup>. As an example, for our purposes we need not look in
detail at legal frameworks when trying to understand financial
concepts, though the two disciplines are deeply intertwined.
</p>
</div>
</div>
<div id="outline-container-orgd3a5007" class="outline-3">
<h3 id="orgd3a5007">Structure</h3>
<div class="outline-text-3" id="text-orgd3a5007">
<p>
An issue which is closely interrelated with the previous one is on how
to strike a balance between computational exploration versus domain
definitions. Too much exploration and you proceed full steam ahead
without knowing the meaning of things; too many boring definitions and
they become just words without bringing any light to the subject under
scrutiny. The sweet spot lies somewhere in the middle.
</p>
<p>
Our approach can be described as follows. We shall try to progress
very slowly and methodically through the concepts in the domain,
building them up as we climb the abstraction ladder but without making
them too dense and technical. We'll make extensive use of Wikipedia
definitions, where possible, but keeping these focused only on the
point at hand rather than exploring the myriad of possibilities around
a theme.
</p>
<p>
Finally, we shall try to marry domain concepts with our chosen
implementations - the computational experiments part - in order to
illustrate their purpose and get a better understanding at what it is
that they are trying to do. So, each post will be focused on one
fairly narrow subject area, start with a bunch of definitions which
are hopefully self-explanatory and then proceed to explore the
available implementations on that topic, or code that we write
ourselves.
</p>
</div>
</div>
<div id="outline-container-org81de0c9" class="outline-3">
<h3 id="org81de0c9">Audience</h3>
<div class="outline-text-3" id="text-org81de0c9">
<p>
The target audience for this material is the fabled <i>homo developus</i>,
that non-existent "standard developer" - in this particular case, one
moderately competent on C++ but unfamiliar with computational
finance. On the "finance" part, if you are already familiar with the
domain, you will no doubt find the content <i>very</i> slow going. I'm
afraid this is by design: the objective is to try to build the
concepts on a solid foundation for those <i>not in the know</i>, so
slowness is unavoidable<sup><a id="fnr.6" class="footref" href="#fn.6">6</a></sup>.
</p>
<p>
With regards to the computational part: the astute reader will likely
point out that there are a great deal of tutorials on QuantLib, ORE
and many other libraries of a similar ilk, and many books have been
written on quantitative finance. One could be forgiven for wondering
if there is a need to pile on more literature onto a seemingly already
crowded subject.
</p>
<p>
In our defence, we are yet to find work that directly targets "plain"
software developers and provides them with a sufficiently broad view
of the domain. In addition, most of the existing material is aimed at
either those with strong mathematical abilities but no domain
knowledge, or its converse, leaving many gaps in the
understanding. What we are instead aiming for is to target those with
strong programming abilities but no particular knowledge of either
computational finance <i>or</i> mathematics. And this leads us nicely to
our next topic.
</p>
</div>
</div>
<div id="outline-container-org04a295b" class="outline-3">
<h3 id="org04a295b">Mathematics</h3>
<div class="outline-text-3" id="text-org04a295b">
<p>
Our assumption is that you, dear reader, are not able to attain deep
levels of understanding by staring at page after page of complex
mathematical formulae. I, for one, certainly cannot. Unfortunately,
non-trivial mathematics is difficult to avoid when covering a subject
matter of this nature so, as a counterweight, we shall strive to use
it sparingly and only from a software engineering <i>application</i>
perspective. Note that this approach is clearly not suitable for the
mathematically savvy amongst us, as they will find it unnecessarily
laboured; then again, our focus lies elsewhere.
</p>
<p>
Our core belief is that an <i>average reader</i> (like me) should be able
to attain a software engineer's intuition of <i>how things work</i> just by
fooling around with software models of formulae. The reason why I am
very confident on this regard is because that's how developers learn:
by coding and seeing what happens. In fact, it is this very tight
feedback loop between having an idea and experimenting with it that
got many of us hooked into programming in the first place, so its a
very powerful tool in the motivational arsenal. And, as it turns out,
these ideas are related to Wolfram's concept of <a href="https://www.wolframscience.com/nks/notes-3-12--history-of-experimental-mathematics/">Experimental
Mathematics</a>. Ultimately, our aspiration is to copy the approach taken
by Klein in <a href="https://codingthematrix.com/">Coding the Matrix</a>, though perhaps that sets the bar a tad
too high. Well, at least you get the spirit of the approach.
</p>
</div>
</div>
<div id="outline-container-org3d4740f" class="outline-3">
<h3 id="org3d4740f">Cryptos</h3>
<div class="outline-text-3" id="text-org3d4740f">
<p>
Another rather peculiar idea we pursued is the use of cryptocurrencies
throughout, to the exclusion of <i>everything else</i>. Whilst very popular
in the media, where they are known as <i>cryptos</i>, in truth
cryptocurrencies still have a limited presence in the "real" world of
finance, and nowhere more so than in derivatives - <i>i.e.</i>, the bulk of
our analysis. So at first blush, this is a most puzzling choice. We
have decided to do so for three reasons.
</p>
<p>
Firstly, just because I wanted to learn more about cryptos. Secondly,
because there is a need to bridge the knowledge gap between these two
distinct worlds of finance; to blend the old with the new if you
will. Personally, I think it will be interesting to see what the
proliferation of derivatives will do to cryptos - but for that we need
to disseminate financial knowledge. Finally, and most important of
all, because in order to properly illustrate all of the concepts we
shall cover, and to drive the open source libraries to their
potential, one needs <i>vast amounts of data of the right kind</i>. Lets
elaborate further on this point.
</p>
<p>
One of the biggest problems with any material in quantitative finance
is in obtaining data sets which are sufficiently rich to cover all of
the concepts being explained. This, in my opinion, is one of the key
shortcomings with most tutorials: they either assume users can source
the data themselves, or provide a small data set to prove a very
specific point but which is insufficient for wider
exploration<sup><a id="fnr.7" class="footref" href="#fn.7">7</a></sup>. This document takes a slightly different
approach. We will base ourselves on a <i>simulated world</i> - a parallel
reality if you'd like, thinly anchored to our reality by freely
available data taken from the crypto markets. We shall then generate
<i>all</i> of the remaining data, to the level of precision, richness and
consistency required both to drive the code samples, but also to allow
for "immersive" exploration. In fact, the <i>very processes for data
generation</i> will be used as a pathway for domain exploration.
</p>
<p>
Of course, generated data is not perfect - <i>i.e.</i>, <i>realistic</i> it is
not, by definition - but our point is to understand the concepts, not
to create new quant models that trade in the real world, so it is
adequate for our needs. In addition, the data sets and code samples,
as well as the means used to derive them shall be part of a git
repository under an open source licence, so they can be extended and
improved over time.
</p>
<p>
If you are not familiar with cryptos, don't worry. For starters, we
can <i>assume</i> the intricate mechanistic details to a large extent - the
blockchain and so forth - and introduce key concepts as required. We
need not concern ourselves with this because there is plenty of freely
available material covering it in excruciating detail, and designed
specifically for software engineers. Instead, we shall treat cryptos
<i>as if they were regular currencies</i>, except where they are just <i>too
different</i> - in which case we'll point out the differences. Its a bit
of a strange approach, but hopefully it will produce the desired
results.
</p>
</div>
</div>
<div id="outline-container-org92cf76f" class="outline-3">
<h3 id="org92cf76f">Non Goals</h3>
<div class="outline-text-3" id="text-org92cf76f">
<p>
If you are trying to learn techniques on how to trade, this is not the
material for you. Even when we discuss trading strategies and other
similar topics, our focus is always on trying to understand how the
machinery works rather than on how to make money with it. Similarly,
if you are a quant or are trying to become one, you are probably
better off reading the traditional books such as Hull or Wilmott
rather than these posts, as our treatment of mathematics will be far
too basic for your requirements. However, if you are an expert in this
subject area, or if you find any mistakes please do point them out.
</p>
</div>
</div>
</div>
<div id="outline-container-orge684ba8" class="outline-2">
<h2 id="orge684ba8">Legalese</h2>
<div class="outline-text-2" id="text-orge684ba8">
<p>
As with anything to do with finance, we need to set out the standard
disclaimers. To make sure these are seen, we shall add them to each
post.
</p>
</div>
<div id="outline-container-org8f32d9b" class="outline-3">
<h3 id="org8f32d9b">Legal Disclaimer</h3>
<div class="outline-text-3" id="text-org8f32d9b">
<p>
All of the content, including source code, is either written by the
author of the posts, or obtained from freely available sites in the
internet, with suitable software licences. All content sources shall
be clearly identified at the point of use. No proprietary information
<i>of any kind</i> - including, but not limited to, source code, text,
market data or mathematical models - shall be used within this
material.
</p>
<p>
All of the views expressed here represent exclusively myself and are
not those of any corporation I may be engaged in commercial activities
with.
</p>
<p>
The information available in these blog posts is for your general
information and use and is not intended to address your particular
requirements. In particular, the information does not constitute any
form of financial advice or recommendation and is not intended to be
relied upon by users in making (or refraining from making) any
investment decisions.<sup><a id="fnr.8" class="footref" href="#fn.8">8</a></sup>
</p>
<p>
All software written by the author for these posts is licensed under
the Gnu GPL v3. As per the licence, it is "distributed in the hope
that it will be useful, but without any warranty; without even the
implied warranty of merchantability or fitness for a particular
purpose. See the GNU General Public License for more details."
</p>
</div>
</div>
</div>
<div id="outline-container-orge15bb5d" class="outline-2">
<h2 id="orge15bb5d">Next</h2>
<div class="outline-text-2" id="text-orge15bb5d">
<p>
With all of the preliminaries out of the way, we can move on to the
meat of the subject. On Part II we shall discuss our first real topic,
and it could not be much more fundamental: <i>Money</i>.
</p>
</div>
</div>
<div id="footnotes">
<h2 class="footnotes">Footnotes: </h2>
<div id="text-footnotes">
<div class="footdef"><sup><a id="fn.1" class="footnum" href="#fnr.1">1</a></sup> <div class="footpara"><p class="footpara">
"We must know, we will know". As per <a href="https://en.wikipedia.org/wiki/David_Hilbert">Wikipedia</a>:
</p>
<blockquote>
<p>
The epitaph on his tombstone in Göttingen consists of the famous lines
he spoke at the conclusion of his retirement address to the Society of
German Scientists and Physicians on 8 September 1930. The words were
given in response to the Latin maxim: "Ignoramus et ignorabimus" or
"We do not know, we shall not know".
</p>
</blockquote></div></div>
<div class="footdef"><sup><a id="fn.2" class="footnum" href="#fnr.2">2</a></sup> <div class="footpara"><p class="footpara">
The <a href="https://www.farnamstreetblog.com/2012/04/learn-anything-faster-with-the-feynman-technique/">Feynman Technique</a> is a well-established learning
methodology. For more details, see <a href="https://www.farnamstreetblog.com/2015/01/richard-feynman-knowing-something/">Richard Feynman: The Difference
Between Knowing the Name of Something and Knowing Something</a>.
</p></div></div>
<div class="footdef"><sup><a id="fn.3" class="footnum" href="#fnr.3">3</a></sup> <div class="footpara"><p class="footpara">
There are no circumstances under which I have seen
software developers lacking confidence. I feel that the motto of our
profession should be the Latin translation of <i>Make up with confidence
that which you lack for in competence.</i>
</p></div></div>
<div class="footdef"><sup><a id="fn.4" class="footnum" href="#fnr.4">4</a></sup> <div class="footpara"><p class="footpara">
An idea that was most likely inspired by Linus' views
on <i>good taste</i>. For details see <a href="https://medium.com/@bartobri/applying-the-linus-tarvolds-good-taste-coding-requirement-99749f37684a">Applying the Linus Torvalds “Good
Taste” Coding Requirement</a>.
</p></div></div>
<div class="footdef"><sup><a id="fn.5" class="footnum" href="#fnr.5">5</a></sup> <div class="footpara"><p class="footpara">
Of course, <i>your</i> intuition is not <i>my</i> intuition. I'm
afraid you will have to take my taste as a given, even where you
disagree. Feel free to make your views heard though.
</p></div></div>
<div class="footdef"><sup><a id="fn.6" class="footnum" href="#fnr.6">6</a></sup> <div class="footpara"><p class="footpara">
As they say in my home country of Angola, <i>malembe
malembe</i>. The expression can be loosely translated to English as
"slowly but surely", or "slowly does it".
</p></div></div>
<div class="footdef"><sup><a id="fn.7" class="footnum" href="#fnr.7">7</a></sup> <div class="footpara"><p class="footpara">
As an example, the latter approach is taken by a library
I respect very much, the <a href="https://github.com/OpenSourceRisk/Engine">Open Source Risk Engine (ORE)</a>.
</p></div></div>
<div class="footdef"><sup><a id="fn.8" class="footnum" href="#fnr.8">8</a></sup> <div class="footpara"><p class="footpara">
This paragraph was obtained from the <a href="https://trulyifa.co.uk/disclaimer/">Truly Independent
Ltd</a> and modified to suit our needs.
</p></div></div>
</div>
</div></div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com2tag:blogger.com,1999:blog-2672427473119923109.post-84952749400759507712018-01-03T07:45:00.000-08:002018-01-03T07:55:24.177-08:00Nerd Food: The Refactoring Quagmire<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: The Refactoring Quagmire</title>
<!-- 2018-01-03 Wed 15:55 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
<a href="https://github.com/DomainDrivenConsulting/dogen/blob/master/doc/agile/v1/sprint_backlog_07.org">The latest Dogen sprint</a> turned out to be a really long and tortuous
one, which is all the more perplexing given the <a href="https://github.com/DomainDrivenConsulting/dogen/blob/master/doc/agile/v1/sprint_backlog_06.org">long</a> <a href="https://github.com/DomainDrivenConsulting/dogen/blob/master/doc/agile/v1/sprint_backlog_05.org">list</a> of <a href="https://github.com/DomainDrivenConsulting/dogen/blob/master/doc/agile/v1/sprint_backlog_04.org">hard</a>
<a href="https://github.com/DomainDrivenConsulting/dogen/blob/master/doc/agile/v1/sprint_backlog_03.org">sprints</a> that preceded it. Clearly, the slope of the curve is
steepening unrelentingly. Experience teaches that whenever you find
yourself wandering over such terrains, it is time to stop and gather
your thoughts; more likely than not, you are going the wrong way -
fast.
</p>
<p>
Thus, for Dogen, this a post of reflection. To the casual reader - if
nothing else - it will hopefully serve as a cautionary tale.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Not Even Wrong</h2>
<div class="outline-text-2" id="text-1">
<p>
If you are one of the lucky few internauts who avidly follows our
release notes, you may recall that the <a href="https://github.com/DomainDrivenConsulting/dogen/releases/tag/v1.0.06">previous sprint</a> had produced a
moment of enlightenment where we finally understood <code>yarn</code> as <i>the</i>
core of Dogen. At the time, it felt like one of those rare <i>eureka</i>
moments, and "the one last great change to the architecture";
afterwards, all would be light. "Famous last words", you may have said
then and, of course, if you did, you were right. But given the
historical context, the optimism wasn't entirely unjustified. To
understand why, we need to quickly recap how the architecture has
evolved over time.
</p>
<p>
Dogen started out divided into three very distinct parts: the
frontends (Dia, JSON), the middle-end (<code>yarn</code>) and the backends (C++,
C#). The "pipeline" metaphor guided our design because we saw Dogen
very much like a compiler, with its frontend, middle-end and backend
stages. This was very handy as it meant we could test all stages of
the pipeline in isolation. Composition was done by orchestrating
frontend, middle-end and backends, at a higher level. This
architecture had very good properties when it came to testability and
debuggability: we'd start by running the entire pipeline and locating
the problem; then, one could easily isolate the issue to a specific
component either by looking at the log file, or by dumping the inputs
and outputs of the different stages and sifting through them. As a
result, bug reproduction was very straightforward since we just needed
to record the inputs and create the test at the right level. Whilst
the names of the models and their responsibilities changed over time,
the overall pipeline architecture remained so since the very early
days of Dogen.
</p>
<p>
In parallel to this, a second trend had emerged over the last ten
sprints or so: we moved more and more functionality from the frontends
and backends to the middle-end. The key objective here was <a href="https://en.wikipedia.org/wiki/Don%2527t_repeat_yourself">DRY</a>: we
soon found a lot of commonalities between frontends, driving us to
create a simple frontend intermediate format so that the work was
carried out only once. Not long after, we discovered that backends
suffered from precisely the same malaise, so the same cure begun to be
applied there too. So far so good, as we were following <a href="http://www.dmi.usherb.ca/~sgiroux/COURS/2008/ift785/fichiers/articles/Roberts_EvolvingFrameworks.pdf">Roberts and
Johnson's</a> sage advice:
</p>
<blockquote>
<p>
People develop abstractions by generalizing from concrete
examples. Every attempt to determine the correct abstractions on paper
without actually developing a running system is doomed to failure. No
one is that smart. A framework is a reusable design, so you develop it
by looking at the things it is supposed to be a design of. The more
examples you look at, the more general your framework will be.
</p>
</blockquote>
<p>
The literature was with us and the wind was on our sails: the concrete
code in the frontends and backends was slowly cleaned up, made general
and moved across to the middle-end. As this process took hold, the
middle-end grew and grew in size and responsibilities, just as
everybody else shed them. Before long, we ended up with one big model,
a couple medium-sized models and lots of very small models:
"modelets", we named them. These were models with very little
responsibility other than gluing together one or two things. The
overhead of maintaining a physical component (e.g. static or dynamic
library) for the sake of one or two classes seemed a tad too
high.
</p>
<p>
As we begun to extrapolate the trend somewhat, a vision suddenly
appeared: why not centralise <b>everything</b> in the middle-end? That is:
</p>
<ul class="org-ul">
<li>place <b>all</b> meta-models and transforms in one single central
location, together with their orchestration; call it the "core
model";
</li>
<li>orchestration becomes either helper code or a transform in its own
right;
</li>
<li>within this "core model", provide interfaces that backends and
frontends implement, injecting them dynamically;
</li>
<li>make these new interfaces appear as transform chains themselves
(mostly).
</li>
</ul>
<p>
In this elegant and clean brave new world, we would no longer have
"ends" as such but something more akin to "plugins", dynamically glued
into the "middle-end" via the magic of dependency injection; the
"middle-end" itself would no longer be a "middle" but <i>the</i> center of
everything. Backends and frontends had to merely implement the
interfaces supplied by the core and the system would just magically
sort itself out. The idea seemed amazing and we quickly moved to
implementation.
</p>
<p>
Alas, in our haste to jump into the fray, we had forgotten to heed
Mencken:
</p>
<blockquote>
<p>
[T]here is always a well-known solution to every human problem — neat,
plausible, and wrong.
</p>
</blockquote>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">The Strange Loop</h2>
<div class="outline-text-2" id="text-2">
<p>
One of the biggest downsides of working alone and in your spare time
is the lack of feedback from other developers. And it's not even just
that other developers will teach you lots of new things. No, most
often than not, they'll simply drag you away from the echo chambers
and tunnels of self-reinforcement you carefully craft and curate for
yourself. You are your own intellectual jailer.
</p>
<p>
In the cold light of day, any developer will tell you that creating
cycles is not a good idea, and should not be done without a great deal
of thought. Yet, we managed to create "circular" dependencies between
all components of the system by centralising all responsibilities into
<code>yarn</code>. Now, you may say that these are not "canonically circular" -
and this is probably why the problem was not picked up in the first
place - because <code>yarn</code> provides interfaces for other models to
implement. Well, <a href="https://www.amazon.co.uk/Large-Scale-C-Software-Design-APC/dp/0201633620">Lakos</a> is very helpful here in explaining what is
going on: our <i>logical design</i> had no cycles - because <code>yarn</code> does not
explicitly call any frontends or backends - but the <i>physical design</i>
did have them. And these came at a cost.
</p>
<p>
For starters, it screwed up <a href="https://the-whiteboard.github.io/coding/debugging/2016/04/07/reasonable-code.html">reasonability</a>. Even though frontends and
backends still had their own models, the net result was that we
jumbled up all of the elements of the pipeline into a single model,
making it really hard to tell what's what. Explaining the system to a
new developer now required saying things such as "ah, don't worry
about that part for now, it belongs to the middle-end, but here we are
dealing only with the backends" - a clear code smell. Once a property
of the architecture, reasonability now had to be conveyed in lossy
natural language. Testability and debuggability got screwed up too
because now everything went through one single central model; if you
needed to test a frontend fix you still required building the backends
and middle-end and initialise them too. Our pursuit of clarity muddied
up the waters.
</p>
<p>
To make matters worse, an even more pertinent question arose: just
when exactly should you stop refactoring? In my two decades of
professional development, I had never encountered this problem. In the
real world, you are fortunate if you get a tiny amount of time
allocated to refactoring - most of the time you need to somehow sneak
it in into some overall estimate and hope no one notices. Like sharks,
Project Managers (PM) are bred to smell refactoring efforts from a
mile a way and know how to trim estimates down to the bone. Even when
you are in a greenfield project or just lucky enough to have an
enlightened PM who will bat for you, you still need to contend with
the realities of corporate development: you need to ship, <i>now</i>. No
one gets away with endless refactoring. No one, that is, other than
the Free and Open Source Software Developer.
</p>
<p>
Like many a spare time project, Dogen is my test bed of ideas around
coding and coding processes; a general sandbox to have fun outside of
work. As such - and very much by design - the traditional feedback
loops that exist in the real world need not apply. I wanted to see
what would happen if you coded without <i>any</i> constraints and, in the
end, what I found out was that if you do not self-impose some kind of
halting machinery, <i>you will refactor on forever</i>. In practice,
physics still apply, so your project will eventually die out because
its energy will dissipate across the many refactoring fronts and
entropy will, as always, triumph. But if you really want to keep it at
bay, at least for a little while, you need to preserve energy by
having one single, consistent vision - "wrong" as it may be according
to some metric or other. For, as Voltaire said and we often forget,
"le mieux est l'ennemi du bien".
</p>
<p>
The trouble is that refactoring is made up of a set of engineering
trade-offs, and when you optimise for one thing you'll inevitably make
something else worse. So, first and foremost, you need to make sure
you understand what your trade-offs are, and prioritise
accordingly. Secondly, looking for a global minima in such a gigantic
multidimensional space is impossible, so you need to make do with
local minima. But how do you known you reach a "good enough" point in
that space? You need some kind of conceptual cost function.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Descending the Gradient</h2>
<div class="outline-text-2" id="text-3">
<p>
So it was that we started by defining the key dimensions across which
we were trying to optimise. This can be phrased slightly differently:
given what we now know about the domain and its implementation, what
are the most important characteristics of an idealised physical and
logical design?
</p>
<p>
After some thinking, the final answer was deceptively simple:
</p>
<ul class="org-ul">
<li>the entities of the logical design (models, namespaces, classes,
methods and the like) should reflect what one reads in the
literature of Model Driven Engineering (MDE). That is, a person
competent on the field should find a code base that talks his or
her language.
</li>
<li>logical and physical design should promote reasonability and
isolation, and thus orchestration should be performed via
composition rather than by circular physical dependencies.
</li>
</ul>
<p>
For now, these are the two fundamental pillars guiding the Dogen
architecture; any engineering trade-offs to be made must ensure these
dimensions take precedence. In other words, we can only optimise away
any "modelets" if they do not impact negatively either of these two
dimensions. If they do, then we must discard this refactoring
option. More generally, it is now possible to "cost" all refactoring
activity - a conceptual refactoring gradient descent if you'd like; it
either brings us closer to the local minima or further away. It gave
us a sieve with which to filter the product and sprint backlogs.
</p>
<p>
To cut a rather long story short, we ended up with a "final" - ha,
ha - set of changes to the architecture to get us closer to the local
minima:
</p>
<ul class="org-ul">
<li><b>move away from sewing terms</b>: from the beginning we had used terms
such as <code>knitter</code>, <code>yarn</code> and so forth. These were… colourful, but
did not add any value and detracted us from the first
dimension. This was a painful decision but clearly required if one
is to comply to point one above: we need to replace all sewing terms
with domain specific vocabulary.
</li>
<li><b>reorganise the models into a pipeline</b>: however, instead of simply
going back to the "modelets" of the past, we need to have a deep
think as to what responsibilities belong at what stage of the
pipeline. Perhaps the "modelets" were warning us of design failures.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Conclusion</h2>
<div class="outline-text-2" id="text-4">
<p>
Its never a great feeling when you end a long and arduous sprint only
to figure out you were going in the wrong direction in design
space. In fact, it is rather frustrating. We have many stories in the
product backlog which are really exciting and which will add real
value to the end users - well, at this point, just us really but hey -
yet we seemed to be lost in some kind of refactoring ground hog day,
with no end in sight. However, the main point of Dogen is to teach,
and learn we undoubtedly did.
</p>
<p>
As with anything in the physical world, nothing in software
engineering exists in splendid perfection like some kind of platonic
solid. Perfection belongs to the realm of maths. In engineering,
something can only be described as "fit for purpose", and to do so
requires to first determine best we can what that purpose might
be. So, before you wonder into a refactoring quagmire of your own
making, be sure to have a very firm idea of what your trade-offs are.
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2018-01-03 Wed 15:55</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 25.2.2 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-74185722109410853902017-03-25T12:34:00.000-07:002017-03-25T13:27:06.629-07:00Nerd Food: Northwind, or Using Dogen with ODB - Part IV<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Northwind, or Using Dogen with ODB - Part IV</title>
<!-- 2017-03-25 Sat 20:26 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
So, dear reader, we meet again for the fourth and final instalment of
our series of posts on using Dogen with ODB! And if you missed an
episode - unlikely as it may be - well, fear not for you can always
catch up! Here are the links: <a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with.html">Part I</a>, <a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with_24.html">Part II</a> and <a href="http://mcraveiro.blogspot.co.uk/2017/03/nerd-food-northwind-or-using-dogen-with.html">Part III</a>. But, if
you are too lazy and need a summary: all we've done thus far is to
install and setup an Oracle Express database, populate it with a
schema (and data) and finally code-generate an ORM model with Dogen
and ODB.
</p>
<p>
I guess it would not be entirely unfair to describe our adventure thus
far as a prelude; if nothing else, it was a character building
experience. But <i>now</i> we can finally enjoy the code.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Building Zango</h2>
<div class="outline-text-2" id="text-1">
<p>
Assuming you have checked out <code>zango</code> as described in <a href="http://mcraveiro.blogspot.co.uk/2017/03/nerd-food-northwind-or-using-dogen-with.html">Part III</a> and you
are sitting on its containing directory, you can "configure" the
project fairly simply:
</p>
<pre class="example">
$ . /u01/app/oracle/product/11.2.0/xe/bin/oracle_env.sh
$ cd zango
$ git pull origin master
$ cd build
$ mkdir output
$ cd output
$ CMAKE_INCLUDE_PATH=/full/path/to/local/include CMAKE_LIBRARY_PATH=/full/path/to/local/lib cmake ../.. -G Ninja
-- The C compiler identification is GNU 6.3.0
<lots of CMake output>
-- Generating done
-- Build files have been written to: /path/to/zango/build/output
</pre>
<p>
As always, do not forget to replace <code>/full/path/to/local</code> with your
path to the directory containing the ODB libraries. If all has gone
according to plan, CMake should have found ODB, Boost, Dogen and all
other dependencies we have carefully and painstakingly setup in the
previous three parts.
</p>
<p>
Once the configuration is done, you can fire up Ninja to build:
</p>
<pre class="example">
$ ninja -j5
[1/100] Building CXX object projects/northwind/src/CMakeFiles/northwind.dir/io/category_id_io.cpp.o
<lots of Ninja output>
[98/100] Linking CXX static library projects/northwind/src/libzango.northwind.a
[99/100] Building CXX object CMakeFiles/application.dir/projects/application/main.cpp.o
[100/100] Linking CXX executable application
</pre>
<p>
That was easy! But what exactly have we just built?
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">The "Application"</h2>
<div class="outline-text-2" id="text-2">
<p>
We've created a really simple application to test drive the northwind
model. Of course, this is really <i>not how your production code should
look like</i>, but it'll do just fine for our purposes. We shall start by
reading a password from the command line and then we use it to
instantiate our Oracle database:
</p>
<div class="org-src-container">
<pre class="src src-c++"> const std::string password(argv[1]);
using odb::oracle::database;
std::unique_ptr<database>
db(new database("northwind", password, "XE", "localhost", 1521));
</pre>
</div>
<p>
We then use this database to read all available customers:
</p>
<div class="org-src-container">
<pre class="src src-c++">std::list<zango::northwind::customers>
load_customers(odb::oracle::database& db) {
odb::oracle::transaction t(db.begin());
std::list<zango::northwind::customers> r;
auto rs(db.query<zango::northwind::customers>());
for (auto i(rs.begin ()); i != rs.end (); ++i)
r.push_back(*i);
return r;
}
</pre>
</div>
<p>
Please note that this is a straightforward use of the ODB API, but
barely scratches the surface of what ODB can do. ODB supports all
sorts of weird and wonderful things, including fairly complex queries
and other great features. If you'd like more details on how to use
ODB, you should read its manual: <a href="http://www.codesynthesis.com/products/odb/doc/manual.xhtml">C++ Object Persistence with ODB</a>. It's
extremely comprehensive and very well written.
</p>
<p>
Once we have the customers in memory, we can start to do things with
them. We can for example serialise them to a Boost serialisation
binary archive and read them back out:
</p>
<div class="org-src-container">
<pre class="src src-c++"> boost::filesystem::path file("a_file.bin");
{
boost::filesystem::ofstream os(file);
boost::archive::binary_oarchive oa(os);
oa << customers;
}
std::cout << "Wrote customers to file: "
<< file.generic_string() << std::endl;
std::list<zango::northwind::customers> customers_from_file;
{
boost::filesystem::ifstream is(file);
boost::archive::binary_iarchive ia(is);
ia >> customers_from_file;
}
</pre>
</div>
<p>
This is where hopefully you should start to see the advantages of
Dogen: without writing any code, we have full serialisation support to
all classes in the model - in addition to ODB support, of course.
</p>
<p>
Another very useful feature is to dump objects into a stream:
</p>
<div class="org-src-container">
<pre class="src src-c++"> for (const auto& c : customers_from_file)
std::cout << "Customer: " << c << std::endl;
</pre>
</div>
<p>
The objects are written in JSON, making it easy to post-process the
output with JSON tools such as <a href="https://github.com/stedolan/jq">JQ</a>, resulting in a nicely formatted
string:
</p>
<pre class="example">
{
"__type__": "zango::northwind::customers",
"customer_id": {
"__type__": "zango::northwind::customer_id",
"value": 90
},
"customer_code": "WILMK",
"company_name": "Wilman Kala",
"contact_name": "Matti Karttunen",
"contact_title": "Owner/Marketing Assistant",
"address": "Keskuskatu 45",
"city": "Helsinki",
"region": "",
"postal_code": "21240",
"country": "Finland",
"phone": "90-224 8858",
"fax": "90-224 8858"
}
</pre>
<p>
Dogen supports dumping arbitrarily-nested graphs, so it's great for
logging program state as you go along. We make extensive use of this
in Dogen, since - <i>of course</i> - we use Dogen to develop Dogen. Whilst
this has proven invaluable, we have also hit some limits. For example,
sometimes you may bump into really large and complex objects and JQ
just won't cut it. But the great thing is that you can always dump the
JSON into PostgreSQL - very easily indeed, given the ODB support - and
then run queries on the object using the power of <a href="http://schinckel.net/2014/05/25/querying-json-in-postgres/">JSONB</a>. With a tiny
bit more bother you can also dump the objects into MongoDB.
</p>
<p>
However, with all of this said, it is also important to notice that we
<b>do not support proper JSON serialisation</b> in Dogen at the
moment. This will be added Real-Soon-Now, as we have a real need for
it in production, but its not there yet. At present all you have is
this debug-dumping of objects into streams which <i>happens</i> to be
JSON. It is not real JSON serialisation. Real JSON support is very
high on our priority list though, so expect it to land in the next few
sprints.
</p>
<p>
Another useful Dogen feature is test data generation. This can be
handy for performance testing, for example. Let's say we want to
generate ~10K customers and see how Oracle fares:
</p>
<div class="org-src-container">
<pre class="src src-c++">std::vector<zango::northwind::customers> generate_customers() {
std::vector<zango::northwind::customers> r;
const auto total(10 * 1000);
r.reserve(total);
zango::northwind::customers_generator g;
for (int i = 0; i < total; ++i) {
const auto c(g());
if (i > 100)
r.push_back(g());
}
return r;
}
</pre>
</div>
<p>
Note that we skipped the first hundred customers just to avoid clashes
with the <code>customer_id</code> primary key. Now, thanks to the magic of ODB we
can easily push this data into the database:
</p>
<div class="org-src-container">
<pre class="src src-c++">void save_customers(odb::oracle::database& db,
const std::vector<zango::northwind::customers>& customers) {
odb::transaction t(db.begin());
for (const auto c : customers)
db.persist(c);
t.commit();
}
</pre>
</div>
<p>
<i>Et voilá</i>, we have lots of customers in the database now:
</p>
<div class="org-src-container">
<pre class="src src-c++">SQL> select count(1) from customers;
COUNT(1)
----------
9990
</pre>
</div>
<p>
To be totally honest, this exercise revealed a shortcoming in Dogen:
since it does not know of the size of fields on the database, the
generated test data may in some cases be too big to fit the database
fields:
</p>
<pre class="example">
Saving customers...
terminate called after throwing an instance of 'odb::oracle::database_exception'
what(): 12899: ORA-12899: value too large for column "NORTHWIND"."CUSTOMERS"."CUSTOMER_CODE" (actual: 6, maximum: 5)
</pre>
<p>
I solved this problem with a quick hack for this article (by removing
the prefix used in the test data) but a proper fix is now sitting in
Dogen's product backlog for implementation in the near future.
</p>
<p>
Finally, just for giggles, I decided to push the data we read from
Oracle into <a href="https://redis.io">Redis</a>, an in-memory cache that seems to be all the rage
amongst the Cool-Kid community. To keep things simple, I used the C
API provided by <a href="https://github.com/redis/hiredis">hiredis</a>. Of course, if this was the real world, I
would have used one of the <a href="https://redis.io/clients#c--">many c++ clients</a> for Redis such as
<a href="https://github.com/mrpi/redis-cplusplus-client">redis-cplusplus-client</a> or <a href="https://github.com/cylix/cpp_redis">cpp redis</a>. As it was, I could not find any
Debian packages for them, so I'll just have to pretend I know C. Since
I'm not much of a C programmer, I decided to do a very bad copy and
paste job from <a href="http://stackoverflow.com/questions/26799074/can-we-set-c-int-array-as-a-keys-value-in-redis-by-hiredis">this Stack Overflow article</a>. The result was this beauty
(forgive me in advance, C programmers):
</p>
<pre class="example">
redisContext *c;
redisReply *reply;
const char *hostname = "localhost";
int port = 6379;
struct timeval timeout = { 1, 500000 }; // 1.5 seconds
c = redisConnectWithTimeout(hostname, port, timeout);
if (c == NULL || c->err) {
if (c) {
std::cerr << "Connection error: " << c->errstr << std::endl;
redisFree(c);
} else {
std::cerr << "Connection error: can't allocate redis context"
<< std::endl;
}
return 1;
}
std::ostringstream os;
boost::archive::binary_oarchive oa(os);
oa << customers;
const auto value(os.str());
const std::string key("customers");
reply = (redisReply*)redisCommand(c, "SET %b %b", key.c_str(),
(size_t) key.size(), value.c_str(), (size_t) value.size());
if (!reply)
return REDIS_ERR;
freeReplyObject(reply);
reply = (redisReply*)redisCommand(c, "GET %b", key.c_str(),
(size_t) key.size());
if (!reply)
return REDIS_ERR;
if ( reply->type != REDIS_REPLY_STRING ) {
std::cerr << "ERROR: " << reply->str << std::endl;
return 1;
}
const std::string redis_value(reply->str, reply->len);
std::istringstream is(redis_value);
std::list<zango::northwind::customers> customers_from_redis;
boost::archive::binary_iarchive ia(is);
ia >> customers_from_redis;
std::cout << "Read from redis: " << customers_from_redis.size()
<< std::endl;
std::cout << "Front customer (redis): "
<< customers_from_redis.front() << std::endl;
freeReplyObject(reply);
</pre>
<p>
And it <i>actually works</i>. Here's the output, with manual formatting of
JSON:
</p>
<pre class="example">
Read from redis: 91
Front customer (redis): {
"__type__": "zango::northwind::customers",
"customer_id": {
"__type__": "zango::northwind::customer_id",
"value": 1
},
"customer_code": "ALFKI",
"company_name": "Alfreds Futterkiste",
"contact_name": "Maria Anders",
"contact_title": "Sales Representative",
"address": "Obere Str. 57",
"city": "Berlin",
"region": "",
"postal_code": "12209",
"country": "Germany",
"phone": "030-0074321",
"fax": "030-0076545"
}
</pre>
<p>
As you can hopefully see, in very few lines of code we managed to
connect to a RDBMS, read some data, push it into a stream, read it and
write into Boost Serialization archives and push it into and out of
Redis. All this in fairly efficient C++ code (and some very dodgy C
code, but we'll keep that one quiet).
</p>
<p>
A final note on the CMake targets. Zango comes with a couple of
targets for Dogen and ODB:
</p>
<ul class="org-ul">
<li><code>knit_northwind</code> generates the Dogen code from the model.
</li>
<li><code>odb_northwind</code> runs ODB against the Dogen model, generating the ODB
sources.
</li>
</ul>
<p>
The ODB target is added automatically by Dogen. The Dogen target was
added manually by yours truly, and it is considered good practice to
have one such target when you use Dogen so that other Dogen users know
how to generate your models. You can, of course, name it what you
like, but in the interest of making everyone's life easier its best if
you follow the convention.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Oracle and Bulk Fetching</h2>
<div class="outline-text-2" id="text-3">
<p>
Whilst I was playing around with ODB and Oracle, I noticed a slight
problem: there is no bulk fetch support in the ODB Oracle wrappers at
present; it works for other scenarios, but not for selects. I reported
this to the main ODB mailing list <a href="http://www.codesynthesis.com/pipermail/odb-users/2017-February/003641.html">here</a>. By the by, the ODB community
is very friendly and their mailing list is a very responsive place to
chat about ODB issues.
</p>
<p>
Anyway, so you can have an idea of this problem, here's a fetch of our
generated customers without prefetch support:
</p>
<pre class="example">
<snip>
Generating customers...
Generated customers. Size: 9899
Saving customers...
Saved customers.
Read generated customers. Size: 9990 time (ms): 263.449
<snip>
</pre>
<p>
Remember the <code>263.449</code> for a moment. Now say you delete all rows we
generated:
</p>
<pre class="example">
delete from customers where customer_id > 100;
</pre>
<p>
Then, say you apply to <code>libodb-oracle</code> the hastily-hacked patch I
mentioned <a href="http://www.codesynthesis.com/pipermail/odb-users/2017-February/003648.html">in that mailing list thread</a>. Of course, I am hand-waving
here greatly, as you need to rebuild the library, install the
binaries, rebuild zango, etc, but you get the gist. At any rate,
here's the patch, hard-coding an unscientifically-obtained-prefetch of
5K rows:
</p>
<pre class="example">
--- original_statement.txt 2017-02-09 15:45:56.585765500 +0000
+++ statement.cxx 2017-02-13 10:18:28.447916100 +0000
@@ -1574,18 +1574,29 @@
OCIError* err (conn_.error_handle ());
+ const int prefetchSize(5000);
+ sword r = OCIAttrSet (stmt_,
+ OCI_HTYPE_STMT,
+ (void*)&prefetchSize,
+ sizeof(int),
+ OCI_ATTR_PREFETCH_ROWS,
+ err);
+
+ if (r == OCI_ERROR || r == OCI_INVALID_HANDLE)
+ translate_error (err, r);
+
// @@ Retrieve a single row into the already bound output buffers as an
// optimization? This will avoid multiple server round-trips in the case
// of a single object load.
//
- sword r (OCIStmtExecute (conn_.handle (),
+ r = OCIStmtExecute (conn_.handle (),
stmt_,
err,
0,
0,
0,
0,
- OCI_DEFAULT));
+ OCI_DEFAULT);
if (r == OCI_ERROR || r == OCI_INVALID_HANDLE)
translate_error (conn_, r);
</pre>
<p>
And now re-run the command:
</p>
<pre class="example">
Generated customers. Size: 9899
Saving customers...
Saved customers.
Read generated customers. Size: 9990 time (ms): 40.85
</pre>
<p>
Magic! We're down to <code>40.85</code>. Now that I have a proper setup, I am
going to start working on upstreaming this patch, so that ODB can
expose the fetch configuration for fetching in a similar manner it
already does for other purposes. If you are interested in the gory
technical details, have a look at <a href="http://www.codesynthesis.com/pipermail/odb-users/2017-February/003651.html">Boris' reply</a>.
</p>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Conclusion</h2>
<div class="outline-text-2" id="text-4">
<p>
Hopefully this concluding part gave you an idea of why you might want
to use Dogen with ODB for your modeling needs. Sadly, its not easy to
frame the discussion adequately, so that you have all the required
context in order to place these two tools in the continuum of tooling;
but I'm hoping this series of articles was useful to at least help you
setup Oracle Express in Debian and get an idea of what you can do with
these two tools.
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2017-03-25 Sat 20:26</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 25.1.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-68106160523640559872017-03-20T04:50:00.000-07:002017-03-20T04:50:38.083-07:00Nerd Food: Northwind, or Using Dogen with ODB - Part III<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Northwind, or Using Dogen with ODB - Part III</title>
<!-- 2017-03-19 Sun 23:07 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<table border="0">
<tr>
<td width="50%"></td>
<td width="50%"><p class="verse" style="text-align:left">
<small>
Optimism is an occupational hazard of programming; feedback is the treatment.
<i>-- <a href="https://books.google.co.uk/books?id=G8EL4H4vf7UC&lpg=PA31&ots=j9AMurgRCr&dq=Optimism%20is%20an%20occupational%20hazard%20of%20programming%3B%20feedback%20is%20the%20treatment.%20Extreme%20Programming%20Explained&pg=PA31#v=onepage&q&f=false">Kent Beck</a></i>
</small>
</p></td>
</tr>
</table>
<p>
Welcome to the third part of a series of N blog posts on using Dogen
with ODB against an Oracle database. If you want more than the TL;DR,
please read <a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with.html">Part I</a> and <a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with_24.html">Part II</a>. Otherwise, the story so far can be
quickly summarised as follows: we got our Oracle Express database
installed and set up by adding the required users; we then built the
ODB libraries and installed the ODB compiler.
</p>
<p>
After this rather grand build up, we shall finally get to look at
Dogen - just about. It now seems clear these series will have to be
extended by at least one or two additional instalments in order to
provide a vaguely sensible treatment of the material I had initially
planned to cover. I wasn't expecting N to become so large, but - like
every good software project - I'm now realising you can only estimate
the size of the series properly once you've actually finished it. And
to rub salt into the wounds, before we can proceed we must start by
addressing some of the instructions in the previous posts which were
<i>not quite right</i>.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Est Humanum Errare?</h2>
<div class="outline-text-2" id="text-1">
<p>
The first and foremost point in the <i>errata</i> agenda is concerned with
the additional Oracle packages we downloaded in <a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with.html">Part I</a>. When I had
originally checked my Oracle XE install, I did not find an <code>include</code>
directory, which led me to conclude that a separate download was
required for driver libraries and header files. I did find this state
of affairs somewhat unusual - but then again, it is Oracle we're
talking about here, so "unusual" is the default behaviour. As it turns
out, I was wrong; the header files are indeed part of the Oracle XE
install, just placed under a rather… shall we say, <i>creative</i>,
location: <code>/u01/app/oracle/product/11.2.0/xe/rdbms/public</code>. The
libraries are there too, under the slightly more conventionally named
<code>lib</code> directory.
</p>
<p>
This is quite an important find because the downloaded OCI driver has
moved on to v12 whereas XE is still on v11. There is backwards
compatibility, of course - and everything should work fine connecting
a v12 client against an v11 database - but it does introduce an extra
layer of complexity: you now need to make sure you do not
simultaneously have both v11 and v12 shared objects in the path when
linking and running or else you will start to get some strange
warnings. As usual, we try our best to confuse only one issue at a
time, so we need to make sure we are making use of v11 and purge all
references to v12; this entails recompiling ODB's oracle support.
</p>
<p>
If you followed the instructions on <a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with_24.html">Part II</a> and you have already
installed the ODB Oracle library, you'll need to remove it first:
</p>
<pre class="example">
rm /full/path/to/local/lib/libodb-oracle* /full/path/to/local/include/odb/oracle
</pre>
<p>
Remember to replace <code>/full/path/to/local</code> with the path to your local
directory. Then, you can build by following the instructions as per
previous post, but with <b>one crucial difference</b> at configure time:
point to the Oracle XE directories instead of the external OCI driver
directories:
</p>
<pre class="example">
. /u01/app/oracle/product/11.2.0/xe/bin/oracle_env.sh
LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/xe/lib CPPFLAGS="-I/full/path/to/local/include -I/u01/app/oracle/product/11.2.0/xe/rdbms/public" LDFLAGS="-L/full/path/to/local/lib -L/u01/app/oracle/product/11.2.0/xe/lib" ./configure --prefix=/full/path/to/local
</pre>
<p>
Again, replacing the paths accordingly. If all goes well, the end
result should be an ODB Oracle library that uses the OCI driver from
Oracle XE. You then just need to make sure you have executed
<code>oracle_env.sh</code> before running your binary, but don't worry too much
because I'll remind you later on. Whilst we're on the subject of
Oracle packages, it's worth mentioning that I did a minor update to
<a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with.html">Part I</a>: you didn't need to download SQLPlus separately either, as it
is also included in XE package. So, in conclusion, after a lot of
faffing, it turns out you can get away with just downloading XE and
nothing else.
</p>
<p>
The other minor alteration to what was laid out on the original posts
is that I removed the need for the <code>basic</code> database schema. In truth,
the entities placed in that schema were not adding a lot of value;
their use cases are already covered by the <code>northwind</code> schema, so I
removed the need for two schemas and collapsed them into one.
</p>
<p>
A final note - not quite an <i>errata per se</i> but still, something
worthwhile mentioning. We didn't do a "proper" Oracle setup, so when
you reboot your box you will find that the service is no longer
running. You can easily restart it from the shell, logged in as root:
</p>
<pre class="example">
# cd /etc/init.d/
# ./oracle-xe start
Starting oracle-xe (via systemctl): oracle-xe.service.
</pre>
<p>
Notice that Debian is actually clever enough to integrate the Oracle
scripts with systemd, so you can use the usual tools to find out more
about this service:
</p>
<pre class="example">
# systemctl status oracle-xe
● oracle-xe.service - SYSV: This is a program that is responsible for taking care of
Loaded: loaded (/etc/init.d/oracle-xe; generated; vendor preset: enabled)
Active: active (exited) since Sun 2017-03-12 15:10:47 GMT; 6s ago
Docs: man:systemd-sysv-generator(8)
Process: 16761 ExecStart=/etc/init.d/oracle-xe start (code=exited, status=0/SUCCESS)
Mar 12 15:10:37 lorenz systemd[1]: Starting SYSV: This is a program that is responsible for taking c…e of...
Mar 12 15:10:37 lorenz oracle-xe[16761]: Starting Oracle Net Listener.
Mar 12 15:10:37 lorenz su[16772]: Successful su for oracle by root
Mar 12 15:10:37 lorenz su[16772]: + ??? root:oracle
Mar 12 15:10:37 lorenz su[16772]: pam_unix(su:session): session opened for user oracle by (uid=0)
Mar 12 15:10:39 lorenz oracle-xe[16761]: Starting Oracle Database 11g Express Edition instance.
Mar 12 15:10:39 lorenz su[16800]: Successful su for oracle by root
Mar 12 15:10:39 lorenz su[16800]: + ??? root:oracle
Mar 12 15:10:39 lorenz su[16800]: pam_unix(su:session): session opened for user oracle by (uid=0)
Mar 12 15:10:47 lorenz systemd[1]: Started SYSV: This is a program that is responsible for taking care of.
Hint: Some lines were ellipsized, use -l to show in full.
</pre>
<p>
With all of this said, lets resume from where we left off.
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Installing the Remaining Packages</h2>
<div class="outline-text-2" id="text-2">
<p>
We still have a number of packages to install, but fortunately the
installation steps are easy enough so we'll cover them quickly in this
section. Let's start with Dogen.
</p>
</div>
<div id="outline-container-sec-2-1" class="outline-3">
<h3 id="sec-2-1">Dogen</h3>
<div class="outline-text-3" id="text-2-1">
<p>
Installing Dogen is fairly straightforward: we can just grab the
latest release from BinTray:
</p>
<ul class="org-ul">
<li><a href="https://dl.bintray.com/domaindrivenconsulting/Dogen/0.99.0/dogen_0.99.0_amd64-applications.deb">dogen 0.99.0 amd64-applications.deb</a>
</li>
</ul>
<p>
As it happens, we must install v99 or above because we did a number of
fixes to Dogen as a result of this series of articles; previous
releases had shortcomings with their ODB support.
</p>
<p>
As expected, the setup is pretty standard-fare debian:
</p>
<pre class="example">
$ wget https://dl.bintray.com/domaindrivenconsulting/Dogen/0.99.0/dogen_0.99.0_amd64-applications.deb -O dogen_0.99.0_amd64-applications.deb
$ sudo dpkg -i dogen_0.99.0_amd64-applications.deb
[sudo] password for USER:
Selecting previously unselected package dogen-applications.
(Reading database ... 551550 files and directories currently installed.)
Preparing to unpack dogen_0.99.0_amd64-applications.deb ...
Unpacking dogen-applications (0.99.0) ...
Setting up dogen-applications (0.99.0) ...
</pre>
<p>
If all has gone according to plan, you should see something along the
lines of:
</p>
<pre class="example">
$ dogen.knitter --version
Dogen Knitter v0.99.0
Copyright (C) 2015-2017 Domain Driven Consulting Plc.
Copyright (C) 2012-2015 Marco Craveiro.
License: GPLv3 - GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
</pre>
</div>
</div>
<div id="outline-container-sec-2-2" class="outline-3">
<h3 id="sec-2-2">Dia</h3>
<div class="outline-text-3" id="text-2-2">
<p>
Dogen has multiple frontends - at the time of writing, JSON and
<a href="https://wiki.gnome.org/Apps/Dia/">Dia</a>. We'll stick with Dia because of its visual nature, but keep in
mind that what you can do with Dia you can also do with JSON.
</p>
<p>
A quick word on Dia for those not in the know, copied <i>verbatim</i> from
its home page:
</p>
<blockquote>
<p>
Dia is a GTK+ based diagram creation program for GNU/Linux, MacOS X,
Unix, and Windows, and is released under the GPL license.
</p>
<p>
Dia is roughly inspired by the commercial Windows program 'Visio,'
though more geared towards informal diagrams for casual use. It can be
used to draw many different kinds of diagrams. It currently has
special objects to help draw entity relationship diagrams, UML
diagrams, flowcharts, network diagrams, and many other diagrams.
</p>
</blockquote>
<p>
Dia does not change very often, which means any old version will
do. You should be able to install dia straight off of package manager:
</p>
<pre class="example">
apt-get install dia
</pre>
</div>
</div>
<div id="outline-container-sec-2-3" class="outline-3">
<h3 id="sec-2-3">Other Dependencies</h3>
<div class="outline-text-3" id="text-2-3">
<p>
I had previously assumed Boost to be installed on <a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with_24.html">Part II</a> but - if
nothing else, purely for the sake of completeness - here are the
instructions to set it up, as well as CMake and Ninja. We will need
these in order to build our application, but we won't dwell on them
<i>too</i> much on them or else this series of posts would go on
forever. Pretty much any recent version of Boost and CMake will do, so
again we'll just stick to vanilla package manager:
</p>
<pre class="example">
# apt-get install cmake ninja-build libboost-all-dev
</pre>
<p>
Mind you, you don't <i>actually</i> need the entirety of Boost for this
exercise, but it's just easier this way.
</p>
</div>
</div>
<div id="outline-container-sec-2-4" class="outline-3">
<h3 id="sec-2-4">Emacs and SQL Plus</h3>
<div class="outline-text-3" id="text-2-4">
<p>
Finally, a couple of lose notes which I might as well add here. If you
wish to use SQLPlus from within Emacs - and you should, since the <a href="http://dbspecialists.com/an-introduction-to-emacs-sqli-mode/">SQLi
mode</a> is just simply glorious - you can configure it to use our Oracle
Express database quite easily:
</p>
<pre class="example">
(add-to-list 'exec-path "/u01/app/oracle/product/11.2.0/xe/bin")
(setenv "PATH" (concat (getenv "PATH") ":/u01/app/oracle/product/11.2.0/xe/bin"))
(setenv "ORACLE_HOME" "/u01/app/oracle/product/11.2.0/xe")
</pre>
<p>
After this you will be able to start SQL Plus from Emacs with the
usual <code>sql-oracle</code> command. I recommend you to do at least a minimal
setup of SQL Plus too, to make it usable:
</p>
<pre class="example">
SQL> set linesize 8192
SQL> set pagesize 50000
</pre>
</div>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Introducing Zango</h2>
<div class="outline-text-2" id="text-3">
<p>
After this excruciatingly long setup process, we can at <i>long last</i>
start to create our very "simple" project. Simple in quotes because it
ended up being a tad more complex than what was originally envisioned, so
it was easier to create a <a href="https://github.com/DomainDrivenConsulting/zango">GitHub repository</a> for it. It would have been
preferable to describe it from first principles, but then the
commentary would literally go on for ever. A compromise had to be
made.
</p>
<p>
In order to follow the remainder of this post please clone <code>zango</code>
from GitHub:
</p>
<pre class="example">
git clone git@github.com:DomainDrivenConsulting/zango.git
</pre>
<p>
Zango is a very small Dogen project that builds with CMake. Here are
some notes on the folder structure to help you navigate:
</p>
<ul class="org-ul">
<li><code>build/cmake</code>: additional CMake modules that are not part of the
standard CMake distribution. We need this for ODB, Oracle and
Dogen.
</li>
<li><code>data</code>: some application data that we will use to populate our
database.
</li>
<li><code>projects</code>: where all the code lives.
</li>
<li><code>projects/input_models</code>: location of the Dogen models - in this
case, we just have one. You could, of course, place it anywhere
you'd like, but traditionally this is where they live.
</li>
<li><code>projects/northwind</code>: code output of the Dogen model. This is the
key project of <code>zango</code>.
</li>
<li><code>projects/application</code>: our little command line driver for the
application.
</li>
</ul>
<p>
Now, before we get to look at the code I'd like to first talk about
Northwind and on the relationship between Dogen and ODB.
</p>
</div>
<div id="outline-container-sec-3-1" class="outline-3">
<h3 id="sec-3-1">Northwind Schema</h3>
<div class="outline-text-3" id="text-3-1">
<p>
Microsoft makes the venerable Northwind database available in
CodePlex, at <a href="https://northwinddatabase.codeplex.com/">this</a> location. I found a useful description of the
Northwind database <a href="http://www.geeksengine.com/article/northwind.html">here</a>, which I quote:
</p>
<blockquote>
<p>
Northwind Traders Access database is a sample database that shipped
with Microsoft Office suite. The Northwind database contains the sales
data for a fictitious company called Northwind Traders, which imports
and exports specialty foods from around the world. You can use and
experiment with Access with Northwind database while you're learning
and develop ideas for Access.
</p>
</blockquote>
<p>
If you <i>really</i> want a thorough introduction to Northwind, you could
do worse than reading this paper: <a href="http://eduglopedia.org/resource/cea008e18dee42899aeff7a0c58dddaf8e9b5bc6/download">Adapting the Access Northwind
Database to Support a Database Course</a>. Having said that, for the
purposes of this series we don't really need to dig <i>that</i> deep. In
fact, I'll just present CodePlex's diagram with the tables and their
relationships to give you an idea of the schema - without any further
commentary - and that's more or less all that needs to be said about
it:
</p>
<img src="https://raw.githubusercontent.com/DomainDrivenConsulting/dogen/master/doc/blog/images/Northwind_A4_size_for_Print.png" width="80%" height="80%"/>
<div>Northwind Schema (C) Microsoft.</div>
<p>
Now, <i>in theory</i>, we could use this image to manually extract all the
required information to create a Dia diagram that follows Dogen's
conventions, code-generate that and Bob's your Uncle. However, <i>in
practice</i> we have a problem: the CodePlex project only contains <a href="https://northwinddatabase.codeplex.com/releases/view/71634">the
SQL statements for Microsoft SQL Server</a>. Part of the point of this
exercise is to show that we can load real data from Oracle, rather
than just generate random data, so it would be nice to load up the
"real" Northwind data from their own tables. This would be more of an
"end-to-end" test, as opposed to using ODB to generate the tables, and
Dogen to generate random data which we can push to the database.
</p>
<p>
However, its not entirely trivial to convert T-SQL into Oracle SQL,
and since this is supposed to be a "quick" project on the side -
focusing on ODB and Dogen - I was keen on not spending time on
unrelated activities such as SQL conversions. Fortunately, I found
exactly what I was looking for: a series of posts from <a href="http://www.geeksengine.com/">GeeksEngine</a>
entitled "Convert MS Access Northwind database to Oracle". For
reference, these are as follows:
</p>
<ul class="org-ul">
<li><a href="http://www.geeksengine.com/article/northwind-oracle.html">How the data types in Access Northwind are converted to Oracle</a>
</li>
<li><a href="http://www.geeksengine.com/article/northwind-oracle-2.html">Building Oracle Northwind database objects</a>
</li>
<li><a href="http://www.geeksengine.com/article/northwind-oracle-3.html">Queries to generate aggregated data for Oracle Northwind database</a>
</li>
</ul>
<p>
If you don't care too much about the details, you can just look at the
Oracle SQL statements, available <a href="http://www.geeksengine.com/lg.php?i=oracle-northwind-sql">here</a> and copied across into the Zango
project. I guess it's still worthwhile mentioning that GeeksEngine has
reduced considerably the number of entities in the schema - for which
they provide a rationale. Before we start an in-depth discussions into
the merits of normalisation and de-normalisation and other DBA level
topics, I have to stop you in your tracks. Please do not get too
hung-up on the "quality" of the database schema of Northwind - either
the Microsoft or the GeeksEngine one. The purpose of this exercise is
merely to demonstrate how Dogen and ODB work together to provide an
ORM solution. From this perspective, any vaguely realistic database
schema is adequate - provided it allows us to test-drive all the
features we're interested in. Whether you agree or not with the
decisions the original creators of this schema made is a completely
different matter, which is well beyond the scope of this series of
posts.
</p>
<p>
Right, so now we need to setup our Northwind schema and populate it
with data. For this you can open a SQL Plus session with user
Northwind as explained previously and then run in the SQL script:
</p>
<pre class="example">
@/path/to/zango/data/Oracle-Northwind.sql
</pre>
<p>
Replacing <code>/path/to</code> with the full path to your Zango checkout. This
executes the GeeksEngine script against your local Oracle XE
database. If all has gone well, you should now have a whole load of
tables and data. You can sanity-check the setup by running the
following SQL:
</p>
<pre class="example">
SQL> select table_name from all_tables where owner = 'NORTHWIND';
TABLE_NAME
------------------------------
ORDER_DETAILS
CATEGORIES
CUSTOMERS
EMPLOYEES
SUPPLIERS
SHIPPERS
PRODUCTS
ORDERS
8 rows selected.
SQL> select employee_id, firstname, lastname from employees where rownum <3;
EMPLOYEE_ID FIRSTNAME LASTNAME
----------- ---------- --------------------
1 Nancy Davolio
2 Andrew Fuller
</pre>
<p>
Now then, let's model these entities in Dogen.
</p>
</div>
</div>
<div id="outline-container-sec-3-2" class="outline-3">
<h3 id="sec-3-2">The Dogen Model for Northwind</h3>
<div class="outline-text-3" id="text-3-2">
<p>
Before we proceed, I'm afraid I must make yet another disclaimer: a
proper explanation on how to use Dia (and UML in general) is outside
the scope of these articles, so you'll see me hand-waving quite a
lot. Hopefully the diagrams are sufficiently self-explanatory for you
to get the idea.
</p>
<p>
The process of modeling is simply to take the entities of the
GeeksEngine SQL schema and to model them in Dia, following Dogen's
conventions: each SQL type is converted to what we deemed to be the
closest C++ type. You can open the diagram from the folder
<code>projects/input_models/northwind.dia</code>, but if you haven't got it
handy, here's a screenshot of most of the UML model:
</p>
<img src="https://raw.githubusercontent.com/DomainDrivenConsulting/dogen/master/doc/blog/images/dia_with_northwind.png" width="90%" height="90%"/>
<div>Dogen Northwind model.</div>
<p>
The first point of note in that diagram is - if you pardon the pun -
the UML note.
</p>
<div class="figure">
<p><img src="https://raw.githubusercontent.com/DomainDrivenConsulting/dogen/master/doc/blog/images/odb_uml_note.png" alt="odb_uml_note.png" />
</p>
<p><span class="figure-number">Figure 1:</span> UML Note from northwind model.</p>
</div>
<p>
This configuration is quite important so we'll discuss it a bit more
detail. All lines starting with <code>#DOGEN</code> are an extension mechanism
used to supply meta-data into Dogen. First, lets have a very quick
look at the model's more "general settings":
</p>
<ul class="org-ul">
<li><code>yarn.dia.comment</code>: this is a special command that tells Dogen to
use this UML note as the source code comments for the namespace of
the model (i.e. <code>northwind</code>). Thus the text "The Northwind model is
a…" will become part of a doxygen comment for the namespace.
</li>
<li><code>yarn.dia.external_modules</code>: this places all types into the
top-level namespace <code>northwind</code>.
</li>
<li><code>yarn.input_language</code>: the notation for types used in this model is
C++. We won't delve on this too much, but just keep in mind that
Dogen supports both C++ and C#.
</li>
<li><code>quilt.cpp.enabled</code>: as we are using C++, we must enable it.
</li>
<li><code>quilt.cpp.hash.enabled</code>: we not require this feature for the
purposes of this exercise.
</li>
<li><code>quilt.csharp.enabled</code>: As this is a C++-only model, we will disable
C#.
</li>
<li><code>annotations.profile</code>: Do not worry too much about this knob, it
just sets a lot of default options for this project such as
copyright notices and so forth.
</li>
</ul>
<p>
As promised, you won't fail to notice we hand-waved quite a lot on the
description of these settings. It is very difficult to explain them
properly them without giving the reader an immense amount of context
about Dogen. This, of course, needs to be done - particularly since we
haven't really spent the required time updating the manual. However,
in the interest of keeping this series of posts somewhat focused on
ODB and ORM, we'll just leave it at that, with a promise to create
Dogen-specific posts on them.
</p>
<p>
Talking about ORM, the next batch of settings is exactly related to
that.
</p>
<ul class="org-ul">
<li><code>yarn.orm.database_system</code>: here, we're stating that we're
interested in both <code>oracle</code> and <code>postgresql</code> databases.
</li>
<li><code>yarn.orm.letter_case</code>: this sets the "case" to use for all
identifiers; either <code>upper_case</code> or <code>lower_case</code>. So if you choose
<code>upper_case</code>, all your table names will be in upper case and
vice-versa. This applies to all columns and object names on the
entirety of this model (e.g. <code>customers</code> becomes <code>CUSTOMERS</code> and so
forth).
</li>
<li><code>yarn.orm.schema_name</code>: finally we set the schema name to
<code>northwind</code>. Remember that we are in upper case, so the name becomes
<code>NORTHWIND</code>.
</li>
</ul>
<p>
In addition to the meta-data, the second point worth noticing is that
there is a large overlap between C++ classes and the entities in the
original diagram. For example, we have <code>customers</code>, <code>suppliers</code>,
<code>employees</code> and so forth - the Object-Relational Mapping is very
"linear". This is a characteristic of the Dogen approach to ORM, but
you do not necessarily need to use ODB in this manner; we discuss this
in the next section.
</p>
<p>
If one is to look at a properties of a few attributes in more detail,
one can see additional Dogen meta-data. Take <code>customer_id</code> in the
<code>customers</code> class:
</p>
<div class="figure">
<p><img src="https://raw.githubusercontent.com/DomainDrivenConsulting/dogen/master/doc/blog/images/odb_customer_id_attributes.png" alt="odb_customer_id_attributes.png" />
</p>
<p><span class="figure-number">Figure 2:</span> Properties of customer<sub>id</sub> in the customer class.</p>
</div>
<p>
The setting <code>yarn.orm.is_primary_key</code> tells Dogen that this attribute
is the primary key of the table. Note that we did not use an <code>int</code> as
the type of <code>customer_id</code> but instead made use of a Dogen feature
called "primitives". Primitives are simple wrappers around builtins
and "core" types such as <code>std::string</code>, intended to have little or no
overhead after the compiler is done with them. They are useful when
you want to use domain concepts to clarify intent. For example,
primitives help making it obvious when you try to use a <code>customer_id</code>
when a <code>supplier_id</code> was called for. It's also worth noticing that
<code>customer_id</code> makes use of <code>yarn.orm.is_nullable</code> - settable to <code>true</code>
or <code>false</code>. It results in Dogen telling ODB if a column can be <code>NULL</code>
or not.
</p>
<p>
As we stated, each of the attributes of these classes has the closest
C++ type we could find that maps to the SQL type used in the database
schema. Of course, different developers can make different choices for
these types. For example, were we to store the picture data rather
than a path to the picture as GeeksEngine decided to do, we would use
a <code>std::vector<char></code> instead of a <code>std::string</code>. In that case, we'd
have to perform some additional mapping too:
</p>
<pre class="example">
#DOGEN yarn.orm.type_override=postgresql,BYTEA
#DOGEN yarn.orm.type_override=oracle,BLOB
</pre>
<p>
This tells Dogen about the mapping of the attribute's type to the SQL
type. Dogen then conveys this information to ODB.
</p>
<p>
Dogen's ORM support is still quite young - literally a couple of
sprints old - so there will be cases where you may need to perform
some customisation which is not yet available in its meta-model. In
these cases, you can bypass Dogen and make use of ODB pragmas
directly. As an example, GeeksEngine Oracle schema named a few columns
in <code>Employees</code> without underscores such as <code>FIRSTNAME</code> and
<code>LASTNAME</code>. We want the C++ classes to have the correct names
(e.g. <code>first_name</code>, <code>last_name</code>, etc) so we simply tell ODB that these
columns have different names in the database. Take last name for
example:
</p>
<div class="figure">
<p><img src="https://raw.githubusercontent.com/DomainDrivenConsulting/dogen/master/doc/blog/images/odb_last_name_attributes.png" alt="odb_last_name_attributes.png" />
</p>
<p><span class="figure-number">Figure 3:</span> Properties of last name in the employee class.</p>
</div>
<p>
A final note on composite keys. Predictably, Dogen follows the ODB
approach - in that primary keys that have more than one column must be
expressed as a class on its own right. In <code>northwind</code>, we use the
postfix <code>_key</code> for these class names in order to make them easier to
identify - e.g. <code>order_details_key</code>. You won't fail to notice that
this class has the flag <code>yarn.orm.is_value</code> set. It tells Dogen - and,
by extension, ODB - that it is not really a full-blown type, which
would map it to a table, but instead should be treated like other
value types such as <code>std::string</code>.
</p>
</div>
</div>
<div id="outline-container-sec-3-3" class="outline-3">
<h3 id="sec-3-3">Interlude: Dogen with ODB vs Plain ODB</h3>
<div class="outline-text-3" id="text-3-3">
<p>
"The technical minutiae is all well and good", the inquisitive reader
will say, "but why Dogen <i>and</i> ODB? Why add yet another layer of
indirection when one can just use ODB?" Indeed, it may be puzzling for
there to be a need for a code-generator which generates code for
another code-generator. "Turtles all the way down" and "We
over-engineered it yet again", the crowd chants from the terraces.
</p>
<p>
Let me attempt to address some of these concerns.
</p>
<p>
First, it is important to understand the argument we're trying to make
here: Dogen models benefit greatly from ODB, but its not necessarily
the case that <i>all</i> ODB users benefit from Dogen. Let's start with a
classic ODB use case, which is to take an existing code base and add
ORM support to it. In this scenario it makes no sense to introduce
Dogen; after all, ODB requires only small changes to the original
source code and has the ability to parse very complex C++. And, of
course, using ODB in this manner also allows one to deal with
impedance mismatches between the relational model and the object model
of your domain.
</p>
<p>
Dogen, on the other hand, exists mainly to support <a href="https://en.wikipedia.org/wiki/Model-driven_software_development">Model Driven
Software Development</a> (MDSD), so the modeling process is the
driver. This means that one is expected to start with a Dogen model,
and to use the traditional MDSD techniques for the management of the
life-cycle of your model - and eventually for the generation of
<i>entire software product lines</i>. Of course, you need not buy into the
whole MDSD sales pitch in order to make use of Dogen, but you should
at least understand it in this context. At a bare minimum, it requires
you to think in terms of Domain Models - as <a href="https://en.wikipedia.org/wiki/Domain-driven_design">Domain Driven Development</a>
defines them - and then in terms of "classes of features" required by
the elements of your domain. These we call "facets" in Dogen
parlance. There are many such facets like <code>io</code>, which is the ability
to dump an object's state into a C++ stream - at present using JSON
notation - or <code>serialization</code> which is the ability to serialise an
object using Boost serialisation. It is in this context that ODB
enters the Dogen world. We could, of course, generate ORM mappings
(and SQL) directly from Dogen. But given what we've seen from ODB, it
seems this would be a very large project - or, conversely, we'd have
very poor support, not dealing with a great number of corner cases. By
generating the very minimal and very non-intrusive code that ODB
needs, we benefit from the years of experience accumulated in this
tool whilst at the same time making life easier for Dogen users.
</p>
<p>
Of course, as with all engineering trade-offs, this one is not without
its disadvantages. When things do go wrong you now have more moving
parts to go through when root-causing: was it an error in the diagram,
or was it Dogen, or was it the mapping between Dogen and ODB or was it
ODB? Fortunately, I found that this situation is minimised by the way
in which you end up using Dogen. For instance, all generated code can
be version-controlled, so you can look at the ODB input files
generated by Dogen and observe how they change with changes in the
Dogen model. The Dogen ODB files should also look very much like
regular hand-crafted ODB files - making use of <code>pragmas</code> and so
forth - and you are also required to run ODB manually against
them. Thus, in practice, I have found troubleshooting straightforward
enough that the layers of indirection end up not constituting a real
problem.
</p>
<p>
Finally, its worth pointing out that the Domain Models Dogen generates
have a fairly straightforward shape to them, making the ODB mapping a
lot more trivial that "general" C++ code would have. It is because of
this that we have <code>orm</code> parameters in Dogen, which can expand to
multiple ODB pragmas - the user should not need to worry about that
expansion.
</p>
</div>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Conclusion</h2>
<div class="outline-text-2" id="text-4">
<p>
This part is already becoming quite large, so I'm afraid we need to
stop it here and continue on Part IV. However, we have managed to
address a few of the mistakes of the Oracle setup of previous parts,
introduced the remaining applications that we need to install and then
discussed Northwind - both in terms of its original intent and also in
terms of the Dogen objectives. Finally we provided an explanation of
how Dogen and ODB fit together in a tooling ecosystem.
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2017-03-19 Sun 23:07</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 25.1.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-76354535361410775592017-02-24T04:18:00.000-08:002017-02-24T20:42:05.530-08:00Nerd Food: Northwind, or Using Dogen with ODB - Part II<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Northwind, or Using Dogen with ODB - Part II</title>
<!-- 2017-02-24 Fri 12:32 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
On <a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with.html">Part I</a> of this series, we got our Oracle Express database up and
running against Debian Testing. It involved quite a bit of fiddling
but we seemed to get there in the end. In Part II we shall now finish
the configuration of the Oracle database and set up the application
dependencies. On Part III we will finally get to the Dogen model, and
start to make use of ODB.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">What's in a Schema?</h2>
<div class="outline-text-2" id="text-1">
<p>
The first thing we need to do to our database is add the "application
users". This is a common approach to most server side apps, where we
tend to have "service users" that login to the database and act upon
user requests on their behalf. We can then use audit tables to stamp
the user actions so we can monitor them. We can also have application
level permissions that stop users from doing silly things. This is of
course a step up from the applications in the nineties, where one
would have one database account for each user - allowing all sorts of
weird and wonderful things such as users connecting directly to
databases via ODBC and Excel or Access. I guess nowadays developers
don't even know someone thought this to be a good idea at one point.
</p>
<p>
When I say "database user", most developers exposed to RDBMS'
immediately associate this to a user account. This is of course how
most databases work, but <i>obviously</i> not so with Oracle. In Oracle,
"users" and "schemas" are conflated, so much so it's hard to tell if
there is any difference between them. For the purist RDBMS user, a
schema is a <i>schema</i> - a collection of tables and other database
objects, effectively a namespace - and a user is a <i>user</i> - a person
(real or otherwise) that owns database objects. In Oracle these two
more or less map to the same concept. So when you create a user, you
have created a schema and you can start adding tables to it; and when
you refer to database objects, you prefix them by the user name just
as you would if they belonged to a schema. And, of course, you can
have users that have no database objects for themselves, but which
were granted permission to access database objects from other users.
</p>
<p>
So our first task is to create two schemas; these are required by the
Dogen model which we will use as our "application". They are:
</p>
<ul class="org-ul">
<li><code>basic</code>
</li>
<li><code>northwind</code>
</li>
</ul>
<p>
As I mentioned before, I had created some fairly basic tests for ODB
support in Dogen. Those entities were placed in the aptly named schema
<code>basic</code>. I then decided to extend the schema with something a bit more
meaty, which is where <code>northwind</code> comes in.
</p>
<p>
For the oldest readers, especially those with a Microsoft background,
Northwind is bound to conjure memories. Many of us learned Microsoft
Access at some point in the nineties, and in those days the samples
were pure gold. I was lucky enough to learn about relational databases
in my high-school days, using <a href="https://en.wikipedia.org/wiki/Clipper_(programming_language)">Clipper</a> and <a href="https://en.wikipedia.org/wiki/DBase">dBASE IV</a>, so the transition
to Microsoft Access was more of an exercise in mapping than learning
proper. And that's where Northwind came in. It was a "large" database,
with forms and queries and tables and all sorts of weird and wonderful
things; every time you needed something done to your database you'd
check first to see how Northwind had done it.
</p>
<p>
Now that we are much older, of course, we can see the flaws of
Northwind and even <a href="http://www.hanselman.com/blog/CommunityCallToActionNOTNorthwind.aspx">call for its abolition</a>. But you must remember that
in the nineties there was no Internet for most of us - even dial-up
was pretty rare where I was - and up-to-date IT books were almost as
scarce, so samples were like gold dust. So for all of these historic
reasons and as an homage to my olden days, I decided to implement the
Northwind schema in Dogen and ODB; it may not cover all corner cases,
but it is certainly a step up on my previous <code>basic</code> tests.
</p>
<p>
Enough about history and motivations. Returning to our SQLPlus from
<a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with.html">Part I</a>, where we were logged in as <code>SYSTEM</code>, we start first by
creating a table space and then the users which will make use of that
table space:
</p>
<pre class="example">
SQL> create tablespace tbs_01 datafile 'tbs_f01.dbf' size 200M online;
Tablespace created.
SQL> create user basic identified by "PASSWORD" default tablespace tbs_01 quota 100M on tbs_01;
User created.
SQL> create user northwind identified by "PASSWORD" default tablespace tbs_01 quota 100M on tbs_01;
User created.
</pre>
<p>
Remember to replace <code>PASSWORD</code> with your own passwords. This is of
course a very simple setup; in the real world you would have to take
great care setting the users and table spaces up, including thinking
about temporary table spaces and so forth. But for our simplistic
purposes this suffices. Now we need to grant these users a couple of
useful privileges - again, for a real setup, you'd need quite a bit
more:
</p>
<pre class="example">
SQL> GRANT create session TO basic;
GRANT create session TO basic;
Grant succeeded.
SQL> GRANT create table TO basic;
GRANT create table TO basic;
Grant succeeded.
SQL> GRANT create session TO northwind;
GRANT create session TO northwind;
Grant succeeded.
SQL> GRANT create table TO northwind;
GRANT create table TO northwind;
Grant succeeded.
</pre>
<p>
If all went well, we should now be able to exit the <code>SYSTEM</code> session,
start a new one with one of these users, and play with a test table:
</p>
<pre class="example">
$ sqlplus northwind@XE
SQL*Plus: Release 11.2.0.2.0 Production on Fri Feb 24 10:20:10 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter password:
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL> create table test ( name varchar(10) );
Table created.
SQL> insert into test(name) values ('kianda');
insert into test(name) values ('kianda');
1 row created.
SQL> select * from test;
NAME
----------
kianda
SQL> grant select on test to basic;
Grant succeeded.
SQL> Disconnected from Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
$ sqlplus basic@XE
SQL*Plus: Release 11.2.0.2.0 Production on Fri Feb 24 10:23:04 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter password:
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL> select * from northwind.test;
NAME
----------
kianda
</pre>
<p>
This all looks quite promising. To recap, we logged in with user
<code>northwind</code>, created a table, inserted some random data and selected
it back; all looked ok. Then for good measure, we granted the rights
to see this test table to user <code>basic</code>; logged in as that user and
selected the test table, with the expected results.
</p>
<p>
At this point we consider our Oracle setup completed and we're ready
to enter the application world.
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Enter ODB</h2>
<div class="outline-text-2" id="text-2">
<p>
Setting up ODB is fairly easy, especially if you are on Debian: you
can simply obtain it from <code>apt-get</code> or <code>synaptic</code>. The only slight
snag is, I could not find the oracle dependencies
(i.e. <code>libodb-oracle</code>). Likely this is because they depend on OCI,
which is non-free, so Debian either does not bother to package it at
all or you need some kind of special (non-free) repo for it. As it
was, instead of losing myself on wild goose chases, I thought easier
to build from source. And since I had to build <i>one</i> from source,
might as well build <i>all</i> (or almost all) to demonstrate the whole
process from scratch as it is pretty straightforward, really.
</p>
<p>
Before we proceed, one warning: best if you either use your package
manager <i>or</i> build from source. You should probably only mix-and-match
if you really know what you are doing; if you do and things get
tangled up, it may take you a long while to figure out the source of
your woes.
</p>
<p>
So, the manual approach. I first started by revisiting my previous
notes on building ODB; as it happens, I had covered installing ODB
from source previously <a href="http://mcraveiro.blogspot.co.uk/2013/03/nerd-food-installing-latest-stable-odb.html">here</a> for version 2.2. However, those
instructions have largely bit-rotted at the Dogen end and things have
changed slightly since that post, so a revisit is worthwhile.
</p>
<p>
As usual, we start by grabbing all of the packages from the main <a href="http://www.codesynthesis.com/products/odb/download.xhtml">ODB
website</a>:
</p>
<ul class="org-ul">
<li><a href="http://www.codesynthesis.com/download/odb/2.4/odb_2.4.0-1_amd64.deb">odb 2.4.0-1 amd64.deb</a>: the ODB compiler itself.
</li>
<li><a href="http://www.codesynthesis.com/download/odb/2.4/libodb-2.4.0.tar.gz">libodb-2.4.0</a>: the main ODB library, required by all backends.
</li>
<li><a href="http://www.codesynthesis.com/download/odb/2.4/libodb-pgsql-2.4.0.tar.gz">libodb-pgsql-2.4.0</a>: the PostgreSQL backend. We don't need it today,
of course, but since PostgreSQL is my DB of choice I always install
it.
</li>
<li><a href="http://www.codesynthesis.com/download/odb/2.4/libodb-oracle-2.4.0.tar.gz">libodb-oracle-2.4.0</a>: the Oracle backend. We will need this one.
</li>
<li><a href="http://www.codesynthesis.com/download/odb/2.4/libodb-boost-2.4.0.tar.gz">libodb-boost-2.4.0</a>: the ODB boost profile. This allows using boost
types in your Dogen model and having ODB do the right thing in terms
of ORM mapping. Our Northwind model does not use boost at present,
but I intend to change it as soon as possible as this is a very
important feature for customers.
</li>
</ul>
<p>
Of course, if you are too lazy to click on buttons, just use <code>wget</code>:
</p>
<pre class="example">
$ mkdir odb
$ cd odb
$ wget http://www.codesynthesis.com/download/odb/2.4/odb_2.4.0-1_amd64.deb -O odb_2.4.0-1_amd64.deb
$ wget http://www.codesynthesis.com/download/odb/2.4/libodb-2.4.0.tar.gz -O libodb-2.4.0.tar.gz
$ wget http://www.codesynthesis.com/download/odb/2.4/libodb-pgsql-2.4.0.tar.gz -O libodb-pgsql-2.4.0.tar.gz
$ wget http://www.codesynthesis.com/download/odb/2.4/libodb-oracle-2.4.0.tar.gz -O libodb-oracle-2.4.0.tar.gz
$ wget http://www.codesynthesis.com/download/odb/2.4/libodb-boost-2.4.0.tar.gz -O libodb-boost-2.4.0.tar.gz
</pre>
<p>
We start with the DEB, as simple as always:
</p>
<pre class="example">
# dpkg -i odb_2.4.0-1_amd64.deb
Selecting previously unselected package odb.
(Reading database ... 549841 files and directories currently installed.)
Preparing to unpack odb_2.4.0-1_amd64.deb ...
Unpacking odb (2.4.0-1) ...
Setting up odb (2.4.0-1) ...
Processing triggers for man-db (2.7.6.1-2) ...
</pre>
<p>
I tend to store locally built software under my home directory, so
that's where we'll place the libraries:
</p>
<pre class="example">
$ mkdir ~/local
$ tar -xaf libodb-2.4.0.tar.gz
$ cd libodb-2.4.0/
$ ./configure --prefix=/full/path/to/local
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-2.4.0'
$ make install
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-2.4.0'
</pre>
<p>
Remember to replace <code>/full/path/to/local</code> with your installation
directory. The process is similar for the other three packages, with
one crucial difference: you need to ensure the environment variables
are set to place all required dependencies in the include and link
path. This is achieved via the venerable environment variables
<code>CPPFLAGS</code> and <code>LDFLAGS</code> (and <code>LD_LIBRARY_PATH</code> as we shall see). You
may bump into <code>--with-libodb</code>. However, be careful; the documentation
states:
</p>
<blockquote>
<p>
If these libraries are not installed and you would like to use their
build directories instead, you can use the <code>--with-libodb</code>, and
<code>--with-boost</code> configure options to specify their locations, for
example:
</p>
<p>
<code>./configure --with-boost=/tmp/boost</code>
</p>
</blockquote>
<p>
So if you did <code>make install</code>, you need the environment variables
instead.
</p>
<p>
Without further ado, here are the shell commands. First boost; do note
I am relying on the presence of Debian's system boost; if you have a
local build of boost, which is not in the flags below, you will also
need to add a path to it.
</p>
<pre class="example">
$ cd ..
$ tar -xaf libodb-boost-2.4.0.tar.gz
$ cd libodb-boost-2.4.0/
$ CPPFLAGS=-I/full/path/to/local/include LDFLAGS=-L/full/path/to/local/lib ./configure --prefix=/full/path/to/local
<snip>
config.status: executing libtool-rpath-patch commands
$ make -j5
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-boost-2.4.0'
$ make install
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-boost-2.4.0'
</pre>
<p>
For PostgreSQL again I am relying on the header files installed in
Debian. The commands are:
</p>
<pre class="example">
$ cd ..
$ tar -xaf libodb-pgsql-2.4.0.tar.gz
$ cd libodb-pgsql-2.4.0/
$ CPPFLAGS=-I/full/path/to/local/include LDFLAGS=-L/full/path/to/local/lib ./configure --prefix=/full/path/to/local
<snip>
config.status: executing libtool-rpath-patch commands
$ make -j5
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-pgsql-2.4.0'
$ make install
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-pgsql-2.4.0'
</pre>
<p>
Finally, Oracle. For this we need to supply the locations of the
downloaded drivers or else ODB will not find the Oracle header and
libraries. If you recall from the previous post, they are located in
<code>/usr/include/oracle/12.1/client64</code> and
<code>/usr/lib/oracle/12.1/client64/lib</code>, so we must augment the flags with
those two paths. In addition, I found <code>configure</code> was failing with
errors finding shared objects, so I added <code>LD_LIBRARY_PATH</code> for good
measure. The end result was as follows:
</p>
<pre class="example">
$ cd ..
$ tar -xaf libodb-oracle-2.4.0.tar.gz
$ cd libodb-oracle-2.4.0
$ LD_LIBRARY_PATH=/usr/lib/oracle/12.1/client64/lib CPPFLAGS="-I/full/path/to/local/include -I/usr/include/oracle/12.1/client64" LDFLAGS="-L/full/path/to/local/lib -L/usr/lib/oracle/12.1/client64/lib" ./configure --prefix=/full/path/to/local
<snip>
config.status: executing libtool-rpath-patch commands
$ make -j5
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-oracle-2.4.0'
$ make install
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-oracle-2.4.0'
</pre>
<p>
And there you are; all libraries built and installed into our local
directory, ready to be used.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Conclusion</h2>
<div class="outline-text-2" id="text-3">
<p>
In this part we've configured the Oracle Express database with the
application users, and we sanity checked the configuration. Once that
was out of the way, we built and installed all of the ODB libraries
required by application code.
</p>
<p>
On Part III we will finally start making use of this setup and attempt
to connect to the Oracle database. Stay tuned!
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2017-02-24 Fri 12:32</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 25.1.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-23287395359159255942017-02-23T15:37:00.000-08:002017-03-25T12:54:44.809-07:00Nerd Food: Northwind, or Using Dogen with ODB - Part I<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Northwind, or Using Dogen with ODB - Part I</title>
<!-- 2017-03-25 Sat 19:54 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
Thanks to my first <a href="https://github.com/DomainDrivenConsulting/dogen">Dogen</a> paying customer, I finally got a chance to
work with <a href="http://www.codesynthesis.com/products/odb/">ODB</a> - <a href="http://www.codesynthesis.com">Code Synthesis'</a> amazingly interesting C++
Object-Relational Mapping tool, built on the back of the GCC plugin
system. I've personally always been in awe of what Boris Kolpackov has
achieved, and, of course, it being a South African company made me all
the more keen to use their wares. More importantly: the product just
seems to rock in terms of polish, features and documentation.
</p>
<p>
Astute readers of this blog will point out that Dogen has been
supporting ODB for quite some time. That is indeed true, but since I
haven't used this feature in anger, I wasn't sure how good the support
really was; our fairly trivial database model (<a href="https://raw.githubusercontent.com/DomainDrivenConsulting/dogen/fe6a0c7b48b49554d3211e1a2c2b07fb0f09d502/projects/input_models/database.dia">Dia</a>) explored only a
small fraction of what is possible. Now that I finally had a chance to
use it in production, I needed to expand the tests and try to
replicate the customer's scenario as close as possible. As always in
these situations, there was a snag: instead of using <a href="https://www.postgresql.org/">PostgreSQL</a> - the
RDBMS I had originally used in my Dogen tests - they were using
Oracle. So my first task was to setup Oracle locally on my beloved
Debian Linux.
</p>
<p>
Never one to miss an opportunity, I decided this adventure was worthy
of a quick blog post; it soon turned out to be a series of posts, if I
was to do any justice to this strange and wild experiment, through all
of its twists and turns. But hopefully it is worth the effort, as it
also demonstrates what you can do with Dogen and ODB. And so, here we
are.
</p>
<p>
The first part of the series deals with just trying to convince Oracle
to run on Debian Testing - something that <i>obviously</i> Oracle does not
support out of the box.
</p>
<p>
Before we proceed, a word to the wise: this is a <b>highly experimental</b>
Oracle setup, which I embarked upon just because I could; follow it at
your own peril, and <b>do not use it anywhere near production</b>. More
generally, if you want to stick to the beaten track, use Oracle on
RHEL, CentOS or - god forbid - even Windows. All of that said, if like
me, you are a Debian person, well, there's not much for it other than
to fire off a VM and start looking for those damned faint tracks in
the jungle.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Alien Worlds</h2>
<div class="outline-text-2" id="text-1">
<p>
The very first stumbling block was Oracle itself. After all, for all
the many years of using this RBDMS at work - more than I care to admit
in polite company - I suddenly realised I actually never used it at
home. Or course, Oracle has supported Linux for a little while now;
and the good news is they have a "free" version available: Oracle
Database Express Edition (XE). A quick glance at the Oracle website
revealed RPM's for 64-bits (Intel only, of course). So before anything
else, I decided to brush up my knowledge of Alien.
</p>
<p>
Alien is a debian package that converts RPMs into DEBs. I've used it
in the past for another (lovely) Oracle technology: Java. It had
worked wonderfully well then so I thought I'd give it a try. The
<a href="https://help.ubuntu.com/community/RPM/AlienHowto">Ubuntu Alien HowTo</a> is pretty straightforward, and so is
<a href="https://wiki.debian.org/Alien">Debian's</a>. Basically, obtain Alien:
</p>
<pre class="example">
sudo apt-get install alien
</pre>
<p>
And then apply it to the RPM in question. So the next quest was
obtaining those darn RPMs.
</p>
<p>
Of course, once you move away from the easy world of Free and Open
Source Software, things start to get a bit more complicated. Those
lovely links you can easy Google for don't actually work until you
sign up for an Oracle developer account, asking <i>all sorts</i> of
personal information. Sadly, even listening to Tesla earnings
conferences requires registering these days. Undaunted, I filled all
required fields, obtained my developer account and returned to
download my loot. For Oracle Express it's rather straightforward:
there is a grand-total of one package for Linux 64-bit, so you can't
really go wrong. Here's the link, just in case:
</p>
<ul class="org-ul">
<li><a href="http://www.oracle.com/technetwork/database/database-technologies/express-edition/downloads/index.html">Oracle Express</a>: download the 64-bit Linux RPM
<code>oracle-xe-11.2.0-1.0.x86_64.rpm.zip</code>.
</li>
</ul>
<p>
It is interesting that they decided to zip the RPM but you can easily
unzip it with the <code>unzip</code> tool. The contents are the RPM Alien
expects, plus a few oracle specific files which I decided to ignore
for now:
</p>
<pre class="example">
$ unzip oracle-xe-11.2.0-1.0.x86_64.rpm.zip
Archive: oracle-xe-11.2.0-1.0.x86_64.rpm.zip
creating: Disk1/
creating: Disk1/upgrade/
inflating: Disk1/upgrade/gen_inst.sql
creating: Disk1/response/
inflating: Disk1/response/xe.rsp
inflating: Disk1/oracle-xe-11.2.0-1.0.x86_64.rpm
</pre>
<p>
From a quick glance at the instructions, it appeared the Oracle
Express package contained just the database server - that meant it did
not include a command line client, or the APIs to build applications
that talk to the database. To be fair, this is not an entirely
uncommon approach; Debian also packages the PostgreSQL server
separately from the development libraries. But behind <code>apt-get</code> and
<code>synaptic</code>, installation of packages is all so trivial. Not so when
you have to go through lots of detailed explanations of different
packages and variations. But; onwards! In the <a href="http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html">Instant client</a> page, I
settled on the downloading the following:
</p>
<ul class="org-ul">
<li><i>Basic</i>: client shared libraries. Package:
<code>oracle-instantclient12.1-basic-12.1.0.2.0-1.x86_64.rpm</code>
</li>
<li><i>SQL Plus</i>: command-line client. Package:
<code>oracle-instantclient12.1-sqlplus-12.1.0.2.0-1.x86_64.rpm</code>
</li>
<li><i>SDK</i>: header files to compile code. Package:
<code>oracle-instantclient12.1-devel-12.1.0.2.0-1.x86_64.rpm</code>
</li>
</ul>
<p>
<b>Update</b>: As it turns out, I was wrong on my original expectations,
and you don't really need the SQL Plus package - its already included
with Oracle Express. But I only figured it out much later, so I'll
leave the steps as I originally followed them.
</p>
<p>
With all of these packages in hand, I swiftly got busy with Alien,
only to also rather swiftly hit an issue:
</p>
<pre class="example">
$ cd Disk1
$ alien --scripts oracle-xe-11.2.0-1.0.x86_64.rpm
Must run as root to convert to deb format (or you may use fakeroot).
</pre>
<p>
Yes, sadly you cannot run alien directly as an unprivileged user. I
did not wish to start reading up on <a href="https://wiki.debian.org/FakeRoot">FakeRoot</a> - seems straightforward
enough, to be fair, but hey - so I took the easy way out and ran all
the Alien commands as root. Note also the <code>--scripts</code> to ensure the
scripts will also get converted across. This will bring us some
other… interesting issues, shall we say, but seems worthwhile doing.
</p>
<p>
Quite a few seconds later (hey, it was a 300 MB RPM!), a nice looking
DEB was generated:
</p>
<pre class="example">
# alien --scripts oracle-xe-11.2.0-1.0.x86_64.rpm
oracle-xe_11.2.0-2_amd64.deb generated
</pre>
<p>
A rather promising start. For good measure, I repeated the process
with all RPMs, all with similar results:
</p>
<pre class="example">
# alien oracle-instantclient12.1-basic_12.1.0.2.0-2_amd64.deb
oracle-instantclient12.1-basic_12.1.0.2.0-2_amd64.deb generated
# alien oracle-instantclient12.1-sqlplus-12.1.0.2.0-1.x86_64.rpm
oracle-instantclient12.1-sqlplus_12.1.0.2.0-2_amd64.deb generated
# alien oracle-instantclient12.1-devel-12.1.0.2.0-1.x86_64.rpm
oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb generated
</pre>
<p>
Voila, all DEBs generated. Of course, as the English love to say, the
proof is in the pudding - whatever that means, exactly. So before one
can celebrate, you should try to install the generated packages. That
can be easily done with our old trusty <code>dpkg</code>:
</p>
<pre class="example">
# dpkg -i oracle-xe_11.2.0-2_amd64.deb
Selecting previously unselected package oracle-xe.
(Reading database ... 564824 files and directories currently installed.)
Preparing to unpack oracle-xe_11.2.0-2_amd64.deb ...
Unpacking oracle-xe (11.2.0-2) ...
Setting up oracle-xe (11.2.0-2) ...
Executing post-install steps...
/var/lib/dpkg/info/oracle-xe.postinst: line 114: /sbin/chkconfig: No such file or directory
You must run '/etc/init.d/oracle-xe configure' as the root user to configure the database.
Processing triggers for libc-bin (2.24-8) ...
Processing triggers for systemd (232-8) ...
Processing triggers for desktop-file-utils (0.23-1) ...
Processing triggers for gnome-menus (3.13.3-8) ...
Processing triggers for mime-support (3.60) ...
Processing triggers for mime-support (3.60) ...
</pre>
<p>
As it turns out, it seems the error for chkconfig is related to
setting up the service to autostart. Since this was not a key
requirement for my purposes, I decided to ignore it. The remaining
RPMs - or should I say DEBs - installed beautifully:
</p>
<pre class="example">
# dpkg -i oracle-instantclient12.1-basic_12.1.0.2.0-2_amd64.deb
Selecting previously unselected package oracle-instantclient12.1-basic.
(Reading database ... 564801 files and directories currently installed.)
Preparing to unpack oracle-instantclient12.1-basic_12.1.0.2.0-2_amd64.deb ...
Unpacking oracle-instantclient12.1-basic (12.1.0.2.0-2) ...
Setting up oracle-instantclient12.1-basic (12.1.0.2.0-2) ...
Processing triggers for libc-bin (2.24-8) ...
</pre>
<pre class="example">
# dpkg -i oracle-instantclient12.1-sqlplus_12.1.0.2.0-2_amd64.deb
Selecting previously unselected package oracle-instantclient12.1-sqlplus.
(Reading database ... 567895 files and directories currently installed.)
Preparing to unpack oracle-instantclient12.1-sqlplus_12.1.0.2.0-2_amd64.deb ...
Unpacking oracle-instantclient12.1-sqlplus (12.1.0.2.0-2) ...
Setting up oracle-instantclient12.1-sqlplus (12.1.0.2.0-2) ...
</pre>
<pre class="example">
# dpkg -i oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb
Selecting previously unselected package oracle-instantclient12.1-devel.
(Reading database ... 567903 files and directories currently installed.)
Preparing to unpack oracle-instantclient12.1-devel_12.1.0.2.0-2_amd64.deb ...
Unpacking oracle-instantclient12.1-devel (12.1.0.2.0-2) ...
Setting up oracle-instantclient12.1-devel (12.1.0.2.0-2) ...
</pre>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Talking to the Oracle</h2>
<div class="outline-text-2" id="text-2">
<p>
So, at this point in time we have a bunch of stuff installed in all
sorts of random (read: Oracle-like) locations. The database itself is
under <code>/u01/app/oracle/product/11.2.0/</code>, and all the other packages
seemed to have gone into <code>/usr/lib/oracle/12.1/client64/</code> and
<code>/usr/include/oracle/12.1/client64/</code>. The first task is now to start
the database server. For this we can rely on the scripts we installed
earlier on. However, before we proceed, one little spoiler: we need to
ensure the scripts can find <code>awk</code> at <code>/bin/awk</code> (these days it lives
in <code>/usr/bin/awk</code>). For this we can do a swift (and brutal) hack:
</p>
<pre class="example">
# ln -s /usr/bin/awk /bin/awk
</pre>
<p>
Now we can configure it. I accepted all of the defaults, and setup a
suitably sensible password:
</p>
<pre class="example">
# cd /etc/init.d/
# /etc/init.d/oracle-xe configure
Oracle Database 11g Express Edition Configuration
-------------------------------------------------
This will configure on-boot properties of Oracle Database 11g Express
Edition. The following questions will determine whether the database should
be starting upon system boot, the ports it will use, and the passwords that
will be used for database accounts. Press <Enter> to accept the defaults.
Ctrl-C will abort.
Specify the HTTP port that will be used for Oracle Application Express [8080]:
Specify a port that will be used for the database listener [1521]:
Specify a password to be used for database accounts. Note that the same
password will be used for SYS and SYSTEM. Oracle recommends the use of
different passwords for each database account. This can be done after
initial configuration:
Confirm the password:
Do you want Oracle Database 11g Express Edition to be started on boot (y/n) [y]:y
y
Starting Oracle Net Listener...Done
Configuring database...
Starting Oracle Database 11g Express Edition instance...Done
Installation completed successfully.
</pre>
<p>
Notice how your port 8080 has been hogged. If you are using it for
other work, you may need to move the Oracle Application Express server
to some other port. At any rate, after this I could indeed see a whole
load of Oracle processes running:
</p>
<pre class="example">
$ ps -ef | grep oracle
oracle 20228 1 0 22:35 ? 00:00:00 /u01/app/oracle/product/11.2.0/xe/bin/tnslsnr LISTENER -inhe
oracle 21251 1 0 22:36 ? 00:00:00 xe_pmon_XE
oracle 21253 1 0 22:36 ? 00:00:00 xe_psp0_XE
oracle 21257 1 0 22:36 ? 00:00:00 xe_vktm_XE
oracle 21261 1 0 22:36 ? 00:00:00 xe_gen0_XE
oracle 21263 1 0 22:36 ? 00:00:00 xe_diag_XE
oracle 21265 1 0 22:36 ? 00:00:00 xe_dbrm_XE
oracle 21267 1 0 22:36 ? 00:00:00 xe_dia0_XE
oracle 21269 1 0 22:36 ? 00:00:00 xe_mman_XE
oracle 21271 1 0 22:36 ? 00:00:00 xe_dbw0_XE
oracle 21273 1 0 22:36 ? 00:00:00 xe_lgwr_XE
...
</pre>
<p>
To the untrained eye, this seems like a healthy start; but for more
details, there are also a bunch of useful logs under the Oracle
directories:
</p>
<pre class="example">
# ls -l /u01/app/oracle/product/11.2.0/xe/config/log
ls -l /u01/app/oracle/product/11.2.0/xe/config/log
total 20
-rw-r--r-- 1 oracle dba 1369 Feb 23 22:36 CloneRmanRestore.log
-rw-r--r-- 1 oracle dba 7377 Feb 23 22:36 cloneDBCreation.log
-rw-r--r-- 1 oracle dba 1278 Feb 23 22:36 postDBCreation.log
-rw-r--r-- 1 oracle dba 227 Feb 23 22:36 postScripts.log
</pre>
<p>
Now, at this point in time, if all had gone according to plan we
should be able to connect to our new instance. A typical trick in
Oracle is to use <code>tnsping</code> to validate the setup. For this we need to
know what to ping, and that is where <a href="https://docs.oracle.com/database/121/NETRF/tnsnames.htm#NETRF260">TNS Names</a> comes in handy:
</p>
<pre class="example">
$ cat /u01/app/oracle/product/11.2.0/xe/network/admin/tnsnames.ora
# tnsnames.ora Network Configuration File:
XE =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = lorenz)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = XE)
)
)
...
</pre>
<p>
The magic word is <code>XE</code> (the net service name, i.e. what we will be
connecting against). Now we can simply do:
</p>
<pre class="example">
$ . /u01/app/oracle/product/11.2.0/xe/bin/oracle_env.sh
$ tnsping XE
TNS Ping Utility for Linux: Version 11.2.0.2.0 - Production on 23-FEB-2017 22:52:04
Copyright (c) 1997, 2011, Oracle. All rights reserved.
Used parameter files:
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = lorenz)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = XE)))
OK (0 msec)
</pre>
<p>
Success! Worth noticing that the first step was to call
<code>oracle_env.sh</code> to bring in all the required environment variables of
our Oracle setup.
</p>
<p>
The final test at this stage is to ensure we can connect with SQL
Plus. For this we will just rely on the <code>SYSTEM</code> user.
</p>
<pre class="example">
$ sqlplus SYSTEM@XE
SQL*Plus: Release 11.2.0.2.0 Production on Thu Feb 23 22:56:31 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter password:
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL> select table_name from all_tables where rownum < 4;
select table_name from all_tables where rownum < 4;
TABLE_NAME
------------------------------
ICOL$
CON$
UNDO$
</pre>
<p>
And there you go. We have an absolutely minimal, bare-bones setup of
Oracle Express running on Debian Linux. Worth bearing in mind that if
you want to make use of SQL Plus from within emacs you must make sure
you start emacs on a shell that has all the variables defined in
<code>oracle_env.sh</code>.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Conclusions</h2>
<div class="outline-text-2" id="text-3">
<p>
In this first part we simply setup Oracle Express, and the client
libraries. We also managed to prove that the setup is vaguely working
by connecting to it first at a low-level via TNS ping and then at a
proper client level using SQL Plus. The next part will wrap things up
with the Oracle setup and then move on to ODB.
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2017-03-25 Sat 19:54</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 25.1.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-16738753504678346212016-06-17T02:58:00.000-07:002016-06-17T02:58:30.867-07:00Nerd Food: Interesting...<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Interesting…</title>
<!-- 2016-06-17 Fri 10:56 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
Time to flush all those tabs again. Some interesting stuff I bumped
into recently-ish.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Finance, Economics, Politics</h2>
<div class="outline-text-2" id="text-1">
<ul class="org-ul">
<li><a href="http://www.strongtowns.org/journal/2016/3/20/dbuidkmm60m63enun84oqs07mktdwq">Understanding Growth, part 1</a>: looks very promising although I've
only started parsing it. Also pointed me to - Tomas Sedlacek and the
<a href="http://www.amazon.co.uk/Economics-Good-Evil-Economic-Gilgamesh/dp/019932218X">Economics of Good and Evil</a>. Bought the book, but still reading
it. Seems very thoughtful.
</li>
<li><a href="http://www.bloomberg.com/features/2016-ev-oil-crisis/">Here’s How Electric Cars Will Cause the Next Oil Crisis</a>: Extremely
interesting take on the relationship between electric cars and the
oil price. Its along the lines of articles posted in the past, to be
fair, but still. Basically, it won't take a huge number of sales of
electric cars to start knocking down the oil price. And with Model 3
<a href="http://www.bloomberg.com/news/features/2016-03-22/how-tesla-model-3-can-complete-its-take-over-of-the-u-s-luxury-market">coming out</a>, this all seems quite ominous to the oil producing
countries. Here we go again, Angola.
</li>
<li><a href="http://www.zdnet.com/article/red-hat-becomes-first-2b-open-source-company/">Red Hat becomes first $2b open-source company</a>: I may not use their
wares any more but RedHat will always be one of my favourite
companies. Really happy to see they are growing nicely and hopefully
continuing all of their incredible investment on Linux.
</li>
<li><a href="https://stratechery.com/2016/the-amazon-tax/">The Amazon Tax</a>: Really, <b>really</b> good article about Amazon and their
strategy. If you read only one, read this. Amazon is amazing - and
its dominance is very worrying because they are so good at
executing! See also Bezos letter.
</li>
<li><a href="https://stratechery.com/2016/its-a-tesla/">It’s a Tesla</a>: Great article about Tesla. Some of the usual Fanboyism
we all know and love, of course, but still a lot of very good
points. The core of the article is a interesting comparison between
Tesla and Apple. By the by, not at all convinced about that
dashboard and the launch ceremony itself was a bit sparse too! But,
Model 3 looks great. I'm officially a Stratechery fanboy now.
</li>
<li><a href="http://www.wired.com/2016/04/googles-alphabet-transition-tougher-b-c/">Google’s Alphabet Transition Has Been Tougher Than A-B-C</a>: Great
article on the pains of moving to a single monolithic structure to
something more distributed. In truth, what would one expect with
such a seismic change? And, also, how come it took Google so long to
make this shift? After all, programmers are supposedly taught how
important separation of concerns is. The other very interesting
point is the CED difficulties. These guys were able founders (at
least able enough to get bought out by Google) but seem to fail
badly at the CEO'ing malarky.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Startups et al.</h2>
<div class="outline-text-2" id="text-2">
<ul class="org-ul">
<li><a href="https://stratechery.com/2015/venture-capital-and-the-internets-impact/">Venture capital and the internet’s impact</a>: From the same guys as the
Amazon post, this is also a very interesting take on VCs and the
internet. Highly recommended.
</li>
<li><a href="http://news.efinancialcareers.com/uk-en/240115/believe-not-want-quit-banking-job-tech-unicorn/">Believe me, you do not want to quit your banking job for a tech
unicorn</a>: Stories from the trenches on how Unicorns are not always
rosy. Of course, given it comes from "eFinacialCareers", one must
assume they are talking their book. Cautionary tale, nonetheless.
</li>
<li><a href="http://obscurehandhelds.com/2016/02/sir-clive-sinclair-revives-the-zx-spectrum/">Sir Clive Sinclair Revives the ZX Spectrum</a>: so the Spectrum is back!
I know I shouldn't - there isn't a single logical reason to back it
up - but I just feel like I need to get me one of these…
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">General Coding</h2>
<div class="outline-text-2" id="text-3">
<ul class="org-ul">
<li><a href="http://www.theregister.co.uk/2016/03/24/water_utility_hacked/">Water treatment plant hacked, chemical mix changed for tap supplies</a>:
this is a tad worrying. Can you imagine the amount of systems out
there with vulnerabilities, etc - many of which are connected to the
internet.
</li>
<li><a href="http://www.metzdowd.com/pipermail/cryptography/2016-March/028824.html">On the Impending Crypto Monoculture</a>: Talking about security, very
worrying news from the crypto front. It seems our foundations are
much less solid than expected - and after all the OpenSSL bugs, this
is a surprising statement indeed. Very interesting email on the
subject. The LWN article is a must read too.
</li>
<li><a href="http://lumiverse.io/video/part-1-data-and-architecture">Neural Networks Demystified - Part 1: Data and Architecture</a>: just
started browsing this in my spare time, but it looks very
promising. For the layperson.
</li>
<li><a href="http://www.telegraph.co.uk/technology/2016/03/24/microsofts-teen-girl-ai-turns-into-a-hitler-loving-sex-robot-wit/">Microsoft deletes 'teen girl' AI after it became a Hitler-loving sex
robot within 24 hours</a>: friggin' hilarious in a <i>funny-not-funny</i>
sort of way. This tweet said it best: "Tay" went from "humans are
super cool" to full nazi in <24 hrs and I'm not at all concerned
about the future of AI. – Gerry
</li>
<li><a href="http://www.beepsend.com/2016/04/05/abandoning-gitflow-github-favour-gerrit/">Abandoning Gitflow and GitHub in favour of Gerrit</a>: I've always
wanted to know more about Gerrit but never seem to find the
time. The article explains it to my required extent, contrasting it
with the model I'm more familiar with - GitHub, forks and pull
requests. I must say, still not convinced about Gerrit, but having
said that, it seems there is definitely scope for some kind of
hybrid between the two. A lot of the issues they mention in the
article are definitely pain points for GitHub users.
</li>
<li><a href="http://githubengineering.com/introducing-dgit/">Introducing DGit</a>: OK this one is a puzzling post, from our friends
at GitHub engineering. I'm not sure I get it at all, but seems
amazing. Basically, they talk about all the hard work they've made
to make git distributed. Fine, I'm jesting - but not totally. The
part that leaves no doubts is that GitHub as a whole is a lot more
reliable after this work and can handle a lot more traffic - <b>without</b>
increasing its hardware requirements. Amazing stuff.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Databases</h2>
<div class="outline-text-2" id="text-4">
<ul class="org-ul">
<li><a href="https://www.citusdata.com/blog/17-ozgun-erdogan/403-citus-unforks-postgresql-goes-open-source">Citus Unforks From PostgreSQL, Goes Open Source</a>: Great news
everyone! Sharding in Postgres just became easier with the open
sourcing of Citus! Also worth watching / reading: <a href="https://www.citusdata.com/blog/15-marco-slot/402-interactive-analytics-github-data-using-postgresql-citus">Interactive
Analytics on GitHub Data using PostgreSQL with Citus</a>. This explains
in a very understandable way how you will use Citus to shard.
</li>
<li><a href="http://blog.2ndquadrant.com/parallel-aggregate/">Parallel Aggregate – Getting the most out of your CPUs</a>: The elephant
just keeps getting better and better. Improved scaling on multi-CPU
for a few scenarios is coming on 9.6.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">C++</h2>
<div class="outline-text-2" id="text-5">
<ul class="org-ul">
<li><a href="https://randomascii.wordpress.com/2016/03/24/compiler-bugs-found-when-porting-chromium-to-vc-2015/">Compiler Bugs Found When Porting Chromium to VC++ 2015</a>: great tales
form the frontline. Also good to hear that MS is really responsive
to bug reports. Can't wait to be able to build my C++ 14 code on
Windows…
</li>
<li><a href="https://github.com/haptork/easylambda">EasyLambda</a>: C++ 14 library for data processing. Based on MPI
though. Still, seems like an interesting find.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">Layperson Science</h2>
<div class="outline-text-2" id="text-6">
<ul class="org-ul">
<li><a href="http://www.fastcompany.com/3042443/mendeley-elsevier-and-the-future-of-scholarly-publishing">The Open Publishing Revolution, Now Behind A Billion-Dollar Paywall</a>:
this is very sad news. How science has regressed yet again, now that
Mendeley has been bought out. This saga gets worse and worse. On the
slightly more positive side: <a href="http://techcrunch.com/2014/03/03/from-crowdfunding-to-open-access-startups-are-experimenting-with-academic-research/">From Crowdfunding To Open Access,
Startups Are Experimenting With Academic Research</a>. But will they
succeed?
</li>
<li><a href="https://www.edge.org/conversation/stephen_wolfram-ai-the-future-of-civilization">AI & The Future Of Civilization</a>: Very interesting chat with
Wolfram. Absurdly long but worth a read.
</li>
<li><a href="https://www.quora.com/What-is-the-best-way-to-explain-the-concept-of-manifold-to-a-novice">What is the best way to explain the concept of manifold to a
novice?</a>: Bumped into this in Quora. If only we had more of these. We
need an entire book of "mathematics for lay people".
</li>
<li><a href="http://kernelmag.dailydot.com/issue-sections/staff-editorials/16335/neuroskeptic-neurohype-brain-training-apps/?utm_content=buffer874e9&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer">Why we’re living in an era of neuroscience hype</a>: One that everyone
interested on the field should read. Interesting take on the wave of
progress on the neuroscience front.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-7" class="outline-2">
<h2 id="sec-7">Other</h2>
<div class="outline-text-2" id="text-7">
<ul class="org-ul">
<li><a href="https://medium.com/@thatdavidhopkins/how-a-tv-sitcom-triggered-the-downfall-of-western-civilization-336e8ccf7dd0#.gjnifjo0k">How a TV Sitcom Triggered the Downfall of Western Civilization</a>: OK,
I got to say that with a click bait title as bad as this, I almost
immediately ignored this article. Somehow I went back to it. Its
very long and a bit crazy but its actually very interesting. Friends
(the sitcom) as the signal of the end.
</li>
</ul>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2016-06-17 Fri 10:56</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com1tag:blogger.com,1999:blog-2672427473119923109.post-76121007595178897802016-06-16T06:13:00.000-07:002016-06-17T02:49:51.219-07:00Nerd Food: The Strange Case of the Undefined References<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: The Strange Case of the Undefined References</title>
<!-- 2016-06-16 Thu 14:12 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
As a kid, I loved reading Sherlock Holmes and Poirot novels. Each book
got me completely spellbound, totally immersed and pretty much unable
to do anything else until I finally found out <i>whodunnit</i>. Somehow,
the culprits were never the characters I suspected of. Debugging and
troubleshooting difficult software engineering problems is a lot like
the plot of a crime novel: in both cases you are trying to form a
mental picture of something that happened, with very incomplete
information - the <i>clues</i>; in both cases, experience and attention to
detail is crucial, with many a wrong path taken before the final
eureka moment; and, in both cases too, there is this overwhelming
sense of urgency in figuring out <i>whodunnit</i>. Of course, unlike a
crime novel, we'd all prefer not having to deal with these kinds of
"interesting" issues, but you don't choose the problems - they choose
you.
</p>
<p>
I recently had to deal with one such problem, which annoyed me to no
end until I finally fixed it. It was so annoying I decided it was
worth blogging about - if nothing else, it may save other people from
the same level of pain and misery.
</p>
<p>
A bit of context for those that are new here. <a href="https://github.com/DomainDrivenConsulting/dogen">Dogen</a> is a pet project
that I've been maintaining for a few years now. Like many other C++
projects, it relies on the foundational <a href="http://www.boost.org/">Boost libraries</a>. To be fair,
we rely on other stuff as well - libraries such as <a href="http://xmlsoft.org/">LibXML2</a> and so on -
but Boost is our core C++ dependency and the only one where latest is
greatest, so it tends to cause us the most problems. I've covered my
past woes in terms of <a href="https://mcraveiro.blogspot.co.uk/2015/12/nerd-food-dogen-package-management-saga.html">dependency management</a> and how happy I was to
find <a href="https://www.conan.io/">Conan</a>. And so it was that life was bliss for a number of builds,
until one day…
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">It All Started With a Warning</h2>
<div class="outline-text-2" id="text-1">
<p>
It was a rainy day and I must have been bored because I noticed a
rather innocuous-looking warning on my Travis build, related to Conan:
</p>
<pre class="example">
CMake Warning (dev) in build/output/conanbuildinfo.cmake:
Syntax Warning in cmake code at
/home/travis/build/DomainDrivenConsulting/dogen/build/output/conanbuildinfo.cmake:142:88
Argument not separated from preceding token by whitespace.
Call Stack (most recent call first):
CMakeLists.txt:30 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
</pre>
<p>
Little did I know that this simple discovery would lead to a sequence
of troublesome events and to many a broken build. I decided to <a href="https://github.com/conan-io/conan/issues/138">report</a>
the problem to the Conan developers who, with their usual promptness,
rolled up their sleeves, quickly bounced ideas back and forth and then
did a sterling job in spinning fixes until we got to the bottom of the
issue. Some of the fixes were to Conan itself, whereas some others
were related to rebuilding <a href="https://www.conan.io/source/Boost/1.60.0/lasote/testing">Boost</a>. In the heat of the investigation, I
bumped into some very troubling - and apparently unrelated - linking
errors:
</p>
<pre class="example">
/home/travis/.conan/data/Boost/1.60.0/lasote/stable/package/ebdc9c0c0164b54c29125127c75297f6607946c5/lib/libboost_log.so: undefined reference to `std::invalid_argument::invalid_argument(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@GLIBCXX_3.4.21'
/home/travis/.conan/data/Boost/1.60.0/lasote/stable/package/ebdc9c0c0164b54c29125127c75297f6607946c5/lib/libboost_log.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
</pre>
<p>
The build was littered with errors such as these. But the most
puzzling thing was that I had changed nothing of consequence on my
side and the Conan guys changed very little at their end too! What on
earth was going on?
</p>
<p>
After quite a lot of thinking, Conan's <i><a href="https://github.com/memsharded">memsharded</a></i> came up a
startling conclusion: we've been hit by one of those rare-but-dreadful
ABI-transitions! His comment is worth <a href="https://github.com/conan-io/conan/issues/138#issuecomment-185163060">reading in full</a>, but the crux of
his findings is as follows (copied <i>verbatim</i>):
</p>
<blockquote>
<ul class="org-ul">
<li>Boost packages, generated with travis use docker to manage different
versions of gcc, as gcc 5.2 or gcc 5.3
</li>
<li>Those docker images are using modern linux distros, e.g. > Ubuntu
15.10
</li>
<li>By default, new modern linux distros have switched to the gcc > 5.1
new C++11 ABI, that is libstdc++ is built with gcc > 5.1, usually
named libcxx11, as well as the rest of the system. The libcxx11 ABI
is incompatible with the old gcc < 5.1 libcxx98 ABI.
</li>
<li>Building in such environment links with the new libcxx11 by default.
</li>
<li>Now, we move to our user, package consumer environment, which could
be an Ubuntu 14.04, or a travis VM (12.04). Those distros use a
libcxx98 libstdc++, as a lot of programs of those distros depends on
the old libcxx98 ABI. It is not simple to replace it for the new
one, requiring to rebuild or reinstall large part of the system and
applications. Maybe it could be installed for dev only, and
specified in the build, but I have not been able yet.
</li>
</ul>
</blockquote>
<p>
Reading the above may have given you that sad, sinking feeling: "what
on earth is he on about, I just want to compile my code!", "Why oh why
is C++ so damn complicated!" and so forth. So, for the benefit of
those not in the know, let me try to provide the required background
to fully grok <i>memsharded</i>'s comment.
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">What's this ABI Malarkey Again?</h2>
<div class="outline-text-2" id="text-2">
<p>
This topic may sound oddly familiar to the faithful reader of Nerd
Food and with good reason: we did cover ABIs in the distant past, at a
slightly lower level. The post in question was <a href="https://mcraveiro.blogspot.co.uk/2012/05/nerd-food-mingw-cygwin-and-wine-up-home.html">On MinGW, Cygwin and
Wine</a> and it does provide some useful context to this discussion, but,
if you want a TL;DR, it basically dealt with kernel space and user
space and with things such as the C library. This time round we will
turn our attention to the C++ Standard Library.
</p>
<p>
In addition to specifying the C++ language, the <a href="https://en.wikipedia.org/wiki/C%252B%252B#Standardization">C++ Standard</a> also
defines the <a href="http://en.wikipedia.org/wiki/Application_programming_interface">API</a> of the C++ Standard Library - the classes and their
methods, the functions and so on. The C++ Standard Library is
responsible for providing a set of services for applications compiled
with a C++ compiler. So far, so similar to the C Standard
Library. Where things begin to differ is in the crucial matter of the
<a href="http://en.wikipedia.org/wiki/Application_binary_interface">ABI</a>. But first, lets get a working definition for ABI, just so we are
all on the same page. For this, we can do worse than using <a href="https://www.amazon.co.uk/Linux-System-Programming-Talking-Directly/dp/1449339530">Linux
System Programming</a>:
</p>
<blockquote>
<p>
Whereas an API defines a source interface, an ABI defines the
low-level binary interface between two or more pieces of software on a
particular architecture. It defines how an application interacts with
itself, how an application interacts with the kernel, and how an
application interacts with libraries. An ABI ensures binary
compatibility, guaranteeing that a piece of object code will function
on any system with the same ABI, without requiring recompilation.
</p>
<p>
ABIs are concerned with issues such as calling conventions, byte
ordering, register use, system call invocation, linking, library
behavior, and the binary object format. The calling convention, for
example, defines how functions are invoked, how arguments are passed
to functions, which registers are preserved and which are mangled, and
how the caller retrieves the return value.
</p>
</blockquote>
<p>
The second paragraph is especially crucial. You see, although both the
C and the C++ Standards are somewhat silent on the matter of
specifying an ABI, C tends to have a <i>de facto</i> standard for <i>a given
OS on a given architecture</i>. This may not sound like much and you may
be saying: "what, wait: the same OS on a different architecture has a
different ABI?" Yep, that is indeed the case. If you think about it,
it makes perfect sense; after all, C was carefully designed to be
equivalent to "portable assembler"; in order to achieve maximum
performance, one must not create artificial layers of indirection on
top of the hardware but instead expose it as is. So, by the same
token, two different C compilers working on the same architecture and
OS will tend to agree on the ABI. The reason why is because the OS
will also follow the hardware where it must, for performance reasons;
and where the OS can make choices, it more or less makes the choice
for everybody else. For example, until recently, if you were on
Windows, it did you no good to compile code into an <a href="https://en.wikipedia.org/wiki/Executable_and_Linkable_Format">ELF</a> binary because
the law of the land was <a href="https://en.wikipedia.org/wiki/Portable_Executable">PE</a>. Things have now <a href="https://blogs.msdn.microsoft.com/wsl/2016/04/22/windows-subsystem-for-linux-overview/">changed dramatically</a>, but
the general point remains: the OS and the hardware rule.
</p>
<p>
C++ inherits much of C's approach to efficiency, so at first blush you
may be fooled into thinking it too would have a <i>de facto</i> ABI
standard ("for a given OS, " etc. etc.). However, there are a few
crucial differences that have grave consequences. Let me point out a
few:
</p>
<ul class="org-ul">
<li>C++'s support for genericity - such as function overloading,
templates, etc - is implemented by using <a href="https://en.wikipedia.org/wiki/Name_mangling">name mangling</a>; however,
each compiler tends to have their own mangling scheme.
</li>
<li>implementation details such as the memory layout of objects in the
C++ Standard Library - in particular, as we shall see,
<code>std::string</code> - are important.
</li>
</ul>
<p>
In the past, compiler vendors tended exacerbate differences such as
these; as it was with the <a href="https://en.wikipedia.org/wiki/Unix_wars">UNIX wars</a>, so too during the "C++ wars" did
it make sense to be as incompatible as possible in the never ending
hunt for monetisation. Thus, ABI specifications were kept internal and
were closely guarded secrets. But since then the world has changed. To
a large extent, C++ lost the huge amounts of funding it once had
during the nineties and part of the naughties, and many vendors either
went under or greatly reduced their efforts in this space. Two
compilers emerged as victors: <a href="https://en.wikipedia.org/wiki/Visual_C%252B%252B">MSVC</a> on the Windows platform and - once
the dust of the <a href="http://www.h-online.com/open/features/GCC-We-make-free-software-affordable-1066831.html%253Fpage=3">EGCS fork</a> finally settled - GCC everywhere else. The
excellent quality of GCC across a vast array of platforms and its
strict standards adherence - coupled with a quick response to the
standardisation efforts - resulted in total domination outside of
Windows. So much so that only recently did it meet a true challenger
in <a href="https://en.wikipedia.org/wiki/Clang">Clang</a>. The brave new world in which we now find ourselves in is one
where C++ ABI standardisation is a real possibility - see <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4028.pdf">Defining a
Portable C++ ABI</a>.
</p>
<p>
But pray forgive the old hand, I digress again. The main point is
that, for a given OS on a given architecture, you normally had to
compile all your code with a single compiler; if you did that, you
were good to go. Granted, GCC never made any official promises to keep
its releases ABI-compatible, but in practice we came to rely on the
fact that new and old releases interoperated just fine since the days
of 3.x. And so did Clang, respecting GCC's ABI so carefully it made us
think of them as one happy family. Then, C++-11 arrived.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Mixing and Matching</h2>
<div class="outline-text-2" id="text-3">
<p>
As described in <a href="http://developers.redhat.com/blog/2015/02/05/gcc5-and-the-c11-abi/">GCC5 and the C++11 ABI</a>, this pleasant state of affairs
was too idyllic to last forever:
</p>
<blockquote>
<p>
[…] [S]ome new complexity requirements in the C++11 standard require
ABI changes to several standard library classes to satisfy, most
notably to <code>std::basic_string</code> and <code>std::list</code>. And since
<code>std::basic_string</code> is used widely, much of the standard library is
affected.
</p>
</blockquote>
<p>
On hindsight, the improvements in the <code>std::string</code> implementation are
great; as a grasshopper, I recall spending hours on end debugging my
code in the long forgotten days of EGGS 2.91, only to find out there
was a weird bug in the <a href="https://en.wikipedia.org/wiki/Copy-on-write">COW</a> implementation for my architecture. That
was the first time - and as it happens, the last time too - I found a
library bug, and it made a strong impression on me, at that young age.
These people were not infallible.
</p>
<p>
These days I sit much higher up in the C++ stack. Like many, I didn't
read that carefully the GCC 5 release notes when it came out, relying
as usual on my distro to do the right thing. And, as usual, the
distros largely did, even though, unbeknown to many, a stir was
happening in their world <sup><a id="fnr.1" name="fnr.1" class="footref" href="#fn.1">1</a></sup>. But hey, who reads distro blogs,
right? Hidden comfortably under my Debian Testing lean-to, I was
blissfully unaware of this transition since my code continued to
compile just fine. Also, where things start to get hairy is when you
need to mix and match compiler versions and build settings - and who
on their right mind does that, right?
</p>
<p>
As it happens, this is a situation in which modern C++ users of Travis
may easily find themselves in, stuck as they are on either on Ubuntu
12.04 (2012) or Ubuntu 14.04 (2014). Nick Sarten's <a href="http://genbattle.bitbucket.org/blog/2016/01/17/c++-travis-ci/">blog post</a> rams the
point home in inimitable fashion:
</p>
<blockquote>
<p>
Hold on, did I say GCC 4.6? Clang 3.4?
WHAT YEAR IS IT?
</p>
</blockquote>
<p>
Yes, what year is it indeed. So it is that most of us rely on PPA's to
bring the C++ environment on Travis up to date, such as the Ubuntu
Toolchain:
</p>
<pre class="example">
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
</pre>
<p>
This always seemed like an innocent thing to do but after my linking
errors and <i>memsharded</i> discoveries, one suddenly started to question
everything: what settings did the PPA use to build? What settings were
used to build the Boost Conan packages? With what compiler? In what
distro? The nightmare was endless. It was clear this was going to lead
to tears before bedtime.
</p>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">The Long Road to a Solution</h2>
<div class="outline-text-2" id="text-4">
<p>
Whilst <i>memsharded</i> honed into the problem pretty quickly - less than
a couple of weeks - a complete solution to my woes was a lot more
elusive. In truth, this is the kind of situation where you need long
spells of concentrated effort, so working in your copious spare time
does not help at all. I first tried the easiest approach: to pray that
it would all go away by itself, given enough time. And, lo and behold,
things did work again, for a little while! And then started to fail
again; the Boost package in Conan got rebuilt and the build broke. And
that way it stayed.
</p>
<p>
Once waiting was no longer an option, I had to take it seriously and
started investigating in earnest. Trouble is, when you lose trust in
the compilation settings you then need to methodically validate
absolutely <i>everything</i>, until you bottom out the problem. And that
takes time. Many things were tried, including:
</p>
<ul class="org-ul">
<li>rebuilding Boost locally, attempting to reproduce the issue - to no
avail.
</li>
<li>rebuilding the Conan Boost packages with the old ABI; a fail (<a href="https://github.com/lasote/conan-boost/issues/12">#12</a>).
</li>
<li>reading up a variety of articles on the subject, most of them linked
in this post.
</li>
<li>building the Boost packages locally and exporting them into Travis
using DropBox's public folders. Another fail, but DropBox was a win.
</li>
<li>obtaining the exact same Ubuntu 14.04 image as Travis is using, use
the compiler from the PPA and export Boost to Travis using DropBox
and replicating the problem locally in a VM. This worked.
</li>
</ul>
<p>
Predictably, the final step is the one I should have tried first, but
one is always lazy. Still, all of this got me wondering why had things
been so complicated. Normally one would be able to <code>ldd</code> or <code>nm -C</code>
the binary and figure out the dependencies, but in this case I seemed
to always be pointing to <code>libstdc++.so.6</code> regardless. Most
puzzling. And then I found the Debian wiki page on <a href="https://wiki.debian.org/GCC5">GCC5</a>, which states:
</p>
<blockquote>
<p>
The good news is, that GCC 5 now provides a stable libcxx11 ABI, and
stable support for C++11 (GCC version before 5 called this supported
experimental). This required some changes in the libstdc++ ABI, and
now libstdc++6 provides a dual ABI, the classic libcxx98 ABI, and the
new libcxx11 (GCC 5 (<< 5.1.1-20) only provides the classic libcxx98
ABI). The bad news is that the (experimental) C++11 support in the
classic libcxx98 ABI and the new stable libcxx11 ABIs are not
compatible, and upstream doesn't provide an upgrade path except for
rebuilding. Note that even in the past there were incompatibilities
between g++ versions, but not as fundamental ones as found in the
g++-5 update to stable C++11 support.
</p>
<p>
Using different libstdc++ ABIs in the same object or in the same
library is allowed, as long as you don't try to pass std::list to
something expecting <code>std::__cxx11::list</code> or vice versa. We should
rebuild everything with g++-5 (once it is the default). Using g++-4.9
as a fallback won't be possible in many cases.
</p>
<p>
libstdc++ (>= 5.1.1-20) doesn't change the soname, provides a dual
ABI. Existing C++98 binary packages will continue to work. Building
these packages using g++-5 is expected to work after build failures
are fixed.
</p>
</blockquote>
<p>
The crux is, of course, all the stuff about a <i>dual ABI</i>. I had never
bumped into the <i>dual ABI</i> beast before, and now that I did I'm not
sure I am entirely pleased. It's probably great when it just works,
but it's tricky to troubleshoot when it doesn't: are you linking
against a <code>libstdc++</code> with dual ABI disabled/unsupported? Or is it
some other error you've introduced? Personally, having a completely
different SO name like <i>memsharded</i> had suggested seems like a less
surprising approach - e.g. call it <code>libcxx11</code> instead of
<code>libstdc++</code>. But, as always, one has to play with the cards that were
dealt so there is no point in complaining.
</p>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">Conclusion</h2>
<div class="outline-text-2" id="text-5">
<p>
The Ubuntu 14.04 build of Boost did get us a <a href="https://travis-ci.org/DomainDrivenConsulting/dogen/builds/137848143">green build again</a>, but
for all the joyous celebrations, there is still a grey cloud hovering
above since the mop-up exercise is not completed. I now need to figure
out how to build Boost with Conan on 14.04 and upload this version
into the package manager's repo. However, for now <i>carpe diem</i>. After
so much unproductive time, there is a real need for a few weeks
(months!) of proper coding - the reason why I have a spare time
project in the first place. But some lessons were learned.
</p>
<p>
Firstly, one cannot but feel truly annoyed at <code>${COSMIC_DEITY}</code> for
having to deal with issues such as this. After all, one of the reasons
I prefer C++ to the languages I use at work (C# and Java) is that it
is usually very transparent; normally I can very quickly reproduce,
diagnose and fix a problem in my code. Of course, lord knows this
statement is not true of <i>all</i> C++ code, but at least it tends to be
valid for most Modern C++ - and over the last five years that's all
the C++ I dealt with in anger. It was indeed rather irritating to find
out that the pain has not yet been removed from the language, and on
occasion, even experienced developers get bitten. Hard.
</p>
<p>
A second point worth of note is that in C++ - more so than in any
other language - one cannot just blindly trust the package
manager. There are just so many configuration knobs and buttons for
that to be possible, and one can easily get bitten by assumptions. The
sad truth is that even when using Conan, one should probably upload
one's own packages built with a well understood configuration. True,
this may cost time - but on the other hand, it will avoid wild goose
chases such as this one.
</p>
<p>
Finally, its also important to note that this whole episode
illustrates the sterling job that package maintainers do in
distributions. Paradoxically, their work is often so good that we tend
to be blissfully unaware of its importance. Articles such as
<a href="http://kmkeen.com/maintainers-matter/">Maintainers Matter</a> take a heightened sense of urgency after an
experience like this.
</p>
<p>
The road was narrow, long and troublesome. But, as with all Poirot
novels, there is always that satisfying feeling of finally finding out
<i>whodunnit</i> in the end.
</p>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">Post Script</h2>
<div class="outline-text-2" id="text-6">
<p>
There is one final twist to this story, which adds insult to injury
and further illustrates <code>${COSMIC_DEITY}</code>'s sense of humour. When I
finally attempted to restore our <a href="https://travis-ci.org/DomainDrivenConsulting/dogen/jobs/137859606">clang builds</a>, I found out that LLVM
has <a href="http://lists.llvm.org/pipermail/llvm-dev/2016-June/100400.html">disabled their APT repo</a> for an unspecified length of time:
</p>
<blockquote>
<p>
> TL;DR: APT repo switched off due to excessive load / traffic
</p>
</blockquote>
<p>
There are no alternatives at present to build with a recent
clang. Sometimes one has the feeling that the universe does not want
to play ball. Stiff upper lip and all that; mustn't grumble.
</p>
</div>
</div>
<div id="footnotes">
<h2 class="footnotes">Footnotes: </h2>
<div id="text-footnotes">
<div class="footdef"><sup><a id="fn.1" name="fn.1" class="footnum" href="#fnr.1">1</a></sup> <p class="footpara">
For example, see <a href="http://allanmcrae.com/2015/06/the-case-of-gcc-5-1-and-the-two-c-abis/">The Case of GCC-5.1 and the Two C++ ABIs</a> to
understand Arch's pains.
</p></div>
</div>
</div></div>
<div id="postamble" class="status">
<p class="date">Created: 2016-06-16 Thu 14:12</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-25946820465804384132016-02-08T14:30:00.000-08:002017-02-24T04:37:31.913-08:00Nerd Food: Interesting...<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Northwind, or Using Dogen with ODB - Part II</title>
<!-- 2017-02-24 Fri 12:37 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
On <a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with.html">Part I</a> of this series, we got our Oracle Express database up and
running against Debian Testing. It involved quite a bit of fiddling
but we seemed to get there in the end. In Part II we shall now finish
the configuration of the Oracle database and set up the application
dependencies. On Part III we will finally get to the Dogen model, and
start to make use of ODB.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">What's in a Schema?</h2>
<div class="outline-text-2" id="text-1">
<p>
The first thing we need to do to our database is add the "application
users". This is a common approach to most server side apps, where we
tend to have "service users" that login to the database and act upon
user requests on their behalf. We can then use audit tables to stamp
the user actions so we can monitor them. We can also have application
level permissions that stop users from doing silly things. This is of
course a step up from the applications in the nineties, where one
would have one database account for each user - allowing all sorts of
weird and wonderful things such as users connecting directly to
databases via ODBC and Excel or Access. I guess nowadays developers
don't even know someone thought this to be a good idea at one point.
</p>
<p>
When I say "database user", most developers exposed to RDBMs
immediately associate this to a user account. This is of course how
most databases work, but <i>obviously</i> not so with Oracle. In Oracle,
"users" and "schemas" are conflated, so much so it's hard to tell if
there is any difference between them. For the purist RDBM user, a
schema is a <i>schema</i> - a collection of tables and other database
objects, effectively a namespace - and a user is a <i>user</i> - a person
(real or otherwise) that owns database objects. In Oracle these two
more or less map to the same concept. So when you create a user, you
have created a schema and you can start adding tables to it; and when
you refer to database objects, you prefix them by the user name just
as you would if they belonged to a schema. And, of course, you can
have users that have no database objects for themselves, but which
were granted permission to access database objects from other users.
</p>
<p>
So our first task is to create two schemas; these are required by the
Dogen model which we will use as our "application". They are:
</p>
<ul class="org-ul">
<li><code>basic</code>
</li>
<li><code>northwind</code>
</li>
</ul>
<p>
As I mentioned before, I had created some fairly basic tests for ODB
support in Dogen. Those entities were placed in the aptly named schema
<code>basic</code>. I then decided to extend the schema with something a bit more
meaty, which is where <code>northwind</code> comes in.
</p>
<p>
For the oldest readers, especially those with a Microsoft background,
Northwind is bound to conjure memories. Many of us learned Microsoft
Access at some point in the nineties, and in those days the samples
were pure gold. I was lucky enough to learn about relational databases
in my high-school days, using <a href="https://en.wikipedia.org/wiki/Clipper_(programming_language)">Clipper</a> and <a href="https://en.wikipedia.org/wiki/DBase">dBASE IV</a>, so the transition
to Microsoft Access was more of an exercise in mapping than learning
proper. And that's where Northwind came in. It was a "large" database,
with forms and queries and tables and all sorts of weird and wonderful
things; every time you needed something done to your database you'd
check first to see how Northwind had done it.
</p>
<p>
Now that we are much older, of course, we can see the flaws of
Northwind and even <a href="http://www.hanselman.com/blog/CommunityCallToActionNOTNorthwind.aspx">call for its abolition</a>. But you must remember that
in the nineties there was no Internet for most of us - even dial-up
was pretty rare where I was - and up-to-date IT books were almost as
scarce, so samples were like gold dust. So for all of these historic
reasons and as an homage to my olden days, I decided to implement the
Northwind schema in Dogen and ODB; it may not cover all corner cases,
but it is certainly a step up on my previous <code>basic</code> tests.
</p>
<p>
Enough about history and motivations. Returning to our SQLPlus from
<a href="http://mcraveiro.blogspot.co.uk/2017/02/nerd-food-northwind-or-using-dogen-with.html">Part I</a>, where we were logged in as <code>SYSTEM</code>, we start first by
creating a table space and then the users which will make use of that
table space:
</p>
<pre class="example">
SQL> create tablespace tbs_01 datafile 'tbs_f01.dbf' size 200M online;
Tablespace created.
SQL> create user basic identified by "PASSWORD" default tablespace tbs_01 quota 100M on tbs_01;
User created.
SQL> create user northwind identified by "PASSWORD" default tablespace tbs_01 quota 100M on tbs_01;
User created.
</pre>
<p>
Remember to replace <code>PASSWORD</code> with your own passwords. This is of
course a very simple setup; in the real world you would have to take
great care setting the users and table spaces up, including thinking
about temporary table spaces and so forth. But for our simplistic
purposes this suffices. Now we need to grant these users a couple of
useful privileges - again, for a real setup, you'd need quite a bit
more:
</p>
<pre class="example">
SQL> GRANT create session TO basic;
GRANT create session TO basic;
Grant succeeded.
SQL> GRANT create table TO basic;
GRANT create table TO basic;
Grant succeeded.
SQL> GRANT create session TO northwind;
GRANT create session TO northwind;
Grant succeeded.
SQL> GRANT create table TO northwind;
GRANT create table TO northwind;
Grant succeeded.
</pre>
<p>
If all went well, we should now be able to exit the <code>SYSTEM</code> session,
start a new one with one of these users, and play with a test table:
</p>
<pre class="example">
$ sqlplus northwind@XE
SQL*Plus: Release 11.2.0.2.0 Production on Fri Feb 24 10:20:10 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter password:
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL> create table test ( name varchar(10) );
Table created.
SQL> insert into test(name) values ('kianda');
insert into test(name) values ('kianda');
1 row created.
SQL> select * from test;
NAME
----------
kianda
SQL> grant select on test to basic;
Grant succeeded.
SQL> Disconnected from Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
$ sqlplus basic@XE
SQL*Plus: Release 11.2.0.2.0 Production on Fri Feb 24 10:23:04 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter password:
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL> select * from northwind.test;
NAME
----------
kianda
</pre>
<p>
This all looks quite promising. To recap, we logged in with user
<code>northwind</code>, created a table, inserted some random data and selected
it back; all looked ok. Then for good measure, we granted the rights
to see this test table to user <code>basic</code>; logged in as that user and
selected the test table, with the expected results.
</p>
<p>
At this point we consider our Oracle setup completed and we're ready
to enter the application world.
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Enter ODB</h2>
<div class="outline-text-2" id="text-2">
<p>
Setting up ODB is fairly easy, especially if you are on Debian: you
can simply obtain it from <code>apt-get</code> or <code>synaptic</code>. The only slight
snag is, I could not find the oracle dependencies
(i.e. <code>libodb-oracle</code>). Likely this is because they depend on OCI,
which is non-free, so Debian either does not bother to package it at
all or you need some kind of special (non-free) repo for it. As it
was, instead of losing myself on wild goose chases, I thought easier
to build from source. And since I had to build <i>one</i> from source,
might as well build <i>all</i> (or almost all) to demonstrate the whole
process from scratch as it is pretty straightforward, really.
</p>
<p>
Before we proceed, one warning: when it comes to the libraries, best
if you either use your package manager <i>or</i> build from source. You
should probably only mix-and-match if you really know what you are
doing; if you do and things get tangled up, it may take you a long
while to figure out the source of your woes. Note also that this
warning applies to the support libraries but not to the ODB compiler
itself.
</p>
<p>
So, the manual approach. I first started by revisiting my previous
notes on building ODB; as it happens, I had covered installing ODB
from source previously <a href="http://mcraveiro.blogspot.co.uk/2013/03/nerd-food-installing-latest-stable-odb.html">here</a> for version 2.2. However, those
instructions have largely bit-rotted at the Dogen end and things have
changed slightly since that post, so a revisit is worthwhile.
</p>
<p>
As usual, we start by grabbing all of the packages from the main <a href="http://www.codesynthesis.com/products/odb/download.xhtml">ODB
website</a>:
</p>
<ul class="org-ul">
<li><a href="http://www.codesynthesis.com/download/odb/2.4/odb_2.4.0-1_amd64.deb">odb 2.4.0-1 amd64.deb</a>: the ODB compiler itself.
</li>
<li><a href="http://www.codesynthesis.com/download/odb/2.4/libodb-2.4.0.tar.gz">libodb-2.4.0</a>: the main ODB library, required by all backends.
</li>
<li><a href="http://www.codesynthesis.com/download/odb/2.4/libodb-pgsql-2.4.0.tar.gz">libodb-pgsql-2.4.0</a>: the PostgreSQL backend. We don't need it today,
of course, but since PostgreSQL is my DB of choice I always install
it.
</li>
<li><a href="http://www.codesynthesis.com/download/odb/2.4/libodb-oracle-2.4.0.tar.gz">libodb-oracle-2.4.0</a>: the Oracle backend. We will need this one.
</li>
<li><a href="http://www.codesynthesis.com/download/odb/2.4/libodb-boost-2.4.0.tar.gz">libodb-boost-2.4.0</a>: the ODB boost profile. This allows using boost
types in your Dogen model and having ODB do the right thing in terms
of ORM mapping. Our Northwind model does not use boost at present,
but I intend to change it as soon as possible as this is a very
important feature for customers.
</li>
</ul>
<p>
Of course, if you are too lazy to click on links, just use <code>wget</code>:
</p>
<pre class="example">
$ mkdir odb
$ cd odb
$ wget http://www.codesynthesis.com/download/odb/2.4/odb_2.4.0-1_amd64.deb -O odb_2.4.0-1_amd64.deb
$ wget http://www.codesynthesis.com/download/odb/2.4/libodb-2.4.0.tar.gz -O libodb-2.4.0.tar.gz
$ wget http://www.codesynthesis.com/download/odb/2.4/libodb-pgsql-2.4.0.tar.gz -O libodb-pgsql-2.4.0.tar.gz
$ wget http://www.codesynthesis.com/download/odb/2.4/libodb-oracle-2.4.0.tar.gz -O libodb-oracle-2.4.0.tar.gz
$ wget http://www.codesynthesis.com/download/odb/2.4/libodb-boost-2.4.0.tar.gz -O libodb-boost-2.4.0.tar.gz
</pre>
<p>
We start with the DEB, as simple as always:
</p>
<pre class="example">
# dpkg -i odb_2.4.0-1_amd64.deb
Selecting previously unselected package odb.
(Reading database ... 549841 files and directories currently installed.)
Preparing to unpack odb_2.4.0-1_amd64.deb ...
Unpacking odb (2.4.0-1) ...
Setting up odb (2.4.0-1) ...
Processing triggers for man-db (2.7.6.1-2) ...
</pre>
<p>
I tend to store locally built software under my home directory, so
that's where we'll place the libraries:
</p>
<pre class="example">
$ mkdir ~/local
$ tar -xaf libodb-2.4.0.tar.gz
$ cd libodb-2.4.0/
$ ./configure --prefix=/full/path/to/local
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-2.4.0'
$ make install
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-2.4.0'
</pre>
<p>
Remember to replace <code>/full/path/to/local</code> with your installation
directory. The process is similar for the other three packages, with
one crucial difference: you need to ensure the environment variables
are set to place all required dependencies in the include and link
path. This is achieved via the venerable environment variables
<code>CPPFLAGS</code> and <code>LDFLAGS</code> (and <code>LD_LIBRARY_PATH</code> as we shall see). You
may bump into <code>--with-libodb</code>. However, be careful; the documentation
states:
</p>
<blockquote>
<p>
If these libraries are not installed and you would like to use their
build directories instead, you can use the <code>--with-libodb</code>, and
<code>--with-boost</code> configure options to specify their locations, for
example:
</p>
<p>
<code>./configure --with-boost=/tmp/boost</code>
</p>
</blockquote>
<p>
So if you did <code>make install</code>, you need the environment variables
instead.
</p>
<p>
Without further ado, here are the shell commands. First boost; do note
I am relying on the presence of Debian's system boost; if you have a
local build of boost, which is not in the flags below, you will also
need to add a path to it.
</p>
<pre class="example">
$ cd ..
$ tar -xaf libodb-boost-2.4.0.tar.gz
$ cd libodb-boost-2.4.0/
$ CPPFLAGS=-I/full/path/to/local/include LDFLAGS=-L/full/path/to/local/lib ./configure --prefix=/full/path/to/local
<snip>
config.status: executing libtool-rpath-patch commands
$ make -j5
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-boost-2.4.0'
$ make install
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-boost-2.4.0'
</pre>
<p>
For PostgreSQL again I am relying on the header files installed in
Debian. The commands are:
</p>
<pre class="example">
$ cd ..
$ tar -xaf libodb-pgsql-2.4.0.tar.gz
$ cd libodb-pgsql-2.4.0/
$ CPPFLAGS=-I/full/path/to/local/include LDFLAGS=-L/full/path/to/local/lib ./configure --prefix=/full/path/to/local
<snip>
config.status: executing libtool-rpath-patch commands
$ make -j5
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-pgsql-2.4.0'
$ make install
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-pgsql-2.4.0'
</pre>
<p>
Finally, Oracle. For this we need to supply the locations of the
downloaded drivers or else ODB will not find the Oracle header and
libraries. If you recall from the previous post, they are located in
<code>/usr/include/oracle/12.1/client64</code> and
<code>/usr/lib/oracle/12.1/client64/lib</code>, so we must augment the flags with
those two paths. In addition, I found <code>configure</code> was failing with
errors finding shared objects, so I added <code>LD_LIBRARY_PATH</code> for good
measure. The end result was as follows:
</p>
<pre class="example">
$ cd ..
$ tar -xaf libodb-oracle-2.4.0.tar.gz
$ cd libodb-oracle-2.4.0
$ LD_LIBRARY_PATH=/usr/lib/oracle/12.1/client64/lib CPPFLAGS="-I/full/path/to/local/include -I/usr/include/oracle/12.1/client64" LDFLAGS="-L/full/path/to/local/lib -L/usr/lib/oracle/12.1/client64/lib" ./configure --prefix=/full/path/to/local
<snip>
config.status: executing libtool-rpath-patch commands
$ make -j5
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-oracle-2.4.0'
$ make install
<snip>
make[1]: Leaving directory '/path/to/build/directory/odb/2.4/libodb-oracle-2.4.0'
</pre>
<p>
And there you are; all libraries built and installed into our local
directory, ready to be used.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Conclusion</h2>
<div class="outline-text-2" id="text-3">
<p>
In this part we've configured the Oracle Express database with the
application users, and we sanity checked the configuration. Once that
was out of the way, we built and installed all of the ODB libraries
required by application code.
</p>
<p>
On Part III we will finally start making use of this setup and attempt
to connect to the Oracle database. Stay tuned!
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2017-02-24 Fri 12:37</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 25.1.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-36700670149856032242016-02-08T13:28:00.000-08:002016-02-08T13:41:26.358-08:00Nerd Food: Tooling in Computational Neuroscience - Part III: Data<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Tooling in Computational Neuroscience - Part III: Data</title>
<!-- 2016-02-08 Mon 21:41 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<table border="0">
<tr>
<td width="50%"></td>
<td width="50%"><p class="verse" style="text-align:left">
<small>
In God we trust; all others must bring data. <i>-- <a href="https://en.wikipedia.org/wiki/W._Edwards_Deming">W. Edwards Deming</a></i>
</small>
</p></td>
</tr>
</table>
<p>
Welcome to yet another instalment in our series of posts about tooling
in Computational Neuroscience. Previously, we have discussed
<a href="http://mcraveiro.blogspot.co.uk/2015/11/nerd-food-tooling-in-computational.html">simulators</a> - a <a href="https://en.wikipedia.org/wiki/Neuron_(software)">popular one</a>, in particular - and <a href="http://mcraveiro.blogspot.co.uk/2015/11/nerd-food-tooling-in-computational_30.html">microscopes</a>. We shall
now talk about <i>data</i> in Computational Neuroscience, a seemingly broad
and somewhat mundane topic but one which is central to any attempt in
understanding the <i>status quo</i> of the discipline. The target audience
remains as it was - the lay person - but I'm afraid things are getting
increasingly technical.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">More Data! We Need More Data!</h2>
<div class="outline-text-2" id="text-1">
<p>
Computational Neuroscience by itself is not particularly interesting
if there are no inputs to the models we carefully craft nor detailed
outputs to allow us to know what the models are doing. Similarly, one
needs to be able to use experimental data to inform our modeling
choices and in order to baseline expectations; if this data is not
available, one cannot tell how close or how far models are from the
real thing. As everywhere else, data is of crucial importance here; we
need lots of it and of many different kinds.
</p>
<p>
Once you need data, you soon need to worry about data representation:
how should information be encoded? Clearly, in order for the data to
be useful in a general sense, it must be accompanied by a formal or
informal specification or else users will not know how to interpret
it. Furthermore, given the highly technical nature of the data in
question, the specification must be very precise or the data becomes
useless or even dangerous; "Was that in microns or nanometres?" is not
the sort of question you want to be asking. In a world where producers
and consumers of data can be anywhere geographically, the
specification assumes an ever larger degree of importance.
</p>
<p>
In summary, it is just not practical to allow everyone to come up with
their own data formats:
</p>
<ul class="org-ul">
<li>writing a clear and concise specification for data interchange is
hard work, and requires a lot of experience in both the domain and
the specification process in general. The first attempts would
probably prove to be incomplete, inconsistent or impractical.
</li>
<li>writing code to read and write files according to a specification
and in multiple programming languages is also demanding engineering
work.
</li>
<li>writing code to convert from one data specification to another is
even more complicated because it requires intimate knowledge of
both.
</li>
<li>some data is generated directly by hardware, making it impractical
to adapt to different requirements.
</li>
</ul>
<p>
Another aspect worth highlighting is the "big data" nature of a lot of
the data sets used in this field. Anything to do with the brain gets
pretty complex pretty quickly, and this manifests itself in the data
dimension by having ever larger data sets with greater levels of
detail. On the plus side, thanks to Moore's Law <a href="https://en.wikipedia.org/wiki/Sigmoid_function">sigmoid</a>, detailed
information at all levels is allowing us to answer questions that were
unanswerable not so long ago. The flip side is that all those details
come at a cost: the data sets are becoming <i>huge</i>. For example, the
resolution of the data coming out of microscopy is now so high that a
single data set can take as much as 500 TB. And of course, not only
are individual data sets getting larger and larger, but we are able to
generate more of them at an ever increasing pace because the processes
are more streamlined. It is a fire-hose of data.
</p>
<p>
All of these difficulties are not unique to Computational Neuroscience
or even to Neuroscience as a whole, but the complexity of the domain
has the effect of greatly exacerbating an already thorny problem.
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Neuroinformatics to the Rescue</h2>
<div class="outline-text-2" id="text-2">
<p>
If you think we're exaggerating then think again. The management of
data in Neuroscience is so complex <i>it is a field on its own right</i>,
with the cool-sounding name of <a href="https://en.wikipedia.org/wiki/Neuroinformatics">Neuroinformatics</a>. Wikipedia tells us
that:
</p>
<blockquote>
<p>
Neuroinformatics is a research field concerned with the organization
of neuroscience data by the application of computational models and
analytical tools. These areas of research are important for the
integration and analysis of increasingly large-volume,
high-dimensional, and fine-grain experimental
data. Neuroinformaticians provide computational tools, mathematical
models, and create interoperable databases for clinicians and research
scientists.
</p>
</blockquote>
<p>
In layman's terms, Neuroinformatics concerns itself with Neuroscience
data and the places where said data is to be stored. It is also
implied that one has to deal with a variety of <i>types</i> of data, e.g.:
data from experiments (of which there can be many kinds), model
inputs, model outputs, the models themselves when viewed as data,
etc. The classification of this data is in itself a Neuroinformatics
task. Finally, Neuroinformatics also is responsible for the tooling
necessary to acquire the data, manipulate it, analyse it, visualise it
and so on. Given such a broad definition, one is forced to conclude
that there is a big overlap between Computational Neuroscience - the
modeling activity - and Neuroinformatics - the management of the data
required by it. This lack of clarity is common in science,
particularly as new fields develop; take for example Mathematics and
Computer Science at its inception.
</p>
<p>
In truth, such definitions and demarcations are only as useful as the
tangible benefits they provide. It is perhaps more fruitful to think
of Neuroinformatics as a hat you don on as and when your Computational
Science work requires; the definition is there then to allow one to be
aware of the separation between the analytic work in modeling and the
data storage / retrieval work. For the purposes of this article, we'll
continue to refer to the "Neuroinformatics Scientist" and the
Computational Neuroscientist personas, but bear in mind they may
resolve to the same person in practice.<sup><a id="fnr.1" name="fnr.1" class="footref" href="#fn.1">1</a></sup>
</p>
<p>
Before we move on, I'd like to point out another interesting challenge
Neuroinformatics has to address, and one that is common to all Medical
Sciences: the need to handle human-derived data very carefully. After
all, making data sets available widely must not have implications for
the original patients, so its often a requirement that the data is
<i>de-identified</i>; in the cases where the data is patient sensitive,
additional requirements may be made to users of the data to avoid
leaking this information, such as requiring a registration, etc. This
illustrates the peculiar nature of Neuroinformatics, with the constant
tension between making data as widely available as possible but at the
same time having to ensure there are no side-effects of doing
so. Presumably, <i>Primum non nocere</i> - first, do no harm.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Databases, Repositories and Archives</h2>
<div class="outline-text-2" id="text-3">
<p>
Thanks to the efforts of Neuroinformatics, there is now a wealth of
Neuroscience data available to all on the Internet. The roots of this
growth were sowed in the nineties when labs started sharing research
results online. Sharing always existed in one way or another, of
course, but the rise of the Internet simply changed the magnitude of
the process. It soon became apparent that there was a need to organise
central repositories of data, and to ensure the consistency of the
shared data. Papers with a distinct Neuroinformatics tone were
written, such as <a href="http://www.ncbi.nlm.nih.gov/pubmed/9821633">An on-line archive of reconstructed hippocampal
neurons</a> (1999). Repositories grew, multiplied, morphed and in many
cases died, as these things do, and the evolutionary process left us
with the survivors. I'd like to highlight some of the ones I have
bumped into so far are (with descriptions in their own words):
</p>
<ul class="org-ul">
<li><a href="https://senselab.med.yale.edu/modeldb/">ModelDB</a>: "ModelDB provides an accessible location for storing and
efficiently retrieving computational neuroscience models. ModelDB is
tightly coupled with NeuronDB. Models can be coded in any language
for any environment. Model code can be viewed before downloading and
browsers can be set to auto-launch the models."
</li>
<li><a href="https://senselab.med.yale.edu/neurondb/">NeuronDB</a>: "NeuronDB provides a dynamically searchable database of
three types of neuronal properties: voltage gated conductances,
neurotransmitter receptors, and neurotransmitter substances. It
contains tools that provide for integration of these properties in a
given type of neuron and compartment, and for comparison of
properties across different types of neurons and compartments."
</li>
<li><a href="http://neuromorpho.org/">NeuroMorpho</a>: "NeuroMorpho.Org is a centrally curated inventory of
digitally reconstructed neurons associated with peer-reviewed
publications. It contains contributions from over 100 laboratories
worldwide and is continuously updated as new morphological
reconstructions are collected, published, and shared. To date,
NeuroMorpho.Org is the largest collection of publicly accessible 3D
neuronal reconstructions and associated metadata."
</li>
<li><a href="http://fcon_1000.projects.nitrc.org/">Functional Connectomes Project</a>: "Following the precedent of full
unrestricted data sharing, which has become the norm in molecular
genetics, the FCP entailed the aggregation and public release (via
www.nitrc.org) of over 1200 resting state fMRI (R-fMRI) datasets
collected from 33 sites around the world."
</li>
<li><a href="https://openfmri.org/">OpenfMRI</a>: "[…] project dedicated to the free and open sharing of
functional magnetic resonance imaging (fMRI) datasets, including raw
data."
</li>
<li><a href="http://www.opensourcebrain.org/projects">Open Source Brain</a>: "resource for sharing and collaboratively
developing computational models of neural systems."
</li>
</ul>
<p>
As you can see from this small list - rather incomplete, I'm sure -
there is a wealth of information out there, covering all sorts of
aspects of the brain. We never had so much data as we do today. And,
in many ways, this is fast becoming a problem. As an example, data
from each of Neuroscience's plethora of divisions and sub-fields is
not designed to talk to each other: Electron Microscopy (EM) data is
disconnected from data obtained by Magnetic Resonance Imaging (MRI),
which is also totally separate from connectome
information<sup><a id="fnr.2" name="fnr.2" class="footref" href="#fn.2">2</a></sup> and so forth. In many cases, these
sub-fields have evolved in fairly separate paths, and developed their
own technical vocabulary in isolation and over long periods of time -
an approach perfectly suitable for a "disconnected" world but less
than ideal for a world where multiple sources of data are required to
make sense of complex phenomena. If one can't even agree on what to
call things, how can one be able to explain them?
</p>
<p>
Thus, the early Neuroinformatics approach is best described as
"evolutionary". It is not as if someone sat down and generated a well
defined set of file formats for data interchange, covering all
different aspects of the areas under study. Instead, what has been
emerging is a multitude of file formats in each sub-field, all calling
out for attention, and all of them designed for the immediate goal at
hand rather than the greater good of Neuroscience.
</p>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Taming the Sea of Data</h2>
<div class="outline-text-2" id="text-4">
<p>
From a Software Engineering perspective, an evolutionary approach
makes perfect sense; after all, the <a href="http://c2.com/cgi/wiki?MakeItWorkMakeItRightMakeItFast">Real Programmers had said</a>: "first
make it work, then make it right, and, finally, make it fast." In many
ways, we are reaching the "make it right" phase, with an increasing
interest in efforts towards the creation of broad standards. There
have been several papers and initiatives on the subject, such as the
Neuroscience Information Framework, or NIF, described in a paper: <a href="http://www.neuinfo.org/about/publications/nif_knowledge_environment.pdf">The
Neuroscience Information Framework: A Data and Knowledge Environment
for Neuroscience</a>. The paper outlined a lot of the problems that are
hampering research, such as:
</p>
<ul class="org-ul">
<li>the need for specialised search engines that are domain aware, and
advanced query tools too;
</li>
<li>the need to aid integration and to provide connectivity across
related data and findings;
</li>
<li>a requirement to provide new and enhanced forms of analysing
existing data, as data reuse is extremely important - new insights
can be obtained on already existing data, often long after the data
was generated, and by using it in ways that were not at all
envisioned by the original authors;
</li>
<li>the need to make contribution to online repositories easier;
lowering the "contribution barrier" is important to increase data
availability but must be done in ways that do not compromise the
quality of the data;
</li>
<li>a requirement to make all code open source such that any lab can
make use of it, and the community as a whole can share the
maintenance load;
</li>
<li>a need for an online repository for all tooling, to avoid
reinventing the wheel;
</li>
<li>the need to create a multi-domain standard vocabulary.
</li>
</ul>
<p>
There are many worthwhile points in this paper, and it is highly
recommended to anyone interested in the subject matter. For instance,
the section discussing the design of the NIF also covers the
requirements for any specification that wishes to solve the problems
outlined above. They are worth highlighting as - in my humble and lay
opinion - they are very well thought out.
</p>
<ul class="org-ul">
<li>The design of such a framework must combine technical specifications
choices and broad community support; "open data, access and
exchange, via open source and platform, aid Framework-enabled open
discover for Neuroscience."
</li>
<li>A common framework would reduce costs and enhance benefits of data
sharing and knowledge sharing; it would "reduce the cost/benefit
ration for data acquisition and utilization."
</li>
<li>The framework must be designed by the broader community and with the
needs of this broader community in mind, and it must build upon
prior development in Neuroinformatics.
</li>
<li>A focus on interoperability is crucial, and it is not a static
target but one that must be looked after over time. In addition,
there is also a need to keep in mind that different resources have
very different interoperability potential. In order to maximise
interoperability, we should aim to standardise as much as possible
all aspects of the process such as user interfaces, terminologies,
formats, etc.
</li>
</ul>
<p>
To the untrained eye, the NIF initiative appears to be a great effort
to solve fundamental problems in the field. It also seems to have
spawned and/or helped popularise many useful and lasting resources
such as <a href="http://neuromorpho.org/">NeuroMorpho</a>. However, the impression one gets from the outside
is that the NIF didn't quite fulfil all of its potential. Having said
that, I am keenly looking for up-to-date documents that describe the
current status across all of its many aspects - alas, I have not yet
succeeded in finding any such document. If indeed it is the case that
the initiative petered out, it did highlight a few potential problems
for anyone working in this space:
</p>
<ul class="org-ul">
<li>large undertakings are hard to pull off; small, organic, incremental
changes are easier to do, but of course, that is why we have the
problems we currently have.
</li>
<li>large initiatives require large amounts of funding; work is
technical and very expensive.
</li>
<li>it is not easy to understand NIFs deliverables from looking at their
documentationa and website. One can clearly see it was an ambitious
project, and one which took on the brunt of the problem areas
highlighted above, but perhaps it needed a slightly more
self-contained view of their achievements rather than a
whole-or-nothing approach. This allows preserving some components
even whilst others are failing to gain traction.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">XML strikes back</h2>
<div class="outline-text-2" id="text-5">
<p>
Another interesting attempt to tackle these problems is what I call
the "XML suite". These are basically a set of different XML-based
standards that are able to interoperate and augment each other, a bit
like a stack of building blocks. You can find more details in this
paper: <a href="http://www.brains-minds-media.org/archive/228#documentContent">XML for Model Specification in Neuroscience</a>. Some of the
components of the XML Suite are (with descriptions on their own words,
copied from the above paper and a link for more details):
</p>
<ul class="org-ul">
<li><a href="http://lems.github.io/LEMS/">LEMS</a>: "the Low Entropy Model Specification […] is being developed
to provide a compact, minimally redundant, human-readable,
human-writable, declarative way of expressing models of biological
systems. It differs from other systems such as CellML or SBML in its
requirement to be human writable and the inclusion of basic physical
concepts such as dimensionality and physical nesting as part of the
language."
</li>
<li><a href="https://www.neuroml.org/">NeuroML</a>: "supports the use of declarative model specifications for
neuroscience modeling efforts at different scales, from
intracellular mechanisms to networks of reconstructed neurons."
</li>
<li><a href="https://www.neuroml.org/">MorphML</a>: "provides a common format for exchange of neuronal
morphology data. It can also be used to specify cell structure for
modeling efforts as part of NeuroML."
</li>
<li><a href="http://neurobot.bio.auth.gr/2006/brainml-a-standard-xml-metaformat-for-exchanging-neuroscience-data/">BrainML</a>: "application for representing time series data, spike
trains, experimental protocols, and other data relevant to
neurophysiology experiments."
</li>
<li><a href="http://www.sbml.org/">SBML</a>: "(Systems Biology Markup Language) is an application for
specifying models of biochemical reaction networks such as metabolic
networks, cell-signaling pathways and gene regulatory networks."
</li>
<li><a href="https://www.cellml.org/">CellML</a>: "is designed for the specification of biological models of
cellular and sub-cellular processes such as calcium dynamics,
metabolic pathways, signal transduction, and electrophysiology."
</li>
<li><a href="https://www.w3.org/Math/">MathML</a>: "provides the means for describing the structure and content
of mathematical notation in order to serve, receive, and process
mathematics on the web. Other XML applications often use MathML
language elements for representing mathematical equations."
</li>
</ul>
<p>
A positive aspect of the XML Suite is its "discrete" nature. Each of
these file formats are free to evolve in isolation, and the nature of
their cooperation is very loose in most cases. For example MathML is
not at all related to Neuroscience and has the support of the Maths
community (to some extent). In addition, the "stacking" approach is
also a very interesting one, allowing a good domain focus. For
example, NeuroML is built on top of LEMS, so in theory each of these
should cover different domains and there should be minimal
redundancy.
</p>
<p>
The key challenge for the XML Suite is for each of their components to
find a sustainable user base and sustainable funding to go along with
it. This is a broader problem of Neuroinformatics: researchers do not
want to spend time on work that is not contributing directly to their
research and so the developer pool to do fundamental work on the file
formats is limited. Once the developer pool becomes too limited, the
file format ends up with a small user base because it is not fit for
purpose, and thus starts a downward spiral. This appears to have been
the fate of projects such as BrainML.
</p>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">Conclusion</h2>
<div class="outline-text-2" id="text-6">
<p>
This post provided an overview of the data landscape in Computational
Neuroscience and introduced the sub-field of Neuroinformatics. We also
looked at some of the available data stores and reviewed a few of the
more popular initiatives to solve the fundamental data problems in the
field.
</p>
<p>
Stay tuned for the next instalment!
</p>
</div>
</div>
<div id="footnotes">
<h2 class="footnotes">Footnotes: </h2>
<div id="text-footnotes">
<div class="footdef"><sup><a id="fn.1" name="fn.1" class="footnum" href="#fnr.1">1</a></sup> <p class="footpara">
For a bit more details on the two fields see
<a href="https://www.maths.nottingham.ac.uk/personal/sc/cnn/CNN2A.pdf">What are Computational Neuroscience and Neuroinformatics?</a>
</p></div>
<div class="footdef"><sup><a id="fn.2" name="fn.2" class="footnum" href="#fnr.2">2</a></sup> <p class="footpara">
"A connectome is a comprehensive map of neural
connections in the brain, and may be thought of as its "wiring
diagram". From <a href="https://en.wikipedia.org/wiki/Connectome">this</a> page.
</p></div>
</div>
</div></div>
<div id="postamble" class="status">
<p class="date">Created: 2016-02-08 Mon 21:41</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-38687670181232373582016-01-18T04:52:00.000-08:002016-01-18T04:52:10.604-08:00Nerd Food: Interesting...<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Interesting…</title>
<!-- 2016-01-18 Mon 12:49 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
Time to flush all those tabs again. Some interesting stuff I bumped
into recently-ish.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Finance, Economics, Politics</h2>
<div class="outline-text-2" id="text-1">
<ul class="org-ul">
<li><a href="https://www.project-syndicate.org/commentary/marginal-pricing-end-of-western-oil-producers-by-anatole-kaletsky-2015-12">Why Big Oil Should Kill Itself</a>: This is a really, really interesting
article. The gist of it is that the entire logic around oil
exploration is now a fallacy and it makes more economic sense to
simply give up looking for oil because all oil that is left is just
to expensive to commercialise. It also has a very interesting take
on the valuation of oil companies (and sources of take overs) but I
won't spoil it for you. If you are into oil (or against it), its a
must read.
</li>
<li><a href="http://krugman.blogs.nytimes.com/2016/01/16/oil-goes-nonlinear/">Oil Goes Nonlinear</a>: Short but thought provoking. I don't tend to
agree with Krugman on a lot of things, but quite like this analysis.
</li>
<li><a href="http://foreignpolicy.com/2015/12/31/africas-boom-is-over/?utm_content=bufferc093b&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer">Africa’s Boom Is Over</a>: And the bad news continue. Totally spot on
analysis of what will befall us.
</li>
<li><a href="http://startupboy.com/2016/01/15/american-spring/">American Spring</a>: interesting take on the state of affairs of
American politics. Not sure I agree with everything, but definitely
food for thought. "Statistically speaking, what are the odds that the
two most qualified candidates to be president out of 300 million
people are siblings? Or married?" Indeed.
</li>
<li><a href="https://www.project-syndicate.org/commentary/sovereign-default-wave-emerging-markets-by-carmen-reinhart-2015-12#Z3gIvCtPUzZmf8PF.01">A Year of Sovereign Defaults?</a>: Very good and very scary. This has to
be on the cards, the only question is the timing.
</li>
<li><a href="https://www.washingtonpost.com/news/wonk/wp/2015/12/30/really-rich-people-are-suddenly-paying-quite-a-bit-more-in-taxes/">Really rich people are suddenly paying quite a bit more in taxes</a>:
some good news on the equality front I guess. But not quite sure it
makes much of a difference in the big scheme of US things.
</li>
<li><a href="http://www.latimes.com/world/mexico-americas/la-fg-argentina-economy-new-president-20151230-story.html">Argentina's 'little trees' getting chopped down by new president</a>:
Seems like Argentina is going to go through yet another turbulent
period, with some good and bad news coming out. Interesting take on
the impact to the less well off of the new policies. The chap is
certainly a doer, it seems: <a href="http://linkis.com/www.economist.com/ne/HiAjy">A fast start</a>.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Startups et al.</h2>
<div class="outline-text-2" id="text-2">
<ul class="org-ul">
<li><a href="https://www.oreilly.com/ideas/the-wtf-economy">The WTF Economy</a>: Tim O'Reilly (of publishing house fame) is setting
up a conference in the future of work. Sounds extremely
interesting. Hopefully, they will have a section dedicated to the
developing world. Source: Tim O'Reilly (Twitter)
</li>
<li><a href="https://medium.com/@jgarzik/bitcoin-is-being-hot-wired-for-settlement-a5beb1df223a#.vn9b7chtk">Bitcoin is Being Hot-Wired for Settlement</a>: Garzik is at it
again. Interesting news of cryptos in the settlement front. Source:
Jeff Garzik (Twitter)
</li>
<li><a href="http://www.wired.com/2015/12/elon-musks-billion-dollar-ai-plan-is-about-far-more-than-saving-the-world/?mbid=social_fb">Elon Musk’s Billion-Dollar AI Plan Is About Far More Than Saving the
World</a>: So it seems Musk and Altman want to ensure AI plays nice. Not
quite sure he's right on this one. Steven Levy's version is
available <a href="https://medium.com/backchannel/how-elon-musk-and-y-combinator-plan-to-stop-computers-from-taking-over-17e0e27dd02a#.f3ydovlgu">here</a>, more of an interview with Musk.
</li>
<li><a href="https://medium.com/backchannel/license-to-not-drive-6dbea84b9c45#.w9jh0xlyw">License to (Not) Drive</a>: Levy gets to try the Google self-driving
cars. Very interesting.
</li>
<li><a href="http://arches.io/2016/01/hire-literally-anyone/">Hire Literally Anyone</a>: Extremely interesting. I always thought the
existing hiring practices are not very well thought out, but this
article makes me realise that the flaws are deeper than I
expected. While we are on this topic, this ain't too bad either: <a href="https://medium.com/swlh/how-to-hire-34f4ded5f176#.mb0pcnhda">How
to Hire</a>.
</li>
<li><a href="https://medium.com/@octskyward/the-resolution-of-the-bitcoin-experiment-dabb30201f7#.vk8lilnkk">The resolution of the Bitcoin experiment</a>: Great - nay - insanely
great analysis on the state of affairs in the BTC world. Spoken with
authority.
</li>
<li><a href="https://tonyarcieri.com/on-the-dangers-of-a-blockchain-monoculture">On the dangers of a blockchain monoculture</a>: Pours more petrol in the
raging BTC fire. Very interesting. Never saw BTC as a monoculture,
but actually it so is.
</li>
<li><a href="http://www.bloomberg.com/news/articles/2015-12-30/the-final-days-of-the-bitcoin-foundation-">The Final Days of the Bitcoin Foundation?</a>: And yet some more on the
BTC impending doom. I just gotta stop reading about it now, the
whole saga is far too depressing. Lets hope the technology survives
where humans failed.
</li>
<li><a href="http://www.coindesk.com/ibm-open-ledger-blockchain/?utm_content=buffer601ae&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer">IBM Talks Open Ledger Project, Bright Future for Blockchain</a>: Still
trying to catch my breath on all the BTC articles coming out, and
lo-and-behold, I missed the whole Open Ledger thing.
</li>
<li><a href="https://www.policyalternatives.ca/publications/monitor/apploitation-city-instaserfs">Apploitation in a city of instaserfs</a>: Scary. Very scary. Reminiscent
of the older Mirani article <a href="http://qz.com/312537/the-secret-to-the-uber-economy-is-wealth-inequality/">The secret to the Uber economy is wealth
inequality</a>. San Francisco is becoming more like Mumbai and that is
not good news.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">General Coding</h2>
<div class="outline-text-2" id="text-3">
<ul class="org-ul">
<li><a href="https://medium.com/@henrikjohansen/feeding-graph-databases-a-third-use-case-for-modern-log-management-platforms-d5dac8a80d53#.tqmmc91uy">Feeding Graph databases - a third use-case for modern log management
platforms</a>: Very interesting ideas on how to use logging data in a
graph database. Sounds extremely counter-intuitive, and then you
start reading at which point its like "Damn, why didn't I think of
that before!". Source: Hacker News
</li>
<li><a href="http://www.agner.org/optimize/blog/read.php?i=417">Moores law hits the roof</a>: Seems like the exponential function is
revealing itself as a sigmoid, as everyone knew it would. Some of
the cracks that are already present in Moore's law. Interesting to
note that a transistor is now only a few silicon atoms wide -
meaning we can't really make it much smaller. Source: Hacker News
</li>
<li><a href="http://robotlolita.me/2016/01/09/no-i-dont-want-to-configure-your-app.html">No, I Don't Want To Configure Your App</a>: Call to arms to get us all
thinking on just how many configuration knobs you need to use
something. Source: Hacker News
</li>
<li><a href="http://jameshunt.us/writings/your-ide-is-killing-you.html">Your IDE Is Killing You</a>: Somewhat preaching to the choir, since I am
an Emacs user of old, but still a very cogent argument on why
relying too much on IDEs is not a good thing. Source: Bruno Antunes
(twitter)
</li>
<li><a href="http://jlongster.com/Starters-and-Maintainers">Starters and Maintainers</a>: The different personas around an open
source project. Interesting, its good to be aware of which hat you
are wearing when.
</li>
<li><a href="https://medium.com/backchannel/i-moved-to-linux-and-it-s-even-better-than-i-expected-9f2dcac3f8fb#.aakpzoln9">I Moved to Linux and It’s Even Better Than I Expected</a>: A feel good
story about the Linux desktop. Given how slowly things are
progressing on that front, we all need one of these some times to
cheer us up. Main value of the article though.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Databases</h2>
<div class="outline-text-2" id="text-4">
<ul class="org-ul">
<li><a href="http://lwn.net/Articles/667946/">Encrypted databases with ZeroDB</a>: I'm not exactly impressed with the
technology itself, but more with the ideas one can extract from
it. Briefly: what if the database only stores encrypted data, which
only each client can decrypt? This is certainly a very useful thing
for certain types of information and a PostgreSQL extension would
be most useful. Source: Hacker News
</li>
<li><a href="http://rachbelaid.com/introduction-to-postgres-physical-storage/">Introduction to PostgreSQL physical storage</a>: Great article on Postgres
low-level details. One to read if you want to get serious about the
Elephant but are not yet in the know.
</li>
<li><a href="http://tech.valgog.com/2012/01/schema-based-versioning-and-deployment.html">Schema based versioning and deployment for PostgreSQL</a>: Tips on how
to manage versions for your stored procs, and also contains links
for table management. For those of us not totally taken by NoSQL.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">C++</h2>
<div class="outline-text-2" id="text-5">
<ul class="org-ul">
<li><a href="http://webcache.googleusercontent.com/search?q=cache:z7PWAldSxdQJ:sourceforge.net/p/sobjectizer/wiki/Lessons%252520learnt%252520from%25252010%252B%252520years%252520with%252520actors%252520in%252520C%252B%252B/+&cd=1&hl=en&ct=clnk&gl=uk">Lessons learnt from 10+ years with actors in C++</a>: The voice of
experience talks about what they learned from using Actors over more
than a decade. Worth reading if you are into that pattern.
</li>
<li><a href="http://blog.scottfrees.com/automating-a-c-program-from-a-node-js-web-app">Automating a C++ program from a Node.js web app</a>: If you are
considering exposing your C++ code into JS, this is a series of
posts to read.
</li>
<li><a href="https://medium.com/swlh/starting-a-tech-startup-with-c-6b5d5856e6de#.tocwuwbe8">Starting a tech startup with C++</a>: lots of libraries I never heard of
and an insight on the performance differences between python and
c++.
</li>
<li><a href="https://medium.com/hacker-daily/writing-high-performance-servers-in-modern-c-7cd00926828#.hksbtpyoh">Writing modern C++ servers using Wangle:</a> The follow up to the
previous post, explaining how to write servers with Facebook
technologies.
</li>
<li><a href="http://www.di.unipi.it/~nids/docs/i_want_my_pony_or_why_you_cannot_have_cpp_exceptions_with_a_stack_trace.html">I want my pony! Or why you cannot have C++ exceptions with a stack
trace</a>: very interesting. Since I started using Boost.Exception I
never missed the stack traces either. Source: Hacker News
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">Layman Science</h2>
<div class="outline-text-2" id="text-6">
<ul class="org-ul">
<li><a href="http://www.forbes.com/sites/startswithabang/2015/12/23/why-string-theory-is-not-science/">Why String Theory Is Not A Scientific Theory</a>: Doesn't say a lot of
new things, but its good to remind ourselves on what exactly do we
mean when we say "Science". This would save us from a lot of grief,
such as considering Economics as a Science.
</li>
<li><a href="https://aeon.co/essays/why-do-scientists-dismiss-the-possibility-of-cold-fusion">The cold fusion horizon</a>: … talking about Science, I was surprised
to find out that people are still talking seriously about cold
fusion. Interesting article, because it takes the flip side of the
Science coin: nothing should <i>not</i> be science unless it is <i>not</i>
using the scientific method. Whilst up til now cold fusion has been
more of a hoax, we should not discredit people who work on it
provided they are following scientific principles. Who knows, they
may be right in the end. Science is all about long-shots.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-7" class="outline-2">
<h2 id="sec-7">Other</h2>
<div class="outline-text-2" id="text-7">
<ul class="org-ul">
<li><a href="http://www.blastr.com/2015-12-23/exit-sandman-neil-gaiman-goes-depth-overture-one-2015s-best-comics">Exit Sandman: Neil Gaiman goes in-depth with Overture, one of 2015's
best comics</a>: For the Sandman fans, the new (and last) Sandman book
is all the rage. A great interview by the man himself.
</li>
<li><a href="https://www.youtube.com/watch?v=MpkqfZ95jtw">Dear Zachary</a>: bumped into this via <i>Wait But Why</i>, and, as usual,
great tip. Fantastic documentary.
</li>
<li><a href="https://www.youtube.com/watch?v=GG9Anstjlro&feature=youtu.be">Solaris</a>: Always wanted to watch this Tarkovsky movie and now it
seems it is available online! This is part of an initiative
described by Open Culture <a href="http://www.openculture.com/2010/07/tarkovksy.html">here</a>. Source: Bruno Antunes (twitter)
</li>
<li><a href="https://www.youtube.com/watch?v=8BoKjQfMihs">Wittgenstein: A Wonderful Life</a>: Found a Wittgenstein documentary,
but sadly haven't had time to watch it just yet. In my watch list
though.
</li>
<li><a href="http://therealnews.com/t2/index.php?option=com_content&task=view&id=31&Itemid=74&jumival=14293">Je ne suis pas Charlie</a>: Haven't yet watched it but seems
thought-provoking. Watch listed.
</li>
<li><a href="https://www.youtube.com/watch?v=THKCteQocns">Tulipa Ruiz - Efêmera - Album Completo</a>: New musical find in the
Brazilian space (Portuguese).
</li>
</ul>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2016-01-18 Mon 12:49</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-82664300196820414782016-01-17T15:38:00.000-08:002019-03-25T08:24:08.471-07:00Nerd Food: On Product Backlog<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: On Product Backlog</title>
<!-- 2016-01-17 Sun 23:55 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<table border="0">
<tr>
<td width="50%"></td>
<td width="50%"><p class="verse" style="text-align:left">
<small>
Would be be good to have a better bug-tracking setup? Yes.
But I think it takes man-power, and it would take something
*fundamentally* better than bugzilla. <i>-- <a href="http://yarchive.net/comp/linux/bug_tracking.html">Linus</a></i>
</small>
</p></td>
</tr>
</table>
<p>
Many developers in large companies tend to be exposed to a strange
variation of agile which I like to call "Enterprise Grade Agile", but
I've also heard it called "Fragile" and, most aptly, "Cargo-Cult
Agile". However you decide to name the phenomena, the gist of it is
that these setups contain nearly all of the ceremony of agile -
including stand-ups, sprint planning, retrospectives and so on - but
none of its spirit. Tweets such as this are great at capturing the
essence of the problem:
</p>
<blockquote class="twitter-tweet" lang="en"><p lang="en" dir="ltr">Top
tip: if you need to bring a notepad to the daily stand up to tell us
what you did yesterday that's too many details</p>— Fran
Buontempo (@fbuontempo) <a
href="https://twitter.com/fbuontempo/status/686856528696086528">January
12, 2016</a></blockquote> <script async
src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>
Once you start having that nagging feeling of doing things "because
you are told to", and once your stand-ups become more of a status
report to the "project manager" and/or "delivery manager" - the
existence of which, in itself, is rather worrying - your Cargo Cult
Agile alarm bells should start ringing. As I see it, agile is a
toolbox with a number of tools, and they only start to add value once
you've adapted them to your personal circumstances. The fitness
function that determines if a tool should be used is how much value it
adds to all (or at least most) of its users. If it does not, the tool
must be further adapted or removed altogether. And, crucially, you
learn about agile tools by using them and by reflecting on the lessons
learned. There is no other way.
</p>
<p>
This post is one such exercise and the tool I'd like to reflect on is
the <i>Product Backlog</i>. Now, before you read through the whole rant,
its probably worth saying that this post takes a slightly narrow and
somewhat "advanced" view of agile, with a target audience of those
already using it. If you require a more introductory approach, you are
probably better off looking at other online resources such as <a href="http://zerodollarbill.blogspot.co.uk/2012/06/how-to-learn-scrum-in-10-minutes-and.html">How to
learn Scrum in 10 minutes and clean your house in the process</a>. Having
said that, I'll try to define terms best I can to make sure we are all
on the same page.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Working Definition</h2>
<div class="outline-text-2" id="text-1">
<p>
Once your company has grokked the basics of agile and starts to move
away from those lengthy specification documents - those that no one
reads properly until implementation and those that never specified
anything the customer wanted, but everything we thought the customer
wanted and then some - you will start to use the product backlog in
anger. And that's when you will realise that it is not quite as simple
as memorising text books.
</p>
<p>
So what do the "text books" say? Let's take a fairly typical
definition - this one from <a href="https://en.wikipedia.org/wiki/Scrum_(software_development)">Scrum</a>:
</p>
<blockquote>
<p>
The agile product backlog in Scrum is a prioritized features list,
containing short descriptions of all functionality desired in the
product. When applying Scrum, it's not necessary to start a project
with a lengthy, upfront effort to document all
requirements. Typically, a Scrum team and its product owner begin by
writing down everything they can think of for agile backlog
prioritization. This agile product backlog is almost always more than
enough for a first sprint. The Scrum product backlog is then allowed
to grow and change as more is learned about the product and its
customers.<sup><a id="fnr.1" name="fnr.1" class="footref" href="#fn.1">1</a></sup>
</p>
</blockquote>
<p>
This is a good working definition, which will suffice for the purposes
of this post. It is deceptively simple. However, as always, one must
remember Yogi Berra: "In theory, there is no difference between theory
and practice. But in practice, there is."
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Potmenkin Product Backlogs</h2>
<div class="outline-text-2" id="text-2">
<p>
Many teams finish reading one such definition, find it amazingly
inspiring, install the "agile plug-in" on their bug-tracking software
of choice and then furiously start typing in those tickets. But if you
look closely, you'd be hard-pressed to find any difference between the
bug tickets of old versus the "stories" in the new and improved
"product backlog" that apparently you are now using.
</p>
<p>
This is a classic management disconnect, whereby a renaming exercise
is applied and suddenly, <a href="https://en.wikipedia.org/wiki/Potemkin_village">Potemkin village-style</a>, we are now in with
the kool kids and our company suddenly becomes a modern and desirable
place to work. But much like Potemkin villages were not designed for
real people to live in, so "Potmenkin Product Backlogs" are not
designed to help you manage the lifecycle of a real product; they are
there to give you the <i>appearance</i> of doing said management, for the
purposes of reporting to the higher eschelons and so that you can tell
stakeholders that "their story has been added to the product backlog
for prioritisation".
</p>
<p>
Alas, very soon you will find that the bulk of the "user stories" are
nothing but glorified one-liners that no one seems to recall what
exactly they're supposed to mean, and those few elaboratedly detailed
tickets end up rotting because they keep being deprioritised and now
describe a world long gone. Soon enough you will find that your sprint
planning meetings will cover less and less of the product backlog -
after all, who is able to prioritise this mess? Some stories don't
even make any sense! The final act is when all stories worked on are
stories raised directly on the sprint backlog, and the product backlog
is nothing but the dumping ground for the stories that didn't make it
on a given sprint. At this stage, the product backlog is in such a
terrible mess that no one looks at it, other than for the occasional
historic search for valuable details on how a bug was
fixed. Eventually the product backlog is zeroed - maybe a dozen or so
of the most recent stories make it through the cull - and the entire
process begins anew. Alas, enlightenment is never achieved, so you are
condemned to repeat this cycle for all eternity.
</p>
<p>
As expected, the Potmenkin Product Backlog adds very little value - in
fact it can be argued that it detracts value - but it must be kept
because "agile requires a product backlog".
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Bug-Trackers: Lessons From History</h2>
<div class="outline-text-2" id="text-3">
<p>
In order to understand the difficulties with a product backlog, we
turn next to their logical predecessors: bug-tracking systems such as
<a href="https://www.bugzilla.org/">Bugzilla</a> or <a href="https://www.atlassian.com/software/jira">Jira</a>. This post starts with a quote from the kernel's
Benevolent Dictator that illustrates the problem with these. Linus has
long taken the approach that there is no need for a bug-tracker in
kernel development, although he does not object if someone wants to
use one for a subsystem. You may think this is a very primitive
approach but in some ways it is also a <i>very</i> modern approach, very
much in line with agile; if you have a bug-tracking system which is
taking time away from developers without providing any value, you
should <i>remove</i> the bug-tracking system. In kernel development, there
simply is no space for ceremony - or, for that matter, for anything
which slows things down<sup><a id="fnr.2" name="fnr.2" class="footref" href="#fn.2">2</a></sup>.
</p>
<p>
All of which begs the question: what makes bug-tracking systems so
useless? From experience, there are a few factors:
</p>
<ul class="org-ul">
<li>they are a "fire and forget" capture system. Most users only care
about entering new data, rather than worrying about the lifecycle of
a ticket. Very few places have some kind of "ticket quality control"
which ensures that the content of the ticket is vaguely sensible,
and those who do suffer from another problem:
</li>
<li>they require dedicated teams. By this I don't just mean running the
bug-tracking software - which you will most likely have to do in a
proprietary shop; I also mean the entire notion of Q&A and Testing
as separate from development, with reams of people dedicated to
setting "environments" up (and keeping them up!), organising
database restores and other such activities that are incompatible
with current best practices of software development.
</li>
<li>they are temples of ceremony: a glance at the myriad of fields you
need to fill in - and the rules and permutations required to get
them exactly right - should be sufficient to put off even the most
ardent believer in process. Most developers end up memorising some
safe incantation that allows them to get on with life, without
understanding the majority of the data they are entering.
</li>
<li>as the underlying product ages, you will be faced with <a href="http://tinyletter.com/programming-beyond-practices/letters/the-sad-graph-of-software-death">the sad graph
of software death</a>. The main problem is that resources get taken away
from systems as they get older, a phenomena that manifests itself as
a growth in the delta between the number of open tickets against the
number of closed tickets. This is actually a <i>really</i> useful metric
but one that is often ignored.<sup><a id="fnr.3" name="fnr.3" class="footref" href="#fn.3">3</a></sup>.
</li>
</ul>
<p>
And what of the newest iterations on this venerable concept such as
<a href="https://guides.github.com/features/issues/">GitHub Issues</a>? Well, clearly they solve a number of the problems
above - such as lowering the complexity and cost barriers - and
certainly they do serve a very useful purpose: they allow the
efficient management of user interactions. Every time I create an
issue - such as this <a href="https://github.com/flycheck/flycheck/issues/852">one</a> - it never ceases to amaze me how easily the
information flows within GitHub projects; one can initiate comms with
the author(s) or other users with <i>zero setup</i> - something that
previously required mailinglist membership, opening an account on a
bug-tracker and so forth. We now take all of this for granted, of
course, but it is important to bear in mind that many open source
projects would probably not even have <i>any</i> form of user interaction
support, were it not for GitHub. After all, most of them are a
one-person shop with very little disposable time, and it makes no
sense to spend part of that time maintaining infrastructure for the
odd person or two who may drop by to chat.
</p>
<p>
However, for all of its glory, it is also important to bear in mind
that GitHub Issues is <b>not</b> a product backlog solution. What I mean by
this is that the product backlog must be owned by the team that owns
the product and, as we shall see, it must be carefully groomed if it
is to be continually useful. This is at loggerheads with allowing free
flow of information from users. Your Issues will eventually be filled
up with user requests and questions which you may not want to address,
or general discussions which may or may not have a story behind
it. They are simply different tools for different jobs, albeit with an
overlap in functionality.
</p>
<p>
So, history tells us what does not work. But is the product backlog
even worth all this hassle?
</p>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Voyaging Through Strange Seas of Thought</h2>
<div class="outline-text-2" id="text-4">
<p>
One of the great things about agile is how much it reflects on itself;
a strange loop of sorts. Presentations such as Kevlin Henney's <a href="http://www.infoq.com/presentations/architecture-uncertainty">The
Architecture of Uncertainty</a> are part of this continual process of
discovery and understanding, and provide great insights about the
fundamental nature of the development process. The product backlog
plays - or should play - a crucial role exactly because of this
uncertain nature of software development. We can explain this by way
of a device.
</p>
<p>
Imagine that you start off by admitting that you know very little
about what it is that you are intending to do and that the problem
domain you are about to explore is vast and complex. In this scenario,
the product backlog is the sum total of the knowledge gained whilst
exploring this space that has yet not been transformed into source
code. Think of it like the explorer's maps in the fifteen-hundreds. In
those days, "users" knew that much of it was incorrect and a great
part was sketchy and ill-defined, but it was all you had. Given that
the odds of success were stacked against you, you'd hold that map
pretty tightly while the storms were raging about you. Those that made
it back would provide corrections and amendments and, over time, the
maps eventually converged with the real geography.
</p>
<p>
The product backlog does something similar, but of course, the space
you are exploring does not have a fixed geometry or topography and
your knowledge of the problem domain can actively <i>change</i> the domain
itself too - an unavoidable consequence of dealing with pure thought
stuff. But the general principle applies. Thus, in the same way <a href="http://www.joelonsoftware.com/articles/fog0000000069.html">a code
base is precious</a> because it embodies the sum total knowledge of a
domain - heck, in many ways it <i>is</i> the sum total knowledge of a
domain! - so the product backlog is precious because it captures all
the known knowledge of these yet-to-be-explored areas. In this light,
you can understand statements such as this:
</p>
<blockquote class="twitter-tweet" lang="en"><p lang="en" dir="ltr">
When your product backlog is empty, your product is dead -
<a href="https://twitter.com/KevlinHenney">@KevlinHenney</a>
<a href="https://twitter.com/hashtag/agileotb?src=hash">#agileotb</a>
</p>— Marc Johnson (@marcjohnson)
<a href="https://twitter.com/marcjohnson/status/507522331900915712">
September 4, 2014</a>
</blockquote>
<script async src="//platform.twitter.com/widgets.js"
charset="utf-8"></script>
<p>
So, if the backlog is this important, how should one manage it?
</p>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">Works For Me, Guv!</h2>
<div class="outline-text-2" id="text-5">
<p>
Up to this point - whilst we were delving into the problem space - we
have been dealing with a fairly general argument, likely applicable to
many. Now, as we enter the solution space, I'm afraid I will have to
move from the general to the particular and talk only about the
specific circumstances of my one-man-project <a href="https://github.com/DomainDrivenConsulting/dogen">Dogen</a>. You can find
Dogen's product backlog <a href="https://github.com/DomainDrivenConsulting/dogen/blob/master/doc/agile/product_backlog.org">here</a>.
</p>
<p>
This may sound like a bit of a cop out, you may say, and not without
reason: how on earth are you supposed to extrapolate conclusions from
a one-person open source project to a team of N working on a
commercial product? However, it is also important to take into account
what I said at the start: agile is what you make of it. I personally
think of it as a) the smallest amount of processes required to make
your development process work smoothly and b) and the continual
improvement of those processes. Thus, there are no one-size-fits-all
solutions; all one can do is to look at others for ideas. So, lets
look at my findings<sup><a id="fnr.4" name="fnr.4" class="footref" href="#fn.4">4</a></sup>.
</p>
<p>
The first and most important thing I did to help me manage my product
backlog was to use a simple text file in <a href="http://orgmode.org/">Org Mode</a> notation. Clearly,
this is not a setup that is workable for a development team much
larger than a set of one, or one that doesn't use Emacs (or <a href="https://github.com/hsitz/VimOrganizer">Vim</a>). But
for my particular circumstances it has worked <i>wonders</i>:
</p>
<ul class="org-ul">
<li>the product backlog is close to the code, so wherever you go, you
take it with you. This means you can always search the product
backlog and - most importantly - add to it <i>wherever</i> you are and
<i>whenever</i> an idea happens to come by. I use this flexibility
frequently.
</li>
<li>the Org Mode interface makes it really easy to move stories up and
down (order is taken to mean priority here) and to create "buckets"
of stories according to whatever categorisation you decide to use,
up to any level of nesting. At some point you end up converging to a
reasonable level of nesting, of course. It is surprising how one can
manage <b>very</b> large amounts of stories thanks to this flexible tree
structure.
</li>
<li>it's trivial to move stories in and out of a sprint, keeping track
of all changes to a story - they are just text that can be copy and
pasted and committed.
</li>
<li>Org Mode provides a very capable <a href="http://orgmode.org/manual/Tags.html">tagging system</a>. I first started by
overusing these, but when tagging got too fine grained it became
unmaintainable. Now we use too few - just <code>epic</code> and <code>story</code> - so
this will have to change again in the near future. For example, it
should be trivial to add tags for different components in the system
or to mark stories as bugs or features, etc. <a href="http://orgmode.org/manual/Tag-searches.html#Tag-searches">Searching</a> then allows
you to see a subset of the stories that match those labels.
</li>
</ul>
<p>
A second decision which has proven to be a very good one has been to
groom the product backlog <i>very often</i>. And by this I don't just mean
a cursory look, but a deep inspection of <i>all</i> stories, fixing them
where required. Again, the choice of format has proved very helpful:
</p>
<ul class="org-ul">
<li>it is easy to mark all stories as "non-reviewed" or some other
suitable tag in Org Mode, and then unmark them as one finishes the
groom - thereby ensuring all stories get some attention. As the
product backlog becomes larger, a full groom could take multiple
sprints, but this is not an issue once you understand its value and
the cost of having it rot.
</li>
<li>because the product backlog is with the code, any downtime can be
used for grooming; those idle weekends or that long wait at the
airport are perfect candidates to get a few stories looked at. Time
spent waiting for the build is also a good candidate.
</li>
<li>you get an HTML representation of the Org Mode file for free in
GitHub, meaning you can read your backlog from your phone. And with
the new editing functionality, you can also edit stories too.
</li>
</ul>
<p>
Thirdly, I decided to take a "multi-pass" approach at managing the
story lifecycle. These are some of the key aspects of this lifecycle
management:
</p>
<ul class="org-ul">
<li>stories can only be captured if they are aligned with the
<a href="https://github.com/DomainDrivenConsulting/dogen/blob/master/doc/agile/vision.org">vision</a>. This filter saves me from adding all sorts of ideas which
are just too "out of the left field" to be of practical use, but
keeps <a href="https://github.com/MASD-Project/dogen/blob/master/doc/agile/product_backlog.org#visionary">those that may sound crazy</a> are but aligned with the vision.
</li>
<li>stories can only be captured if there is no "prior art". I always
perform a number of searches in the backlog to look for anything
which covers similar ground. If found, I append to that.
</li>
<li>new stories tend to start with very little content - just the
minimum required to allow resetting state back to the idea I was
trying to capture. Due to this, very little gets lost. At this
point, we have a "proto-story".
</li>
<li>as time progresses, I end up having more ideas on this space, and I
update the story with those ideas - mainly bullet points with one
liners and links.
</li>
<li>at some point the story begins to mature; there is enough on it that
we can convert the "proto-story" to a full blown story. After a
number of grooms, the story becomes fully formed and is then a
candidate to be moved to a sprint backlog for implementation. It may
stay in this state <i>ad-infinitum</i>, with periodic updates just to
make sure it does not rot.
</li>
<li>A candidate story can still get refined: trimmed in scope,
re-targeted, or even cancelled because it no longer fits with the
current architecture or even the vision. Cancelled stories are
important because we may come back to them - its just very unlikely
that we do.
</li>
<li>every sprint has a "sprint mission"<sup><a id="fnr.5" name="fnr.5" class="footref" href="#fn.5">5</a></sup>. When we start to
move stories into the sprint backlog, we look for those which
resonate with the sprint mission. Not all of them are fully formed,
and the work on the sprint can entail the analysis required to
create a full blown story. But many will be implementable directly
off of the product backlog.
</li>
<li>some times I end up finding related threads in multiple stories and
decide to merge them. Merging of related stories is done by simply
copying and pasting them into a single story; over time, with the
multiple passes done in the grooms, we end up again with a single
consistent story.
</li>
</ul>
<p>
What all of this means is that a story can evolve over time in the
product backlog, only to become the exact thing you need at a given
sprint; at that point you benefit from the knowledge and insight
gained over that long period of time. Some stories in Dogen's backlog
have been there for years, and when I finally get to them, I find them
extremely useful. Remember: they are a map to the unknown space you
are exploring.
</p>
<p>
With all of this machinery in place, we've ended up with a very useful
product backlog for Dogen - one that certainly adds a lot of
value. Don't take me wrong, the cost of maintenance is high and I'd
rather be coding instead of maintaining the product backlog,
especially given the limited resources. But I keep it because I can
see on a daily basis how much it improves the overall quality of the
development process. It is a price I find worth paying, given what I
get in return.
</p>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">Final Thoughts</h2>
<div class="outline-text-2" id="text-6">
<p>
This post was an attempt to summarise some of the thoughts I've been
having on the space of product backlogs. One of its main objectives
was to try to convey the importance of this tool, and to provide ideas
on how you can improve the management of your own product backlog by
discussing the approach I have taken with Dogen.
</p>
<p>
If you have any suggestions or want to share your own tips on how to
manage your product backlog please reach me on the comments section -
there is always space for improvement.
</p>
</div>
</div>
<div id="footnotes">
<h2 class="footnotes">Footnotes: </h2>
<div id="text-footnotes">
<div class="footdef"><sup><a id="fn.1" name="fn.1" class="footnum" href="#fnr.1">1</a></sup> <p class="footpara">
Source: <a href="https://www.mountaingoatsoftware.com/agile/scrum/product-backlog">Scrum Product Backlog</a>, Mountain Goat Software.
</p></div>
<div class="footdef"><sup><a id="fn.2" name="fn.2" class="footnum" href="#fnr.2">2</a></sup> <p class="footpara">
A topic which I covered some time ago here: <a href="http://mcraveiro.blogspot.co.uk/2008/06/nerd-food-on-evolutionary-methodology.html">On
Evolutionary Methodology</a>. It is also interesting to see how the kernel
processes are organised for speed: <a href="http://lwn.net/Articles/670209/">How 4.4's patches got to the
mainline</a>.
</p></div>
<div class="footdef"><sup><a id="fn.3" name="fn.3" class="footnum" href="#fnr.3">3</a></sup> <p class="footpara">
Another topic which I also covered here some time
ago: <a href="http://mcraveiro.blogspot.co.uk/2007/05/nerd-food-on-maintenance.html">On Maintenance</a>.
</p></div>
<div class="footdef"><sup><a id="fn.4" name="fn.4" class="footnum" href="#fnr.4">4</a></sup> <p class="footpara">
I am self-plagiarising a little bit here and
rehashing some of the arguments I've used before in <a href="http://mcraveiro.blogspot.co.uk/2014/09/nerd-food-dogen-lessons-in-incremental.html">Lessons in
Incremental Coding</a>, mainly from section DVCS to the Core.
</p></div>
<div class="footdef"><sup><a id="fn.5" name="fn.5" class="footnum" href="#fnr.5">5</a></sup> <p class="footpara">
See the <a href="https://github.com/DomainDrivenConsulting/dogen/blob/master/doc/agile/sprint_backlog_78.org">current sprint backlog</a> for an example.
</p></div>
</div>
</div></div>
<div id="postamble" class="status">
<p class="date">Created: 2016-01-17 Sun 23:55</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-43930784583928938262015-12-22T06:01:00.000-08:002015-12-22T06:01:23.584-08:00Nerd Food: Dogen: The Package Management Saga<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Dogen: The Package Management Saga</title>
<!-- 2015-12-22 Tue 14:00 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
We've just gone past <a href="https://github.com/DomainDrivenConsulting/dogen">Dogen's</a> <a href="https://github.com/DomainDrivenConsulting/dogen/blob/master/doc/agile/sprint_backlog_75.org">Sprint 75</a>, so I guess it's time for one
of those "reminiscing posts" - something along the lines of what we
did for <a href="http://mcraveiro.blogspot.co.uk/2014/09/nerd-food-dogen-lessons-in-incremental.html">Sprint 50</a>. This one is a bit more practical though; if you are
only interested in the practical side, keep scrolling until you see
"Conan".
</p>
<p>
So, package management. Like any other part-time C++ developer whose
professional mainstay is C# and Java, I have keenly felt the need for
a package manager when in C++-land. The problem is less visible when
you are working with mature libraries and dealing with just Linux, due
to the huge size of the package repositories and the great tooling
built around them. However, things get messier when you start to go
cross-platform, and messier still when you are coding on the bleeding
edge of C++: either the package you need is not available in the
distro's repos or even <a href="https://launchpad.net/ubuntu/+ppas">PPA's</a>; or, when it is, its rarely at the
version you require.
</p>
<p>
Alas, for all our sins, that's exactly where we were when <a href="https://github.com/DomainDrivenConsulting/dogen">Dogen</a> got
started.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">A Spoonful of Dogen History</h2>
<div class="outline-text-2" id="text-1">
<p>
Dogen sprung to life just a tad after C++-0x became <a href="https://en.wikipedia.org/wiki/C%252B%252B11">C++-11</a>, so we
experienced first hand the highs of a quasi-new-language followed by
the lows of feeling the brunt of the bleeding edge pain. For starters,
<i>nothing</i> we ever wanted was available out of the box, on any of the
platforms we were interested in. Even Debian testing was a bit
behind - probably stalled due to a compiler transition or other, but I
can't quite recall the details. In those days, Real Programmers were
Real Programmers and mice were mice: we had to <a href="http://mcraveiro.blogspot.co.uk/2012/06/nerd-food-c-11-with-gcc.html">build and install the
C++ compilers ourselves</a> and, even then, C++-11 support was new, a bit
flaky and limited. We then had to use those compilers to compile all
of the dependencies in C++-11 mode.
</p>
</div>
<div id="outline-container-sec-1-1" class="outline-3">
<h3 id="sec-1-1">The PFH Days</h3>
<div class="outline-text-3" id="text-1-1">
<p>
After doing this manually once or twice, it soon stopped being
fun. And so we solved this problem by creating the PFH - the Private
Filesystem Hierarchy - a gloriously over-ambitious name to describe a
set of wrapper scripts that helped with the process of downloading
tarballs, unpacking, building and finally installing them into
well-defined locations. It worked well enough in the confines of its
remit, but we were often outside those, having to apply out-of-tree
patches, adding new dependencies and so on. We also didn't use Travis
in those days - not even sure it existed, but if it did, the rigmarole
of the bleeding edge experience would certainly put a stop to any
ideas of using it. So we used a local install of CDash with a number
of build agents on OSX, Windows (MinGW) and Linux (32-bit and
64-bit). Things worked beautifully when nothing changed and the setup
was stable; but, every time a new version of a library - or god
forbid, of a compiler - was released, one had that sense of dread: do
I <b>really</b> need to upgrade?
</p>
<p>
Since one of the main objectives of Dogen was to learn about C++-11,
one has to say that the pain was worth it. But all of the moving parts
described above were not ideal and they were certainly not the thing
you want to be wasting your precious time on when it is very
scarce. They were certainly not scalable.
</p>
</div>
</div>
<div id="outline-container-sec-1-2" class="outline-3">
<h3 id="sec-1-2">The Good Days and the Bad Days</h3>
<div class="outline-text-3" id="text-1-2">
<p>
Things improved slightly for a year or two when distros started to
ship C++-11 compliant compilers and recent boost versions. It was all
so good we were able to move over to <a href="https://travis-ci.org/DomainDrivenConsulting/dogen">Travis</a> and ditch almost all of
our private infrastructure. For a while things looked really
good. However, due to Travis' <a href="https://wiki.ubuntu.com/LTS">Ubuntu LTS</a> policy, we were stuck with a
rapidly ageing Boost version. At first PPAs were a good solution for
this, but soon these became stale too. We also needed to get latest
CMake as there are a lot of developments on that front, but we
certainly could not afford (time-wise) to revert back to the bad old
days of the PFH. At the same time, it made no sense to freeze
dependencies in time, providing a worse development experience. So the
only route left was to break Travis and hope that some solution would
appear. Some alternatives were tried such as <a href="https://drone.io/github.com/DomainDrivenConsulting/dogen">Drone.io</a> but nothing was
successful.
</p>
<p>
There was nothing else for it; what was needed was a package manager
to manage the development dependencies.
</p>
</div>
</div>
<div id="outline-container-sec-1-3" class="outline-3">
<h3 id="sec-1-3">Nuget Hopes Dashed</h3>
<div class="outline-text-3" id="text-1-3">
<p>
Having used <a href="https://www.nuget.org/">Nuget</a> in anger for both C# and C++ projects, and given
Microsoft's recent change of heart with regards to open source, I was
secretly hoping that Nuget would get some traction in the wider C++
world. To recap, Nuget worked <a href="http://mcraveiro.blogspot.co.uk/2014/05/nerd-food-using-mono-in-anger-part-ii_3422.html">well enough in Mono</a>; in addition, C++
support for Windows was added <a href="http://blogs.msdn.com/b/vcblog/archive/2013/04/26/nuget-for-c.aspx">early on</a>. It was somewhat limited and a
bit quirky at the start, but it kept on getting better, to the point
of usability. Trouble was, their focus was just Visual Studio.
</p>
<p>
Alas, nothing much ever came from my Nuget hopes. However, there have
been a couple of recent announcements from Microsoft that make me
think that they will eventually look into this space:
</p>
<ul class="org-ul">
<li><a href="http://blogs.msdn.com/b/vcblog/archive/2015/12/04/introducing-clang-with-microsoft-codegen-in-vs-2015-update-1.aspx">Clang with Microsoft CodeGen in VS 2015 Update 1</a>
</li>
<li><a href="http://blogs.msdn.com/b/vcblog/archive/2015/12/15/support-for-android-cmake-projects-in-visual-studio.aspx">Support for Android CMake projects in Visual Studio</a>
</li>
</ul>
<p>
Surely the logical consequence is to be able to manage packages in a
consistent way across platforms? We can but hope.
</p>
</div>
</div>
<div id="outline-container-sec-1-4" class="outline-3">
<h3 id="sec-1-4">Biicode Comes to the Rescue?</h3>
<div class="outline-text-3" id="text-1-4">
<p>
Nuget did not pan out but what did happen was even more unlikely: some
crazy-cool Spaniards decided to create a stand alone package
manager. Being from the same peninsula, I felt compelled to use their
wares, and was joyful as they went from strength to strength -
including the success of their <a href="https://www.biicode.com/biicode-open-source-challenge">open source campaign</a>. And I loved the
fact that it integrated really well with <a href="https://cmake.org">CMake</a>, and that <a href="https://www.jetbrains.com/clion/">CLion</a>
provided Biicode integration very early on.
</p>
<p>
However, my biggest problem with Biicode was that it was just too
complicated. I don't mean to say the creators of the product didn't
have very good reasons for their technical choices - lord knows
creating a product is hard enough, so I have nothing but praise to
anyone who tries. However, for me personally, I never had the time to
understand why Biicode needed its own version of CMake, nor did I want
to modify my CMake files too much in order to fit properly with
Biicode and so on. Basically, I needed a solution that worked well and
required minimal changes at my end. Having been brought up with Maven
and Nuget, I just could not understand why there wasn't a simple
"packages.xml" file that specified the dependencies and then some
non-intrusive CMake support to expose those into the CMake
files. As you can see from some of <a href="http://forum.biicode.com/t/building-out-of-tree-using-biicode/460">my posts</a>, it just seemed it
required "getting" Biicode in order to make use of it, which for me
was not an option.
</p>
<p>
Another thing that annoyed me was the difficulty on knowing what the
"real" version of a library was. I wrote, at the time:
</p>
<blockquote>
<p>
One slightly confusing thing about the process of adding dependencies
is that there may be more than one page for a given dependency and it
is not clear which one is the "best" one. For RapidJson there are
three options, presumably from three different Biicode users:
</p>
<ul class="org-ul">
<li><a href="https://www.biicode.com/fenix/rapidjson">fenix</a>: authored on 2015-Apr-28, v1.0.1.
</li>
<li><a href="https://www.biicode.com/hithwen/rapidjson">hithwen</a>: authored 2014-Jul-30
</li>
<li><a href="https://www.biicode.com/denis/rapidjson">denis</a>: authored 2014-Oct-09
</li>
</ul>
<p>
The "fenix" option appeared to be the most up-to-date so I went with
that one. However, this illustrates a deeper issue: how do you know
you can trust a package? In the ideal setup, the project owners would
add Biicode support and that would then be the one true
version. However, like any other project, Biicode faces the initial
adoption conundrum: people are not going to be willing to spend time
adding support for Biicode if there aren't a lot of users of Biicode
out there already, but without a large library of dependencies there
is nothing to draw users in. In this light, one can understand that it
makes sense for Biicode to allow anyone to add new packages as a way
to bootstrap their user base; but sooner or later they will face the
same issues as all distributions face.
</p>
<p>
A few features would be helpful in the mean time:
</p>
<ul class="org-ul">
<li>popularity/number of downloads
</li>
<li>user ratings
</li>
</ul>
<p>
These metrics would help in deciding which package to depend on.
</p>
</blockquote>
<p>
For all these reasons, I never found the time to get Biicode setup and
these stories lingered in Dogen's backlog. And the build continued to
be red.
</p>
<p>
Sadly Biicode the company <a href="http://blog.biicode.com/biicode-just-the-company-post-mortem/">didn't make it either</a>. I feel very sad for
the guys behind it, because their heart was on the right place.
</p>
<p>
Which brings us right up to date.
</p>
</div>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Enter Conan</h2>
<div class="outline-text-2" id="text-2">
<p>
When I was a kid, we were all big fans of Conan. No, not <a href="https://en.wikipedia.org/wiki/Conan_the_Barbarian">the
barbarian</a>, the Japanese Manga <a href="https://en.wikipedia.org/wiki/Future_Boy_Conan">Future Boy Conan</a>. For me the name Conan
will always bring back great memories of this show, which we watched
in the original Japanese with Portuguese subtitles. So I was secretly
pleased when I found <a href="https://www.conan.io/">conan.io</a>, a new package management system for
C++. The guy behind it seems to be one of the original Biicode
developers, so a lot of lessons from Biicode were learned.
</p>
<p>
To cut a short story short, the great news is I managed to add Conan
support to Dogen in roughly <a href="https://github.com/DomainDrivenConsulting/dogen/blob/master/doc/agile/sprint_backlog_77.org#add-support-for-conanio">3 hours</a> and with very minimal knowledge
about Conan. This to me was a litmus test of sorts, because I have
very little interest in package management - creating my own product
has proven to be challenging enough, so the last thing I need is to
divert my energy further. The other interesting thing is that roughly
half of that time was taken by trying to get Travis to behave, so its
not quite fair to impute it to Conan.
</p>
</div>
<div id="outline-container-sec-2-1" class="outline-3">
<h3 id="sec-2-1">Setting Up Dogen for Conan</h3>
<div class="outline-text-3" id="text-2-1">
<p>
So, what changes did I do to get it all working? It was a very simple
3-step process. First I installed Conan using a Debian package from
<a href="https://www.conan.io/downloads">their site</a>.
</p>
<p>
I then created a <code>conanfile.txt</code> on my top-level directory:
</p>
<pre class="example">
[requires]
Boost/1.60.0@lasote/stable
[generators]
cmake
</pre>
<p>
Finally I modified my top-level <code>CMakeLists.txt</code>:
</p>
<pre class="example">
# conan support
if(EXISTS "${CMAKE_BINARY_DIR}/conanbuildinfo.cmake")
message(STATUS "Setting up Conan support.")
include("${CMAKE_BINARY_DIR}/conanbuildinfo.cmake")
CONAN_BASIC_SETUP()
else()
message(STATUS "Conan build file not found, skipping include")
endif()
</pre>
<p>
This means that it is entirely possible to build Dogen without Conan,
but if it is present, it will be used. With these two changes, all
that was left to do was to build:
</p>
<pre class="example">
$ cd dogen/build/output
$ mkdir gcc-5-conan
$ conan install ../../..
$ make -j5 run_all_specs
</pre>
<p>
<i>Et voila</i>, I had a brand spanking new build of Dogen using
Conan. Well, actually, <i>not quite</i>. I've omitted a couple of problems
that are a bit of a distraction on the Conan success story. Let's look
at them now.
</p>
</div>
</div>
<div id="outline-container-sec-2-2" class="outline-3">
<h3 id="sec-2-2">Problems and Their Solutions</h3>
<div class="outline-text-3" id="text-2-2">
<p>
The first problem was that Boost 1.59 does not appear to have an
overridden <code>FindBoost</code>, which means that I was not able to link. I
moved to Boost 1.60 - which I wanted to do any way - and it worked out
of the box.
</p>
<p>
The second problem was that Conan seems to get confused with <a href="https://ninja-build.org/manual.html">Ninja</a>, my
build system of choice. For whatever reason, when I use the Ninja
generator, it fails like so:
</p>
<pre class="example">
$ cmake ../../../ -G Ninja
$ ninja -j5
$ ninja: error: '~/.conan/data/Boost/1.60.0/lasote/stable/package/ebdc9c0c0164b54c29125127c75297f6607946c5/lib/libboost_system.so', needed by 'stage/bin/dogen_utility_spec', missing and no known rule to make it
</pre>
<p>
This is very strange because boost system is clearly available in the
Conan download folder. Using make solved this problem. I am going to
open a ticket on the Conan GitHub project to investigate this.
</p>
<p>
The third problem is more boost related than anything else. Boost
Graph has not been as well maintained as it should, really. Thus users
now find themselves carrying patches, and all because no one seems to
be able to apply them upstream. Dogen is in this situation as we've
hit the issue described here: <a href="http://stackoverflow.com/questions/25395805/compile-error-with-boost-graph-1-56-0-and-g-4-6-4">Compile error with boost.graph 1.56.0
and g++ 4.6.4.</a> Sadly this is still present on Boost 1.60; the patch
exists in Trac but remains unapplied (<a href="https://svn.boost.org/trac/boost/ticket/10382">#10382</a>). This is a tad worrying
as we make a lot of use of Boost Graph and intend to increase the
usage in the future.
</p>
<p>
At any rate, as you can see, none of the problems were showstoppers,
nor can they all be attributed to Conan.
</p>
</div>
</div>
<div id="outline-container-sec-2-3" class="outline-3">
<h3 id="sec-2-3">Getting Travis to Behave</h3>
<div class="outline-text-3" id="text-2-3">
<p>
Once I got Dogen building locally, I then went on a mission to
convince Travis to use it. It was painful, but mainly because of the
lag between commits and hitting an error. The core of the changes to
my YML file were as follows:
</p>
<pre class="example">
install:
<snip>
# conan
- wget https://s3-eu-west-1.amazonaws.com/conanio-production/downloads/conan-ubuntu-64_0_5_0.deb -O conan.deb
- sudo dpkg -i conan.deb
- rm conan.deb
<snip>
script:
- export GIT_REPO="`pwd`"
- cd ${GIT_REPO}/build
- mkdir output
- cd output
- conan install ${GIT_REPO}
- hash=`ls ~/.conan/data/Boost/1.60.0/lasote/stable/package/`
- cd ~/.conan/data/Boost/1.60.0/lasote/stable/package/${hash}/include/
- sudo patch -p0 < ${GIT_REPO}/patches/boost_1_59_graph.patch
- cmake ${GIT_REPO} -DWITH_MINIMAL_PACKAGING=on
- make -j2 run_all_specs
<snip>
</pre>
<p>
I probably should have a bash script by know, given the size of the
YML, but hey - if it works. The changes above deal with installation
of the package, applying the boost patch and using Make instead of
Ninja. Quite trivial in the end, even though it required a lot of
iterations to get there.
</p>
</div>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Conclusions</h2>
<div class="outline-text-2" id="text-3">
<p>
Having a red build is a very distressful event for a developer, so you
can imagine how painful it has been to have red builds for <i>several
months</i>. So it is with unmitigated pleasure that I got to see <a href="https://travis-ci.org/DomainDrivenConsulting/dogen/builds/98304957">build
#628</a> in a shiny emerald green. As far as that goes, it has been an
unmitigated success.
</p>
<p>
In a broader sense though, what can we say about Conan? There are many
positives to take home, even at this early stage of Dogen usage:
</p>
<ul class="org-ul">
<li>it is a lot less intrusive than Biicode and easier to setup. Biicode
was very well documented, but it was easy to stray from the beaten
track and that then required reading a lot of different wiki
pages. It seems easier to stay on the beaten track with Conan.
</li>
<li>as with Biicode, it seems to provide solutions to Debug/Release and
multi-platforms and compilers. We shall be testing it on Windows
soon and reporting back.
</li>
<li>hopefully, since it started Open Source from the beginning, it will
form a community of developers around the source with the know-how
required to maintain it. It would also be great to see if a business
forms around it, since someone will have to pay the cloud bill.
</li>
</ul>
<p>
In terms of negatives:
</p>
<ul class="org-ul">
<li>I still believe the most scalable approach would have been to extend
Nuget for the C++ Linux use case, since Microsoft is willing to take
patches and since they foot the bill for the public repo. However,
I can understand why one would prefer to have total control over the
solution rather than depend on the whims of some middle-manager in
order to commit.
</li>
<li>it seems publishing packages requires getting down into
Python. Haven't tried it yet, but I'm hoping it will be made as easy
as importing packages with a simple text file. The more complexity
around these flows the tool adds, the less likely they are to be
used.
</li>
<li>there still are no "official builds" from projects. As explained
above, this is a chicken and egg problem, because people are only
willing to dedicate time to it once there are enough users
complaining. Having said that, since Conan is easy to setup, one
hopes to see some adoption in the near future.
</li>
<li>even when using a GitHub profile, one still has to define a Conan
specific password. This was not required with Biicode. Minor pain,
but still, if they want to increase traction, this is probably an
unnecessary stumbling block. It was sufficient to make me think
twice about setting up a login, for one.
</li>
</ul>
<p>
In truth, these are all very minor negative points, but still worth
making them. All and all, I am quite pleased with Conan thus far.
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-12-22 Tue 14:00</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-39693496686659127022015-12-21T15:32:00.000-08:002015-12-21T15:32:29.361-08:00Nerd Food: Interesting...<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Interesting…</title>
<!-- 2015-12-21 Mon 23:31 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
Time to flush all those tabs again. Some interesting stuff I bumped
into recently-ish.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Finance, Economics, Politics</h2>
<div class="outline-text-2" id="text-1">
<ul class="org-ul">
<li><a href="http://qz.com/564513/a-not-so-brief-history-of-the-fall-and-fall-of-the-nigerian-naira/">A (not so) brief history of the fall and fall of the Nigerian naira</a>:
Very good read for Angolans; if nothing else, it makes us understand
that our precious Kwanza behaves in many ways like any other
petro-currency. Source: <a href="https://twitter.com/KingDouyeAlfred/status/675280673561710592">King Alfred (twitter)</a>.
</li>
<li><a href="https://www.youtube.com/watch?v=SRjwZvLo_hI">#ThisIsACoup - Episode 1- "Angela, suck our balls"</a>: A rather
political take on the recent-ish financial mess in Greece. On a
similar vein, BBC's <a href="http://www.bbc.co.uk/programmes/b06s1s5x">A Greek Drama</a> is worth a listen.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Startups et al.</h2>
<div class="outline-text-2" id="text-2">
<ul class="org-ul">
<li><a href="http://techcrunch.com/2015/12/16/brazils-congress-has-shut-down-whatsapp-tonight-and-the-rest-of-the-social-web-could-be-next/?ncid=rss#.xiisrb:Oo9T">Brazilian Judge Shuts Down WhatsApp And Brazil’s Congress Wants To
Shut Down The Social Web Next</a>: One of the most enlightened internet
countries decides to shut it all down. Sad day for the Internet and
for all Portuguese speakers. Source: Hacker News (twitter)
</li>
<li><a href="http://www.wired.com/2015/12/bitcoins-creator-satoshi-nakamoto-is-probably-this-unknown-australian-genius/">Bitcoin’s Creator Satoshi Nakamoto Is Probably This Unknown
Australian Genius</a>: So they found Satoshi (again). Hesitated in
adding this link, to be totally honest - there have been far too
many fakes to recount and the whole process is such a media circus
that its best avoiding it altogether. But after reading it -
questionable media behaviour notwhitstanding - it does appear to
provide some insights into these bitcoin early days. Useful to
anyone who likes BTC. There is also the <a href="http://gizmodo.com/this-australian-says-he-and-his-dead-friend-invented-bi-1746958692">Gizmodo report</a>, with
additional evidence. This is all getting a bit too much for my
liking though.
</li>
<li><a href="https://blog.jolla.com/jolla-back-business/">Jolla is back in business!</a>: Good to hear Jolla is still going. Now
that my Firefox OS phone is no longer supported, I am keen on
getting a Jolla. Source: Hacker News (twitter)
</li>
<li><a href="https://bitcoinmagazine.com/articles/tech-and-banking-giants-join-forces-with-the-linux-foundation-to-create-new-open-source-blockchain-hyperledger-1450384716">Tech and Banking Giants Join Forces with the Linux Foundation to
Create New Open Source Blockchain 'Hyperledger'</a>: In truth, hard not
to be sceptical - even though it's coming from the Linux
Foundation. I guess - in this world of <a href="http://radar.oreilly.com/2015/01/blockchain-scalability.html">scalability wars</a> - this must
come as good news. However, I still think there is a lot of
misunderstanding around Bitcoin and the Blockchain, and there are
far too many "AOLs" out there trying to create their gated
communities, failing to understand history (again). Not quite sure
on which side of the fence to place this initiative but, alas, I'm
more inclined towards the AOL side.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">General Coding</h2>
<div class="outline-text-2" id="text-3">
<ul class="org-ul">
<li><a href="http://spectrum.ieee.org/view-from-the-valley/computing/software/yahoos-engineers-move-to-coding-without-a-net">Yahoo’s Engineers Move to Coding Without a Net</a>: How removing a
testing team can help reduce the bug count and ramp up
productivity. Source: <a href="https://twitter.com/newsycombinator/status/675420147365060608">Hacker News (twitter)</a>
</li>
<li><a href="http://githubengineering.com/move-fast/">Move Fast and Fix Things</a>: An <i>incredible</i> tale of real engineering
from the GitHub guys with lots of take-ins - <a href="https://github.com/github/scientist">Scientist</a> is a pretty
neat idea, for one. Worth a read and a re-read. Logically related to
the previous article. Source: Hacker News (twitter)
</li>
<li><a href="https://medium.com/@thi.ng/the-jacob-s-ladder-of-coding-4b12477a26c1#.v80mhs3cv">The Jacob’s Ladder of coding</a>: Reminiscences on our beloved
profession of coding. Long and deep, so still parsing.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Databases</h2>
<div class="outline-text-2" id="text-4">
<ul class="org-ul">
<li><a href="https://wiki.postgresql.org/wiki/What's_new_in_PostgreSQL_9.5">What's new in PostgreSQL 9.5</a>: The RC's are starting and 9.5 looks to
continue the trend of amazing Postgres releases. My only missing
wish is for native (and full) support for bitemporality really,
though to be fair <a href="http://pgxn.org/dist/temporal_tables/">Temporal Tables</a> is probably enough for <a href="http://clarkdave.net/2015/02/historical-records-with-postgresql-and-temporal-tables-and-sql-2011/">my needs</a>.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">C++</h2>
<div class="outline-text-2" id="text-5">
<ul class="org-ul">
<li><a href="http://www.agner.org/optimize/optimizing_cpp.pdf">Optimizing software in C++</a>: One to bookmark now but to digest
later. A whole load of stuff on optimisation.
</li>
<li><a href="http://blogs.msdn.com/b/vcblog/archive/2015/12/15/support-for-android-cmake-projects-in-visual-studio.aspx">Support for Android CMake projects in Visual Studio</a>: So, as if the
latest patches to Clang hadn't been enough, MS now decides to add
support for CMake in Visual Studio. A bit embryonic, and a bit too
android focused, but surely it should be extensible for more regular
C++ use. Whats going on at MS? This is all far too cool to be true.
</li>
<li><a href="http://probablydance.com/2015/12/19/quickly-loading-things-from-disk/">Quickly Loading Things From Disk</a>: interesting analysis about the
state of affairs of serialisation in C++. I'll probably require a
few passes to fully digest it.
</li>
<li><a href="https://www.youtube.com/watch?v=FYtBv_OosYw">Beyond ad-hoc automation: leveraging structured platforms</a>: I've been
consuming this presentation slowly but steadily. It deals with a lot
of the questions we all have about the new world of containers and
microservices, and it seems vital to learn from experience before
one finds oneself in a much bigger mess than the monolith could ever
get you into. Bridget Kromhout talks intelligently about the
subject.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">Layman Science</h2>
<div class="outline-text-2" id="text-6">
<ul class="org-ul">
<li><a href="http://wavewatching.net/2014/03/31/the-church-of-d-wave/">The Church of D-Wave</a>: So is D-Wave a quantum computer or not? It
appears the verdict is "not", even with the 2X and the <a href="http://googleresearch.blogspot.co.uk/2015/12/when-can-quantum-annealing-win.html">Google paper</a>.
</li>
<li><a href="https://www.youtube.com/watch?v=qZM9JREjnp4">Intelligence and the Brain</a>: Oldish but still very good and
relevant. Another high-level introduction to HTM.
</li>
<li><a href="https://www.newscientist.com/article/dn28452-nasa-probe-shows-how-solar-burps-may-have-stripped-mars-of-water/">NASA probe shows how solar burps may have stripped Mars of water</a>:
How the sun could be responsible for stripping water away from the
red planet.
</li>
<li><a href="https://www.youtube.com/watch?v=GcQWZG50zX0">Artificial Intelligence Through Hierarchical Temporal Memory</a>:
Continuing my adventures in the HTM space, Dr. Paul Cottrell is my
latest find. I'm still not totally sure I understand all concepts in
this video but what I do understand - assuming they have succeeded
in doing what he describes - seem mondo-cool. Basically, it's all
about the application of HTM to Finance and trading. He also
introduces the idea of adding sub-cortical machinery to HTM (which
is just cortical); a most puzzling concept. Once I finish parsing
this video, I intend to move to <a href="https://www.youtube.com/watch?v=dDD7D-fm7Wc&feature=youtu.be">Neuroscience Foundation For
Artificial Intelligence</a>.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-7" class="outline-2">
<h2 id="sec-7">Other</h2>
<div class="outline-text-2" id="text-7">
<ul class="org-ul">
<li><a href="https://www.youtube.com/watch?v=6FFAQuJZmOA">Benjamin Clementine - Le Ring - Live</a>: Haven't totally made up my
mind about Benjamin Clementine, but certainly a very interesting
performance.
</li>
</ul>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-12-21 Mon 23:31</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-54245394557466205932015-12-11T05:13:00.000-08:002015-12-11T07:48:07.726-08:00Nerd Food: Pull Request Driven Development<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Pull Request Driven Development</title>
<!-- 2015-12-11 Fri 13:12 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
Being in this game for the best part of twenty years, I must confess
that its not often I find something that revolutionises my coding
ways. I do tend to try a lot of things, but most of them end up
revealing themselves as fads or are incompatible with my flow. For
instance, I never managed to get <a href="https://en.wikipedia.org/wiki/Behavior-driven_development">BDD</a> to work for me, try as I might. I
will keep trying because it sounds really useful, but it hasn't
clicked just yet.
</p>
<p>
Having said all of that, these moments of enlightenment do
occasionally happen, and when they do, nothing beats that
life-changing feeling. "Pull Request Driven Development" (or PRDD) is
my latest find. I'll start by confessing that "PRDD" as a name was
totally made up for this post and hopefully you can see its rather
tongue in cheek. However, the benefits of this approach are very
real. In fact, I've been using PRDD for a while now but I just never
really noticed its presence creeping in. Today, as I introduced a new
developer to the process, I finally had the eureka moment and saw just
how brilliant it has been thus far. It also made me realise that some
people are not aware of this great tool in the developer's arsenal.
</p>
<p>
But first things first. In order to explain what I mean by PRDD, I
need to provide a bit of context. Everyone is migrating to git these
days, even those of us locked behind corporate walls; in our
particular case, the migration path implied exposure to <a href="https://en.wikipedia.org/wiki/Stash_(software)">Git Stash</a>. For
those not in the know, picture it as an expensive and somewhat less
featureful version of <a href="https://github.com/">GitHub</a>, but with most of the core functionality
there. Of course, I'm sure GitHub is not that cheap for enterprises
either, but hey at least its the tool everyone uses. Anyway -
grumbling or not - we moved to Stash and all development started to
revolve around Pull Requests (PRs), raised for each new <a href="https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow">feature</a>.
</p>
<p>
Not long after PRs were introduced, a particularly interesting habit
started to appear: developers begun opening the PRs earlier and
earlier during the feature cycle rather than waiting to the very
end. Taking this approach to the limit, the idea is that when you
start to work on a new feature, you raise the ticket and the PR
<i>before you write any code at all</i>. In practice - due to Stash's
anachronisms - you need to push at least one commit, but the general
notion is valid. This was never mandated anywhere, and there was no
particular coordination. I guess one possible explanation for this
behaviour is that one wants to get rid of the paperwork as quickly as
possible to get to the coding. At any rate, the causes may be obscure
but the emerging behaviour was not.
</p>
<p>
When you combine early PRs with the <a href="https://sethrobertson.github.io/GitBestPractices/">commit early and commit often</a>
approach - which you should be using anyway - the PR starts to become
a living document; people see your development work as it progresses
and they start commenting on it and possibly even sending you patches
<i>as you go along</i>. In a way, this is an enabler for a very efficient
kind of peer programming - particularly if you have a tightly knit
team - because it gives you maximum parallelism but in a very subtle,
non-noticeable way. The main author of the PR is coding as she would
normally be, but whenever there is a lull in development - those
moments where you'd be browsing the web for five minutes or so - you
can quickly check for any comments on your PR and react to
those. Similarly, other developers can carry on doing their own work
and browse the PRs on their downtime; this allows them to provide
feedback whenever it is convenient <i>to them</i>, and to choose the format
of the feedback - lengthy or quick, as time permits.
</p>
<p>
Quick feedback is many a times invaluable in large code bases because
everyone tends to know their own little corner of the code and only
very few old hands know how it all hangs together. Thus, seemingly
trivial one liners such as "have you considered using API xyz instead
of rolling your own" or "don't forget to do abc when you do that"
could save you <i>many</i> hours of pain and enable knowledge to be
transferred organically - something that no number of wiki pages could
hope to achieve in a million years because its very difficult to find
these pearls in a sea of uncurated content. And because you committed
early and often, each commit is very small and very easy to parse in a
small interval of time, so people are much more willing to review - as
opposed to that several Kb (or even Mb!) patch that you will have to allocate a day or two
for. Further: if you <a href="http://chris.beams.io/posts/git-commit/">take your commit message seriously</a> - as, again,
you should - you will find that the number of reviewers will grow
rapidly simply because developers are nosy and opinionated.
</p>
<p>
Note that this review process involves no vague meetings and no
lengthy and unfocused email chains; it is very high-quality because it
is (or can be) very focused to specific lines of code; it causes no
unwanted disruptions because you review where and when you choose to
review; reviewers can provide examples and even fix things themselves
if they so choose; it is totally inclusive because anyone who wants to
participate can, but no one is forced to; and it equalises local and
remote developers because they all have access to the same data
(modulus some IRL conversations that always take place) - an important
feature in this world of near-shoring, off-shoring and
home-working. Most importantly, instead of finding out some
fundamental errors of approach at the end of an intense period of
coding, you now have timely feedback. This saves an <i>enormous</i> amount
of time - an advantage that anyone who has been through lengthy code
reviews and then spent a week or two reacting to that feedback can
appreciate.
</p>
<p>
I am now a believer in PRDD. So much so that whenever I go back to
work on legacy projects in svn, I find myself cringing all the way to
the end of the feature. It just feels so nineties.
</p>
<p>
<b>Update:</b> As I finished penning this post and started reflecting about it
it suddenly dawned on me that a lot of things we now take for granted are only
possible because of git. And I don't mean DVCS', I specifically mean git. For
example PRDD is made possible to a large extent because committing in git is a
reversible process and history can be fluid if required. This means that people
are not afraid of committing, which in turn enables a lot of the goodness I described
above. Many DVCS' didn't like this way of viewing history - and to be fair, I know of
very few people that liked the idea until they started using it. Once you figure out
what it is good for (and not so good for), it suddenly becomes an amazing tool. Git
is full of little decisions like this that at first sight look either straight insane
or just not particularly useful but then turn out to change entire development flows.
</p>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-12-11 Fri 13:12</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-21008615932466798552015-12-09T04:50:00.000-08:002015-12-09T07:02:05.166-08:00Nerd Food: Interesting...<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Interesting…</title>
<!-- 2015-12-09 Wed 12:49 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
Time to flush all those tabs again. Some interesting stuff I bumped
into recently-ish.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Finance, Economics, Politics</h2>
<div class="outline-text-2" id="text-1">
<ul class="org-ul">
<li><a href="https://www.propublica.org/article/debt-collection-lawsuits-squeeze-black-neighborhoods">The Color of Debt: How Collection Suits Squeeze Black Neighborhoods</a>:
Another great example of how markets are not so efficient for
certain things and not exactly fair.
</li>
<li><a href="http://www.economist.com/news/briefing/21678215-world-entering-third-stage-rolling-debt-crisis-time-centred-emerging">Pulled back in</a>: The Economist's take on how the emerging markets
credit bubble will play out. Not sure if I agree with their
analysis, but its certainly very worrying to see so much EM debt
piling up in such a volatile world.
</li>
<li><a href="http://www.politico.com/magazine/story/2015/11/isil-whos-calling-the-shots-213360">ISIL: Who’s Calling the Shots?</a>: Interesting analysis, much better
than the usual superficial take one is used to from mass-media.
</li>
<li><a href="http://www.newyorker.com/magazine/2015/08/31/the-other-france">The Other France</a>: As with the previous article, I cannot help but be
surprised - even more so with this one. Truly, an amazing, in-depth
job. Surprisingly good coming from the American media. If you have
watched <a href="http://www.imdb.com/title/tt0113247/">La Haine</a>, read this. If you have read this, watch La Haine.
</li>
<li><a href="https://www.youtube.com/watch?v=iavquu6PP9g">Elon Musk talks Climate Change and Carbon Tax at the Sorbone
(12.2.15)</a>: Musk raises some interesting points, as usual, such as
why we need to tax carbon or else.
</li>
<li><a href="http://www.theguardian.com/society/2015/nov/29/future-of-work-gig-sharing-economy-juggling-jobs">'My father had one job in his life, I've had six in mine, my kids
will have six at the same time'</a>: The Guardian's take in this new
world of job displacement and "job context switching". Very
interesting.
</li>
<li><a href="https://www.youtube.com/watch?v=LZbsxs_VGtQ&feature=youtu.be">Ship it! QuantLib, IPython Notebook, and Docker</a>: QuantLib conference
is over, and sadly there are very few videos. This one bucks the
trend. The ever informative Luigi talks about how QuantLib is
moving with the times.
</li>
<li><a href="http://www.theguardian.com/business/2015/dec/07/morgan-stanley-axes-400-bankers-bonds-jobs?CMP=Share_AndroidApp_Seesmic">Morgan Stanley axes 400 bankers as bond-trading income dives</a>: The
contraction of the traditional banking industry continues, even as
cryptos are growing insanely.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Startups et al.</h2>
<div class="outline-text-2" id="text-2">
<ul class="org-ul">
<li><a href="http://blog.erratasec.com/2015/12/tesla-is-copying-apples-business-model.html#.VmL8A3VX9hG">Tesla is copying Apple's business model</a>: very interesting comparison
between Tesla and Apple's businesses. I don't fully agree with the
article, but to be fair it does raise a number of interesting
points. I definitely think that when Tesla can deliver mass-market
quantities they will dominate sales in a similar fashion to the
iPhone.
</li>
<li><a href="http://www.theguardian.com/technology/2015/nov/29/arm-cambridge-britain-tech-company-iphone">ARM: Britain's most successful tech company you've never heard of</a>:
Short history of ARM. It would be great to have a book about these
guys!
</li>
<li><a href="https://medium.com/backchannel/doordash-wants-to-own-the-last-mile-27c03098a657">DoorDash Wants to Own the Last Mile</a>: interesting story of a startup
that focuses on "last mile" delivery.
</li>
<li><a href="https://www.bitpesa.co">BitPesa</a>: cool African start-up in the BitCoin / MPesa space.
</li>
<li><a href="http://ventureburn.com/2015/10/lulalend-true-fintech-company-mixing-tech-finance/">LulaLend</a>: Another cool African start-up that is doing well in the
payments space.
</li>
<li><a href="https://www.youtube.com/watch?v=SqEo107j-uw">Elon Musk and Y Combinator President on Thinking for the Future</a>:
Altman and Musk discuss the future. Shame the presenter is not a bit
geekier or it could have been one of the best.
</li>
<li><a href="https://www.youtube.com/watch?v=WwrEQklDoyE">Elon Musk with his Brother Kimbal Musk on a panel</a>: Since we're doing
the Musk fanboy thing, here's a great panel with Elon and his
brother. A more personal view of his achievements.
</li>
<li><a href="http://www.bloomberg.com/news/articles/2015-11-24/jeff-bezos-vs-elon-musk-a-thrilling-new-space-race">Jeff Bezos vs. Elon Musk: A Thrilling, New Space Race</a>: More Musk
fanboying; lets go all the way and read up on the latest about
the space race. Very interesting.
</li>
<li><a href="https://www.youtube.com/watch?v=u6IZRjP39do">Tesla Shareholders Meeting June 2015</a>: Final Musk fanboying. I think Tesla is one of the
few companies where non-shareholders tune in just to listen and get
inspiration. Elon, nerdy and awkward but great and inspiring as
always. Choice quote: "I'd expect SpaceX to go public once we get
regular flights to mars." - very few people could get away with a
statement like that.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">General Coding</h2>
<div class="outline-text-2" id="text-3">
<ul class="org-ul">
<li><a href="http://www.nytimes.com/2015/11/13/technology/gene-amdahl-pioneer-of-mainframe-computing-dies-at-92.html?smprod=nytcore-ipad&smid=nytcore-ipad-share">Gene Amdahl, Pioneer of Mainframe Computing, Dies at 92</a>: I've heard
the name a lot but never really read about the man.
</li>
<li><a href="http://jvns.ca/blog/2015/11/21/why-you-should-understand-a-little-about-tcp/">Why you should understand (a little) about TCP</a>: The new generation
discovers the joys of understanding low-level protocols. And Nagle
(yes, he of <a href="https://en.wikipedia.org/wiki/Nagle%2527s_algorithm">Nagle Algorithm</a> fame) replies on that thread.
</li>
<li><a href="https://www.youtube.com/channel/UCvq_RgZp3kljp9X8Io9Z1DA">systemd.conf</a>: Videos from the conference. Have watched a couple,
seemed like a lively conference. Hard to imagine an init system with
its own conference though!
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Databases</h2>
<div class="outline-text-2" id="text-4">
<ul class="org-ul">
<li><a href="http://blog.2ndquadrant.com/when-are-we-going-to-contribute-bdr-to-postgresql/">When are we going to contribute BDR to PostgreSQL?</a>: For those (like
me) who keep moaning about the lack of <a href="https://en.wikipedia.org/wiki/Multi-master_replication">BDR</a> in Postgres, a great
explanation of how the patchset is being merged. Great work by the
2nd Quadrant guys.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">C++</h2>
<div class="outline-text-2" id="text-5">
<ul class="org-ul">
<li><a href="http://blog.llvm.org/2015/11/new-elf-linker-from-llvm-project.html">New ELF Linker from the LLVM Project</a>: LLVM keeps on delivering! Now
a new ELF linker. To be totally honest, I haven't even started using
<a href="https://en.wikipedia.org/wiki/Gold_(linker)">Gold</a> in anger - I get the feeling the LLVM linker is going to be
transitioned in much quicker than Gold.
</li>
<li><a href="http://blogs.msdn.com/b/vcblog/archive/2015/12/04/introducing-clang-with-microsoft-codegen-in-vs-2015-update-1.aspx">Clang with Microsoft CodeGen in VS 2015 Update 1</a>: OMG, OMG how cool
is this - MSFT decided to create a backend for Clang that is totally
compatible with MSVC <span class="underline">AND</span> open source it! This is just insane. This
means for example that you now can develop C++ on Windows without
ever having to use MSVC and Visual Studio. It also means you can
cross-compile from Linux into Windows with 100% certainty things
will work. It means that projects like <a href="https://www.winehq.org/">Wine</a> and <a href="https://www.reactos.org/">ReactOS</a> can start
thinking about a migration path into Clang (not quite as simple as
it may sound but surely makes sense). <a href="https://www.jetbrains.com/clion/">CLion</a> with Clang on Windows
will rock. The possibilities are just endless. I never quite
understood what C2 was all about until I read this announcement -
<a href="http://www.theregister.co.uk/2015/10/21/microsoft_promises_clang_for_windows_in_november_visual_c_update/">suddenly it all makes sense</a>. This is <i>fantastic</i> news.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">Layman Science</h2>
<div class="outline-text-2" id="text-6">
<ul class="org-ul">
<li><a href="http://www.wired.com/brandlab/2015/05/jeff-hawkins-firing-silicon-brain/">Jeff Hawkins on Firing Up the Silicon Brain</a>: OK, let me totally
honest: I <span class="underline">love</span> Jeff Hawkins. I read On Intelligence far too many
times to count and would be lying if I didn't admit that it had a
little bit to do with my forays into Computational
Neuroscience. So as you can imagine, I'm rather excited about <a href="https://en.wikipedia.org/wiki/Hierarchical_temporal_memory">HTM</a>
and <a href="https://en.wikipedia.org/wiki/Numenta">Numenta's</a> latest developments. This article is a good catch-up,
if slightly high-level. If you want something slightly more
technical but still very approachable, <a href="https://www.youtube.com/watch?v=6ufPpZDmPKA">Principles of Hierarchical
Temporal Memory (HTM): Foundations of Machine Intelligence</a> is a must
watch.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-7" class="outline-2">
<h2 id="sec-7">Other</h2>
<div class="outline-text-2" id="text-7">
<ul class="org-ul">
<li><a href="https://www.youtube.com/watch?v=lWJkrP4WPFw">NoiseRV Live</a>: Still discovering this Portuguese musician, but love
his work. Great concert. Could do a little bit less talking between
songs, but still - artists prerogative and all that.
</li>
<li><a href="http://bff.fm/broadcasts/4499">Warm Focus: Winging It</a>: Interesting set of "intelligent dance music"
as we used to call it back in the day.
</li>
<li><a href="https://overcast.fm/+Bj7wZZ3Mg">Mosaic - The “First” Web Browser</a>: Super-cool podcasts about internet
history. It would be great to have something like this for UNIX!
</li>
<li><a href="https://www.youtube.com/watch?v=0va3F2PWBJc">Jackson C. Frank (1965)</a>: Tragic musician from the 60s. Great tunes.
</li>
<li><a href="http://www.gutenberg.org/files/15000/15000-h/vol1.html">Reason in common sense</a>: Always wanted to read Santayana
properly. Started, but I guess it will be a <span class="underline">very</span> long
exercise. Interesting, if somewhat strange book.
</li>
<li><a href="https://www.youtube.com/watch?v=OSD1mud8JZ8">Ceu - jazz baltica Live (2010)</a>: New find, Brazilian musician Ceu.
</li>
</ul>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-12-09 Wed 12:49</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-51619286794282373152015-11-30T15:12:00.000-08:002019-01-21T08:22:42.975-08:00Nerd Food: Tooling in Computational Neuroscience - Part II: Microscopy<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Tooling in Computational Neuroscience - Part II: Microscopy</title>
<!-- 2015-11-30 Mon 23:12 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p class="verse" style="text-align:right">
<small>
Research is what I'm doing when I don't know what I'm doing.
<br>
<i>Wernher von Braun</i>
</small>
</p>
<p>
Welcome to the second instalment of our second series on Computational
Neuroscience for lay people. You can find the first post of the
previous series <a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks.html">here</a>, and the first post of the current series
<a href="http://mcraveiro.blogspot.co.uk/2015/11/nerd-food-tooling-in-computational.html">here</a>. As you'd expect, this second series is slightly more advanced,
and, as such, it is peppered with unavoidable technical jargon. Having
said that, we shall continue to pursue our ambitious target of making
things as easy to parse as possible (but no easier). If you read the
first series, the second should hopefully make some sense.<sup><a id="fnr.1" name="fnr.1" class="footref" href="#fn.1">1</a></sup>
</p>
<p>
Our <a href="http://mcraveiro.blogspot.co.uk/2015/11/nerd-food-tooling-in-computational.html">last post</a> discussed <a href="https://en.wikipedia.org/wiki/Computational_neuroscience">Computational Neuroscience</a> as a discipline,
and the kind of things one may want to do in this field. We also spoke
about models and their composition, and the desirable properties of a
platform that runs simulations of said models. However, it occurred to
me that we should probably build some kind of "end-to-end"
understanding; that is, by starting with the simulations and models we
are missing a vital link with the physical (i.e. non-computational)
world. To put matters right, this part attempts to provide a
high-level introduction on how data is acquired from the real world
and can then be used - amongst other things - to inform the modeling
process.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Macro and Micro Microworlds</h2>
<div class="outline-text-2" id="text-1">
<p>
For the purposes of this post, the data gathering process starts with the
microscope. Of course, keep in mind that we are focusing only on the
<i>morphology</i> at present - the shape and the structures that make up
the neuron - so we are ignoring other important activities in the
lab. For instance, one can conduct experiments to measure voltage in a
neuron, and these measurements provide data for the functional aspects
of the model. Alas, we will skip these for now, with the promise of
returning to them at a later date<sup><a id="fnr.2" name="fnr.2" class="footref" href="#fn.2">2</a></sup>.
</p>
<p>
So, microscopes then. <a href="https://en.wikipedia.org/wiki/Microscopy">Microscopy</a> is the technical name for the
observation work done with the microscope. Because neurons are so
small - some 4 to 100 microns in size - only certain types of
microscopes are suitable to perform neuronal microscopy. To make
matters worse, the sub-structures inside the neuron are an important
area of study and they can be ridiculously small: a <a href="https://en.wikipedia.org/wiki/Dendritic_spine">dentritic spine</a> -
the minute protrusions that come out of the dendrites - can be as tiny
as 500 nanometres; the lipid bylayer itself is only 2 or 3 nanometres
thick, so you can imagine how incredibly small ion channels and pumps
are. Yet these are some of the things we want to observe and
measure. Lets call this the "micro" work. On the other hand, we also
want to understand connectivity and other larger structures, as well
as perform observations of the evolution of the cell and so on. Lets
call this the "macro" work. These are not technical terms, by the by,
just so we can orient ourselves. So, how does one go about observing
these differently sized microworlds?
</p>
<div class="figure">
<p><img src="http://www.pnas.org/content/106/39/16877/F1.large.jpg" alt="F1.large.jpg" width="300px" height="300px" />
</p>
<p><span class="figure-number">Figure 1:</span> Example of measurements one may want to perform on a dendrite. Source: <a href="http://www.pnas.org/content/106/39/16877.abstract">Reversal of long-term dendritic spine alterations in Alzheimer disease models</a></p>
</div>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Optical Microscopy</h2>
<div class="outline-text-2" id="text-2">
<p>
The "macro" work is usually done using the <a href="https://en.wikipedia.org/wiki/Optical_microscope">Optical</a> "family" of
microscopes, which is what most of us think of when hearing the word
microscope. As it was with <a href="https://en.wikipedia.org/wiki/Microscope">Van Leeuwenhoek's</a> tool in the sixteen
hundreds, so it is that today's optical microscopes still rely on
light and lenses to perform observations. Needless to say, things did
evolve a fair bit since then, but standard optical microscopy has not
completely removed the shackles of its limitations. These are of three
kinds, as Wikipedia helpfully <a href="https://en.wikipedia.org/wiki/Microscopy#Optical_microscopy">tells us</a>: a) the objects we want to
observe must be dark or strongly refracting - a problem, since the
internal structures of the cell are transparent; b) visible light's
<a href="https://en.wikipedia.org/wiki/Diffraction-limited_system">diffraction limit</a> means that we cannot go much lower than 200
nanometres - pretty impressive, but unfortunately not quite low enough
for detailed sub-structure analysis; and c) out of focus light hampers
image clarity.
</p>
<p>
Workarounds to these limitations have been found in the guise of
<i>techniques</i>, with the aim of augmenting the abilities of standard
optical microscopy. There are many of these techniques. There is the
<a href="https://en.wikipedia.org/wiki/Confocal_microscopy">Confocal Microscopy</a><sup><a id="fnr.3" name="fnr.3" class="footref" href="#fn.3">3</a></sup> - improving resolution and contrast;
the <a href="https://en.wikipedia.org/wiki/Fluorescence_microscope">Fluorescence microscope</a>, which uses a <i><a href="https://en.wikipedia.org/wiki/Microscopy#Sub-diffraction_techniques">sub-diffraction technique</a></i>
to reconstruct some of the detail that is missing due to diffraction;
or the incredible-looking movies produced by <a href="http://blogs.scientificamerican.com/expeditions/journey-through-the-brain-multiphoton-microscopy/">Multiphoton
Microscopy</a>. And of course, it is possible to combine multiple
techniques in a single microscope, as is the case with the <a href="https://en.wikipedia.org/wiki/Multiphoton_fluorescence_microscope">Multiphoton
Fluorescence Microscopes</a> (MTMs) and many others.
</p>
<p>
In fact, given all of these developments, it seems there is no sign of
optical microscopy dying out. Presumably some of this is due to the
relative lower cost of this approach as well as to the ease of use. In
addition, optical microscopy is complementary to the other more
expensive types of microscopes; it is the perfect tool for "macro"
work that can then help to point out where to do "micro" work. For
example, you can use an optical microscope to assess the larger
structures and see how they evolve over time, and eventually decide on
specific areas that require more detailed analysis. And when you do,
you need a <i>completely</i> different kind of microscope.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Electron Microscopy</h2>
<div class="outline-text-2" id="text-3">
<p>
When you need <i>really</i> high-resolution, there is only one tool to turn
to: the <a href="https://en.wikipedia.org/wiki/Electron_microscope">Electron Microscope (EM)</a>. This crazy critter can provide
<i>insane</i> levels of magnification by using a beam of electrons instead
of visible light. Just how insane, you ask? Well, if you think that an
optical microscope lives in the range of 1500x to 2000x - that is, can
magnify a sample up to two thousand times - an EM can magnify as much
as 10 <span class="underline">million times</span>, and provide a sub-nanometre
resolution<sup><a id="fnr.4" name="fnr.4" class="footref" href="#fn.4">4</a></sup>. It is mind boggling. If fact, we've already
seen images of atoms using EM in <a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks_31.html">part II</a>, but perhaps it wasn't easy
to appreciate just how amazing a feat that is.
</p>
<p>
Of course, EM is itself a family - and a large one at that, with many
and diverse members. As with optical microscopy, each member of the
family specialises on a given technique or combination of
techniques. For example, the <a href="https://en.wikipedia.org/wiki/Scanning_electron_microscope">Scanning Electron Microscope</a> (SEM)
performs a scan of the object under study, and has a resolution of 1
nanometre or higher; the <a href="https://en.wikipedia.org/wiki/Scanning_confocal_electron_microscopy">Scanning Confocal Electron Microscope (SCEM)</a>
uses the same confocal technique mentioned above to provide higher
depth resolution; and <a href="https://en.wikipedia.org/wiki/Transmission_electron_microscopy">Transmission Electron Microscopy</a> (TEM) has the
ability to penetrate inside the specimen during the imagining process,
given samples with thickness of 100 nanometres or less.
</p>
<p>
A couple of noteworthy points are required at this juncture. First,
whilst some of these EM techniques may sound new and exciting, most
have been around for a <i>very</i> long time; it just seems they keep
getting better and better as they mature. For example, TEM was used in
the fifties to show that neurons communicate over synaptic junctions
but its still wildly popular today. Secondly, its important to
understand that the entire imaging process is not at all trivial -
certainly not for TEM, nor EM in general and probably not for Optical
Microscopy either. It just is a <i>very</i> labour intensive and <i>very</i>
specialised process - most likely done by an expert human
neuroanatomist - and the difficulties range from the chemical
preparation of the samples all the way up to creating the images. The
end product may give the impression it was easy to produce, but easy
it was not.
</p>
<p>
At any rate, whatever the technical details, the fact is that the
imagery that results from all these advances is truly evocative -
haunting, even. Take this image produced by SEM:
</p>
<div class="figure">
<p><img src="http://ucsdnews.ucsd.edu/news_uploads/2014_06_26_human_ipsc_derived_neuron_deerinck.jpg" alt="2014_06_26_human_ipsc_derived_neuron_deerinck.jpg" width="300px" height="300px" />
</p>
<p><span class="figure-number">Figure 2:</span> Human neuron. <a href="http://ucsdnews.ucsd.edu/pressrelease/new_reprogramming_method_makes_better_stem_cells">Source: New Reprogramming Method Makes Better Stem Cells</a></p>
</div>
<p>
Personally, I think it is incredibly beautiful; simultaneously
awe-inspiring and depressing because it really conveys the messiness
and complexity of wetware. By way of contrast, look at the neatness of
man-made micro-structures:
</p>
<div class="figure">
<p><img src="http://m.eet.com/media/1118299/bluegeneq%20x%20420.jpg" alt="bluegeneq%20x%20420.jpg" width="300px" height="300px" />
</p>
<p><span class="figure-number">Figure 3:</span> The BlueGene/Q chip. Source: <a href="http://www.eetimes.com/document.asp?doc_id=1260096">IBM plants transactional memory in CPU</a></p>
</div>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Stacks and Stacks of 'Em</h2>
<div class="outline-text-2" id="text-4">
<p>
Technically, pictures like the ones above are called <a href="https://en.wikipedia.org/wiki/Micrograph">micrographs</a>. As
you can see in the neuron micrograph, these images provide a great
visual description of the topology of the object we are trying to
study. You also may notice a slight coloration of the cell in that
picture. This is most likely due to the fact that the people doing the
analysis <a href="https://en.wikipedia.org/wiki/Staining">stain</a> the neuron to make it easier to image. Now, in
practice - at least as far as I have seen, which is not very far at
all, to be fair - 2D grayscale images are preferred by researchers to
the nice, Public Relations friendly pictures like the one above; those
appear to be more useful for magazine covers. The working micrographs
are not quite as exciting to the untrained eye but very useful to the
professionals. Here's an example:
</p>
<div class="figure">
<p><img src="http://www.leet.it/home/giusti/website/lib/exe/fetch.php?w=900&tok=d88a10&media=wiki:biomed-neurons.jpg" alt="fetch.php?w=900&tok=d88a10&media=wiki:biomed-neurons.jpg" width="600px" height="200px" />
</p>
<p><span class="figure-number">Figure 4:</span> The left-hand side shows the original micrograph. On the right-hand side it shows the result of processing it with machine learning. Source: <a href="http://papers.nips.cc/paper/4741-deep-neural-networks-segment-neuronal-membranes-in-electron-microscopy-images.pdf">Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images</a></p>
</div>
<p>
Let's focus on the left-hand side of this image for the moment. It was
taken using <i>ssTEM</i> - serial-section TEM, an evolutionary step in
TEM. The <i>ss</i> part of ssTEM is helpful in creating <i>stacks</i> of images,
which is why you see the little drawings on the left of the picture;
they are there to give you the idea that the top-most image is one of
30 in a stack<sup><a id="fnr.5" name="fnr.5" class="footref" href="#fn.5">5</a></sup>. The process of producing the images above was
as follows: they started off with a neuronal tissue sample, which is
prepared for observation. The sample had 1.5 micrometres and was then
sectioned into 30 slices of 50 nanometres. Each of these slices was
imaged, at a resolution of 4x4 nanometres per pixel.
</p>
<p>
As you can imagine, this work is extremely sensitive to measurement
error. The trick is to ensure there is some kind of visual continuity
between images so that you can recreate a 3D model from the 2D
slices. This means for instance that if you are trying to figure out
connectivity, you need some way to relate a dendrite to it's soma and
say to the axon of the neuron it connects to - and that's one of the
reasons why the slices have to be so thin. It would be no good if the
pictures miss this information out as you will not be able to recreate
the connectivity faithfully. This is actually really difficult to
achieve in practice due to the minute sizes involved; a slight tremor
that displaces the sample by some nanometres would cause shifts in
alignment; even with the high-precision the tools have, you can
imagine that there is always some kind of movement in the sample's
position as part of the slicing process.
</p>
<p>
Images in a stack are normally stored using traditional formats such
as <a href="https://en.wikipedia.org/wiki/Tagged_Image_File_Format">TIFF</a><sup><a id="fnr.6" name="fnr.6" class="footref" href="#fn.6">6</a></sup>. You can see an example of the raw images in a stack
<a href="https://github.com/unidesigner/groundtruth-drosophila-vnc/tree/master/stack2/raw">here</a>. Its worth noticing that, even though the images are 2D
grey-scale, since the pixel size is only a few nanometres wide (4x4 in
this case), the full size of an image is very large. Indeed, the
latest generation of microscopes produce stacks on the 500 Terabyte
range, making the processing of the images a "big-data" challenge.
</p>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">What To Do Once You Got the Images</h2>
<div class="outline-text-2" id="text-5">
<p>
But back to the task at hand. Once you have the stack, the next
logical step is to try to figure out what's what: which objects are in
the picture. This is called segmentation and labelling, presumably
because you are breaking the one big monolithic picture into discrete
objects and give them names. Historically, segmentation has been
done manually, but its a painful, slow and error-prone process. Due to
this, there is a lot of interest in automation, and it has recently
become feasible to do so - what with the abundance of cheap computing
resources as well as the advent of "useful" <a href="https://en.wikipedia.org/wiki/Machine_learning">machine learning</a> (rather
than the theoretical variety). Cracking this puzzle is gaining
traction amongst the programming herds, as you can see by the
popularity of challenges such as this one: <a href="http://fiji.sc/Segmentation_of_neuronal_structures_in_EM_stacks_challenge_-_ISBI_2012">Segmentation of neuronal
structures in EM stacks challenge - ISBI 2012</a>. It is from this
challenge we sourced the stack and micrograph above; the right-hand
side is the finished product after machine learning processing.
</p>
<p>
There are also open source packages to help with segmentation. A
couple of notable contenders are <a href="http://fiji.sc/Fiji">Fiji</a> and <a href="http://ilastik.org/">Ilastik</a>. Below is a
screenshot of Ilastik.
</p>
<div class="figure">
<p><img src="https://raw.githubusercontent.com/ilastik/ilastik.github.io/master/gallery/Figure-2-a.png" alt="Figure-2-a.png" width="400px" height="300px" />
</p>
<p><span class="figure-number">Figure 5:</span> Source: <a href="http://ilastik.org/gallery.html#">Ilastik gallery</a>.</p>
</div>
<p>
An activity that naturally follows on from segmentation and labelling
is <a href="https://en.wikipedia.org/wiki/Neuronal_tracing">reconstruction</a>. The objective of reconstruction is to try to
"reconstruct" morphology given the images in the stack. It could
involve inferring the missing bits of information by mathematical
means or any other kind of analysis which transforms the set of
discrete objects spotted by segmentation into something looking more
like a bunch of connected neurons.
</p>
<p>
Once we have a reconstructed model, we can start performing
<i>morphometric analysis</i>. As wikipedia tells us, <a href="https://en.wikipedia.org/wiki/Morphometrics">Morphometry</a> is "the
quantitative analysis of form"; as you can imagine, there are a lot of
useful things one may want to measure in the brain structures and
sub-structures such as lengths, volumes, surface area and so on. Some
of these measurements can of course be done in 2D, but life is made
easier if the model is available in 3D. One such tool is
<a href="http://wiki.blender.org/index.php/Extensions:2.6/Py/Scripts/Neuro_tool">NeuroMorph</a>. It is an open source extension written in Python for the
popular open source 3D computer graphics software <a href="https://en.wikipedia.org/wiki/Blender_(software)">Blender</a>.
</p>
<div class="figure">
<p><img src="https://en.blender.org/uploads/9/98/NeuroMorph_screenshot.png" alt="NeuroMorph_screenshot.png" width="300px" height="300px" />
</p>
<p><span class="figure-number">Figure 6:</span> Source: <a href="http://figshare.com/articles/Segmented_anisotropic_ssTEM_dataset_of_neural_tissue/856713">Segmented anisotropic ssTEM dataset of neural tissue</a></p>
</div>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">Conclusion</h2>
<div class="outline-text-2" id="text-6">
<p>
This post was a bit of a world-wind tour of some of the sources of
real world data for Computational Neuroscience. As I soon found out,
each of these sections could have easily been ten times bigger and
still not provide you with a proper overview of the landscape; having
said that, I hope that the post at least gives some impression of the
terrain and its main features.
</p>
<p>
From a software engineering perspective, its worth pointing out the
lack of standardisation in information exchange. In an ideal world,
one would want a pipeline with components to perform each of the steps
of the complete process, from data acquisition off of a microscope
(either opitical or EM), to segmentation, labelling, reconstruction
and finally morphometric analysis. This would then be used as an input
to the models. Alas, no such overarching standard appears to exist.
</p>
<p>
One final point in terms of Free and Open Source Software (FOSS). On
one hand, it is encouraging to see the large number of FOSS tools and
programs being used. Unfortunately - at least for the lovers of Free
Software - there are also some proprietary tools that are widely used
such as <a href="http://www.mbfbioscience.com/neurolucida">NeuroLucida</a>. Since the software is so specialised, the fear is
that in the future, the better funded commercial enterprises will take
over more and more of the space.
</p>
<p>
That's all for now. Don't forget to tune in for the next instalment!
</p>
</div>
</div>
<div id="footnotes">
<h2 class="footnotes">Footnotes: </h2>
<div id="text-footnotes">
<div class="footdef"><sup><a id="fn.1" name="fn.1" class="footnum" href="#fnr.1">1</a></sup> <p class="footpara">
As it happens, what we are doing here is to apply a
well-established learning methodology called the <a href="https://www.farnamstreetblog.com/2012/04/learn-anything-faster-with-the-feynman-technique/">Feynman Technique</a>. I
was blissfully unaware of its existence all this time, even though
<a href="https://en.wikipedia.org/wiki/Richard_Feynman">Feynman</a> is one of my heroes and even though I had read a fair bit
about the man. On this topic (and the reason why I came to know about
the Feynman Technique), its worth reading <a href="https://www.farnamstreetblog.com/2015/01/richard-feynman-knowing-something/">Richard Feynman: The
Difference Between Knowing the Name of Something and Knowing
Something</a>, where Feynman discusses his disappointment with science
education in Brazil. Unfortunately the Portuguese and the Brazilian
teaching systems have a lot in common - or at least they did when I
was younger.
</p></div>
<div class="footdef"><sup><a id="fn.2" name="fn.2" class="footnum" href="#fnr.2">2</a></sup> <p class="footpara">
Nor is the microscope the only way to figure out
what is happening inside the brain. For example, there are
<a href="https://en.wikipedia.org/wiki/Neuroimaging">neuroimagining</a> techniques which can provide data about both structure
and function.
</p></div>
<div class="footdef"><sup><a id="fn.3" name="fn.3" class="footnum" href="#fnr.3">3</a></sup> <p class="footpara">
Patented by <a href="https://en.wikipedia.org/wiki/Marvin_Minsky">Marvin Minsky</a>, no less - yes, he of Computer
Science and AI fame!
</p></div>
<div class="footdef"><sup><a id="fn.4" name="fn.4" class="footnum" href="#fnr.4">4</a></sup> <p class="footpara">
And, to be fair, sub-nanometre just doesn't quite
capture just how low these things can go. For an example, read
<a href="http://www.ncbi.nlm.nih.gov/pubmed/21844593">Electron microscopy at a sub-50 pm resolution</a>.
</p></div>
<div class="footdef"><sup><a id="fn.5" name="fn.5" class="footnum" href="#fnr.5">5</a></sup> <p class="footpara">
For a more technical but yet short and understandable take,
read <a href="http://www.jneurosci.org/content/26/47/12101.full">Uniform Serial Sectioning for Transmission Electron Microscopy</a>.
</p></div>
<div class="footdef"><sup><a id="fn.6" name="fn.6" class="footnum" href="#fnr.6">6</a></sup> <p class="footpara">
On the topic of formats: its probably time we mention the
<a href="https://www.openmicroscopy.org/site">Open Microscopy Environment</a> (OME). The microscopy world is dominated
by hardware and as such its the perfect environment for corporations,
their proprietary formats and expensive software packages. The OME
guys are trying to buck the trend by creating a suite of open source
tools and protocols, and by looking at some of <a href="http://help.openmicroscopy.org/viewing-data.html#screen">their stuff</a>, they seem
to be doing alright.
</p></div>
</div>
</div></div>
<div id="postamble" class="status">
<p class="date">Created: 2015-11-30 Mon 23:12</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-12314254157654773052015-11-11T10:01:00.000-08:002015-11-16T06:48:17.651-08:00Nerd Food: Tooling in Computational Neuroscience - Part I: NEURON<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Tooling in Computational Neuroscience - Part I</title>
<!-- 2015-11-11 Wed 17:59 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
In the <a href="http://mcraveiro.blogspot.co.uk/2015/09/nerd-food-neurons-for-computer-geeks_16.html">previous series of posts</a> we did a build up of theory - right up
to the point where we were just about able to make sense of <i>Integrate
and Fire</i> - one of the simpler families of neuron models. The series
used a <a href="https://en.wikipedia.org/wiki/Reductionism">reductionist</a> approach - or bottom up, if you
prefer<sup><a id="fnr.1" name="fnr.1" class="footref" href="#fn.1">1</a></sup>. We are now starting a new series with the
opposite take, this time coming at it from the top. The objective is
to provide a (very) high-level overview - in laymen's terms, still -
of a few of the "platforms" used in computational neuroscience. As
this is a rather large topic, we'll try to tackle a couple of
platforms each post, discussing a little bit of their history, purpose
and limitations - whilst trying to maintain a focus on <i>file formats</i>
or DSLs. "File formats" may not sound particularly exciting at first
glance, but it is important to keep in mind that these are instances
of meta-models of the problem domain in question, and as such, their
expressiveness is very important. Understand those and you've
understood a great deal about the domain and about the engineering
choices of those involved.
</p>
<p>
But first, let's introduce <i>Computational Neuroscience</i>.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Computers and the Brain</h2>
<div class="outline-text-2" id="text-1">
<p>
<a href="http://mcraveiro.blogspot.co.uk/2015/09/nerd-food-neurons-for-computer-geeks_7.html">Part V</a> of our previous series discussed some of the reasons why one
would want to model neurons (section <i>Brief Context on
Modeling</i>). What we did not mention is that there is a whole
scientific discipline dedicated to this endeavour, called
Computational Neuroscience. Wikipedia has a <a href="https://en.wikipedia.org/wiki/Computational_neuroscience">pretty good working
definition</a>, which we will take wholesale. It states:
</p>
<blockquote>
<p>
Computational neuroscience […] is the study of brain function in
terms of the information processing properties of the structures that
make up the nervous system. It is an interdisciplinary science that
links the diverse fields of neuroscience, cognitive science, and
psychology with electrical engineering, computer science, mathematics,
and physics.
</p>
<p>
Computational neuroscience is distinct from psychological
connectionism and from learning theories of disciplines such as
machine learning, neural networks, and computational learning theory
in that it emphasizes descriptions of functional and biologically
realistic neurons (and neural systems) and their physiology and
dynamics. These models capture the essential features of the
biological system at multiple spatial-temporal scales, from membrane
currents, proteins, and chemical coupling to network oscillations,
columnar and topographic architecture, and learning and memory.
</p>
<p>
These computational models are used to frame hypotheses that can be
directly tested by biological or psychological experiments.
</p>
</blockquote>
<p>
Lots of big words, of course, but hopefully they make <i>some</i> sense
after the previous posts. If not, don't despair; what they all hint at
is an "interdisciplinary" effort to create biologically plausible
models, and to use these to provide insights on how the brain is
performing certain functions. Think of the Computational
Neuroscientist as the right-hand person of the Neuroscientist - the
"computer guy" to the "business guy", if you like. The Neuroscientist
(particularly the experimental Neuroscientist) gets his or her hands
messy with wetware and experiments, which end up providing data and a
better biological understanding; the Computational Neuroscientist
takes these and uses them to make improved computer models, which are
used to test hypothesis or to make new ones, which can then validated
by experiments and so on, in a virtuous feedback loop.<sup><a id="fnr.2" name="fnr.2" class="footref" href="#fn.2">2</a></sup>
Where the "interdisciplinary" part comes in is that many of the people
doing the role of "computer guys" are actually not computer scientists
but instead come from a variety of backgrounds such as biology,
physics, chemistry and so on. This variety adds a lot of value to the
discipline because the brain is such a complex organ; understanding it
requires all kinds of skills - and then some.
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">It's Models All the Way Down</h2>
<div class="outline-text-2" id="text-2">
<p>
At the core, then, the work of the Computational Neuroscientist is to
create models. Of course, <a href="http://mcraveiro.blogspot.co.uk/2015/09/nerd-food-neurons-for-computer-geeks_7.html">as we already seen</a>, one does not just walk
straight into Mordor and starts creating the "most biologically
plausible" model of the brain possible; all models must have a scope
as narrow as possible, if they are to become a) understandable and b)
computationally feasible. Thus engineering trade-offs are crucial to
the discipline.
</p>
<p>
Also, it is important to understand that creating a model does not
always imply writing things from scratch. Instead, most practitioners
rely on a wealth of software available, all with different advantages
and disadvantages.
</p>
<p>
At this juncture you are probably wondering just what exactly are
these "models" we speak so much of. Are they just equations like IaF?
Well, yes and no. As it happens, all models have roughly the following
structure:
</p>
<ul class="org-ul">
<li><i>a morphology definition</i>: we've already <a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks.html">spoken a bit</a> about
morphology; think of it as the definition of the entities that exist
in your model, their characteristics and relationships. This is
actually closer to what we, computer scientists think the word
modeling means. For example, the morphology defines how many neurons
you have, how many axons and dendrites, connectivity, spatial
positioning and so on.
</li>
<li><i>a functional, mathematical or physical definition</i>: I've heard it
named in many ways, but fundamentally, what it boils down to is the
definition of the equations that your model requires. For example,
are you modeling electrical properties or reaction/diffusion?
</li>
</ul>
<p>
For the simpler models, the morphology gets somewhat obscured - after
all, in LIF, there is very little information about a neuron because
all we are interested in are the spikes. For other models, a lot of
morphological details are required.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">The Tooling Landscape</h2>
<div class="outline-text-2" id="text-3">
</div><div id="outline-container-sec-3-1" class="outline-3">
<h3 id="sec-3-1">Idealised…</h3>
<div class="outline-text-3" id="text-3-1">
<p>
It is important to keep in mind that these models are to be used in a
<i>simulation</i>; that is, we are going to run the program for a period of
time (hours or days) and observe different aspects of its
behaviour. Thus the functional definition of the model provides the
equations that describe the dynamics of the system being simulated and
the morphology will provide some of the inputs for those equations.
</p>
<p>
From here one can start sketch the requirements for a system for the
Computational Neuroscientist:
</p>
<ul class="org-ul">
<li>a platform of some kind to provide simulation control: starting,
stopping, re-running, storing the results and so on. As the
simulations can take a long time to run, the data sets can be quite
large - on the hundreds of gigs range - so efficiently handling of
the output data is a must.
</li>
<li>some kind of DSL that provides a user friendly way to define their
models, ideally with a graphical user interface that helps author
the DSL. The DSL must cover the two aspects we mention above.
</li>
<li>efficient libraries of numerical routines to help solve the
equations. The libraries must be exposed in someway to the DSL so
that users can make use of these when defining the functional
aspects of the model.
</li>
</ul>
<p>
Architecturally, the ability to use a cluster or GPUs would of course
be very useful, but we shall ignore those aspects for now. Given this
idealised platform, we can now make a bit more sense of what actually
exists in the wild.
</p>
</div>
</div>
<div id="outline-container-sec-3-2" class="outline-3">
<h3 id="sec-3-2">… vs Actual</h3>
<div class="outline-text-3" id="text-3-2">
<p>
The multidisciplinary nature of Computational Neuroscience poses some
challenges when it comes to software development: as mentioned, many
of the practitioners in the field do not have a Software Engineering
background; of those that do have, most tend not to have strong
biology and neuroscience backgrounds. As a result, the landscape is
fragmented and the quality is uneven. On one side, most of the
software is open source, making reuse a lot less of a problem. On the
other hand, things such as continuous integration, version control,
portability, user interface guide lines, organised releases, packaging
and so on are still lagging behind most "regular" Free and Open Source
projects<sup><a id="fnr.3" name="fnr.3" class="footref" href="#fn.3">3</a></sup>.
</p>
<p>
In some ways, to enter Computational Neuroscience is a bit like
travelling in time to a era before git, before GitHub, before Travis
and all other things we take for granted. Not everywhere, of course,
but still in quite a few places, particularly with the older and more
popular projects. One cannot help but get the feeling that the field
could do with some of the general energy we have in the FOSS
community, but the technical barriers to contributing tend to be large
since the domain is so complex.
</p>
<p>
So after all of this boring introductory material, we can finally look
at our first system.
</p>
</div>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">NEURON</h2>
<div class="outline-text-2" id="text-4">
<p>
Having to choose, one feels compelled to start with <a href="http://www.neuron.yale.edu/neuron/">NEURON</a> - the most
venerable of the lot, with roots in the 80s<sup><a id="fnr.4" name="fnr.4" class="footref" href="#fn.4">4</a></sup>. NEURON is a
simulation environment with great depth of functionality and a
comprehensive user manual published as a (non-free) <a href="http://ebooks.cambridge.org/ebook.jsf?bid=CBO9780511541612">book</a>. For the less
wealthy, an <a href="http://www.neuron.yale.edu/neuron/static/papers/hbtnn2/overviewforhbtnn2e.pdf">overview paper</a> is available, as are many other <a href="http://www.neuron.yale.edu/neuron/docs">online
resources</a>. The software itself is fully open source, with a <a href="http://www.neuron.yale.edu/hg/neuron/nrn/file/5b5889f69d6e/src">public
mercurial repo</a>.
</p>
<p>
As with many of the older tools in this field, NEURON development has
not quite kept up the pace with the latest and greatest. For instance,
it still has a Motif'esque look to its UI but, alas, do not be
fooled - its not Motif but <a href="https://en.wikipedia.org/wiki/InterViews">InterViews</a> - a technology I never heard of,
but seems to have been popular in the 80's and early 90's. One fears
that NEURON may just be the last widely used program relying on
InterViews - and the fact that they carry <a href="http://www.neuron.yale.edu/hg/neuron/iv/file/91e22c4a0a0c/README">their own fork of it</a> does
not make me hopeful.
</p>
<div class="figure">
<p><img src="http://www.neuron.yale.edu/neuron/static/docs/cbtut/stylized/figs/subset0.gif" alt="subset0.gif" width="400px" height="300px" />
</p>
<p><span class="figure-number">Figure 1:</span> Source: NEURON Cell Builder</p>
</div>
<p>
However, once one goes past these layers of legacy, the domain
functionality of the tool is very impressive. This goes some way to
explain why so many people rely on it daily and why so many papers
have been written using it - over 600 papers at the last count.
</p>
<p>
Whilst NEURON is vast, we are particularly interested in only two
aspects of it: <i>hoc</i> and <i>mod</i> (in its many incarnations). These are
the files that can be used to define models.
</p>
</div>
<div id="outline-container-sec-4-1" class="outline-3">
<h3 id="sec-4-1">Hoc</h3>
<div class="outline-text-3" id="text-4-1">
<p>
<a href="https://en.wikipedia.org/wiki/Hoc_(programming_language)">Hoc</a> has a fascinating history and a pedigree to match. It is actually
the creation of Kernighan and Pike, two UNIX luminaries, and has as
contenders tools like bc and dc and so on. NEURON took hoc and
extended it both in terms of syntax as well as the number of available
functions; <a href="http://www.neuron.yale.edu/neuron/static/docs/refman/hoc.html">NEURON Hoc</a> is now an interpreted object oriented language,
albeit with some limitations such as lack of inheritance. Programs
written in hoc execute in an interpreter called <code>oc</code>. There are a few
variations of this interpreter, with different kinds of libraries made
available to the user (UI, neuron modeling specific functionality,
etc) but the gist of it is the same, and the strong point is the
interactive development with rapid feedback. On the GUI versions of
the interpreter, the script can specify it's UI elements including
input widgets for parameters and widgets to display the output. Hoc is
then used as a mix between model/view logic and morphological definition
language.
</p>
<p>
To get a feel for the language, here's a very simple sample <a href="http://www.neuron.yale.edu/neuron/static/docs/elementarytools/writcode.htm">from the
manual</a>:
</p>
<pre class="example">
create soma // model topology
access soma // default section = soma
soma {
diam = 10 // soma dimensions in um
L = 10/PI // surface area = 100 um^2
}
</pre>
</div>
</div>
<div id="outline-container-sec-4-2" class="outline-3">
<h3 id="sec-4-2">NMODL</h3>
<div class="outline-text-3" id="text-4-2">
<p>
The second language supported by NEURON is <a href="http://www.neuron.yale.edu/neuron/static/docs/help/neuron/nmodl/nmodl.html">NMODL</a> - The NEURON extended
MODL (Model Description Language). NMODL is used to specify a physical
model in terms of equations such as simultaneous nonlinear algebraic
equations, differential equations and so on. In practice, there are
actually different versions of NMODL for different NEURON versions,
but to keep things simple I'll just abstract these complexities and
refer to them as one entity<sup><a id="fnr.5" name="fnr.5" class="footref" href="#fn.5">5</a></sup>.
</p>
<p>
As intimated above, NMODL is a descendant of MODL. As with Hoc, the
history of MODL is quite interesting; it was a language was defined by
the National Biomedical Simulation Resource to specify models for use
with SCoP - the Simulation Control Program<sup><a id="fnr.6" name="fnr.6" class="footref" href="#fn.6">6</a></sup>. From what I
can gather of SCoP, its main purpose was to make life easier when
creating simulations, providing an environment where users could focus
on what they were trying to simulate rather than nitty-gritty
implementation specific details.
</p>
<p>
NMODL took MODL syntax and extended it with the primitives required by
its domain; for instance, it added the NEURON block to the language,
which allows multiple instances of "entities". As with MODL, NMODL is
translated into efficient C code and linked against supporting
libraries that provide the numerics; the NMODL translator to C also
had to take into account the requirement of linking against NEURON
libraries rather than SCoP.
</p>
<p>
The below is a snippet of NMODL code, copied from the <a href="http://ebooks.cambridge.org/ebook.jsf?bid=CBO9780511541612">NEURON book</a>
(chapter 9, listing 9.1):
</p>
<pre class="example">
NEURON {
SUFFIX leak
NONSPECIFIC_CURRENT i
RANGE i, e, g
}
PARAMETER {
g = 0.001 (siemens/cm2) < 0, 1e9 >
e = -65 (millivolt)
}
ASSIGNED {
i (milliamp/cm2)
v (millivolt)
}
</pre>
<p>
NMODL and hoc are used together to form a model; hoc to provide the
UI, parameters and morphology and NMODL to provide the physical
modeling. The website <a href="https://senselab.med.yale.edu/modeldb/default.cshtml">ModelDB</a> provides a database of models in a
variety of platforms with the main objective of making research
reproducible. <a href="https://senselab.med.yale.edu/modeldb/showModel.cshtml?model=83319&file=/destexhe_benchmarks/NEURON/README">Here</a> you can see an example of a production NEURON model
in its full glory, with a mix of hoc and NMODL files - as well as a
few others such as session files, which we can ignore for our
purposes.
</p>
</div>
</div>
<div id="outline-container-sec-4-3" class="outline-3">
<h3 id="sec-4-3">Thoughts</h3>
<div class="outline-text-3" id="text-4-3">
<p>
NEURON is more or less a standard in Computational Neuroscience -
together with a few other tools such as GENESIS, which we shall cover
later. Embedded deeply in it source code is the domain logic learned
painstakingly over several decades. Whilst software engineering-wise
it is creaking at the seams, finding a next generation heir will be a
non-trivial task given the features of the system, the amount of
models that exist out there, and the knowledge and large community
that uses it.
</p>
<p>
Due to this, a solution that a lot of next-generation tools have
developed is to use NEURON as a backend, providing a shiny modern
frontend and then generating the appropriate hoc and NMODL required by
NEURON. This is then executed in a NEURON environment and the results
are sent back to the user for visualisation and processing using
modern tools. Le Roi Est Mort, Vive Le Roi!
</p>
</div>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">Conclusions</h2>
<div class="outline-text-2" id="text-5">
<p>
In this first part we've outlined what Computational Neuroscience is
all about, what we mean by a model in this context and what services
one can expect from a platform in this domain. We also covered the
first of such platforms. Tune in for the next instalment where we'll
cover more platforms.
</p>
</div>
</div>
<div id="footnotes">
<h2 class="footnotes">Footnotes: </h2>
<div id="text-footnotes">
<div class="footdef"><sup><a id="fn.1" name="fn.1" class="footnum" href="#fnr.1">1</a></sup> <p class="footpara">
I still owe you the final post of that series, coming
out soon, hopefully.
</p></div>
<div class="footdef"><sup><a id="fn.2" name="fn.2" class="footnum" href="#fnr.2">2</a></sup> <p class="footpara">
Of course, once you scratch the surface, things get a
bit murkier. <a href="http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1000078">Erik De Schutter</a> states:
</p>
<blockquote>
<p>
[…] The term is often used to denote theoretical approaches in
neuroscience, focusing on how the brain computes information. Examples
are the search for “the neural code”, using experimental, analytical,
and (to a limited degree) modeling methods, or theoretical analysis of
constraints on brain architecture and function. This theoretical
approach is closely linked to systems neuroscience, which studies
neural circuit function, most commonly in awake, behaving intact
animals, and has no relation at all to systems biology. […]
Alternatively, computational neuroscience is about the use of
computational approaches to investigate the properties of nervous
systems at different levels of detail. Strictly speaking, this implies
simulation of numerical models on computers, but usually analytical
models are also included […], and experimental verification of
models is an important issue. Sometimes this modeling is quite data
driven and may involve cycling back and forth between experimental and
computational methods.
</p>
</blockquote></div>
<div class="footdef"><sup><a id="fn.3" name="fn.3" class="footnum" href="#fnr.3">3</a></sup> <p class="footpara">
This is a problem that has not gone unnoticed; for
instance, this paper provides an interesting and thorough review of
the state onion in Computational Neuroscience: <a href="http://arxiv.org/pdf/1205.3025.pdf">Current practice in
software development for computational neuroscience and how to improve
it.</a> In particular, it explains the dilemmas faced by the maintainers
of neuroscience packages.
</p></div>
<div class="footdef"><sup><a id="fn.4" name="fn.4" class="footnum" href="#fnr.4">4</a></sup> <p class="footpara">
The early story of NEURON is available <a href="http://neuron.duke.edu/userman/4/neuron.html">here</a>; see also the
<a href="http://www.scholarpedia.org/article/Neuron_simulation_environment">scholarpedia page</a>.
</p></div>
<div class="footdef"><sup><a id="fn.5" name="fn.5" class="footnum" href="#fnr.5">5</a></sup> <p class="footpara">
See the <a href="http://www.neuron.yale.edu/neuron/static/docs/help/neuron/nmodl/nmodl.html">NMODL page</a> for details, in the history
section.
</p></div>
<div class="footdef"><sup><a id="fn.6" name="fn.6" class="footnum" href="#fnr.6">6</a></sup> <p class="footpara">
As far as I can see, in the SCoP days MODL it was just
called the <a href="http://www.neuron.yale.edu/ftp/ted/neuron/scop/scopman.html">SCoP Language</a>, but as the related paper is under a paywall
I can't prove it either way. Paper: SCoP: An interactive simulation
control program for micro- and minicomputers, from <a href="http://link.springer.com/article/10.1007/BF02459691">Springer</a>.
</p></div>
</div>
</div></div>
<div id="postamble" class="status">
<p class="date">Created: 2015-11-11 Wed 17:59</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-5646887969684789292015-11-09T15:58:00.001-08:002015-11-10T09:54:42.640-08:00Nerd Food: Interesting...<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Interesting…</title>
<!-- 2015-11-09 Mon 23:57 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
Time to flush all those tabs again. Some interesting stuff I bumped
into recently-ish.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Finance</h2>
<div class="outline-text-2" id="text-1">
<ul class="org-ul">
<li><a href="https://medium.com/backchannel/how-bitcoins-blockchain-could-power-an-alternate-internet-bb501855af67">There’s a blockchain for that</a>!: The rise and rise of the
blockchain…
</li>
<li><a href="http://www.joecoin.com/2015/02/crypto-20-and-other-misconceptions.html">Crypto 2.0–And Other Misconceptions</a>: … but maybe we are getting a
bit ahead of ourselves, and bitcoin is really what matters!
</li>
<li><a href="http://www.michaelnielsen.org/ddi/how-the-bitcoin-protocol-actually-works/">How the Bitcoin protocol actually works</a>: Gory details of bitcoin's
internals. Must say that the <a href="http://shop.oreilly.com/product/0636920032281.do">Bitcoin book</a> already provides a pretty
good explanation, but interesting nonetheless.
</li>
<li><a href="http://blogs.wsj.com/moneybeat/2015/11/03/bitbeat-bitcoin-surges-past-400-on-back-of-the-new-shining-star/">BitBeat: Bitcoin Surges Past $400 on Back of the New ‘Shining Star’</a>:
Bitcoin is up again! Rollercoaster ride?
</li>
<li><a href="http://modval.org/home/">ModVal</a>: How cool is that, a site for financial models.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Startups et. al.</h2>
<div class="outline-text-2" id="text-2">
<ul class="org-ul">
<li><a href="http://www.newyorker.com/magazine/2015/02/23/shape-things-come">The Shape of Things to Come</a>: Jonathan Ive seems like a cool fellow,
actually. Even though I'm not much of an apple fan.
</li>
<li><a href="http://blog.ycombinator.com/yc-continuity-fund">Y Combinator Posthaven</a>: thoughts from the combinator. 1000 companies
funded!
</li>
<li><a href="https://medium.com/@EskoKilpi/movement-of-thought-that-led-to-airbnb-and-uber-9d4da5e3da3a#.d2dvu2nub">The Future of Firms. Is There an App for That?</a>: What does it mean to
be a company in this world of change?
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">General Coding</h2>
<div class="outline-text-2" id="text-3">
<ul class="org-ul">
<li><a href="https://medium.com/@landongn/12-years-later-what-i-ve-learned-about-being-a-software-engineer-d6e334d6e8a3">What I’ve learned so far about software development</a>: Tales from the
trenches.
</li>
<li><a href="https://codewords.recurse.com/issues/two/git-from-the-inside-out">Git from the inside out</a>: Gory details about git. And I mean <b>really</b>
gory and <b>really</b> detailed. As with bitcoin, the <a href="http://shop.oreilly.com/product/9780596520137.do">git book</a> was
already pretty detailed but interesting nonetheless.
</li>
<li><a href="http://chris.beams.io/posts/git-commit/">How to Write a Git Commit Message</a>: jeez, each to their own! Here's a
true geek of git commit messages. But very useful though.
</li>
<li><a href="http://www.benstopford.com/2015/04/28/elements-of-scale-composing-and-scaling-data-platforms/">Elements of Scale: Composing and Scaling Data Platforms</a>: Interesting
take on data, should help navigate the SQL/NoSQL debate.
</li>
<li><a href="http://apihandyman.io/do-you-really-know-why-you-prefer-rest-over-rpc/">Do you really know why you prefer REST over RPC?</a>: Title says it all,
interesting take on the RESTification of the world.
</li>
<li><a href="http://highscalability.com/blog/2015/10/12/making-the-case-for-building-scalable-stateful-services-in-t.html">Making The Case For Building Scalable Stateful Services In The
Modern Era</a>: Instead of knee-jerk reactions about statefulness, think
deeply before you decide. With presentation: <a href="https://www.youtube.com/watch?v=H0i_bXKwujQ">"Building Scalable
Stateful Services" by Caitie McCaffrey</a>
</li>
<li><a href="https://www.youtube.com/watch?v=9RMOc0SwRro">"Apache Kafka and the Next 700 Stream Processing Systems" by Jay
Kreps</a>: Improved my understanding of Kafka somewhat. And the title
made me curious, so I ended up reading <a href="http://www.cs.cmu.edu/~crary/819-f09/Landin66.pdf">The Next 700 Programming
Languages</a>.
</li>
<li><a href="http://www.theatlantic.com/technology/archive/2015/11/where-was-the-internet-born/413221/">The Room Where the Internet Was Born</a>: A quest for understanding the
cloud. Rather long but great for those that like computer history.
</li>
<li><a href="http://www.aaron-gray.com/a-criticism-of-scrum/">A Criticism of Scrum</a>: quite well thought out actually. I love agile
but I must say, I agree with many of the points made. I guess in the
end, the key is not to fall in love with ceremony.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Databases</h2>
<div class="outline-text-2" id="text-4">
<ul class="org-ul">
<li><a href="http://gotocon.com/dl/goto-berlin-2013/slides/HenningJacobs_and_ValentineGogichashvili_WhyZalandoTrustsInPostgreSQL.pdf">Why Zalando trusts in PostgreSQL</a>: Great to see how the elephant is
used in anger. Picked up a few tips.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">C++</h2>
<div class="outline-text-2" id="text-5">
<ul class="org-ul">
<li><a href="https://code.facebook.com/posts/1661982097368498">Futures for C++11 at Facebook</a>: futures are all the rage…
</li>
<li><a href="http://instagram-engineering.tumblr.com/post/121930298932/c-futures-at-instagram">C++ Futures at Instagram</a>: … everywhere!
</li>
<li><a href="https://www.livecoding.tv/videos/c-cplusplus/?sort=newest">Livestream</a>: WOT, C++ live coding? Man this is a weird notion!
</li>
<li><a href="https://www.youtube.com/watch?v=nXaxk27zwlk&feature=youtu.be">CppCon 2015: Chandler Carruth "Tuning C++: Benchmarks, and CPUs, and
Compilers! Oh My!</a>: Chandler at his best - hilarious but extremely
informative.
</li>
<li><a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4367.html">Comparison in C++</a>: A weird but weirdly useful paper. Deep thinking
about comparisons. For good measure, you should also watch the
presentation: <a href="https://www.youtube.com/watch?v=fi0CQ7laiXE">CppCon 2015: Lawrence Crowl "Comparison is not simple,
but it can be simpler"</a>
</li>
<li><a href="https://www.youtube.com/watch?v=mFUXNMfaciE">CppCon 2015: Eric Niebler "Ranges for the Standard Library"</a>: Ranges,
ranges, ranges!! Can't wait! For good measure, you can also read <a href="http://bartoszmilewski.com/2014/10/17/c-ranges-are-pure-monadic-goodness/">C++
Ranges are Pure Monadic Goodness</a>.
</li>
<li><a href="https://cryptocoding.net/index.php/Coding_rules">Crypto coding rules</a>: haven't parsed the entire document, but already
found a few very useful points.
</li>
<li><a href="https://github.com/emil-e/rapidcheck">RapidCheck</a>: Didn't know of QuickCheck or RapidCheck, but this may
come in handy at some point…
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">Layman Science</h2>
<div class="outline-text-2" id="text-6">
<ul class="org-ul">
<li><a href="https://nolaymanleftbehind.wordpress.com/2011/07/10/linear-algebra-what-matrices-actually-are/">Linear Algebra</a>: What matrices actually are: an attempt to make
matrices accessible.
</li>
<li><a href="http://betterexplained.com/articles/a-gentle-introduction-to-learning-calculus/">A Gentle Introduction To Learning Calculus</a>: Helpful when trying to
remember all the stuff one did all those years ago…
</li>
<li><a href="http://betterexplained.com/articles/a-visual-intuitive-guide-to-imaginary-numbers/">A Visual, Intuitive Guide to Imaginary Numbers</a>: … continued.
</li>
<li><a href="http://esciencecommons.blogspot.co.uk/2011/01/new-theories-reveal-nature-of-numbers.html">New theories reveal the nature of numbers</a>: A <a href="http://www.imdb.com/title/tt0787524/">Ramanujan movie</a> is due
out soon, and these are the guys doing the maths behind the
scenes. Can't wait for the movie!
</li>
<li><a href="http://v.cx/2010/04/feynman-brazil-education">Richard Feynman on education in Brazil</a>: Hilarious, but somewhat
reminiscent of my own education, in a different country but
culturally very similar and certainly with a very similar approach
to science.
</li>
<li><a href="http://neuralnetworksanddeeplearning.com/chap1.html">Using neural nets to recognize handwritten digits</a>: nice introduction
to neural nets with a good example.
</li>
<li><a href="http://nautil.us/issue/15/turbulence/your-brain-is-on-the-brink-of-chaos">Your Brain Is On the Brink of Chaos</a>: <b>Very</b> interesting. This is
particularly interesting because I have been reading up on the
delicate balance between inhibitory and excitatory neurons, but the
literature always gives you this static impression of balance. If
you assume chaos on the other hand…
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-7" class="outline-2">
<h2 id="sec-7">Other</h2>
<div class="outline-text-2" id="text-7">
<ul class="org-ul">
<li><a href="http://www.newyorker.com/magazine/2015/06/01/extreme-city-specter">Extreme City</a>: Luanda, my beloved, just keeps on getting crazier and
crazier. Interesting - if somewhat expat-oriented - take on the
city.
</li>
<li><a href="http://www.slate.com/blogs/schooled/2015/09/17/kiera_wilmot_arrest_florida_teenager_reacts_to_ahmed_mohamed_story.html">This Florida Teenager Knows What Ahmed Mohamed Is Going Through. It
Happened to Her in 2013</a>: sad, really. Whatever the real truth was
about Ahmed.
</li>
<li><a href="http://www.nytimes.com/2015/10/11/opinion/sunday/will-you-ever-be-able-to-upload-your-brain.html?_r=1">Will You Ever Be Able to Upload Your Brain?</a>: er., spoiler alert -
not really. Interesting though.
</li>
<li><a href="https://dl.dropboxusercontent.com/u/50282823/Flash%2520BBG%25202015-%2520October-Taleb.pptx.pdf">Dealing with “power laws” with upper (lower) bound</a>: most certainly
not for laypeople. Taleb is back at it. Would be great to have this
translated to laymen's maths.
</li>
<li><a href="https://www.youtube.com/watch?v=5mcyUUf20Ng&feature=youtu.be&list=PL37ZVnwpeshH37NxpV6XbgdDpY-w48hMd">Lieke Boon: Unconscious Bias</a>: we're all guilty: How to be a bit more
aware of your own the biases is my take on it.
</li>
</ul>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-11-09 Mon 23:57</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-65752721171991645342015-09-16T10:06:00.000-07:002018-03-22T03:46:27.820-07:00Nerd Food: Neurons for Computer Geeks - Part VI: LIF At Long Last!<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Neurons for Computer Geeks - Part VI: Integrate and Fire!</title>
<!-- 2015-09-16 Wed 18:05 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js"></script>
<script type="text/javascript">
<!--/*--><![CDATA[/*><!--*/
MathJax.Hub.Config({
// Only one of the two following lines, depending on user settings
// First allows browser-native MathML display, second forces HTML/CSS
// config: ["MMLorHTML.js"], jax: ["input/TeX"],
jax: ["input/TeX", "output/HTML-CSS"],
extensions: ["tex2jax.js","TeX/AMSmath.js","TeX/AMSsymbols.js",
"TeX/noUndefined.js"],
tex2jax: {
inlineMath: [ ["\\(","\\)"] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"], ["\\begin{displaymath}","\\end{displaymath}"] ],
skipTags: ["script","noscript","style","textarea","pre","code"],
ignoreClass: "tex2jax_ignore",
processEscapes: false,
processEnvironments: true,
preview: "TeX"
},
showProcessingMessages: true,
displayAlign: "center",
displayIndent: "2em",
"HTML-CSS": {
scale: 100,
availableFonts: ["STIX","TeX"],
preferredFont: "TeX",
webFont: "TeX",
imageFont: "TeX",
showMathMenu: true,
},
MMLorHTML: {
prefer: {
MSIE: "MML",
Firefox: "MML",
Opera: "HTML",
other: "HTML"
}
}
});
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
Welcome to part VI of a multi-part series on modeling neurons. In <a href="http://mcraveiro.blogspot.co.uk/2015/09/nerd-food-neurons-for-computer-geeks_7.html">part
V</a> we added a tad more theory to link electricity with neurons, and
also tried to give an idea of just how complex neurons are. Looking
back on that post, I cannot help but notice I skipped one bit that is
rather important to understanding Integrate-and-Fire (IAF) models. So
lets look at that first and then return to our trail.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Resting Potential and Action Potential</h2>
<div class="outline-text-2" id="text-1">
<p>
We have spoken <a href="http://mcraveiro.blogspot.co.uk/2015/09/nerd-food-neurons-for-computer-geeks_7.html">before</a> about the <a href="https://en.wikipedia.org/wiki/Membrane_potential">membrane potential</a> and the resting
membrane potential, but we did so with such a high degree of
hand-waving it now warrants revisiting. When we are talking about the
<i>resting</i> membrane potential we mean just that - the value for the
membrane potential when nothing much is happening. That is the magical
circa -65mv we discussed before - with all of the related explanations
on conventions around negative voltages. However, time does not stand
still and things happen. The cell receives input from other neurons,
and this varies over time. Some kinds of inputs can cause events to
trigger on the receiving neuron: active ion channels may get opened or
shut, ions move around, concentrations change and so forth, and thus,
the cell will change its membrane potential in response. When these
changes result in a higher voltage - such as moving to -60mv - we say
a <i>depolarisation</i> is taking place. Conversely, when the voltage
becomes more negative, we say <i>hyperpolarisation</i> is occurring.
</p>
<p>
Now, it may just happen that there is a short-lived but "strong" burst
of depolarisation, followed by equally rapid hyperpolarisation - and,
as a result of which, the Axon's terminal decides to release
<i>neurotransmitters</i> into the synapse (well, into the <i>synaptic gap</i> or
<i>synaptic cleft</i> to be precise). This is called an <i><a href="https://en.wikipedia.org/wiki/Action_potential">action potential</a></i>,
and it is also known by many other names such as "nerve impulses" or
"spikes". When you hear that "a neuron has fired" this means that an
action potential has just been emitted. If you record the neuron's
behaviour over time you will see a <i>spike train</i> - a plot of the
voltage over time, clearly showing the spikes. Taking a fairly random
example:
</p>
<div class="figure">
<p><img src="https://upload.wikimedia.org/wikipedia/commons/1/1b/Simulation_of_hrose_neuron.png" alt="Simulation_of_hrose_neuron.png" width="300px" height="300px" />
</p>
<p><span class="figure-number">Figure 1:</span> Source: Wikipedia, <a href="https://en.wikipedia.org/wiki/Neural_oscillation">Neural oscillation</a></p>
</div>
<p>
One way of picturing this is as a kind of "chain-reaction" whereby
something triggers the voltage of the neuron to rise, which triggers a
number of gates to open, which then trigger the voltage to rise and so
on, until some kind of magic voltage threshold is reached where the
inverse occurs: the gates that were causing the voltage to rise shut
and some other gates that cause the voltage to decrease open, and so
on, until we fall back down to the resting membrane potential. The
process feeds back on itself, first as a positive feedback and then as
a negative feedback. In the case of the picture above, something else
triggers us again and again, until we finally come to rest.
</p>
<p>
This spiking or firing behaviour is what we are trying to model.
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Historical Background</h2>
<div class="outline-text-2" id="text-2">
<p>
As it happens, we are not the first ones to try to do so. A couple of
years after Einstein's <i>annus mirabilis</i>, a french chap called <a href="https://en.wikipedia.org/wiki/Louis_Lapicque">Louis
Lapicque</a> was also going through his own personal moment of
inspiration, the output of which was the seminal <a href="http://www.snv.jussieu.fr/brette/papers/Lap07.pdf">Recherches
quantitatives sur l'excitation électrique des nerfs traitée comme une
polarisation</a>. It is summarised <a href="http://neurotheory.columbia.edu/~larry/AbbottBrResBul99.pdf">here</a> in fairly accessible English by
Abbot.
</p>
<p>
Lapicque had the insight of imagining the neuron as an RC circuit,
with the membrane potential's behaviour explained as the interplay
between capacitor and resistor; the action potential is then the
capacitor reaching a threshold followed by a discharge. Even with our
faint understanding of the subject matter, one cannot but appreciate
Lapique's brilliance to have the ability to reach these conclusions
in 1907. Of course, he also had to rely on the work of many others to
get there, let's not forget.
</p>
<p>
This model is still considered a useful model today, even though we
know so much more about neurons now - a great example of what we
mentioned <a href="http://mcraveiro.blogspot.co.uk/2015/09/nerd-food-neurons-for-computer-geeks_7.html">before</a> in terms of the choices of the level of detail when
modeling. Each model is designed for a specific purpose and it should
be as simple as possible for the stated end (but no simpler). As <a href="http://neurotheory.columbia.edu/~larry/AbbottBrResBul99.pdf">Abbot</a>
says:
</p>
<blockquote>
<p>
While Lapicque, because of the limited knowledge of his time, had no
choice but to model the action potential in a simple manner, the
stereotypical character of action potentials allows us, even today, to
use the same approximation to avoid computation of the voltage
trajectory during an action potential. This allows us to focus both
intellectual and computation resources on the issues likely to be most
relevant in neural computation, without expending time and energy on
modeling a phenomenon, the generation of action potentials, that is
already well understood.
</p>
</blockquote>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">The IAF Family</h2>
<div class="outline-text-2" id="text-3">
<p>
Integrate-and-Fire is actually a family of models - related because
all of them follow Lapicque's original insights. Over time, people
have addressed shortcomings in the model by adding more parameters and
modifying it slightly and from this other models were born.
</p>
<p>
In general, models in the IAF family are single neuron models with a
number of important properties (as per <a href="http://cns-classes.bu.edu/cn510/Papers/Izhikevich_Ch8.pdf">Izhikevich</a>):
</p>
<ul class="org-ul">
<li>The spikes are all or none; that is, we either spike or we
don't. This is a byproduct of the way spikes are added to the model,
as we shall see later. This also means all spikes are identical
because the are all created the same way.
</li>
<li>The threshold for the spike is well defined and there is no
ambiguity as to whether the neuron will fire or not.
</li>
<li>It is possible to add a <i>refractory period</i>, similarly to how we add
the spike. The refractory period is a time during which the neuron
is less excitable (e.g. ignores inputs) and occurs right after the
spike.
</li>
<li>Positive currents are used as excitatory inputs and negative currents
as inhibitory inputs.
</li>
</ul>
<p>
But how do the members of this family look like? We will take a few
examples from <a href="https://en.wikipedia.org/wiki/Biological_neuron_model">Wikipedia</a> to make a family portrait and then focus on
LIF.
</p>
</div>
<div id="outline-container-sec-3-1" class="outline-3">
<h3 id="sec-3-1">IAF: Integrate-and-Fire</h3>
<div class="outline-text-3" id="text-3-1">
<p>
This the Lapicque model. It is also called a "perfect" or "non-leaky"
neuron. The formula is as follows:
</p>
\begin{align}
I(t) = C_m \frac{dV_m(t)}{dt}
\end{align}
<p>
The <i>m</i>'s are there to signify <i>membrane</i>, nothing else. Note that its
the job of the user to determine θ - that is the point at which
the neuron spikes - and then to reset everything to zero and start
again. If you are wondering why it's called "integrate", that's
because the differential equation must be integrated before we can
compare the current value to a threshold and then, if we're passed it,
well - fire!. Hence Integrate-and-Fire.
</p>
<p>
Wikipedia states this in a classier way, of course:
</p>
<blockquote>
<p>
[This formula] is just the time derivative of the law of capacitance, Q =
CV. When an input current is applied, the membrane voltage increases
with time until it reaches a constant threshold Vth, at which point a
delta function spike occurs and the voltage is reset to its resting
potential, after which the model continues to run. The firing
frequency of the model thus increases linearly without bound as input
current increases.
</p>
</blockquote>
</div>
</div>
<div id="outline-container-sec-3-2" class="outline-3">
<h3 id="sec-3-2">Integrate-and-Fire with Refractory Period</h3>
<div class="outline-text-3" id="text-3-2">
<p>
It is possible to extend IAF to take the refractory period into
account. This is done by adding a period of time <i>t ref</i> during which
the neuron does not fire.
</p>
</div>
</div>
<div id="outline-container-sec-3-3" class="outline-3">
<h3 id="sec-3-3">LIF: Leaky Integrate-and-Fire</h3>
<div class="outline-text-3" id="text-3-3">
<p>
One of the problems of IAF is that it will "remember" stimulus,
regardless of the time that elapses between stimuli. By way of
example: if a neuron gets some input below the firing threshold at
some time (say <i>ta</i>), then nothing for a long period of time and then
subsequent stimulus at say <i>tb</i>, this will cause the neuron to fire
(assuming the two inputs together are above the threshold). In the
real world, neurons "forget" about below-threshold stimulus after
certain amount of time has elapsed. This problem is solved in LIF by
adding a <i>leak</i> term to IAF. The Wikipedia's formula is like so:
</p>
\begin{align}
I_m(t) - \frac{V_m(t)}{R_m} = C_m \frac{dV_m(t)}{dt}
\end{align}
<p>
We will discuss it in detail later on.
</p>
</div>
<div id="outline-container-sec-3-3-1" class="outline-4">
<h4 id="sec-3-3-1">Interlude: Leaky Integrators and Low-Pass Filters</h4>
<div class="outline-text-4" id="text-3-3-1">
<p>
<b>Update</b>: this section got moved here from an earlier post.
</p>
<p>
Minor detour into the world of "Leaky Integrators". As it turns out,
mathematicians even have a name to describe functions like the one
above: they are called <i><a href="https://en.wikipedia.org/wiki/Leaky_integrator">Leaky Integrators</a></i>. A leaky integrator is
something that takes an input and "integrates" - that is, sums it over
a range - but by doing so, starts "leaking" values out. In order
words, a regular sum of values over a range should just result in an
ever growing output. With a leaky integrator, we add up to a point,
but then we start leaking, resetting the value of the sum back to
where we started off.
</p>
<p>
It turns out these kind of functions have great utility. For example,
imagine that you have a range of inputs varying from some arbitrary
low number to some other arbitrary high-number. When you supply these
inputs to a leaky integrator, it can be used to "filter out" the high
numbers; input numbers higher than a certain cut-off point just result
in zeros in the output. This is known as a <i><a href="https://en.wikipedia.org/wiki/Low-pass_filter">low-pass filter</a></i>. One can
conceive of a function that acted in the opposite way - a <i>high-pass
filter</i>.
</p>
</div>
</div>
</div>
<div id="outline-container-sec-3-4" class="outline-3">
<h3 id="sec-3-4">Exponential Integrate-and-Fire</h3>
<div class="outline-text-3" id="text-3-4">
<p>
In this model, spike generation is exponential:
</p>
\begin{align}
\frac{dX}{dt} = \Delta_\tau exp(\frac{X - X_t}{\Delta_\tau})
\end{align}
<p>
Wikipedia explains it as follows:
</p>
<blockquote>
<p>
where X is the membrane potential, X<sub>T</sub> is the membrane potential
threshold, and Δ<sub>T</sub> is the sharpness of action potential
initiation, usually around 1 mV for cortical pyramidal neurons. Once
the membrane potential crosses X<sub>T</sub>, it diverges to infinity in finite
time.
</p>
</blockquote>
</div>
</div>
<div id="outline-container-sec-3-5" class="outline-3">
<h3 id="sec-3-5">Others</h3>
<div class="outline-text-3" id="text-3-5">
<p>
We could continue and look into other IAF models, but you get the
point. Each model has limitations, and as people work through those
limitations - e.g. try to make the spike trains generated by the model
closer to those observed in reality - they make changes to the model
and create new members of the IAF family.
</p>
</div>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Explaining the LIF Formula</h2>
<div class="outline-text-2" id="text-4">
<p>
Let's look at a slightly more familiar formulation of LIF:
</p>
\begin{align}
\tau_m \frac{dv}{dt} = -v(t) + RI(t)
\end{align}
<p>
By now this should make vague sense, but lets do it step by step
breakdown just to make sure we are all on the same page. First, we
know that the current of the RC circuit is defined like so:
</p>
\begin{align}
I(t) = I_R + I_C
\end{align}
<p>
From Ohm's Law we also know that:
</p>
\begin{align}
I_R = \frac {v}{R}
\end{align}
<p>
And from the <a href="http://mcraveiro.blogspot.co.uk/2015/09/nerd-food-neurons-for-computer-geeks_5.html">rigmarole of the capacitor</a> we also know that:
</p>
\begin{align}
I_C = C \frac{dv}{dt}
\end{align}
<p>
Thus its not much of a leap to say:
</p>
\begin{align}
I(t) = \frac {v(t)}{R} + C \frac{dv}{dt}
\end{align}
<p>
Now, if we now multiply both sides by R, we get:
</p>
\begin{align}
RI(t) = v(t) + RC \frac{dv}{dt}
\end{align}
<p>
Remember that RC is τ, the <a href="https://en.wikipedia.org/wiki/RC_time_constant">RC time constant</a>; in this case, we are
dealing with the membrane so hence the <i>m</i>. With that, the rest of the
rearranging to the original formula should be fairly obvious.
</p>
<p>
Also, if you recall, we mentioned <a href="https://en.wikipedia.org/wiki/Leaky_integrator">Leaky Integrators</a> before. You should
hopefully be able to see the resemblance between these and our first
formula.
</p>
<p>
Note that we did not model spikes explicitly with this
formula. However, when it comes to implementing it, all that is
required is to look for a threshold value for the membrane potential -
called the <i>spiking threshold</i>; when that value is reached, we need to
reset the membrane potential back to a lower value - the <i>reset
potential</i>.
</p>
<p>
And with that we have enough to start thinking about code…
</p>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">Method in our Madness</h2>
<div class="outline-text-2" id="text-5">
<p>
.. Or so you may think. First, a quick detour on discretisation. As it
happens, computers are rather fond of discrete things rather than the
continuous entities that inhabit the world of calculus. Computers are
very much of the same opinion as <a href="https://en.wikipedia.org/wiki/George_Berkeley">the priest</a> <a href="http://www.maths.tcd.ie/pub/HistMath/People/Berkeley/Analyst/Analyst.pdf">who said</a>:
</p>
<blockquote>
<p>
And what are these same evanescent Increments? They are neither finite
Quantities nor Quantities infinitely small, nor yet nothing. May we
not call them the Ghosts of departed Quantities?
</p>
</blockquote>
<p>
So we cannot directly represent differential equations in the
computer - not even the simpler ordinary differential equations
(ODEs), with their single independent variable. Instead, we need to
approximate them with a <i>method</i> for <i>numerical integration</i> of the
ODE. Remember: when we say <i>integration</i> we just mean "summing".
</p>
<p>
Once we enter the world of <i>methods</i> and <i>numerical analysis</i> we are
much closer to our ancestral home of Software Engineering. The job of
numerical analysis is to look for ways in which one can make discrete
approximations of the problems in mathematical analysis - like, say,
calculus. The little recipes they come up with are called <i>numerical
methods</i>. A method is nothing more than an algorithm, a set of steps
used iteratively. One such method is the <a href="https://en.wikipedia.org/wiki/Euler_method">Euler Method</a>: "[a] numerical
procedure for solving ordinary differential equations (ODEs) with a
given initial value", as Wikipedia tells us, and as it happens that is
exactly what we are trying to do.
</p>
<p>
So how does the Euler method work? Very simply. First you know that:
</p>
\begin{align}
y(t_0) = y_0 \\
y'(t) = f(t, y(t))
\end{align}
<p>
That is, at the beginning of time we have a known value. Then, for all
other <i>t</i>'s, we use the current value in <i>f</i> in order to be able to
compute the next value. Lets imagine that our steps - how much we are
moving forwards by - are of a size <i>h</i>. You can then say:
</p>
\begin{align}
t_{n+1} = t_n + h \\
y_{n+1} = y_n + h * f(x_n, t_n)
\end{align}
<p>
And that's it. You just need to know where you are right now, by how
much you need to scale the function - e.g. the step size - and then
apply the function to the current values of <i>x</i> and <i>t</i>.
</p>
<p>
In code:
</p>
<div class="org-src-container">
<pre class="src src-c++">template<typename F>
void euler(F f, double y0, double start, double end, double h) {
double y = y0;
for (auto t(start); t < end; t += h) {
y += h * f(t, y, h);
}
}
</pre>
</div>
<p>
We are passing <i>h</i> to the function <i>F</i> because it needs to know about
the step size, but other than that it should be a pretty clean mapping
from the maths above.
</p>
<p>
This method is also known as <i>Forward Euler</i> or <i>Explicit Euler</i>.
</p>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">What next?</h2>
<div class="outline-text-2" id="text-6">
<p>
And yet again, we run out of time yet again before we can get into
serious coding. In the next instalment we shall cover the
implementation of the LIF model.
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-09-16 Wed 18:05</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-28882220543045544592015-09-07T09:13:00.000-07:002015-09-07T09:29:04.948-07:00Nerd Food: Neurons for Computer Geeks - Part V: Yet More Theory<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Neurons for Computer Geeks - Part V: Yet More Theory</title>
<!-- 2015-09-07 Mon 17:12 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
Welcome to part V of a multi-part series on modeling neurons. In <a href="http://mcraveiro.blogspot.co.uk/2015/09/nerd-food-neurons-for-computer-geeks_5.html">part
IV</a> we introduced the RC Circuit by making use of the foundations we
painstakingly laid in previous posts. In truth, we could now move on
to code and start looking at the <a href="http://icwww.epfl.ch/~gerstner/SPNM/node26.html">Leaky Integrate-and-Fire</a> (LIF) model,
since we've already covered most required concepts. However, we are
going to do just a little bit more theory before we get to that.
</p>
<p>
The main reason for this detour is that I do not want to give you the
impression neurons are <i>easy</i>; if there is one thing that they are
<b>not</b> is <i>easy</i>. So we're going to resume our morphological and electrical exploits
to try to provide a better account of the complexity inside the
neuron, hopefully supplying enough context to appreciate the
simplifications done in LIF.
</p>
<p>
The content of this post is highly inspired from <a href="http://www.cambridge.org/us/academic/subjects/life-sciences/neuroscience/principles-computational-modelling-neuroscience">Principles of
Computational Modelling in Neuroscience</a>, a book that is a must read
introduction if you decide to become serious on this subject. If so,
you may also want to check the Gerstner videos: <a href="http://klewel.com/conferences/epfl-neural-networks/index.php?talkID=1">Neural networks and
biological modeling</a>.
</p>
<p>
But you need not worry, casual reader. Our feet are firmly set in
layman's land and we'll remain so until the end of the series.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Brief Context on Modeling</h2>
<div class="outline-text-2" id="text-1">
<p>
Before we get into the subject matter proper, I'd like us to ponder a
few "meta-questions" in terms of modeling.
</p>
</div>
<div id="outline-container-sec-1-1" class="outline-3">
<h3 id="sec-1-1">Why Model?</h3>
<div class="outline-text-3" id="text-1-1">
<p>
A layperson may think that we model neurons because we want to build a
"computer brain": one that is similar to a real brain, with its
amazing ability to learn, and one which at some point may even <i>think
and be conscious</i>. Hopefully, after you finish this series of posts,
you will appreciate the difficulty of the problem and see that it's
not very likely we'll be able to make a "realistic" "computer brain"
any time soon - for sensible values of "realistic", "computer brain"
and "any time soon".
</p>
<p>
Whilst we have good models that explain part of the behaviour of the
neuron and good models for neural networks too, it is not the case
that we can put all of these together to form some kind of "unified
neuron model", multiply it by 80 billion, add a few quadrillion
synapses and away we go: artificial consciousness. Given what we know
at the moment, this approach is far too computationally demanding to
be feasible. Things would change if there was a massive leap in
computational power, of course, but not if they stay at present
projections - even with <a href="https://en.wikipedia.org/wiki/Moore%2527s_law">Moore's Law</a>.
</p>
<p>
So if we are not just trying to build a computer brain, then why
bother? Well, if you set your sights a little lower, computational
models are actually amazingly useful:
</p>
<ul class="org-ul">
<li>one can use code to explore a small portion of the problem domain,
making and validating predictions using computer models, and then
test those predictions in the lab with real wetware. The iterative
process is orders of magnitude faster.
</li>
<li>computer models are now becoming quite sophisticated, so in some
cases they are good representations of biological processes. This
tends to be the case for small things such as individual cells or
smaller. As computers get faster and faster according to <a href="https://en.wikipedia.org/wiki/Moore%2527s_law">Moore's
Law</a>, the power and scope of these models grows too.
</li>
<li>distributing work with Free and Open Source Software licences means
it is much easier for researchers to reproduce each others work, as
well as for them to explore avenues not taken by those who did the
work originally, speeding things up considerably. Standing on the
shoulders of giants and all that.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-1-2" class="outline-3">
<h3 id="sec-1-2">What Tools Do We Model With?</h3>
<div class="outline-text-3" id="text-1-2">
<p>
The focus of these posts is on writing models from scratch, but that's
not how most research is conducted. In the real world, people try
their best to reuse existing infrastructure - of which there is
plenty. For example there is <a href="https://www.neuron.yale.edu/neuron/">NEURON</a>, <a href="http://neuralensemble.org/PyNN/">PyNN</a>, <a href="http://briansimulator.org/">Brian</a> and much more. Tools
and processes have evolved around these ecosystems, and there is a
push to try to standardise around the more successful frameworks.
</p>
<p>
There is also a push to find some kind of standard "language" to
describe models so that we can all share information freely without
having to learn the particulars of each others representations. The
world is not quite there yet, but initiatives such as <a href="https://www.neuroml.org/">NeuroML</a> are
making inroads in this direction.
</p>
<p>
However, the purpose of our this series is simplification, so we will
swerve around all of this. Perhaps material for another series.
</p>
</div>
</div>
<div id="outline-container-sec-1-3" class="outline-3">
<h3 id="sec-1-3">At What Level Should One Model?</h3>
<div class="outline-text-3" id="text-1-3">
<p>
A related question to the previous ones - and one that is not normally
raised in traditional software engineering, but is very relevant in
biology - is the level of detail at which one should model.
</p>
<p>
Software Engineers tend to believe there is <i>a model</i> for <i>a problem</i>,
and once you understand enough about the problem domain you will come
up with it and <i>all will be light</i>. Agile and sprints are just a way
to converge to it, to the perfection that exists somewhere in the
platonic cloud. Eric Evans with <a href="https://domainlanguage.com/ddd/">DDD</a> started to challenge that
assumption somewhat by making us reflect on just what it is that we
mean by "model" and "modeling", but, in general, we have such an
ingrained belief in this idea that is very hard to shake it off or to
even realise the belief is there in the first place. Most of us still
think of the code representation of the domain model as <i>the model</i> -
rather than accept it is one of a multitude of possible
representations, each suitable for a given purpose.
</p>
<p>
Alas, all of this becomes incredibly obvious when you are faced with a
problem like modeling a neuron or a network of neurons. Here, there is
just no such thing as the "right model"; only a set of models at a
different perspectives, each with a different set of trade-offs, and
any of them only make sense in the context of what one is trying to
study. It may make sense to model neurons like networks, ignoring the
finer details of each one and looking at their behaviour as a group,
or it may make sense to model individual bits of the neuron as an
entity. What makes it "right" or "wrong" is what it is that we are
using the model for and how much computational power one has at one's
disposal.
</p>
<p>
Having said all of that, lets resume our morphology adventures.
</p>
</div>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Electricity and Neurons</h2>
<div class="outline-text-2" id="text-2">
<p>
We started off with <a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks.html">an overview of the neuron</a> and then moved over to
<a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks_31.html">lots</a> and <a href="http://mcraveiro.blogspot.co.uk/2015/09/nerd-food-neurons-for-computer-geeks_5.html">lots</a> of electricity; now it's time to see how those two fit
together.
</p>
<p>
As we explained in <a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks.html">part I</a>, there is a electric potential difference
between the inside of the cell and the outside, called the <i>membrane
potential</i>. The convention to compute this potential is to subtract
the potential inside the cell to the potential outside the cell;
current is positive when there is a flow of positive charge from the
inside to the outside and negative otherwise. Taken into account these
definitions, one should be able to make sense of the <i>resting membrane
potential</i>: it is around -65mv. But how does this potential change?
</p>
</div>
<div id="outline-container-sec-2-1" class="outline-3">
<h3 id="sec-2-1">Ion Channels</h3>
<div class="outline-text-3" id="text-2-1">
<p>
<a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks_31.html">Earlier</a>, we spoke about ions - atoms that either lost or gained
electrons and so are positively or negatively charged. We also said
that, in general, the cell's membrane is impermeable, but there are
tiny gaps in the membrane which allow things in and out of the
cell. Now we can expand a bit further. <i>Ion channels</i> are one such
gap, and they have that name because they let ions through. There are
<i>many</i> kinds of ion channels. One way of naming them is to use the ion
they are most permeable to - but of course, this being biology, the
ion channels don't necessarily always have a major ion they are
permeable to.
</p>
<p>
Another useful categorisation distinguishes between <i>passive</i> and
<i>active</i> ion channels. Active channels are those that change their
permeability depending on external factors such as the membrane
potential, the concentration of certain ions, and so on. For certain
values they are open - i.e. permeable - whereas for other values they
are closed, not allowing any ions through. Passive channels are
simpler, they just have a fixed permeability behaviour.
</p>
<p>
There are also <i>ionic pumps</i>. These are called pumps because they take
one kind of ion out, exchanging it for another kind. For instance, the
sodium-potassium pump pushes potassium into the cell and expels sodium
out. A pump has a <i>stoichiometry</i>, which is a fancy word to describe
the ratio of ions being pumped in and out.
</p>
</div>
</div>
<div id="outline-container-sec-2-2" class="outline-3">
<h3 id="sec-2-2">Complexity Starts To Emerge</h3>
<div class="outline-text-3" id="text-2-2">
<p>
As you can imagine, the key to understating electric behaviour is
understanding how these pesky ions move around. Very simplistically,
ions tend to move for two reasons: because there is a potential
difference between the inside and the outside of the cell, or because
of the <i>concentration gradient</i> of said ion. The concentration
gradient just means that, left to their own devices, concentration
becomes uniform over time. For example, if you drop some ink in a
glass of water, you will start by seeing the ink quite clearly; given
enough time, the ink will diffuse in the water, making it all
uniformly coloured. The same principle applies to ions - they want to
be uniformly concentrated.
</p>
<p>
It should be fairly straightforward to work out that a phenomenal
number of permutations is possible here. Not only do we have a great
number of channels, all with different properties - some switching on
and off as properties change around the cell - but we also have the
natural flow of ions being affected by the membrane's potential and
the concentration gradient, all of which are changing over time. To
make matters worse, factors interact with each other such that even if
you have simple models to explain each aspect individually, the
overall behaviour is still incredibly complex.
</p>
<p>
Now imagine more than <a href="http://pubs.acs.org/doi/abs/10.1021/jp0120662">50 thousand</a> such ion channels - of over one hundred (known) types - in just a single neuron and you are starting to get an idea of the magnitude of the task.
</p>
</div>
</div>
<div id="outline-container-sec-2-3" class="outline-3">
<h3 id="sec-2-3">Equivalent Circuit for a Patch of Membrane</h3>
<div class="outline-text-3" id="text-2-3">
<p>
But lets return to simplicity. The very clever people determined that
it is possible to model the behaviour of ions and its electric effects
by thinking of it as an electric circuit. Taking a patch of
membrane as an example, it can be visualised as an electric circuit
like so:
</p>
<div class="figure">
<p><img src="https://upload.wikimedia.org/wikipedia/commons/e/e5/Cell_membrane_equivalent_circuit.svg" alt="Cell_membrane_equivalent_circuit.svg" width="500px" height="500px" />
</p>
<p><span class="figure-number">Figure 1:</span> Source: Wikipedia, <a href="https://en.wikipedia.org/wiki/Membrane_potential">Membrane Potential</a></p>
</div>
<p>
What this diagram tells us is that the membrane itself acts as a
capacitor, with its capacitance determined by the properties of the
<i>lipid bilayer</i>. We didn't really discuss the lipid bilayer before so
perhaps a short introduction is in order. The membrane is made up of
two sheets of lipids (think fatty acids), which when layered so, have
interesting properties: the outside of the sheets are impermeable to
most things such as water molecules and ions. The membrane itself is
pretty thin, at around 5nm.
</p>
<p>
The membrane capacitance is considered constant. We then have a series
of ion channels: sodium, potassium, chlorine, calcium. Each of these
can be thought of as a pairing of a resistor with variable conductance
coupled with a battery. Note that the resistor and the battery are in
series, but the ion channels themselves form a parallel circuit. The
voltages for each pathway are determined by the different
concentrations of the ion inside and outside the cell.
</p>
<p>
If we further assume fixed ion concentrations and passive ion
channels, we can perform an additional simplification on the circuit
above and we finally end up with an RC Circuit:
</p>
<div class="figure">
<p><img src="https://upload.wikimedia.org/wikipedia/commons/b/b6/Cell_membrane_reduced_circuit.svg" alt="Cell_membrane_reduced_circuit.svg" width="300px" height="300px" />
</p>
<p><span class="figure-number">Figure 2:</span> Source: Wikipedia, <a href="https://en.wikipedia.org/wiki/Membrane_potential">Membrane Potential</a></p>
</div>
<p>
The circuit now has one resistance, which we call the membrane
resistance, and a membrane battery.
</p>
</div>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">What next?</h2>
<div class="outline-text-2" id="text-3">
<p>
Hopefully you can start to see both the complexity around modeling
neurons and the necessity to create simpler models to make them
computationally feasible - just look at the amount of simplification
that was required for us to get to an RC Circuit!
</p>
<p>
But at least we can now look forward to implementing LIF.
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-09-07 Mon 17:12</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-73310818946026064512015-09-05T10:57:00.000-07:002018-03-22T03:48:05.100-07:00Nerd Food: Neurons for Computer Geeks - Part IV: More Electricity<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Neurons for Computer Geeks - Part IV: More Electricity</title>
<!-- 2015-09-05 Sat 18:56 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js"></script>
<script type="text/javascript">
<!--/*--><![CDATA[/*><!--*/
MathJax.Hub.Config({
// Only one of the two following lines, depending on user settings
// First allows browser-native MathML display, second forces HTML/CSS
// config: ["MMLorHTML.js"], jax: ["input/TeX"],
jax: ["input/TeX", "output/HTML-CSS"],
extensions: ["tex2jax.js","TeX/AMSmath.js","TeX/AMSsymbols.js",
"TeX/noUndefined.js"],
tex2jax: {
inlineMath: [ ["\\(","\\)"] ],
displayMath: [ ['$$','$$'], ["\\[","\\]"], ["\\begin{displaymath}","\\end{displaymath}"] ],
skipTags: ["script","noscript","style","textarea","pre","code"],
ignoreClass: "tex2jax_ignore",
processEscapes: false,
processEnvironments: true,
preview: "TeX"
},
showProcessingMessages: true,
displayAlign: "center",
displayIndent: "2em",
"HTML-CSS": {
scale: 100,
availableFonts: ["STIX","TeX"],
preferredFont: "TeX",
webFont: "TeX",
imageFont: "TeX",
showMathMenu: true,
},
MMLorHTML: {
prefer: {
MSIE: "MML",
Firefox: "MML",
Opera: "HTML",
other: "HTML"
}
}
});
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
<a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks.html">Part I</a> of this series looked at a neuron from above; <a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks_31.html">Part II</a> attempted
to give us the fundamental building blocks in electricity required to
get us on the road to modeling neurons. We did a quick interlude with
a bit of coding in <a href="http://mcraveiro.blogspot.co.uk/2015/09/nerd-food-neurons-for-computer-geeks_4.html">part III</a> but now, sadly, we must return to boring
theory once more.
</p>
<p>
Now that we grok the basics of electricity, we need to turn our
attention to the <i>RC circuit</i>. As we shall see, this circuit is of
particular interest when modeling neurons. The RC circuit is so called
because it is a circuit, and it is composed of a Resistor and a
Capacitor. We've already got some vague understanding of circuits and
resistors, so lets start by having a look at this new crazy critter,
<i>the capacitor</i>.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">Capacitors</h2>
<div class="outline-text-2" id="text-1">
<p>
Just like the battery is a source of current, one can think of the
capacitor as a temporary store of current. If you plug a capacitor
into a circuit with just a battery, it will start to "accumulate"
charge over time, up to a "maximum limit". But how exactly does this
process work?
</p>
<p>
<a href="https://www.khanacademy.org/science/physics/circuits-topic/circuits-with-capacitors/v/capacitors-and-capacitance">In simple terms</a>, the capacitor is made up of two metal plates, one of
which will connect to the positive end of the battery and another
which connects to the negative end. At the positive end, the metal
plate will start to lose negative charges because these are attracted
to the positive end of the battery. This will make this metal plate
positively charged. Similarly, at the negative end, the plate will
start to accumulate negative charges. This happens because the
electrons are repelled by the negative end of the battery. Whilst this
process is taking place, the capacitor is <i>charging</i>.
</p>
<p>
At some point, the process reaches a kind of equilibrium, whereby the
electrons in the positively charged plate are attracted equally to the
plate as they are to the positive end of the battery, and thus stop
flowing. At this point we say the capacitor is <i>charged</i>. It is
interesting to note that both plates of the capacitor end up with the
same "total" charge but different signs (i.e. <code>-q</code> and <code>+q</code>).
</p>
</div>
<div id="outline-container-sec-1-1" class="outline-3">
<h3 id="sec-1-1">Capacitance</h3>
<div class="outline-text-3" id="text-1-1">
<p>
We mentioned a "maximum limit". A few things control this limit: how
big the plates are, how much space there is between them and the kind
of material we place between them, if any. The bigger the plates and
the closer they are - without touching - the more you can store in the
capacitor. The material used for the plates is, of course, of great
importance too - it must be some kind of metal good at conducting.
</p>
<p>
In a more technical language, this notion of a limit is captured by
the concept of <i>capacitance</i>, and is given by the following formula:
</p>
\begin{align}
C = \frac{q}{V}
\end{align}
<p>
Lets break it down to its components to see what the formula is trying
to tell us. The role of <code>V</code> is to inform us about the potential
difference between the two plates. This much is easy to grasp; since
one plate is positively charged and other negatively charged, it is
therefore straightforward to imagine that a charge will have a
different electric potential in each plate, and thus that there will
be an electric potential difference between them. <code>q</code> tells us about
the magnitude of the charges that we placed on the plates -
i.e. ignoring the sign. It wouldn't be to great a leap to conceive
that plates with a larger surface area would probably have more
"space" for charges and so a larger <code>q</code> - and vice-versa.
</p>
<p>
Capacitance is then the ratio between these two things; a measure of
how much electric charge one can store for a given potential
difference. It may not be very obvious from this formula, but
capacitance is constant. That is to say, a given capacitor has a
capacitance, influenced by the properties described above. This
formula does not describe the discharging or charging process - but of
course, capacitance is used in the formulas that describe those.
</p>
<p>
Capacitance is measured in SI units of Farads, denoted by the letter
<code>F</code>. A farad is 1 coulomb over 1 volt:
</p>
\begin{align}
1F = \frac{C}{V}
\end{align}
</div>
</div>
<div id="outline-container-sec-1-2" class="outline-3">
<h3 id="sec-1-2">Capacitors and Current</h3>
<div class="outline-text-3" id="text-1-2">
<p>
After charging a capacitor, one may be tempted to discharge it. For
that one could construct a simple circuit with just the capacitor.
Once the circuit is closed, the negative charges will start to flow to
the positively charged plate, at full speed - minus the resistance of
the material. Soon enough both plates would be made neutral. At first
glance, this may appear to be very similar to our previous circuit
with a battery. However, there is one crucial difference: the battery
circuit had a constant voltage and a constant current (for a
theoretical battery) whereas a circuit with a discharging capacitor
has voltage and current that <i>decay</i> over time. By "decaying", all we
really mean is that we start at some arbitrarily high value and we
move towards zero over a period of time. This makes intuitive sense:
you cannot discharge the capacitor forever; and, as you discharge it,
the voltage starts to decrease - for there are less charges in the
plates and so less potential difference - and similarly, so does the
current - for there is less "pressure" to make the charges flow.
</p>
<p>
This intuition is formally captured by the following equation:
</p>
\begin{align}
I(t) = C \frac{dV(t)}{dt}
\end{align}
<p>
I'm rather afraid that, at this juncture, we have no choice but to
introduce Calculus. A proper explanation of Calculus a tad outside the
remit of these posts, so instead we will have to make do with some
common-sense but extremely hand-waved interpretations of the ideas
behind it. If you are interested in a light-hearted but still
comprehensive treatment of the subject, perhaps <a href="http://betterexplained.com/articles/a-gentle-introduction-to-learning-calculus/">A Gentle Introduction
To Learning Calculus</a> may be to your liking.
</p>
<p>
Let's start by taking a slightly different representation of the
formula above and then compare these two formulas.
</p>
\begin{align}
i = C \frac{dv}{dt}
\end{align}
<p>
In the first case we are talking about the current <code>I</code>, which normally
is some kind of average current over some unspecified period. Up to
now, <i>time</i> didn't really matter - so we got away with just talking
about <code>I</code> in these general terms. This was the case with the Ohm's Law
in <a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks_31.html">part II</a>. However, as we've seen, it is not so with capacitors - so
we need to make the current specific to a point in time. For that we
supply an "argument" to I - <code>I(t)</code>; here, a mathematician would say
that that <code>I</code> is a function of <i>time</i>. In the second case, we make use
of <code>i</code>, which is the <i>instantaneous current</i> through the
capacitor. The idea is that, somehow, we are able to know - for any
point in time - what the instantaneous current is.
</p>
<p>
How we achieve that is via the magic of Calculus. The expression
<code>dv/dt</code> in the second formula provides us with the instantaneous rate
of change of the voltage over time. The same notion can be applied to
<code>V</code>, as per first formula.
</p>
<p>
These formulas may sound awfully complicated, but what they are trying
to tell us is that the capacitor's current has the following
properties:
</p>
<ul class="org-ul">
<li>it varies as a "function" of time; that is to say, different
time points have different currents. Well, that's pretty consistent
with our simplistic notion of a decaying current.
</li>
<li>it is "scaled" by the capacitor's capacitance <code>C</code>; "bigger"
capacitors can hold on to higher currents for longer when compared
to "smaller" capacitors.
</li>
<li>the change in electric potential difference varies as a function of
time. This is subtle but also makes sense: we imagined some kind of
decay for our voltage, but there was nothing to say the decay would
remain <i>constant</i> until we reached zero. This formula tells us it
does not; voltage may decrease faster or slower at different points
in time.
</li>
</ul>
</div>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Circuits: Parallel and Series</h2>
<div class="outline-text-2" id="text-2">
<p>
The RC circuit can appear in a parallel or series form, so its a good
time to introduce these concepts. One way we can connect circuits is
in <i>series</i>; that is, all components are connected along a single
path, such that the current flows through <i>all</i> of them, one after the
other. If any component fails, the flow will cease.
</p>
<p>
This is best understood by way of example. Lets imagine the <a href="http://www.physicsclassroom.com/class/circuits/Lesson-4/Two-Types-of-Connections">canonical
example</a> of a battery - our old friend the 1.5V AA battery - and a
three small light bulbs. A circuit that connects them in series would
be made up of a cable segment plugged onto one of the battery's
terminals - say <code>+</code>, then connected to the first light bulb. A second
cable segment would then connect this light bulb to another light
bulb, followed by another segment and another light bulb. Finally, a
cable segment would connect the light build to the other battery
terminal - say <code>-</code>. Graphically - and pardoning my inability to use
<a href="https://wiki.gnome.org/Apps/Dia/">Dia</a> to create circuit diagrams - it would look more or less like this:
</p>
<div class="figure">
<p><img src="https://github.com/mcraveiro/neurite/raw/master/doc/blog/images/series_circuit.png" alt="series_circuit.png" width="300px" height="200px" />
</p>
<p><span class="figure-number">Figure 1:</span> Series circuit. Source: Author</p>
</div>
<p>
This circuit has a few interesting properties. First, if any of the
light bulbs fail, all of them will stop working because the circuit is
no longer closed. Second, if one were to add more and more light
bulbs, the brightness of each light bulb will start to decrease. This
is because each light bulb is in effect a resistor - the light shining
being a byproduct of said resistance - and so they are each decreasing
the current. So it is that in a series circuit the total resistance is
given by the sum of all individual resistances, and the current is the
same for all elements.
</p>
<p>
Parallel circuits are a bit different. The idea is that two or more
components are connected to the circuit <i>in parallel</i>, i.e. there are
two or more paths along which the current can flow at the same
time. So we'd have to modify our example to have a path to each of the
light bulbs which exists in parallel to the main path - quite
literally a segment of cable that connects the other segments of
cable, more or less like so:
</p>
<div class="figure">
<p><img src="https://github.com/mcraveiro/neurite/raw/master/doc/blog/images/parallel_circuit.png" alt="parallel_circuit.png" width="300px" height="200px" />
</p>
<p><span class="figure-number">Figure 2:</span> Parallel circuit. Source: Author</p>
</div>
<p>
Here you can see that if a bulb fails, there is still a closed loop in
which current can flow, so the other bulbs should be unaffected. This
also means that the voltage is the same for all components in the
circuit. Current and resistance are now "relative" to each component,
and it is possible to compute the overall current for the circuit via
<a href="https://en.wikipedia.org/wiki/Kirchhoff%2527s_circuit_laws#Kirchhoff.27s_current_law_.28KCL.29">Kirchhoff's Current Law</a>. Simplifying it, it means that the current for
the circuit is the sum of all currents flowing through each component.
</p>
<p>
This will become significant later on when we finally return to the
world of neurons.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">The RC Circuit</h2>
<div class="outline-text-2" id="text-3">
<p>
With all of this we can now move to the <i>RC circuit</i>. In its simplest
form, the circuit has a source of current with a resistor and a
capacitor:
</p>
<div class="figure">
<p><img src="https://upload.wikimedia.org/wikipedia/commons/a/a4/Discharging_capacitor.svg" alt="Discharging_capacitor.svg" width="300px" height="300px" />
</p>
<p><span class="figure-number">Figure 3:</span> Source: Wikipedia, <a href="https://en.wikipedia.org/wiki/RC_circuit">RC circuit</a></p>
</div>
<p>
Let's try to understand how the capacitor's voltage will behave over
time. This circuit is rather similar to the one we analysed when
discussing capacitance, with the exception that we now have a resistor
as well. But in order to understand this, we must return to
Kirchhoff's current law, which we hand-waved a few paragraphs
ago. Wikipedia tells us that:
</p>
<blockquote>
<p>
The algebraic sum of currents in a network of conductors meeting at a
point is zero.
</p>
</blockquote>
<p>
One way to understand this statement is to think that the total
quantity of current entering a junction point must be identical to the
total quantity leaving that junction point. If we consider entering to
be positive and leaving to be negative, that means that adding the two
together must yield zero.
</p>
<p>
Because of Kirchhoff's law, we can state that, for the positive
terminal of the capacitor:
</p>
\begin{align}
i_c(t) + i_r(t) = 0
\end{align}
<p>
That is: at any particular point in time <i>t</i>, the current flowing
through the capacitor added to the current flowing through the
resistor must sum to zero. However, we can now make use of the
previous formulas; after all, our section on capacitance taught us
that:
</p>
\begin{align}
i_c(t) = C \frac{dv(t)}{dt}
\end{align}
<p>
And making use of Ohm's Law we can also say that:
</p>
\begin{align}
i_r(t) = \frac{v(t)}{R}
\end{align}
<p>
So we can expand the original formula to:
</p>
\begin{align}
C \frac{dv(t)}{dt} + \frac{v(t)}{R}
\end{align}
<p>
Or:
</p>
\begin{align}
C \frac{dV}{dt} + \frac{V}{R}
\end{align}
<p>
I'm not actually going to follow the remaining steps to compute <code>V</code>,
but you can see them <a href="http://www.digilentinc.com/classroom/realanalog/text/Chapter_2p4p2.pdf">here</a> and they are fairly straighforward, or at
least as straightforward as calculus gets. The key point is, when you
solve the differential equation for <code>V</code>, you get:
</p>
\begin{align}
V(t) = V_0e^{-\frac{t}{RC}}
\end{align}
<p>
With <code>V0</code> being voltage when time is zero. This is called the
circuit's <i>natural response</i>. This equation is <i>very important</i>. Note
that we are now able to describe the behaviour of voltage over time
with just a few inputs: the starting voltage, the time, the resistance
and the capacitance.
</p>
<p>
A second thing falls off of this equation: the RC Time constant, or
τ. It is given by:
</p>
\begin{align}
\tau = RC
\end{align}
<p>
The Time Constant is described in a very useful way <a href="http://www.tpub.com/neets/book2/3d.htm">in this page</a>, so
I'll just quote them and their chart here:
</p>
<blockquote>
<p>
The time required to charge a capacitor to 63 percent (actually 63.2
percent) of full charge or to discharge it to 37 percent (actually
36.8 percent) of its initial voltage is known as the TIME CONSTANT
(TC) of the circuit.
</p>
</blockquote>
<div class="figure">
<p><img src="http://www.tpub.com/neets/book2/32NE0159.GIF" alt="32NE0159.GIF" />
</p>
<p><span class="figure-number">Figure 4:</span> The RC Time constant. Source: <a href="http://www.tpub.com/neets/book2/1.htm">Concepts of alternating current</a></p>
</div>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">What next?</h2>
<div class="outline-text-2" id="text-5">
<p>
Now we understand the basic behaviour of the RC Circuit, together with
a vague understanding of the maths that describe it, we need to return
to the neuron's morphology. Stay tuned.
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-09-05 Sat 18:56</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-26147927614975655232015-09-04T07:56:00.000-07:002015-09-05T07:31:33.571-07:00Nerd Food: Neurons for Computer Geeks - Part III: Coding Interlude<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Neurons for Computer Geeks - Part III: Coding Interlude</title>
<!-- 2015-09-04 Fri 17:16 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
If you are anything like me, the first two parts of this series have
already bored you silly with theory (<a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks.html">Part I</a>, <a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks_31.html">Part II</a>) and you are now
hankering for some code - any code - to take away the pain. So part
III is here to do exactly that. However, let me prefix that grandiose
statement by saying this is not the best code you will ever
see. Rather, its just a quick hack to introduce a few of the
technologies we will make use of for the remainder of these series,
namely:
</p>
<ul class="org-ul">
<li><a href="http://www.cmake.org/">CMake</a> and <a href="http://martine.github.io/ninja/">Ninja</a>: this is how we will build our code.
</li>
<li><a href="http://www.webtoolkit.eu/wt">Wt</a>: provides a quick way to knock-up a web frontend for C++ code.
</li>
<li><a href="http://www.boost.org/">Boost</a>: in particular <a href="http://www.boost.org/doc/libs/1_59_0/doc/html/boost_units.html">Boost Units</a> and later on <a href="http://www.boost.org/doc/libs/1_59_0/libs/numeric/odeint/doc/html/index.html">Boost OdeInt</a>. Provides
us with the foundations for our numeric work.
</li>
</ul>
<p>
What I mean by a "quick hack" is: there is no validation, no unit
tests, no "sound architecture" and none of the things you'd expect
from production code. But it should serve as an introduction to
modeling in C++.
</p>
<p>
All the code is available in GitHub under <a href="https://github.com/mcraveiro/neurite">neurite</a>. Lets have a quick
look at the project structure.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">CMake</h2>
<div class="outline-text-2" id="text-1">
<p>
We just took a slimmed down version of the <a href="https://github.com/DomainDrivenConsulting/dogen">Dogen</a> build system to build
this code. We could have gotten away with a much simpler CMake setup,
but I intend to use it for the remainder of this series so that's why
its a bit more complex than what you'd expect. It is made up of the
following files:
</p>
<ul class="org-ul">
<li>Top-level <code>CMakeLists.txt</code>: ensures all of the dependencies can be
found and configured for building, sets up the version number and
debug/release builds.
</li>
<li><code>build/cmake</code>: any Find* scripts that are not supplied with the
CMake distribution. We Google for these and copied them here.
</li>
<li><code>projects/CMakeLists.txt</code>: sets up all of the compiler and linker
flags we need to build the project. Uses pretty aggressive flags
such as <code>-Wall</code> and <code>-Werror</code>.
</li>
<li><code>projects/ohms_law/src/CMakeLists.txt</code>: our actual project, the bit
that matters for this article.
</li>
</ul>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2"><code>ohms_law</code> Project</h2>
<div class="outline-text-2" id="text-2">
<p>
The project is made up of two classes, in files <code>calculator.[hc]pp</code>
and <code>view.[hc]pp</code>. The names are fairly arbitrary but they try to
separate View from Model: the user interface is in <code>view</code> and the
"number crunching" is in <code>calculator</code>.
</p>
</div>
<div id="outline-container-sec-2-1" class="outline-3">
<h3 id="sec-2-1">The View</h3>
<div class="outline-text-3" id="text-2-1">
<p>
Lets have a quick look at <code>view</code>. In the header file we simply define
a Wt application with a few widgets:
</p>
<pre class="example">
class view : public Wt::WApplication {
public:
view(const Wt::WEnvironment& env);
private:
Wt::WLineEdit* current_;
Wt::WLineEdit* resistance_;
Wt::WText* result_;
};
</pre>
<p>
It is implemented in an equally trivial manner. We just setup the
widgets and hook them together. Finally, we create a trivial event
handler that performs the "computations" when the button is clicked.
</p>
<pre class="example">
view::view(const Wt::WEnvironment& env) : Wt::WApplication(env) {
setTitle("Ohm's Law Calculator");
root()->addWidget(new Wt::WText("Current: "));
current_ = new Wt::WLineEdit(root());
current_->setValidator(new Wt::WDoubleValidator());
current_->setFocus();
root()->addWidget(new Wt::WText("Resistance: "));
resistance_ = new Wt::WLineEdit(root());
resistance_->setValidator(new Wt::WDoubleValidator());
Wt::WPushButton* button = new Wt::WPushButton("Calculate!", root());
button->setMargin(5, Wt::Left);
root()->addWidget(new Wt::WBreak());
result_ = new Wt::WText(root());
button->clicked().connect([&](Wt::WMouseEvent&) {
const auto current(boost::lexical_cast<double>(current_->text()));
const auto resistance(boost::lexical_cast<double>(resistance_->text()));
calculator c;
const auto voltage(c.voltage(resistance, current));
const auto s(boost::lexical_cast<std::string>(voltage));
result_->setText("Voltage: " + s);
});
}
</pre>
</div>
</div>
<div id="outline-container-sec-2-2" class="outline-3">
<h3 id="sec-2-2">The Model</h3>
<div class="outline-text-3" id="text-2-2">
<p>
The model is equally as simple as the view. It is made up of a single
class, <code>calculator</code>, whose job is to compute the voltage using Ohm's
Law. It does this by making use of Boost Units. This is obviously not
necessary, but we wanted to take the opportunity to explore this
library as part of this series of articles.
</p>
<pre class="example">
double calculator::
voltage(const double resistance, const double current) const {
boost::units::quantity<boost::units::si::resistance>
R(resistance * boost::units::si::ohms);
boost::units::quantity<boost::units::si::current>
I(current * boost::units::si::amperes);
auto V(R * I);
return V.value();
}
</pre>
</div>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Compiling and Running</h2>
<div class="outline-text-2" id="text-3">
<p>
If you are on a debian-based distribution, you can do the following
steps to get the code up and running. First install the dependencies:
</p>
<pre class="example">
$ sudo apt-get install libboost-all-dev witty-dev ninja-build cmake clang-3.5
</pre>
<p>
Then obtain the source code from GitHub:
</p>
<pre class="example">
$ git clone https://github.com/mcraveiro/neurite.git
</pre>
<p>
Now you can build it:
</p>
<pre class="example">
cd neurite
mkdir output
cd output
cmake ../ -G Ninja
ninja -j5
</pre>
<p>
If all went according to plan, you should be able to run it:
</p>
<pre class="example">
$ stage/bin/neurite_ohms_law --docroot . --http-address 0.0.0.0 --http-port 8080
</pre>
<p>
Now using a web browser such as chrome, connect to
<a href="http://127.0.0.1:8080">http://127.0.0.1:8080</a> and you should see a "shiny" Ohm's Law
calculator! Sorry, just had to be done to take away the boredom a
little bit. Lets proceed with the more serious matters at hand, with
the promise that the real code will come later on.
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-09-04 Fri 17:16</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-38167396234980454342015-08-31T11:38:00.000-07:002015-09-14T09:39:05.248-07:00Nerd Food: Neurons for Computer Geeks - Part II: The Shocking Complexity of Electricity<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Neurons for Computer Geeks - Part II: The Shocking Complexity of Electricity</title>
<!-- 2015-08-31 Mon 19:27 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
In <a href="http://mcraveiro.blogspot.co.uk/2015/08/nerd-food-neurons-for-computer-geeks.html">part I</a> we started to describe the basic morphology of the
neuron. In order to continue, we now need to take a detour around the
world of electricity. If you are an electricity nerd, I apologise in
advance; this is what happens when a computer scientist escapes into
your realm, I'm afraid.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">"Honor the charge they made!"</h2>
<div class="outline-text-2" id="text-1">
<p>
First and foremost, we need to understand the concept of <i>charge</i>. It
is almost a tautology that atoms are made up of "sub-atomic"
particles. These are the <i>proton</i>, the <i>neutron</i> and the
<i>electron</i>. The neutron is not particularly interesting right now;
however the electron and the proton are, and all because they have a
magical property called <i>charge</i>. For our purposes, it suffices to
know that "charge" means that certain sub-atomic particles attract or
repeal each other, according to a well defined set of rules.
</p>
<p>
You can think of a charge as a property attached to the sub-atomic
particle, very much like a person has a weight or height, but with a
side-effect; it is as if this property makes people push or hug each
other when they are in close proximity, and they do so with the same
strength when at the same distance. This "strength" is the <i>electric
force</i>. How they decide whether to hug or push the next guy is based
on the "sign" of the charge - that is, positive or negative - with
respect to their own charge "sign". Positives push positives away but
hug negatives and vice-versa.
</p>
<p>
For whatever historical reasons, very clever people decided that an
electron has one negative unit of charge and a proton has a positive
unit of charge. The sign is, of course, rather arbitrary. We could
have just as well said that protons are red and electrons are blue or
some other suitably binary-like convention to represent these
permutations. Just because protons and electrons have the same
charge, it does not follow that they are similar in other respects. In
fact, they are <i>very</i> different creatures. For example, the electron is
very "small" when compared to the proton - almost 2000 times
"smaller". The relevance of this "size" difference will become
apparent later on. Physicists call this "size" <i>mass</i>, by the by.
</p>
<p>
As it happens, all of these sub-atomic crazy critters are rather
minute entities. So small in fact that it would be really cumbersome
if we had to talk about charges in terms of the charge of an electron;
the numbers would just be too big and unwieldy. So, the very clever
people came up with a sensible way to bundle up the charges of the
sub-atomic particles in bigger numbers, much like we don't talk about
millimetres when measuring the distance to the Moon. However, unlike
the nice and logical metric system, with its neat use of the decimal
system, physicists came up instead with the <i>Coulomb</i>, or <i>C</i>, one
definition of which is:
</p>
<ul class="org-ul">
<li>1 Coloumb (1C) = 6.241 x 10<sup>18</sup> protons
</li>
<li>-1 Coloumb (-1C) = 6.241 x 10<sup>18</sup> electrons
</li>
</ul>
<p>
This may sound like a <i>very</i> odd choice - hey, why not just 1 x 10<sup>20</sup>
or some other "round" number? - but just like a <a href="http://www.quora.com/Why-is-a-kilogram-equal-to-1000-grams-but-a-kilobyte-equals-1024-bytes">kilobyte is 1024 bytes
rather 1000</a>, this wasn't done by accident either. In fact, all related
<a href="https://en.wikipedia.org/wiki/International_System_of_Units">SI units</a> were carefuly designed to work together and make calculations
as easy as possible.
</p>
<p>
Anyway, whenever you see <code>q</code> or <code>Q</code> in formulas it normally refers to
a charge in Coulombs.
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">Units, Dimensions, Measures, Oh My!</h2>
<div class="outline-text-2" id="text-2">
<p>
Since we are on the subject of SI, this is probably a good point to
talk about units, dimensions, measurements, magnitudes, conversions
and other such exciting topics. Unfortunately, these are important to
understand how it all hangs together.
</p>
<p>
A number such as <code>1A</code> makes use of the SI <i>unit of measure</i> "Ampere"
and it exists in a <i>dimension</i>: the dimension of all units which can
talk about electric charges. This is very much in the same way we can
talk about time in seconds or minutes - we are describing points in
the time dimension, but using different <i>units of measure</i> - or just
<i>units</i>, because we're lazy. A <i>measurement</i> is the recording of a
quantity with a unit in a dimension. Of course, it would be too simple
to call it a "quantity", so instead physicists, mathematicians and the
like call it <i>magnitude</i>. But for the lay person, its not too bad an
approximation to replace "magnitude" with "quantity".
</p>
<p>
Finally, it is entirely possible to have <i>compound dimensional units</i>;
that is, one can have a unit of measure that refers to more than one
dimension, such as say "10 kilometres per second".
</p>
<p>
I won't discuss conversions just now, but you can easily imagine that
formulas that contain multiple units may provide ways to convert from
one unit to another. This will become relevant later on.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">Go With the Flow</h2>
<div class="outline-text-2" id="text-3">
<p>
Now we have a way of talking about charge, and now we know these
things can move - since they attract and repel each other - the next
logical thing is to start to imagine <i>current</i>. The name sounds
magical, but in reality it is akin to a current in a river: you are
just trying to figure out how much water is coming past you every
second (or in some other suitable unit in the time dimension). The
exact same exercise could be repeated for the number of cars going
past in a motorway or the number of runners across some imaginary
point in a track. For our electric purposes, current tells you how
many charges have zipped past over a period of time.
</p>
<p>
In terms of SI units, current is measured in <i>Amperes</i>, which have the
symbol <i>A</i>; an Ampere tells us how many Coloumbs have flown past in a
second. Whenever you see <code>I</code> in formulas it normally refers to
current.
</p>
<p>
Now lets see how these two things - Coulombs and Amperes - could work
together. Lets imagine an arbitrary "pipe" between two imaginary
locations, one side of which with a pile of positive charges and, on the
other side, a pile of negative charges - both measured in
Coulombs, naturally. In this <i>extraordinarily</i> simplified and
non-existing world, the negative charges would "flow" down the pipe,
attracted by the positive charges. Because the positive charges are so
huge they won't budge, but the negative charges - the lighter
electrons - would zip across to meet them. The number of charges you
see going past in a time tick is the current.
</p>
</div>
</div>
<div id="outline-container-sec-4" class="outline-2">
<h2 id="sec-4">Resist!</h2>
<div class="outline-text-2" id="text-4">
<p>
Going back to our example of current in a river, one can imagine that
some surfaces are better at allowing water to flow than others; for
example, a river out in the open is a lot less "efficient" at flowing
than say a plastic pipe designed for that purpose. One reason is that
the river has to deal with twists and turns as it finds a path over
the landscape whereas the pipe could be laid out as straight as
possible; but it is also that the rocks and other elements of the
landscape slow down water, whereas a nice flat pipe would have no such
impediments. If one were to take these two extremes - a plastic pipe
designed for maximum water flow versus a landscape - one could see
that they affect flow differently; and one could be tempted to name
the property of "slowing down the flow" <i>resistance</i>, because it
describes how much "resistance" these things are offering to the
water. If you put up a barrier to avoid flooding, you probably would
want it to "resist" water quite a lot rather than allow it to flow;
and you can easily imagine that sand and sandbags "resist" water in
very different ways.
</p>
<p>
Resistance is a fundamental concept in the electrical world. The gist
of it is similar to the contrived examples above, in that not all
materials behave the same way with regards to allowing charges to
flow. Some allow them to flow freely nearly at maximum speed whereas
others do not allow them to flow at all.
</p>
<p>
Since we are dealing with physics, it is of course possible to measure
resistance. We do so in SI units of <i>Ohms</i>, denoted by the Greek
letter upper-case Ω.
</p>
<p>
As we shall see, not all materials are nicely behaved when it comes to
resistance.
</p>
</div>
</div>
<div id="outline-container-sec-5" class="outline-2">
<h2 id="sec-5">You've Got Potential Baby!</h2>
<div class="outline-text-2" id="text-5">
<p>
Lets return to our non-existing "pipe that allows charges to flow"
scenario, and take it one step further. Imagine that for whatever
reason our pipe becomes clogged up with a blockage somewhere in the
middle. Nothing could actually flow due to this blockage so our
current drops to zero.
</p>
<p>
According to the highly simplified rules that we have learned thus
far, we do know that - were there to be no blockage - there <i>would</i> be
movement (current). That is, the setup of the two bundles in space is
such that, given the right conditions, we would start to see things
flowing. But, alas, we do not have the right conditions because the
pipe is blocked; hence no flow. You could say this setup has "the
potential" to get some flow going, if only we could fix the blockage.
</p>
<p>
In the world of electricity, this idea is captured by a few related
concepts. If we highly simplify them, they amount to this:
</p>
<ul class="org-ul">
<li><i>electric potential</i>: the idea that depending where you place a
charge in space, it may have different "potential" to generate
energy. We'll define energy a bit better latter on, but for now a
layman's idea of it suffices. By way of an example: if you place a
positive charge next to a lump of positive charges and let it go, it
will move a certain distance away from the lump. Before you let the
charge go, you know the charge has potential to move away. You can
also see that the charge will move by different amounts depending
how close you place it to the lump; the closer you place it, the
more it will move. When we are thinking of electric potential, we
think of just one charge.
</li>
<li><i>electric potential energy</i>: clearly it would be possible to move
two or three charges too, as we did for the one; and clearly they
should produce more energy than a single charge. So one simple way
of understanding electric potential energy is to think of it as the
case of electric potential that deals with the total number of
charges we're interested in, rather than just one.
</li>
</ul>
<p>
Another way of imagining these two concepts is to think that electric
potential is a good way to measure things when you don't particularly
care about the number of charges involved; it is as if you scaled
everything to just one unit of charge. Electric potential energy is
more when you are thinking of a system with an actual number of
charges. But both concepts deal with the notion that placing a charge
at different points in space may have an impact in the energy you can
get out of it.
</p>
<p>
Having said all of that we can now start to think about <i>electric
potential difference</i>. It uses the same approach as electric
potential, in that everything is scaled to just one unit of charge,
but as the name implies, it provides a measurement of <i>the difference</i>
between the electric potential of two points. Electric potential
difference is more commonly known as <i>voltage</i>. Interestingly, it is
also known as <i>electric pressure</i>, and this may be the most meaningful
of its names; this is because when there is an electric potential
difference, it applies "pressure" on charges which force them to move.
</p>
<p>
The SI unit <i>Volt</i> is used to measure electric potential, electric
potential energy and electric potential difference amongst other
things. This may sound a bit weird at first, but it is just because
one is unfamiliar with these concepts. Take <i>time</i>, for example: we
use minutes as a unit of measure of all sorts of things (duration of a
football game, time it takes for the moon to go around the earth,
etc.). We did not invent a new unit for each phenomenon because we
recognised - at some point - that we were dealing with points in the
same dimension.
</p>
</div>
</div>
<div id="outline-container-sec-6" class="outline-2">
<h2 id="sec-6">Quick Conceptual Mop-Up</h2>
<div class="outline-text-2" id="text-6">
<p>
Before we move over to the formulas, it may be best to tie up a few
loose ends. These are not strictly necessary, but just make the
picture a bit more complete and moves us to a more realistic model -
if still very simplistic.
</p>
<p>
First, we should start with atoms; we mentioned charges but skipped
them. Atoms are (mostly) a stable arrangement of charges, placed in
such a way that the atoms themselves are neutral - i.e. contain
exactly the same amount of negative and positive charges. We mentioned
before that protons and electrons don't really get along, and neutrons
are kind of just there, hanging around. In truth, neutrons and protons
also really get along, via the aptly named <i>nuclear force</i>; this is
what binds them together in the nucleus of the atom. Electrons are
attracted to protons and live their existences in a "cloud" around the
nucleus. Note that the nucleus is more than 99% of the mass of the
atom, which gives you an idea of just how small electrons are.
</p>
<p>
The materials we will deal with in our examples are made of atoms, as
are, well, quite a few things in the universe. These materials are
themselves stable arrangements of atoms, just like atoms are stable
arrangements of protons, neutrons and electrons. As you can see in the
picture, these look like lattices of some kind.
</p>
<div class="figure">
<p><img src="https://sciencemonday.files.wordpress.com/2011/09/carbon-atoms.jpg" alt="carbon-atoms.jpg" />
</p>
<p><span class="figure-number">Figure 1:</span> Microscopic View of Carbon Atoms. Source: <a href="https://sciencemonday.wordpress.com/2011/09/04/quantum-physics-the-brink-of-knowing-something-wonderful/">Quantum Physics: The Brink of Knowing Something Wonderful</a></p>
</div>
<p>
In practice, copper wires are made up of a great many things rather
than just atoms of copper. One such "kind of thing" is the <i>unbound
electrons</i> - or free-moving electrons; basically electrons are not
trapped into an atom. As we mentioned before, electrons are the ones
doing most of the moving. Left to their own devices, electrons in a
conducting material will just move around, bumping into atoms in a
fairly random way. However, lets say you take one end of a copper wire
and plug it to the <code>+</code> side of a regular AA battery and then take
other end and plug it to the <code>-</code> side of the battery. According to all
we've just learned, its easy to imagine what will happen: the
electrons stored in the <code>-</code> side will zip across the copper to meet
their proton friends at the other end. This elemental construction,
with its circular path, is called a <i>circuit</i>. What you've done is to
upset the neutral balance of the copper wire and got all the electrons
to move in a coordinated way (rather than random) from the <code>-</code> side to
the <code>+</code> side.
</p>
<p>
It is at this juncture that we must introduce the concept of ions. An
<i>ion</i> is basically an atom that is no longer neutral - either because
it has more protons than electrons (called a <i>cation</i>) or more
electrons than protons (called an <i>anion</i>). In either case, this comes
about because the atom has gained or lost some electrons. Ions will
become of great interest when we return to the neuron.
</p>
<p>
One final word on resistance and its sister concept of <i>conductance</i>:
</p>
<ul class="org-ul">
<li><i>Resistance</i> is in effect a <a href="http://education.jlab.org/qa/current_02.html">byproduct of the way the electrons are
arranged in the electron cloud</a> and is related to the ionisation
mentioned above; certain arrangements just don't allow electrons to
flow across.
</li>
<li><i>Conductance</i> is the inverse of resistance. When you talk about
resistance you are focusing on the material's ability to impair
movement of charges; when you talk about conductance you are
focusing on the material's ability to let charge flow through.
</li>
</ul>
<p>
The reason we choose copper or other metals for our examples is
because they are good at <i>conducting</i> these pesky electrons.
</p>
</div>
</div>
<div id="outline-container-sec-7" class="outline-2">
<h2 id="sec-7">Ohm's Law</h2>
<div class="outline-text-2" id="text-7">
<p>
We have now introduced all the main actors required for one of the
main parts in the play: Ohm's Law. It can be stated very easily:
</p>
<pre class="example">
V = R x I
</pre>
<p>
And here's a picture to aid intuition.
</p>
<div class="figure">
<p><img src="http://i.stack.imgur.com/4KhUg.jpg" alt="4KhUg.jpg" />
</p>
<p><span class="figure-number">Figure 2:</span> Source: <a href="http://physics.stackexchange.com/questions/161650/could-someone-intuitively-explain-to-me-ohms-law">Could someone intuitively explain to me Ohm's law?</a></p>
</div>
<p>
The best way to understand this law is to create a simple circuit.
</p>
<div class="figure">
<p><img src="https://upload.wikimedia.org/wikipedia/commons/b/b4/Ohm's_Law_with_Voltage_source_TeX.svg" alt="Ohm's_Law_with_Voltage_source_TeX.svg" />
</p>
<p><span class="figure-number">Figure 3:</span> Simple electrical circuit. Source: Wikipedia, <a href="https://en.wikipedia.org/wiki/Electrical_network">Electrical network</a></p>
</div>
<p>
On the left we have a voltage source, which could be our 1.5V AA
battery. On the right of the diagram we have a <i>resistor</i> - an
electric component that is designed specifically to "control" the flow
of the electric current. Without the resistor, we would be limited by
how much current the battery can pump out and how much "natural"
resistance the copper wire has, which is not a lot since it is very
good at conducting. The resistor gives us a way to limit current flow
from these theoretical maximum limitations.
</p>
<p>
Even if you are not particularly mathematically oriented, you can
easily see that Ohm's Law gives us a nice way to find any of these
three variables, given the other two. That is to say:
</p>
<pre class="example">
R = V / I
I = V / R
</pre>
<p>
These tell us many interesting things such as: for the same
resistance, current increases as the voltage increases. For good
measure, we can also find out the conductance too:
</p>
<pre class="example">
G = I / V = 1 / R
</pre>
<p>
It is important to notice that not everything obeys Ohm's law -
i.e. behave in a straight line. The conductors that obey this law are
called <i>ohmic conductors</i>. Those that do not are called <i>non-ohmic
conductors</i>. There are also things that obey to Ohm's Law, for the
most part. These are called <i>quasi-ohmic</i>.
</p>
</div>
<div id="outline-container-sec-7-1" class="outline-3">
<h3 id="sec-7-1">What next?</h3>
<div class="outline-text-3" id="text-7-1">
<p>
We have already run out of time for this instalment but there are
still some more fundamental electrical concepts we need to
discuss. The next part will finish these and start to link them back
to the neuron.
</p>
</div>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-08-31 Mon 19:27</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0tag:blogger.com,1999:blog-2672427473119923109.post-60243757825193841822015-08-31T09:26:00.000-07:002015-09-14T09:32:51.669-07:00Nerd Food: Neurons for Computer Geeks - Part I: A Neuron From Up On High<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Nerd Food: Neurons for Computer Geeks - Part I: A Neuron From Up On High</title>
<!-- 2015-08-31 Mon 17:25 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="generator" content="Org-mode" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center; }
.todo { font-family: monospace; color: red; }
.done { color: green; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.right { margin-left: auto; margin-right: 0px; text-align: right; }
.left { margin-left: 0px; margin-right: auto; text-align: left; }
.center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: visible;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline;}
pre.src-sh:before { content: 'sh'; }
pre.src-bash:before { content: 'sh'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-R:before { content: 'R'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-java:before { content: 'Java'; }
pre.src-sql:before { content: 'SQL'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.right { text-align: center; }
th.left { text-align: center; }
th.center { text-align: center; }
td.right { text-align: right; }
td.left { text-align: left; }
td.center { text-align: center; }
dt { font-weight: bold; }
.footpara:nth-child(2) { display: inline; }
.footpara { display: block; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
/*]]>*/-->
</style>
<script type="text/javascript">
/*
@licstart The following is the entire license notice for the
JavaScript code in this tag.
Copyright (C) 2012-2013 Free Software Foundation, Inc.
The JavaScript code in this tag is free software: you can
redistribute it and/or modify it under the terms of the GNU
General Public License (GNU GPL) as published by the Free Software
Foundation, either version 3 of the License, or (at your option)
any later version. The code is distributed WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
As additional permission under GNU GPL version 3 section 7, you
may distribute non-source (e.g., minimized or compacted) forms of
that code without the copy of the GNU GPL normally required by
section 4, provided you include this license notice and a URL
through which recipients can access the Corresponding Source.
@licend The above is the entire license notice
for the JavaScript code in this tag.
*/
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.cacheClassElem = elem.className;
elem.cacheClassTarget = target.className;
target.className = "code-highlighted";
elem.className = "code-highlighted";
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(elem.cacheClassElem)
elem.className = elem.cacheClassElem;
if(elem.cacheClassTarget)
target.className = elem.cacheClassTarget;
}
/*]]>*///-->
</script>
</head>
<body>
<div id="content">
<p>
As any computer geek would tell you, computer science is great in and
of itself and many of us could live long and contented lives inside
that box. But things certainly tend to become interesting when there
is a whole problem domain to model, and doubly so when that domain is
outside of our comfort zone. As it happens, I have managed to step
outside said zone - rather, quite far outside - so it seemed like a
good idea to chronicle these adventures here.
</p>
<p>
The journey we are about to embark starts with a deceptively simple
mission: to understand how one can use computers to model neurons. The
intended audience of these posts is anyone who loves coding but has no
idea about electricity, circuits, cells and so on - basically someone
very much like me. We shall try to explain, at least to a degree, all
of the required core concepts in order to start coding. As it turns
out, there are quite a few.
</p>
<p>
But hey, as <a href="http://skeptics.stackexchange.com/questions/8742/did-einstein-say-if-you-cant-explain-it-simply-you-dont-understand-it-well-en">they say</a>, "If you can't explain something to a six
year-old, you really don't understand it yourself". So lets see if I
got it or not.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1">I'm a Cell, Get Me Out Of Here!</h2>
<div class="outline-text-2" id="text-1">
<p>
A neuron is a <i>cell</i>, so it makes sense to start with cells. Cells are
a basic building block in biology and can be considered as the
smallest unit of a living organism - at least for our purposes, if
nothing else. The key idea behind a cell is as obvious as you'd like:
there is the inside, the outside, and the thing that separates
both.
</p>
<p>
Of course, this being biology, we need to give it complicated
names. Accordingly, the inside of the cell is the <i>cytoplasm</i> and the
thing that separates the cell from the outside world is the
<i>membrane</i>. You can think of it as a <i>tiny</i> roundy-box-like thing,
with some gooey stuff inside. The material of the box is the
membrane. The gooey stuff is the cytoplasm. When we start describing
the different cellular structures - as we are doing here - we are
talking about the cell's <i>morphology</i>.
</p>
<p>
Living beings are made up of many, many cells - according to some
estimates, a human body would have several trillion - and cells
themselves come in many, <i>many</i> kinds. Fortunately, we are interested
in just one kind: the <i>neuron</i>.
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2">The Neuron Cell</h2>
<div class="outline-text-2" id="text-2">
<p>
The neuron is a nerve cell. Of course, there are many, <i>many</i> kinds of
neurons - nature just seems to love complexity - but they all share
things in common, and those things define their neuron-ness.
</p>
<p>
Unlike the "typical" cell we described above (i.e. "roundy-box-like
thing"), the neuron is more like a roundy-box-like thing with some
branches coming out of it. The box-like thing is the cell body and is
called <i>soma</i>. There are two types of branches: axons and dendrites. A
<i>dendrite</i> tends to be short, and it branches like a tree with a very
small trunk. The <i>axon</i> tends to be long and it also branches off like
a tree, but with a very long trunk. As we said, there are many kinds
of neurons, but a fair generalisation is that they tend to have few
axons (one or maybe a couple) and many dendrites (in the thousands).
</p>
<div class="figure">
<p><img src="http://usercontent1.hubimg.com/8808542_f520.jpg" alt="8808542_f520.jpg" />
</p>
<p><span class="figure-number">Figure 1:</span> Source: <a href="http://mariexotoni.hubpages.com/hub/What-is-a-Neuron2#">What is a Neuron?</a></p>
</div>
<p>
This very basic morphology is already sufficient to allows to start to
think of a neuron as a "computing device" - a strange kind of device
where the dendrites provide inputs and the axon outputs. The neuron
receives all these inputs, performs some kind of computation over
them, and produces an output.
</p>
<p>
The next logical question for a computer scientist is, then: "where do
the outputs come from and where do they go?". Imagining an idealised
neuron, the dendrites would be "connecting" to other dendrites or to
axons. At this juncture (pun not intended), we need to expand on what
exactly these "connections" are. In truth, its not that the axon binds
directly to the dendrite; there is always a gap between them. But this
gap is a special kind of gap, first because it is a very small gap and
second because it is one over which things can travel, from the axon
into the dendrite. This kind of connectivity between neurons is called
a <i>synapse</i>.
</p>
<p>
From this it is an easy leap to imagine that these sets of neurons
connected to other neurons begin to form "networks" of connectivity,
and these networks will also have computational-device-like
properties, just like a neuron. These are called <i>neural
networks</i>. Our brain happens to be one of these "neural networks", and
a pretty large one at that: it can have <a href="http://www.nature.com/scitable/blog/brain-metrics/are_there_really_as_many">as many as 80-100 billion
neurons</a>, connected over some 1 quadrillion synapses. In these days of
financial billions and trillions, it is easy to be fooled into
thinking 100 billion is not a very large number, so to get a sense of
perspective lets compare it to another large network. The biggest and
fastest growing human-made network is the Internet, estimated to have
<a href="http://www.gartner.com/newsroom/id/2905717">some 5 billion connected devices</a> but less than <a href="http://bgp.potaroo.net/">600k connections in its
core</a> - and yet we are already <a href="http://research.dyn.com/2014/08/internet-512k-global-routes/">creacking at the seams</a>.
</p>
</div>
</div>
<div id="outline-container-sec-3" class="outline-2">
<h2 id="sec-3">The Need To Go Lower</h2>
<div class="outline-text-2" id="text-3">
<p>
Alas, we must dig deeper before we start to understand how these
things behave in groups. Our skimpy first pass at the neuron
morphology left a lot of details out, which are required to understand
how they behave. As we explained, neurons have axons and dendrites,
and these are responsible for hooking them together. However, what is
interesting is what they talk about once they are hooked.
</p>
<p>
A neuron is can be thought of as an <i>electrical device</i>, and much of
its power (sorry!) stems from this. In general, as computer
scientists, we don't like to get too close to the physical messiness
of the world of hardware; we deem it sufficient to understand some
high-level properties, but rarely do we want to concern ourselves with
transistors or even - regrettably - registers or pipelines in the
CPU. With neurons, we can't get away with it. We need to understand
the hardware - or better, the wetware - and for that we have to go
<i>very</i> low-level.
</p>
<p>
We started off by saying cells have a membrane that separates the
outside world from the cytoplasm. That was a tad of an
oversimplification; after all, if the membrane did not allow anything
in, how would the cell continue to exist - or even come about in the
first place? In practice these membranes are permeable - or to be
precise, <i>semi-permeable</i>. This just means that it allows some stuff
in and some stuff out, under controlled circumstances. This is how a
cell gets energy <i>in</i> to do its thing and how it expels its unwanted
content <i>out</i>. Once things started to move in and out selectively,
something very interesting can start to happen: the build up of
"electric potential". However, rather unfortunately, in order to
understand what we mean by this, we need to cover the fundamentals of
electricity.
</p>
<p>
Onward and downwards we march. Stay tuned for Part II.
</p>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="date">Created: 2015-08-31 Mon 17:25</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>Marco Craveirohttp://www.blogger.com/profile/01039195055988254979noreply@blogger.com0