emacs/var/elfeed/db/data/e0/e070cf8c3d94b7da6350be24c86f338d8b9fe750

<p>The <a href="https://fennel-lang.org">Fennel programming
    language</a> recently celebrated its fifth birthday, and we ran
  <a href="https://fennel-lang.org/2021">a survey</a> to learn more
  about the community and what has been working well and what
  hasn't. Fennel's approach has always been one of simplicity; not
  just in the conceptual footprint of the language, but in reducing
  dependencies and moving parts, and using on a runtime that fits in
  under 200kb. In order to reflect this, the Fennel web site is hosted
  as static files on the same Apache-backed shared hosting account
  I've been using for this blog since 2005.</p>

<p>Of course, <a href="https://git.sr.ht/~technomancy/fennel-lang.org/tree/main/item/main.fnl">generating
  HTML from lisp code</a> is one of the oldest tricks in the
  book[<a href="https://technomancy.us#fn1">1</a>], so I won't bore anyone with the details there. But what
  happens when you want to mix in something that <em>isn't</em>
  completely static, like this survey? Well, that's where it gets interesting.</p>

<blockquote>
  I put the shotgun in an Adidas bag and padded it out with four pairs
 of tennis socks, not my style at all, but that was what I was aiming
 for: If they think you're crude, go technical; if they think you're
 technical, go crude. I'm a very technical boy. So I decided to get as
 crude as possible. These days, though, you have to be pretty
 technical before you can even aspire to crudeness.[<a href="https://technomancy.us#fn2">2</a>]
</blockquote>

<p>When I was in school, I learned how to write and deploy Ruby web
  programs. The easiest way to get that set up was
  using <a href="https://en.wikipedia.org/wiki/Common_Gateway_Interface">CGI</a>. A CGI script is just a process which is launched by the
  web server in such a way that the request comes in on stdin and
  environment variables and the response is sent over stdout. But
  larger Ruby programs tended to have very slow boot times, which
  didn't fit very well with CGI's model of launching a process afresh
  for every request that came in, and eventually other models replaced
  CGI. Most people regard CGI as somewhat outmoded and obsolete, but
  it fits Fennel's ethos nicely and complements a mostly-static-files
  approach.</p>

<p>So the
  survey <a href="https://git.sr.ht/~technomancy/fennel-lang.org/tree/3bed58d0007ac8f9616486ef20094cffc2c10562/item/survey/survey.fnl#L39">generates
    an HTML form</a> in a static file which points to a CGI script as
  its <tt>action</tt>. The CGI script looks like this, but it gets
  compiled to Lua as part of the deploy process to keep the
  dependencies light.</p>

<pre class="code">(<span class="keyword">let</span> [contents (<span class="type">io.read</span> <span class="string">"*all"</span>)
      date (<span class="type">io.popen</span> <span class="string">"date --rfc-3339=ns"</span>)
      id (<span class="keyword">:</span> (date<span class="builtin">:read</span> <span class="string">"*a"</span>) <span class="builtin">:sub</span> 1 -2)]
  (<span class="keyword">with-open</span> [raw (<span class="type">io.open</span> (<span class="keyword">..</span> <span class="string">"responses/"</span> id <span class="string">".raw"</span>) <span class="builtin">:w</span>)]
    (raw<span class="builtin">:write</span> contents))
  (<span class="builtin">print</span> <span class="string">"status: 301 redirect"</span>)
  (<span class="builtin">print</span> <span class="string">"Location: /survey/thanks.html\n"</span>))</pre>

<p>As you can see, all this does is read the request body
  using <tt>io.read</tt>, create a file with the current timestamp
  as the filename (we shell out to <tt>date</tt> because the
  built-in <tt>os.time</tt> function lacks subsecond resolution) and
  prints out a canned response redirecting the browser to another
  static HTML page. We could have printed HTML for the response body,
  but why complicate things?</p>

<p>At this point we're all set as far as gathering data goes. But what
  do we do with these responses? Well, a typical approach would be to
  write them to a database rather than the filesystem, and to create
  another script which reads from the database whenever it gets an
  HTTP request and emits HTML which summarizes the results. You could
  certainly do this in Fennel
  using <a href="https://openresty.org/en/postgres-nginx-module.html">nginx
  and its postgres</a> module, but it didn't feel like a good fit for
  this. A database has a lot of moving parts and complex features
  around consistency during concurrent writes which are simply
  astronomically unlikely[<a href="https://technomancy.us#fn3">3</a>] to happen in this
  case.</p>

<p>At this point I think it's time to take a look at the <tt>Makefile</tt>:</p>

<pre class="code"><span class="makefile-targets">upload</span>: index.html save.cgi thanks.html 2021.html
    <span class="makefile-targets">rsync -rAv </span><span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="constant"><span class="makefile-targets">^</span></span></span><span class="makefile-targets"> fennel-lang.org</span>:fennel-lang.org/survey/

<span class="makefile-targets">index.html</span>: survey.fnl questions.fnl
    ../fennel/fennel --add-fennel-path <span class="string">"../?.fnl"</span> $<span class="constant">&lt;</span> &gt; <span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="makefile-targets"><span class="constant">@</span></span></span>

<span class="makefile-targets">save.cgi</span>: save.fnl
    echo <span class="string">"#!/usr/bin/env lua"</span> &gt; <span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="makefile-targets"><span class="constant">@</span></span></span>
    ../fennel/fennel --compile $<span class="constant">&lt;</span> &gt;&gt; <span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="makefile-targets"><span class="constant">@</span></span></span>
    chmod 755 <span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="makefile-targets"><span class="constant">@</span></span></span>

<span class="makefile-targets">pull</span>:
    <span class="makefile-targets">rsync -rA fennel-lang.org</span>:fennel-lang.org/survey/responses/ responses/

<span class="makefile-targets">2021.html</span>: summary.fnl chart.fnl questions.fnl responses/* commentary/2021/*
    ../fennel/fennel --add-fennel-path <span class="string">"../?.fnl"</span> $<span class="constant">&lt;</span> &gt; <span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="makefile-targets"><span class="constant">@</span></span></span></pre>

<p>So the <tt>pull</tt> target takes all the raw response files from
  the server and brings them into my local checkout of the web site on
  my laptop. The <tt>2021.html</tt> target runs
  the <tt>summary.fnl</tt> script locally to read thru all the
  responses, parse them, aggregate them, and emit static HTML
  containing inline SVG charts. Then the <tt>upload</tt> task puts the
  output back on the server. Here's the code which takes that raw form
  data from the CGI script and turns it into a data structure[<a href="https://technomancy.us#fn4">4</a>]:</p>

<pre class="code"><span class="region">(</span><span class="keyword"><span class="region">fn</span></span><span class="region"> </span><span class="function-name"><span class="region">parse</span></span><span class="region"> [contents] </span><span class="comment"><span class="region">; for form-encoded data
</span></span><span class="region">  (</span><span class="keyword"><span class="region">fn</span></span><span class="region"> </span><span class="function-name"><span class="region">decode</span></span><span class="region"> [str] (str</span><span class="builtin"><span class="region">:gsub</span></span><span class="region"> </span><span class="string"><span class="region">"%%(%x%x)"</span></span><span class="region"> (</span><span class="keyword"><span class="region">fn</span></span><span class="region"> [v] (</span><span class="type"><span class="region">string.char</span></span><span class="region"> (</span><span class="builtin"><span class="region">tonumber</span></span><span class="region"> v 16)))))
  (</span><span class="keyword"><span class="region">let</span></span><span class="region"> [out {}]
    (</span><span class="keyword"><span class="region">each</span></span><span class="region"> [k v (contents</span><span class="builtin"><span class="region">:gmatch</span></span><span class="region"> </span><span class="string"><span class="region">"([^&amp;=]+)=([^&amp;=]+)"</span></span><span class="region">)]
      (</span><span class="keyword"><span class="region">let</span></span><span class="region"> [key (decode (k</span><span class="builtin"><span class="region">:gsub</span></span><span class="region"> </span><span class="string"><span class="region">"+"</span></span><span class="region"> </span><span class="string"><span class="region">" "</span></span><span class="region">))]
        (</span><span class="keyword"><span class="region">when</span></span><span class="region"> (</span><span class="keyword"><span class="region">not</span></span><span class="region"> (</span><span class="keyword"><span class="region">.</span></span><span class="region"> out key))
          (</span><span class="keyword"><span class="region">tset</span></span><span class="region"> out key []))
        (</span><span class="type"><span class="region">table.insert</span></span><span class="region"> (</span><span class="keyword"><span class="region">.</span></span><span class="region"> out key) (</span><span class="keyword"><span class="region">pick-values</span></span><span class="region"> 1 (decode (v</span><span class="builtin"><span class="region">:gsub</span></span><span class="region"> </span><span class="string"><span class="region">"+"</span></span><span class="region"> </span><span class="string"><span class="region">" "</span></span><span class="region">))))))
    out))</span></pre>

<p>The final piece I want to mention is the charts in the survey
  results. I wasn't sure how I'd visualize the results, but I had some
  experience writing SVG from
  my <a href="https://gitlab.com/technomancy/atreus/-/blob/master/case/case.rkt">programmatically
  generated keyboard cases</a> I had constructed on my laser
  cutter. If you've never looked closely at SVG before, it's a lot
  more accessible than you might expect. This code takes the data from
  the previous function after it's been aggregated by response count
  and emits a bar chart with counts for each response. Here's an
  example of one of the charts; inspect the source to see how it looks
  if you're curious:</p>

<svg class="chart" height="105" width="900" xmlns="http://www.w3.org/2000/svg">
  <g class="bar">
    <rect height="20" width="430" y="0"></rect>
    <text dy="0.35em" x="435" y="12">Linux-based (43)</text>
  </g>
  <g class="bar">
    <rect height="20" width="110" y="21"></rect>
    <text dy="0.35em" x="115" y="33">MacOS (11)</text>
  </g>
  <g class="bar">
    <rect height="20" width="60" y="42"></rect>
    <text dy="0.35em" x="65" y="54">Windows (6)</text>
  </g>
  <g class="bar">
    <rect height="20" width="40" y="63"></rect>
    <text dy="0.35em" x="45" y="75">Other BSD-based (4)</text>
  </g>
  <desc id="desc-5">Linux-based: 43, MacOS: 11, Windows: 6, Other
    BSD-based: 4</desc>
</svg>

<p>I had never tried putting SVG directly into HTML before, but I
  found you can just embed an &lt;svg&gt; element like any other. The
  &lt;desc&gt; elements even allow it to be read by a screen reader.</p>

<pre class="code">(<span class="keyword">fn</span> <span class="function-name">bar-rect</span> [answer count i]
  (<span class="keyword">let</span> [width (<span class="keyword">*</span> count 10)
        y (<span class="keyword">*</span> 21 (<span class="keyword">-</span> i 1))]
    [<span class="builtin">:g</span> {<span class="builtin">:class</span> <span class="builtin">:bar</span>}
     [<span class="builtin">:rect</span> {<span class="keyword">:</span> width <span class="builtin">:height</span> 20 <span class="keyword">:</span> y}]
     [<span class="builtin">:text</span> {<span class="builtin">:x</span> (<span class="keyword">+</span> 5 width) <span class="builtin">:y</span> (<span class="keyword">+</span> y 12) <span class="builtin">:dy</span> <span class="string">"0.35em"</span>}
      (<span class="keyword">..</span> answer <span class="string">" ("</span> count <span class="string">")"</span>)]]))

(<span class="keyword">fn</span> <span class="function-name">bar</span> [i data ?sorter]
  <span class="comment-delimiter">;; </span><span class="comment">by default, sort in descending order of count of responses, but
</span>  <span class="comment-delimiter">;; </span><span class="comment">allow sorting to be overridden, for example with the age question
</span>  <span class="comment-delimiter">;; </span><span class="comment">the answers should be ordered by the age, not response count.
</span>  (<span class="keyword">fn</span> <span class="function-name">count-sorter</span> [k1 k2]
    (<span class="keyword">let</span> [v1 (<span class="keyword">.</span> data k1) v2 (<span class="keyword">.</span> data k2)]
      (<span class="keyword">if</span> (<span class="keyword">=</span> v1 v2) (<span class="keyword">&lt;</span> k1 k2) (<span class="keyword">&lt;</span> v2 v1))))
  (<span class="keyword">let</span> [sorter (<span class="keyword">or</span> ?sorter count-sorter)
        answers (<span class="keyword">doto</span> (<span class="keyword">icollect</span> [k (<span class="builtin">pairs</span> data)] k) (<span class="type">table.sort</span> sorter))
        svg [<span class="builtin">:svg</span> {<span class="builtin">:class</span> <span class="builtin">:chart</span> <span class="builtin">:role</span> <span class="builtin">:img</span>
                   <span class="builtin">:aria-label</span> <span class="string">"bar graph"</span> <span class="builtin">:aria-describedby</span> (<span class="keyword">..</span> <span class="string">"desc-"</span> i)
                   <span class="builtin">:width</span> 900 <span class="builtin">:height</span> (<span class="keyword">*</span> 21 (<span class="keyword">+</span> 1 (<span class="keyword">length</span> answers)))}]
        descs []]
    (<span class="keyword">each</span> [i answer (<span class="builtin">ipairs</span> answers)]
      (<span class="type">table.insert</span> svg (bar-rect answer (<span class="keyword">.</span> data answer) i))
      (<span class="type">table.insert</span> descs (<span class="keyword">..</span> answer <span class="string">": "</span> (<span class="keyword">.</span> data answer))))
    (<span class="type">table.insert</span> svg [<span class="builtin">:desc</span> {<span class="builtin">:id</span> (<span class="keyword">..</span> <span class="string">"desc-"</span> i)} (<span class="type">table.concat</span> descs <span class="string">", "</span>)])
    svg))

{<span class="keyword">:</span> bar}</pre>

<p>In the end, other than the
  actual <a href="https://git.sr.ht/~technomancy/fennel-lang.org/tree/main/item/survey/questions.fnl">questions</a>
  of the survey, all the code clocked in at just over 200 lines. If
  you're curious to read thru the whole thing you can find it
  in <a href="https://git.sr.ht/~technomancy/fennel-lang.org/tree/main/item/survey/">the
  survey/ subdirectory of the fennel-lang.org repository</a>.</p>

<p>As you can see
  from <a href="https://fennel-lang.org/survey/2021">reading the
    results</a>, one of the things people wanted to see more of with Fennel
  was some detailed example code. So hopefully this helps with that,
  and people can learn both about how the code is put together and the
  unusual approach to building it out.</p>

<hr />

<p>[<a name="fn1">1</a>] In fact,
  the <a href="https://git.sr.ht/~technomancy/fennel-lang.org/tree/main/item/html.fnl">HTML
    generator code</a> which is used for Fennel's web site was written
  in 2018 at <a href="https://conf.fennel-lang.org/2018">the first FennelConf</a>.</p>

<p>[<a name="fn2">2</a>] from <i>Johnny Mnemonic</i> by William Gibson</p>

<p>[<a name="fn3">3</a>] If we <em>had</em> used <tt>os.time</tt>
  with its second-level granularity instead of <tt>date</tt> with
  nanosecond precision then concurrent conflicting writes would have
  moved from astronomically unlikely to merely very, very unlikely,
  with the remote possibility of two responses overwriting each other
  if they arrived within the same second. We had fifty responses over
  a period of 12 days, so this never came close to happening, but in
  other contexts it could have, so choose your data storage mechanism
  to fit the problem at hand.</p>

<p>[<a name="fn4">4</a>] This code is actually taken from the code I
  wrote a couple years ago to handle signups
  for <a href="https://conf.fennel-lang.org/2019">FennelConf 2019</a>.
  If I wrote it today I would have made it use the <tt>accumulate</tt>
  or <tt>collect</tt> macros.</p>