chris/emacs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

trying to fix

Browse source
This commit is contained in:
Chris Cochrun 2022-01-03 12:41:35 -06:00
parent fa407dfeb6
commit e013d7569e
22945 changed files with 447936 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

216
var/elfeed/db/data/e0/e070cf8c3d94b7da6350be24c86f338d8b9fe750 Normal file
View file

@ -0,0 +1,216 @@
<p>The <a href="https://fennel-lang.org">Fennel programming
language</a> recently celebrated its fifth birthday, and we ran
<a href="https://fennel-lang.org/2021">a survey</a> to learn more
about the community and what has been working well and what
hasn't. Fennel's approach has always been one of simplicity; not
just in the conceptual footprint of the language, but in reducing
dependencies and moving parts, and using on a runtime that fits in
under 200kb. In order to reflect this, the Fennel web site is hosted
as static files on the same Apache-backed shared hosting account
I've been using for this blog since 2005.</p>
<p>Of course, <a href="https://git.sr.ht/~technomancy/fennel-lang.org/tree/main/item/main.fnl">generating
HTML from lisp code</a> is one of the oldest tricks in the
book[<a href="https://technomancy.us#fn1">1</a>], so I won't bore anyone with the details there. But what
happens when you want to mix in something that <em>isn't</em>
completely static, like this survey? Well, that's where it gets interesting.</p>
<blockquote>
I put the shotgun in an Adidas bag and padded it out with four pairs
of tennis socks, not my style at all, but that was what I was aiming
for: If they think you're crude, go technical; if they think you're
technical, go crude. I'm a very technical boy. So I decided to get as
crude as possible. These days, though, you have to be pretty
technical before you can even aspire to crudeness.[<a href="https://technomancy.us#fn2">2</a>]
</blockquote>
<p>When I was in school, I learned how to write and deploy Ruby web
programs. The easiest way to get that set up was
using <a href="https://en.wikipedia.org/wiki/Common_Gateway_Interface">CGI</a>. A CGI script is just a process which is launched by the
web server in such a way that the request comes in on stdin and
environment variables and the response is sent over stdout. But
larger Ruby programs tended to have very slow boot times, which
didn't fit very well with CGI's model of launching a process afresh
for every request that came in, and eventually other models replaced
CGI. Most people regard CGI as somewhat outmoded and obsolete, but
it fits Fennel's ethos nicely and complements a mostly-static-files
approach.</p>
<p>So the
survey <a href="https://git.sr.ht/~technomancy/fennel-lang.org/tree/3bed58d0007ac8f9616486ef20094cffc2c10562/item/survey/survey.fnl#L39">generates
an HTML form</a> in a static file which points to a CGI script as
its <tt>action</tt>. The CGI script looks like this, but it gets
compiled to Lua as part of the deploy process to keep the
dependencies light.</p>
<pre class="code">(<span class="keyword">let</span> [contents (<span class="type">io.read</span> <span class="string">"*all"</span>)
date (<span class="type">io.popen</span> <span class="string">"date --rfc-3339=ns"</span>)
id (<span class="keyword">:</span> (date<span class="builtin">:read</span> <span class="string">"*a"</span>) <span class="builtin">:sub</span> 1 -2)]
(<span class="keyword">with-open</span> [raw (<span class="type">io.open</span> (<span class="keyword">..</span> <span class="string">"responses/"</span> id <span class="string">".raw"</span>) <span class="builtin">:w</span>)]
(raw<span class="builtin">:write</span> contents))
(<span class="builtin">print</span> <span class="string">"status: 301 redirect"</span>)
(<span class="builtin">print</span> <span class="string">"Location: /survey/thanks.html\n"</span>))</pre>
<p>As you can see, all this does is read the request body
using <tt>io.read</tt>, create a file with the current timestamp
as the filename (we shell out to <tt>date</tt> because the
built-in <tt>os.time</tt> function lacks subsecond resolution) and
prints out a canned response redirecting the browser to another
static HTML page. We could have printed HTML for the response body,
but why complicate things?</p>
<p>At this point we're all set as far as gathering data goes. But what
do we do with these responses? Well, a typical approach would be to
write them to a database rather than the filesystem, and to create
another script which reads from the database whenever it gets an
HTTP request and emits HTML which summarizes the results. You could
certainly do this in Fennel
using <a href="https://openresty.org/en/postgres-nginx-module.html">nginx
and its postgres</a> module, but it didn't feel like a good fit for
this. A database has a lot of moving parts and complex features
around consistency during concurrent writes which are simply
astronomically unlikely[<a href="https://technomancy.us#fn3">3</a>] to happen in this
case.</p>
<p>At this point I think it's time to take a look at the <tt>Makefile</tt>:</p>
<pre class="code"><span class="makefile-targets">upload</span>: index.html save.cgi thanks.html 2021.html
<span class="makefile-targets">rsync -rAv </span><span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="constant"><span class="makefile-targets">^</span></span></span><span class="makefile-targets"> fennel-lang.org</span>:fennel-lang.org/survey/
<span class="makefile-targets">index.html</span>: survey.fnl questions.fnl
../fennel/fennel --add-fennel-path <span class="string">"../?.fnl"</span> $<span class="constant">&lt;</span> &gt; <span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="makefile-targets"><span class="constant">@</span></span></span>
<span class="makefile-targets">save.cgi</span>: save.fnl
echo <span class="string">"#!/usr/bin/env lua"</span> &gt; <span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="makefile-targets"><span class="constant">@</span></span></span>
../fennel/fennel --compile $<span class="constant">&lt;</span> &gt;&gt; <span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="makefile-targets"><span class="constant">@</span></span></span>
chmod 755 <span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="makefile-targets"><span class="constant">@</span></span></span>
<span class="makefile-targets">pull</span>:
<span class="makefile-targets">rsync -rA fennel-lang.org</span>:fennel-lang.org/survey/responses/ responses/
<span class="makefile-targets">2021.html</span>: summary.fnl chart.fnl questions.fnl responses/* commentary/2021/*
../fennel/fennel --add-fennel-path <span class="string">"../?.fnl"</span> $<span class="constant">&lt;</span> &gt; <span class="makefile-targets"><span class="makefile-targets">$</span></span><span class="makefile-targets"><span class="makefile-targets"><span class="constant">@</span></span></span></pre>
<p>So the <tt>pull</tt> target takes all the raw response files from
the server and brings them into my local checkout of the web site on
my laptop. The <tt>2021.html</tt> target runs
the <tt>summary.fnl</tt> script locally to read thru all the
responses, parse them, aggregate them, and emit static HTML
containing inline SVG charts. Then the <tt>upload</tt> task puts the
output back on the server. Here's the code which takes that raw form
data from the CGI script and turns it into a data structure[<a href="https://technomancy.us#fn4">4</a>]:</p>
<pre class="code"><span class="region">(</span><span class="keyword"><span class="region">fn</span></span><span class="region"> </span><span class="function-name"><span class="region">parse</span></span><span class="region"> [contents] </span><span class="comment"><span class="region">; for form-encoded data
</span></span><span class="region"> (</span><span class="keyword"><span class="region">fn</span></span><span class="region"> </span><span class="function-name"><span class="region">decode</span></span><span class="region"> [str] (str</span><span class="builtin"><span class="region">:gsub</span></span><span class="region"> </span><span class="string"><span class="region">"%%(%x%x)"</span></span><span class="region"> (</span><span class="keyword"><span class="region">fn</span></span><span class="region"> [v] (</span><span class="type"><span class="region">string.char</span></span><span class="region"> (</span><span class="builtin"><span class="region">tonumber</span></span><span class="region"> v 16)))))
(</span><span class="keyword"><span class="region">let</span></span><span class="region"> [out {}]
(</span><span class="keyword"><span class="region">each</span></span><span class="region"> [k v (contents</span><span class="builtin"><span class="region">:gmatch</span></span><span class="region"> </span><span class="string"><span class="region">"([^&amp;=]+)=([^&amp;=]+)"</span></span><span class="region">)]
(</span><span class="keyword"><span class="region">let</span></span><span class="region"> [key (decode (k</span><span class="builtin"><span class="region">:gsub</span></span><span class="region"> </span><span class="string"><span class="region">"+"</span></span><span class="region"> </span><span class="string"><span class="region">" "</span></span><span class="region">))]
(</span><span class="keyword"><span class="region">when</span></span><span class="region"> (</span><span class="keyword"><span class="region">not</span></span><span class="region"> (</span><span class="keyword"><span class="region">.</span></span><span class="region"> out key))
(</span><span class="keyword"><span class="region">tset</span></span><span class="region"> out key []))
(</span><span class="type"><span class="region">table.insert</span></span><span class="region"> (</span><span class="keyword"><span class="region">.</span></span><span class="region"> out key) (</span><span class="keyword"><span class="region">pick-values</span></span><span class="region"> 1 (decode (v</span><span class="builtin"><span class="region">:gsub</span></span><span class="region"> </span><span class="string"><span class="region">"+"</span></span><span class="region"> </span><span class="string"><span class="region">" "</span></span><span class="region">))))))
out))</span></pre>
<p>The final piece I want to mention is the charts in the survey
results. I wasn't sure how I'd visualize the results, but I had some
experience writing SVG from
my <a href="https://gitlab.com/technomancy/atreus/-/blob/master/case/case.rkt">programmatically
generated keyboard cases</a> I had constructed on my laser
cutter. If you've never looked closely at SVG before, it's a lot
more accessible than you might expect. This code takes the data from
the previous function after it's been aggregated by response count
and emits a bar chart with counts for each response. Here's an
example of one of the charts; inspect the source to see how it looks
if you're curious:</p>
<svg class="chart" height="105" width="900" xmlns="http://www.w3.org/2000/svg">
<g class="bar">
<rect height="20" width="430" y="0"></rect>
<text dy="0.35em" x="435" y="12">Linux-based (43)</text>
</g>
<g class="bar">
<rect height="20" width="110" y="21"></rect>
<text dy="0.35em" x="115" y="33">MacOS (11)</text>
</g>
<g class="bar">
<rect height="20" width="60" y="42"></rect>
<text dy="0.35em" x="65" y="54">Windows (6)</text>
</g>
<g class="bar">
<rect height="20" width="40" y="63"></rect>
<text dy="0.35em" x="45" y="75">Other BSD-based (4)</text>
</g>
<desc id="desc-5">Linux-based: 43, MacOS: 11, Windows: 6, Other
BSD-based: 4</desc>
</svg>
<p>I had never tried putting SVG directly into HTML before, but I
found you can just embed an &lt;svg&gt; element like any other. The
&lt;desc&gt; elements even allow it to be read by a screen reader.</p>
<pre class="code">(<span class="keyword">fn</span> <span class="function-name">bar-rect</span> [answer count i]
(<span class="keyword">let</span> [width (<span class="keyword">*</span> count 10)
y (<span class="keyword">*</span> 21 (<span class="keyword">-</span> i 1))]
[<span class="builtin">:g</span> {<span class="builtin">:class</span> <span class="builtin">:bar</span>}
[<span class="builtin">:rect</span> {<span class="keyword">:</span> width <span class="builtin">:height</span> 20 <span class="keyword">:</span> y}]
[<span class="builtin">:text</span> {<span class="builtin">:x</span> (<span class="keyword">+</span> 5 width) <span class="builtin">:y</span> (<span class="keyword">+</span> y 12) <span class="builtin">:dy</span> <span class="string">"0.35em"</span>}
(<span class="keyword">..</span> answer <span class="string">" ("</span> count <span class="string">")"</span>)]]))
(<span class="keyword">fn</span> <span class="function-name">bar</span> [i data ?sorter]
<span class="comment-delimiter">;; </span><span class="comment">by default, sort in descending order of count of responses, but
</span> <span class="comment-delimiter">;; </span><span class="comment">allow sorting to be overridden, for example with the age question
</span> <span class="comment-delimiter">;; </span><span class="comment">the answers should be ordered by the age, not response count.
</span> (<span class="keyword">fn</span> <span class="function-name">count-sorter</span> [k1 k2]
(<span class="keyword">let</span> [v1 (<span class="keyword">.</span> data k1) v2 (<span class="keyword">.</span> data k2)]
(<span class="keyword">if</span> (<span class="keyword">=</span> v1 v2) (<span class="keyword">&lt;</span> k1 k2) (<span class="keyword">&lt;</span> v2 v1))))
(<span class="keyword">let</span> [sorter (<span class="keyword">or</span> ?sorter count-sorter)
answers (<span class="keyword">doto</span> (<span class="keyword">icollect</span> [k (<span class="builtin">pairs</span> data)] k) (<span class="type">table.sort</span> sorter))
svg [<span class="builtin">:svg</span> {<span class="builtin">:class</span> <span class="builtin">:chart</span> <span class="builtin">:role</span> <span class="builtin">:img</span>
<span class="builtin">:aria-label</span> <span class="string">"bar graph"</span> <span class="builtin">:aria-describedby</span> (<span class="keyword">..</span> <span class="string">"desc-"</span> i)
<span class="builtin">:width</span> 900 <span class="builtin">:height</span> (<span class="keyword">*</span> 21 (<span class="keyword">+</span> 1 (<span class="keyword">length</span> answers)))}]
descs []]
(<span class="keyword">each</span> [i answer (<span class="builtin">ipairs</span> answers)]
(<span class="type">table.insert</span> svg (bar-rect answer (<span class="keyword">.</span> data answer) i))
(<span class="type">table.insert</span> descs (<span class="keyword">..</span> answer <span class="string">": "</span> (<span class="keyword">.</span> data answer))))
(<span class="type">table.insert</span> svg [<span class="builtin">:desc</span> {<span class="builtin">:id</span> (<span class="keyword">..</span> <span class="string">"desc-"</span> i)} (<span class="type">table.concat</span> descs <span class="string">", "</span>)])
svg))
{<span class="keyword">:</span> bar}</pre>
<p>In the end, other than the
actual <a href="https://git.sr.ht/~technomancy/fennel-lang.org/tree/main/item/survey/questions.fnl">questions</a>
of the survey, all the code clocked in at just over 200 lines. If
you're curious to read thru the whole thing you can find it
in <a href="https://git.sr.ht/~technomancy/fennel-lang.org/tree/main/item/survey/">the
survey/ subdirectory of the fennel-lang.org repository</a>.</p>
<p>As you can see
from <a href="https://fennel-lang.org/survey/2021">reading the
results</a>, one of the things people wanted to see more of with Fennel
was some detailed example code. So hopefully this helps with that,
and people can learn both about how the code is put together and the
unusual approach to building it out.</p>
<hr />
<p>[<a name="fn1">1</a>] In fact,
the <a href="https://git.sr.ht/~technomancy/fennel-lang.org/tree/main/item/html.fnl">HTML
generator code</a> which is used for Fennel's web site was written
in 2018 at <a href="https://conf.fennel-lang.org/2018">the first FennelConf</a>.</p>
<p>[<a name="fn2">2</a>] from <i>Johnny Mnemonic</i> by William Gibson</p>
<p>[<a name="fn3">3</a>] If we <em>had</em> used <tt>os.time</tt>
with its second-level granularity instead of <tt>date</tt> with
nanosecond precision then concurrent conflicting writes would have
moved from astronomically unlikely to merely very, very unlikely,
with the remote possibility of two responses overwriting each other
if they arrived within the same second. We had fifty responses over
a period of 12 days, so this never came close to happening, but in
other contexts it could have, so choose your data storage mechanism
to fit the problem at hand.</p>
<p>[<a name="fn4">4</a>] This code is actually taken from the code I
wrote a couple years ago to handle signups
for <a href="https://conf.fennel-lang.org/2019">FennelConf 2019</a>.
If I wrote it today I would have made it use the <tt>accumulate</tt>
or <tt>collect</tt> macros.</p>